Digital breast artificial intelligence auxiliary diagnosis system based on multi-modal fusion

Abstract

The application relates to the technical field of disease auxiliary diagnosis, in particular to a digital breast artificial intelligence auxiliary diagnosis system based on multi-modal fusion, which comprises the following steps: a multi-modal data acquisition module is used for acquiring digital breast image data and rehabilitation scheme data of a patient; an image quality correction module is used for performing acquisition quality evaluation and correction on the digital breast image data; a feature modeling and evaluation module is used for performing feature extraction on the corrected digital breast image data and the rehabilitation scheme data, including an image-drug action mechanism cooperation unit and an image-rehabilitation double-modal co-learning unit; and a rehabilitation effect prediction module is used for constructing a rehabilitation effect prediction model and outputting a rehabilitation prediction result under the influence of a scheme-specific scheme for the current rehabilitation scheme of the patient. The digital breast image and the patient rehabilitation scheme are deeply fused, and the accuracy and reliability of breast tumor rehabilitation prediction are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of disease-assisted diagnosis technology, and in particular to a digital breast cancer artificial intelligence-assisted diagnosis system based on multimodal fusion. Background Technology

[0002] Breast cancer is a highly heterogeneous disease characterized by variations in biological features, and it poses a significant threat to the physical and mental health of women worldwide. In the diagnosis and treatment of breast cancer, imaging techniques such as digital mammography (DM), breast ultrasound, and breast magnetic resonance imaging (MRI) are crucial tools for clinically assessing tumor burden and guiding subsequent rehabilitation decisions.

[0003] With the widespread application of various precision rehabilitation strategies such as chemotherapy, targeted therapy, and immunotherapy, the mechanisms of action and biological effects of different rehabilitation methods on tumor cells vary significantly, resulting in complex and dynamic changes in tumor imaging. However, existing digital breast cancer AI-assisted diagnostic systems are mainly based on static image features and lack the ability to integrate multi-source data such as rehabilitation plans and drug mechanisms of action, making it difficult to reflect the heterogeneity of tumor biological responses during the rehabilitation process.

[0004] Meanwhile, in actual clinical settings, patients undergo multiple rounds of imaging examinations during the recovery period. The comparison of images from different periods can reflect the recovery trend of the tumor. However, side effects such as tissue edema and skin inflammation that occur during the recovery process can interfere with image quality, such as reduced contrast, increased noise, and blurred tissue boundaries, further weakening the stability of image features.

[0005] Therefore, existing technologies ignore the interference of side effects on the quality of digital imaging and the significant differences in the mechanisms of action and biological effects of different rehabilitation methods on tumor cells, resulting in insufficient accuracy in predicting rehabilitation for breast diseases. Summary of the Invention

[0006] To overcome the defects and shortcomings of existing technologies, this invention provides a digital breast AI-assisted diagnostic system based on multimodal fusion. By deeply integrating digital breast images with patient rehabilitation plans, it effectively improves the accuracy and reliability of breast tumor rehabilitation prediction.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a digital breast cancer artificial intelligence-assisted diagnostic system based on multimodal fusion, including a multimodal data acquisition module, an image quality correction module, a feature modeling and evaluation module, and a rehabilitation effect prediction module;

[0009] The multimodal data acquisition module is used to acquire digital breast imaging data and rehabilitation plan data of the patient during the rehabilitation period.

[0010] The image quality correction module is used to acquire acquisition operation metadata and, based on the acquisition operation metadata and rehabilitation program side effect data, to assess and correct the acquisition quality of digital breast imaging data.

[0011] The feature modeling and evaluation module is used to extract features from the corrected digital breast imaging data and rehabilitation program data. It includes an image-drug mechanism synergy unit for evaluating changes in image phenotype and an image-rehabilitation bimodal co-learning unit for evaluating changes in rehabilitation trends.

[0012] The rehabilitation effect prediction module is used to construct a rehabilitation effect prediction model based on changes in imaging phenotypic patterns and changes in rehabilitation trends, and output rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term rehabilitation prediction indicators and long-term rehabilitation prediction indicators.

[0013] Furthermore, the multimodal data acquisition module is used to acquire digital breast imaging data and rehabilitation plan data of the patient during the rehabilitation period. The digital breast imaging data includes digital mammography, ultrasound, magnetic resonance imaging and infrared thermal imaging. The rehabilitation plan data includes rehabilitation type, drug type, drug dosage, drug component target data, rehabilitation period, immune activation indicators and rehabilitation plan side effect data. The rehabilitation type can be any one of chemotherapy, radiotherapy, targeted therapy and immunotherapy.

[0014] Furthermore, the specific execution steps of the image quality correction module include:

[0015] Acquire acquisition operation metadata during the digital breast imaging process. Acquisition operation metadata includes compression thickness, compression force, tube voltage, tube current exposure, and glandular density.

[0016] Extract the side effect data of the rehabilitation program from the rehabilitation program data. The side effect data of the rehabilitation program includes the degree of tissue edema and the level of skin reaction.

[0017] Based on the collected operation metadata and rehabilitation program side effect data, combined with multilayer perceptron, the image noise level, contrast reduction degree and tissue overlap blur are predicted;

[0018] Adaptive correction is performed on digital breast imaging data based on image quality assessment results. The adaptive correction includes adaptive filtering based on image noise level, contrast enhancement based on contrast reduction degree, and image sharpening based on tissue overlap blur.

[0019] Furthermore, the specific execution steps of the imaging-drug action mechanism synergistic unit include:

[0020] Based on the drug component target data in the rehabilitation program data, a molecular signaling pathway feature set corresponding to the drug is constructed. The molecular signaling pathway feature set includes the gene expression profile of the drug target, the receptor activation mode, and the response characteristics of downstream signaling pathway nodes.

[0021] Multi-scale texture and morphological features were extracted from the corrected digital breast imaging data to generate an image texture atlas that describes the microstructural state of tumor tissue.

[0022] Based on molecular signaling pathway feature sets and image texture maps, and combined with cross-modal Transformer, the correlation strength between image features and drug action signals is evaluated, drug response phenotype regions within tumor areas are identified, and a synergistic response feature vector of image and drug action is generated.

[0023] Furthermore, the specific execution steps of the image-rehabilitation bimodal co-learning unit include:

[0024] Extract drug types, drug dosages, rehabilitation cycles, and immune activation indicators from rehabilitation program data and construct a rehabilitation feature vector;

[0025] Image features were extracted from the corrected digital breast imaging data to obtain image feature vectors that characterize tumor morphological and structural changes, blood perfusion characteristics, and tissue heterogeneity.

[0026] Based on deep canonical correlation analysis, rehabilitation feature vectors and imaging feature vectors are mapped to a unified latent space and a fused feature vector is generated, thereby achieving feature alignment between rehabilitation intervention factors and imaging phenotypes.

[0027] Based on the fusion feature vector of continuous rehabilitation cycle, combined with long short-term memory neural network to learn the evolution trajectory of fusion features in latent space, a rehabilitation trend feature vector is generated.

[0028] Furthermore, the specific execution steps of the rehabilitation effect prediction module include:

[0029] Obtain the collaborative response feature vector generated by the image-drug action mechanism collaborative unit and the rehabilitation trend feature vector generated by the image-rehabilitation dual-modal co-learning unit;

[0030] A rehabilitation effect prediction model is constructed based on a time series graph neural network. By performing time-series correlation modeling on the synergistic response feature vector and the rehabilitation trend feature vector, short-term and long-term rehabilitation prediction indicators for patients under the current rehabilitation plan are generated. The short-term rehabilitation prediction indicators include the lesion volume shrinkage rate and the metabolic activity decline rate, while the long-term rehabilitation prediction indicators include the probability of complete pathological remission and the probability of progression-free survival.

[0031] Secondly, the present invention provides a digital breast cancer AI-assisted diagnostic method based on multimodal fusion, comprising:

[0032] Acquire digital breast imaging data and rehabilitation plan data of patients during the rehabilitation period;

[0033] Acquire metadata of the acquisition operation and, based on the metadata and side effect data of the rehabilitation program, assess and correct the acquisition quality of digital breast imaging data.

[0034] Feature extraction was performed on the corrected digital breast imaging data and rehabilitation program data to assess changes in imaging phenotypes and rehabilitation trends.

[0035] A rehabilitation outcome prediction model is constructed based on changes in imaging phenotypic patterns and rehabilitation trends. The model outputs rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term and long-term rehabilitation prediction indicators.

[0036] Thirdly, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes a digital breast cancer artificial intelligence-assisted diagnosis method based on multimodal fusion by calling the computer program stored in the memory.

[0037] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform a digital breast cancer AI-assisted diagnostic method based on multimodal fusion.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] (1) By constructing a side effect-driven image quality correction mechanism, this invention achieves adaptive image quality enhancement based on the acquired operation metadata and rehabilitation plan side effect data. It can effectively reduce noise, improve contrast and highlight the clarity of lesion edges, and effectively improve the reliability and consistency of multimodal images in multi-cycle rehabilitation comparative analysis.

[0040] (2) This invention integrates digital breast imaging data and rehabilitation program data through collaborative modeling of imaging-drug action mechanism and the introduction of imaging-rehabilitation dual-modal co-learning mechanism. It dynamically captures the phenotypic evolution of tumor tissue during the rehabilitation cycle and accurately models the changes in efficacy by combining the action mechanism of different rehabilitation programs, which effectively improves the accuracy and reliability of breast tumor rehabilitation prediction. Attached Figure Description

[0041] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 This is a schematic diagram of the structure of the digital breast cancer AI-assisted diagnostic system based on multimodal fusion provided in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the digital breast cancer AI-assisted diagnosis method based on multimodal fusion provided in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0046] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a digital breast cancer AI-assisted diagnostic system based on multimodal fusion provided in an embodiment of the present invention, including:

[0047] The multimodal data acquisition module 210 is used to acquire digital breast imaging data and rehabilitation plan data of patients during the rehabilitation period;

[0048] The image quality correction module 220 is used to acquire acquisition operation metadata and, based on the acquisition operation metadata and rehabilitation program side effect data, to assess and correct the acquisition quality of digital breast imaging data.

[0049] The feature modeling and evaluation module 230 is used to extract features from the corrected digital breast imaging data and rehabilitation program data, including an image-drug mechanism synergy unit for evaluating changes in image phenotype and an image-rehabilitation bimodal co-learning unit for evaluating changes in rehabilitation trends.

[0050] The rehabilitation effect prediction module 240 is used to construct a rehabilitation effect prediction model based on changes in imaging phenotypic patterns and changes in rehabilitation trends, and output rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term rehabilitation prediction indicators and long-term rehabilitation prediction indicators.

[0051] In this embodiment of the invention, the multimodal data acquisition module 210 is used to acquire digital breast imaging data and rehabilitation plan data of the patient during the rehabilitation period. The digital breast imaging data includes digital mammography, ultrasound, magnetic resonance imaging and infrared thermal imaging. The rehabilitation plan data includes rehabilitation type, drug type, drug dosage, drug component target data, rehabilitation period, immune activation indicators and rehabilitation plan side effect data. The rehabilitation type is any one of chemotherapy, radiotherapy, targeted therapy and immunotherapy.

[0052] In this embodiment of the invention, the image quality correction module 220 is used to acquire acquisition operation metadata and, based on the acquisition operation metadata and rehabilitation program side effect data, to evaluate and correct the acquisition quality of digital breast imaging data.

[0053] Traditional image quality control methods typically rely solely on device-side parameters such as compression thickness, tube voltage, and exposure to estimate image quality. However, in real-world rehabilitation scenarios, treatments like chemotherapy, radiotherapy, and immunotherapy can lead to physiological changes such as tissue edema, fibrosis, and skin reactions. These changes significantly affect breast tissue thickness, density distribution, and local scattering characteristics, resulting in increased image noise, decreased contrast, and blurred boundaries. For example, tissue edema during chemotherapy rehabilitation can temporarily increase local breast tissue thickness, leading to enhanced soft tissue scattering and significantly higher noise levels and decreased tissue layer contrast. Radiotherapy-induced skin reactions (such as erythema and inflammation) can increase the heterogeneity of the breast surface tissue, causing blurred edges or distorted textures. Immunotherapy-induced immune infiltration can locally alter glandular density distribution, resulting in abnormal local contrast. If these phenomena are not addressed, they will directly affect the stability and accuracy of subsequent image feature extraction. The specific execution steps of the image quality correction module include:

[0054] Acquire acquisition operation metadata during the digital breast imaging process. Acquisition operation metadata includes compression thickness, compression force, tube voltage, tube current exposure, and glandular density.

[0055] Extract the side effect data of the rehabilitation program from the rehabilitation program data. The side effect data of the rehabilitation program includes the degree of tissue edema and the level of skin reaction.

[0056] Based on acquisition operation metadata and rehabilitation program side effect data, combined with a multilayer perceptron (MLP), this study predicts image noise levels, contrast degradation, and tissue overlap blurring. Specifically, a MLP model consisting of an input layer, a hidden layer, and a regression output layer is constructed. The input layer receives acquisition operation metadata, rehabilitation program side effect data, and corresponding image quality labels. The hidden layer uses the ReLU activation function combined with batch normalization to enhance the model's nonlinear expressiveness and training stability. The output layer uses linear activation to simultaneously regress the three quality indicators: image noise level, contrast degradation, and tissue overlap blurring. During the MLP model training process, digital breast images with known image quality labels are collected. The corresponding acquisition operation metadata and rehabilitation program side effect data are used as inputs. Mean squared error is used as the loss function, and the Adam optimizer is used for iterative training to minimize the error between the predicted quality indicators and the true quality labels. After training, the MLP can predict the impact of tissue changes caused by different rehabilitation stages on the image imaging process, achieving joint modeling of rehabilitation and imaging conditions. This provides a quantitative reference for subsequent adaptive filtering, contrast enhancement, and sharpening, thereby effectively improving the accuracy and specificity of image quality correction.

[0057] Adaptive correction is performed on digital breast imaging data based on image quality assessment results. The adaptive correction includes adaptive filtering based on image noise level, contrast enhancement based on contrast reduction degree, and image sharpening based on tissue overlap blur.

[0058] In this embodiment of the invention, the feature modeling and evaluation module 230 is used to extract features from the corrected digital breast imaging data and rehabilitation program data, including an image-drug mechanism synergy unit for evaluating changes in image phenotype and an image-rehabilitation bimodal co-learning unit for evaluating changes in rehabilitation trends.

[0059] By integrating the molecular signaling pathway characteristics of drug targets with imaging phenotypic changes in tumor regions across modalities through an image-drug action mechanism synergistic unit, this approach can accurately characterize the true effects of different rehabilitation programs on tumor tissue microstructure. Traditional methods rely solely on superficial features such as image texture, morphology, or density, failing to reflect the changes in molecular signaling pathways initiated at the cellular level by drugs such as chemotherapy, targeted therapy, or immunotherapy, thus resulting in a lack of biological interpretability in imaging features. Therefore, this invention incorporates the drug target gene expression profile, receptor activation mode, and downstream signaling pathway node response characteristics into the image feature modeling process, enabling imaging phenotypic changes to directly correspond to the molecular responses produced by the drug, thus forming a comprehensive assessment in rehabilitation evaluation. A closed-loop feature expression system from molecular mechanisms to imaging manifestations is established. For example, in HER2-positive breast cancer patients, targeted drugs such as trastuzumab (Herceptin) significantly inhibit HER2 receptor activation and trigger changes in the downstream MAPK / PI3K signaling pathway, leading to a decrease in tumor cell proliferation rate and a rearrangement of cell density structure. The imaging-drug mechanism synergistic unit can identify microstructural changes in the tumor region caused by HER2 pathway inhibition through image texture atlases, such as decreased local texture roughness, enhanced glandular regularity, and increased sharpness of necrotic area boundaries. These changes are then mapped across modalities with the drug signaling pathway response characteristics to form an interpretable drug response phenotype region. The system can not only identify imaging changes but also determine whether these changes are truly caused by molecular regulation of the drug target, thus avoiding misinterpreting non-recovery changes as therapeutic improvement. The specific execution steps of the imaging-drug mechanism synergistic unit include:

[0060] Based on the drug component target data in the rehabilitation program data, a molecular signaling pathway feature set corresponding to the drug is constructed. The molecular signaling pathway feature set includes the gene expression profile of the drug target, the receptor activation mode, and the response characteristics of downstream signaling pathway nodes.

[0061] Multi-scale texture and morphological features were extracted from the corrected digital breast imaging data to generate an image texture atlas that describes the microstructural state of tumor tissue.

[0062] Based on molecular signaling pathway feature sets and image texture maps, combined with a cross-modal Transformer, the correlation strength between image features and drug action signals is evaluated. This identifies drug response phenotypic regions within tumor areas and generates synergistic response feature vectors between images and drug action. The cross-modal Transformer is used to achieve deep fusion between drug molecular signaling pathway features and digital breast image texture maps. Specifically, its construction includes: constructing drug molecular feature sequences based on drug target gene expression profiles, receptor activation patterns, and downstream signal node response features; and constructing image feature sequences based on multi-scale texture and morphological features of the image texture map. The cross-modal Transformer consists of an image feature encoder, a molecular... The system consists of a signal feature encoder and a multi-head cross-modal attention module. The image feature encoder and molecular signal feature encoder adopt stacked self-attention structures to capture the texture relationships within the image and the node dependencies within the drug signaling pathway. The multi-head cross-modal attention module is used to establish feature-level correspondences between image features and molecular signal features, and to enhance the representation of salient regions of drug regulatory signals in the image space. During the training phase, image annotations of real drug response regions are used as supervision information. At the same time, a cross-modal matching loss function based on cosine similarity and a cross-modal contrast loss function based on InfoNCE function are constructed to enable the model to learn the differences in image phenotypes caused by different drug mechanisms and suppress spurious changes caused by non-drug factors.

[0063] In real-world clinical settings, even with similar imaging phenotypes, different types of rehabilitation (such as chemotherapy, targeted therapy, and immunotherapy) lead to significantly different tumor response patterns due to their varying mechanisms of action. The imaging-rehabilitation bimodal co-learning unit, by simultaneously considering changes in rehabilitation parameters and imaging phenotypes, enables the system to establish a true causal relationship between rehabilitation intervention and imaging changes, thereby providing higher reliability and individualization for rehabilitation trend assessment. By mapping rehabilitation features and imaging features to a unified latent space, deep alignment of cross-modal data is achieved. For example, during immunotherapy, even if the tumor exhibits pseudo-enlargement in the short term, the increase in immune activation indicators and the stability of the intra-image perfusion pattern can jointly indicate that rehabilitation is effective. The imaging-rehabilitation bimodal co-learning unit, by establishing a synergistic relationship between rehabilitation features and imaging features, avoids misjudgments caused by solely relying on imaging, achieving rehabilitation trend judgments that better reflect clinical reality. The specific execution steps of the imaging-rehabilitation bimodal co-learning unit include:

[0064] Extract drug types, drug dosages, rehabilitation cycles, and immune activation indicators from rehabilitation program data and construct a rehabilitation feature vector;

[0065] Image feature extraction was performed on the corrected digital breast imaging data to obtain image feature vectors that characterize tumor morphological and structural changes, blood perfusion characteristics, and tissue heterogeneity. The image feature extraction was performed using a convolutional neural network.

[0066] Based on deep canonical correlation analysis, rehabilitation feature vectors and imaging feature vectors are mapped to a unified latent space and a fused feature vector is generated, realizing feature alignment between rehabilitation intervention factors and imaging phenotypes. Specifically, deep canonical correlation analysis (DCCA) constructs two independent deep neural networks as encoders for rehabilitation feature vectors and imaging feature vectors. The rehabilitation feature encoder receives structured rehabilitation feature vectors, while the imaging feature encoder receives imaging feature vectors extracted from corrected digital breast images. During the training phase, paired rehabilitation-imaging data from the same patient are used as input. The encoder parameters of the rehabilitation feature encoder and the imaging feature encoder are optimized through the DCCA model. The optimization objective is to maximize the canonical correlation between the outputs of the two encoders. The total correlation between the two latent variables is calculated through singular value decomposition, and the correlation loss is maximized using gradient descent, so that the encoder learns a representation that can reflect the deep association between the two modalities.

[0067] Based on the fusion feature vectors of continuous rehabilitation cycles, a long short-term memory (LSTM) neural network is used to learn the evolution trajectory of fusion features in the latent space, generating a rehabilitation trend feature vector. The LSTM is used to perform time-series modeling of the fusion feature vector sequence of continuous rehabilitation cycles to capture the dynamic impact of rehabilitation intervention on changes in image phenotypes. Specifically, a three-layer LSTM unit is constructed, each including an input gate, a forget gate, and an output gate to control the retention and updating of information in the time series. The fusion feature vectors generated based on deep canonical correlation analysis are input into the network in the order of the rehabilitation cycles. The network parameters are trained using backpropagation combined with time step unfolding (BPTT), while cross-validation is used to prevent overfitting. After training, the LSTM neural network can learn the evolution law of fusion features in the latent space and generate a rehabilitation trend feature vector.

[0068] In this embodiment of the invention, the rehabilitation effect prediction module 240 is used to construct a rehabilitation effect prediction model based on changes in image phenotypic patterns and changes in rehabilitation trends, and output rehabilitation prediction results under the specific influence of the current rehabilitation plan for the patient, including short-term rehabilitation prediction indicators and long-term rehabilitation prediction indicators.

[0069] A rehabilitation effect prediction model is constructed based on changes in imaging phenotypic features and rehabilitation trend changes. This model can simultaneously capture the immediate response characteristics of tumor regions to rehabilitation drugs and the cumulative efficacy changes of patients over multiple rehabilitation cycles. Imaging phenotypic changes reflect short-term alterations in tissue structure, texture distribution, and metabolic characterization after drug targets act on tumor cells, while rehabilitation trend changes reveal the long-term dynamic correlation between rehabilitation intensity, immune activation level, and disease progression. By performing time-series correlation modeling on the synergistic response feature vector and trend feature vector, a comprehensive quantification of individualized rehabilitation effects for patients is achieved. The specific execution steps of the rehabilitation effect prediction module include:

[0070] Obtain the collaborative response feature vector generated by the image-drug action mechanism collaborative unit and the rehabilitation trend feature vector generated by the image-rehabilitation dual-modal co-learning unit;

[0071] A rehabilitation effect prediction model is constructed based on a time series graph neural network. By performing time-series correlation modeling on the synergistic response feature vector and the rehabilitation trend feature vector, short-term and long-term rehabilitation prediction indicators for patients under the current rehabilitation plan are generated. The short-term rehabilitation prediction indicators include the lesion volume shrinkage rate and the metabolic activity decline rate. The long-term rehabilitation prediction indicators include the probability of complete pathological remission and the probability of progression-free survival. The rehabilitation effect prediction model is constructed based on a time series graph neural network (TGNN). The core is to model the synergistic response feature vector and the rehabilitation trend feature vector extracted in each rehabilitation cycle into a time series graph structure. Specifically, the continuous rehabilitation cycle of each patient is defined as a time series node. The time series node contains the concatenation of the synergistic response feature vector and the rehabilitation trend feature vector corresponding to the rehabilitation cycle. The time series nodes are connected to each other through two types of edges: (1) time series edges, which connect the nodes of adjacent cycles in the order of rehabilitation time to capture the continuous evolution of the therapeutic effect; (2) feature similarity edges, which establish connections after performing k-nearest neighbor search on the nodes in the feature space to capture non-time series but phenotypic similar rehabilitation responses. The training model involves inputting a temporal graph structure into a temporal graph neural network. By combining a gated graph neural network with temporal convolution, it simultaneously performs graph space neighborhood information aggregation and temporal dimension dependency learning. During training, corresponding patient data is used to train protocol-specific sub-models for different rehabilitation types. The optimization objective is to minimize the joint loss function between short-term rehabilitation prediction indicators, long-term rehabilitation prediction indicators, and the actual clinical endpoint. The joint loss function can be the sum of mean squared error loss and cross-entropy loss. The mean squared error term is used for regression tasks to optimize short-term rehabilitation prediction indicators, and the cross-entropy loss term is used for classification tasks to optimize long-term rehabilitation prediction indicators.

[0072] Please see Figure 2 , Figure 2 This is a schematic diagram of the overall process of the digital breast cancer AI-assisted diagnosis method based on multimodal fusion provided in this embodiment of the invention, which specifically includes the following steps:

[0073] S100. Obtain the patient's digital breast imaging data and rehabilitation plan data during the rehabilitation period. The digital breast imaging data includes digital mammography, ultrasound, magnetic resonance imaging, and infrared thermal imaging. The rehabilitation plan data includes rehabilitation type, drug type, drug dosage, drug component target data, rehabilitation period, immune activation indicators, and rehabilitation plan side effect data. The rehabilitation type can be any one of chemotherapy, radiotherapy, targeted therapy, and immunotherapy.

[0074] S200. Acquire acquisition operation metadata and, based on the acquisition operation metadata and rehabilitation program side effect data, assess and correct the acquisition quality of digital breast imaging data. Specific steps include:

[0075] Acquire acquisition operation metadata during the digital breast imaging process. Acquisition operation metadata includes compression thickness, compression force, tube voltage, tube current exposure, and glandular density.

[0076] Extract the side effect data of the rehabilitation program from the rehabilitation program data. The side effect data of the rehabilitation program includes the degree of tissue edema and the level of skin reaction.

[0077] Based on the collected operation metadata and rehabilitation program side effect data, combined with multilayer perceptron, the image noise level, contrast reduction degree and tissue overlap blur are predicted;

[0078] Adaptive correction is performed on digital breast imaging data based on image quality assessment results. The adaptive correction includes adaptive filtering based on image noise level, contrast enhancement based on contrast reduction degree, and image sharpening based on tissue overlap blur.

[0079] S300: Feature extraction is performed on the corrected digital breast imaging data and rehabilitation program data to assess changes in imaging phenotypes and rehabilitation trends. The specific assessment steps for changes in imaging phenotypes include:

[0080] Based on the drug component target data in the rehabilitation program data, a molecular signaling pathway feature set corresponding to the drug is constructed. The molecular signaling pathway feature set includes the gene expression profile of the drug target, the receptor activation mode, and the response characteristics of downstream signaling pathway nodes.

[0081] Multi-scale texture and morphological features were extracted from the corrected digital breast imaging data to generate an image texture atlas that describes the microstructural state of tumor tissue.

[0082] Based on molecular signaling pathway feature sets and image texture maps, and combined with cross-modal Transformer, the correlation strength between image features and drug action signals is evaluated, drug response phenotype regions within tumor areas are identified, and a synergistic response feature vector of image and drug action is generated.

[0083] The specific steps for assessing changes in rehabilitation trends include:

[0084] Extract drug types, drug dosages, rehabilitation cycles, and immune activation indicators from rehabilitation program data and construct a rehabilitation feature vector;

[0085] Image features were extracted from the corrected digital breast imaging data to obtain image feature vectors that characterize tumor morphological and structural changes, blood perfusion characteristics, and tissue heterogeneity.

[0086] Based on deep canonical correlation analysis, rehabilitation feature vectors and imaging feature vectors are mapped to a unified latent space and a fused feature vector is generated, thereby achieving feature alignment between rehabilitation intervention factors and imaging phenotypes.

[0087] Based on the fusion feature vector of continuous rehabilitation cycle, combined with long short-term memory neural network to learn the evolution trajectory of fusion features in latent space, a rehabilitation trend feature vector is generated.

[0088] S400. Based on changes in imaging phenotypic patterns and rehabilitation trends, a rehabilitation outcome prediction model is constructed, outputting rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term and long-term rehabilitation prediction indicators. Specific steps include:

[0089] Obtain the collaborative response feature vector generated by the image-drug action mechanism collaborative unit and the rehabilitation trend feature vector generated by the image-rehabilitation dual-modal co-learning unit;

[0090] A rehabilitation effect prediction model is constructed based on a time series graph neural network. By performing time-series correlation modeling on the synergistic response feature vector and the rehabilitation trend feature vector, short-term and long-term rehabilitation prediction indicators for patients under the current rehabilitation plan are generated. The short-term rehabilitation prediction indicators include the lesion volume shrinkage rate and the metabolic activity decline rate, while the long-term rehabilitation prediction indicators include the probability of complete pathological remission and the probability of progression-free survival.

[0091] The parameters and steps in the above-described digital breast AI-assisted diagnosis method based on multimodal fusion of the present invention can be referred to the parameters and steps of each unit module in the above-described embodiments of the digital breast AI-assisted diagnosis system based on multimodal fusion, which are used to implement the corresponding functions, and will not be repeated here.

[0092] Please refer to Figure 3The embodiments of the present invention also provide an electronic device, the electronic device including a processor, an internal bus, a network interface, an internal memory, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement any of the digital breast artificial intelligence-assisted diagnosis methods based on multimodal fusion as described in the embodiments of the present invention.

[0093] The processor can be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement any of the digital breast AI-assisted diagnostic methods based on multimodal fusion as described in the embodiments of the present invention.

[0094] The processor can also be an integrated circuit electronic device with signal processing capabilities. In implementation, each step of any of the multimodal fusion-based digital breast cancer AI-assisted diagnostic methods in this invention can be completed through integrated logic circuits in the processor's hardware or through software instructions.

[0095] The aforementioned processor can also be a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the functions required by the units included in the data processing apparatus of the embodiments of this invention, or executes any of the digital breast cancer AI-assisted diagnostic methods based on multimodal fusion in the embodiments of this invention.

[0096] An internal bus may include a pathway for transmitting information between the aforementioned components. The internal bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Internal buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 3 The symbol is represented by a single double arrow, but this does not mean that there is only one bus or one type of bus.

[0097] A network interface is used to enable data interaction between electronic devices and external communication networks. The network interface can be an Ethernet interface, an optical fiber interface, or a wireless communication interface.

[0098] Internal memory provides temporary data read and write space for electronic devices. Internal memory can adopt random access memory (RAM), cache, or other storage structures with fast read and write capabilities. Internal memory exchanges data with the processor at high speed through an internal bus.

[0099] The memory is used to store the operating system, computer program, and database of the electronic device for a long time, ensuring that the system can maintain the integrity and reliability of the data in the event of power failure or restart. The operating system can be embedded Linux, real-time operating system (RTOS), or other software platform that can manage system resources and support secure communication protocols. The computer program is used to implement any of the digital breast AI-assisted diagnostic methods based on multimodal fusion as described in the embodiments of the present invention. The database is used to store the running data of the computer program.

[0100] This invention provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described in the above embodiments, which is a digital breast AI-assisted diagnostic method based on multimodal fusion.

[0101] In this embodiment of the invention, the computer-readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium can be a portable computer disk, a hard disk, a USB flash drive, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), lectern random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, optical disk, magnetic disk, mechanical encoding device, or any combination thereof.

[0102] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0103] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions claimed in this invention.

Claims

1. A digital breast cancer AI-assisted diagnostic system based on multimodal fusion, characterized in that, It includes a multimodal data acquisition module, an image quality correction module, a feature modeling and evaluation module, and a rehabilitation effect prediction module; The multimodal data acquisition module is used to acquire digital breast imaging data and rehabilitation plan data of the patient during the rehabilitation period. The image quality correction module is used to acquire acquisition operation metadata and, based on the acquisition operation metadata and rehabilitation program side effect data, to assess and correct the acquisition quality of digital breast imaging data. The feature modeling and evaluation module is used to extract features from the corrected digital breast imaging data and rehabilitation program data. It includes an image-drug mechanism synergy unit for evaluating changes in image phenotype and an image-rehabilitation bimodal co-learning unit for evaluating changes in rehabilitation trends. The rehabilitation effect prediction module is used to construct a rehabilitation effect prediction model based on changes in imaging phenotypic patterns and changes in rehabilitation trends, and output rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term rehabilitation prediction indicators and long-term rehabilitation prediction indicators. The specific execution steps of the image quality correction module include: Acquire acquisition operation metadata during the digital breast imaging process. Acquisition operation metadata includes compression thickness, compression force, tube voltage, tube current exposure, and glandular density. Extract the side effect data of the rehabilitation program from the rehabilitation program data. The side effect data of the rehabilitation program includes the degree of tissue edema and the level of skin reaction. Based on the collected operation metadata and rehabilitation program side effect data, combined with multilayer perceptron, the image noise level, contrast reduction degree and tissue overlap blur are predicted; Adaptive correction is performed on digital breast imaging data based on image quality assessment results. Adaptive correction includes adaptive filtering based on image noise level, contrast enhancement based on contrast reduction degree, and image sharpening based on tissue overlap blur. The specific execution steps of the imaging-drug action mechanism synergistic unit include: Based on the drug component target data in the rehabilitation program data, a molecular signaling pathway feature set corresponding to the drug is constructed. The molecular signaling pathway feature set includes the gene expression profile of the drug target, the receptor activation mode, and the response characteristics of downstream signaling pathway nodes. Multi-scale texture and morphological features were extracted from the corrected digital breast imaging data to generate an image texture atlas that describes the microstructural state of tumor tissue. Based on molecular signaling pathway feature sets and image texture maps, and combined with cross-modal Transformer, the correlation strength between image features and drug action signals is evaluated, drug response phenotype regions within tumor areas are identified, and a synergistic response feature vector of image and drug action is generated. The specific execution steps of the image-rehabilitation bimodal co-learning unit include: Extract drug types, drug dosages, rehabilitation cycles, and immune activation indicators from rehabilitation program data and construct a rehabilitation feature vector; Image features were extracted from the corrected digital breast imaging data to obtain image feature vectors that characterize tumor morphological and structural changes, blood perfusion characteristics, and tissue heterogeneity. Based on deep canonical correlation analysis, rehabilitation feature vectors and imaging feature vectors are mapped to a unified latent space and a fused feature vector is generated, thereby achieving feature alignment between rehabilitation intervention factors and imaging phenotypes. Based on the fusion feature vector of continuous rehabilitation cycle, combined with long short-term memory neural network, the evolution trajectory of fusion features in latent space is learned to generate rehabilitation trend feature vector. The specific execution steps of the rehabilitation effect prediction module include: Obtain the collaborative response feature vector generated by the image-drug action mechanism collaborative unit and the rehabilitation trend feature vector generated by the image-rehabilitation dual-modal co-learning unit; A rehabilitation effect prediction model is constructed based on a time series graph neural network. By performing time-series correlation modeling on the synergistic response feature vector and the rehabilitation trend feature vector, short-term and long-term rehabilitation prediction indicators for patients under the current rehabilitation plan are generated. The short-term rehabilitation prediction indicators include the lesion volume shrinkage rate and the metabolic activity decline rate, while the long-term rehabilitation prediction indicators include the probability of complete pathological remission and the probability of progression-free survival.

2. The digital breast cancer AI-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The multimodal data acquisition module is used to acquire digital breast imaging data and rehabilitation plan data of patients during the rehabilitation period. The digital breast imaging data includes digital mammography, ultrasound, magnetic resonance imaging and infrared thermal imaging. The rehabilitation plan data includes rehabilitation type, drug type, drug dosage, drug component target data, rehabilitation period, immune activation indicators and rehabilitation plan side effect data. The rehabilitation type can be any one of chemotherapy, radiotherapy, targeted therapy and immunotherapy.

3. A digital breast cancer AI-assisted diagnostic method based on multimodal fusion, applied to the digital breast cancer AI-assisted diagnostic system based on multimodal fusion as described in any one of claims 1-2, characterized in that, The method includes: Acquire digital breast imaging data and rehabilitation plan data of patients during the rehabilitation period; Acquire metadata of the acquisition operation and, based on the metadata and side effect data of the rehabilitation program, assess and correct the acquisition quality of digital breast imaging data. Feature extraction was performed on the corrected digital breast imaging data and rehabilitation program data to assess changes in imaging phenotypes and rehabilitation trends. A rehabilitation outcome prediction model is constructed based on changes in imaging phenotypic patterns and rehabilitation trends. The model outputs rehabilitation prediction results under the specific influence of the patient's current rehabilitation plan, including short-term and long-term rehabilitation prediction indicators.

4. An electronic device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes the digital breast AI-assisted diagnosis method based on multimodal fusion as described in claim 3 by calling the computer program stored in the memory.

5. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the digital breast cancer AI-assisted diagnostic method based on multimodal fusion as described in claim 3.

Images