Breast cancer multi-mode image collaborative diagnosis system and method thereof

Abstract

The invention relates to the technical field of medical information systems, in particular to a breast cancer multi-mode image collaborative diagnosis system and method, and the system comprises an image collection module which collects the image data of a mammary gland molybdenum target mode, an ultrasonic mode and a magnetic resonance mode of a patient; the feature extraction module is in communication connection with the image acquisition module; the data processing module is in communication connection with the feature extraction module; the data analysis module is in communication connection with the data processing module and receives the initial diagnosis result sent by the data processing module; analyzing the initial diagnosis result to generate a diagnosis analysis result; the judgment module is in communication connection with the data analysis module and receives the diagnosis analysis result sent by the data analysis module; determining a treatment suggestion according to the diagnosis analysis result; the automatic examination module is in communication connection with the judgment module and triggers re-examination according to the treatment suggestions; and outputting a final diagnosis result and a treatment suggestion, making full use of the advantages of each mode, and significantly improving the feature extraction effect.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information systems, and in particular to a breast cancer multimodal imaging collaborative diagnosis system and method thereof. Background Art

[0002] As one of the most common malignant tumors in women, early diagnosis and precise treatment of breast cancer are crucial to improving the survival rate and quality of life of patients. With the continuous advancement of medical imaging technology, various imaging methods such as molybdenum target X-ray, ultrasound examination and magnetic resonance imaging are playing an increasingly important role in the diagnosis of breast cancer. However, a single imaging modality is often difficult to fully reflect the pathological changes of breast tissue, which can easily lead to missed diagnosis or misdiagnosis.

[0003] In recent years, multimodal image fusion diagnosis has gradually become a research hotspot. Existing multimodal breast imaging diagnostic systems usually use simple methods such as feature splicing or weighted averaging for feature fusion, which makes it difficult to fully explore the complementary information between different modalities. In addition, these systems often lack adaptive learning capabilities and find it difficult to continuously optimize diagnostic performance based on new data. In clinical practice, factors such as the time interval between different modality examinations and individual differences among patients also pose challenges to multimodal fusion diagnosis.

[0004] In the existing technology, some systems try to use deep learning methods to improve diagnostic accuracy, but the following problems often exist: first, for modalities with small amounts of data (such as magnetic resonance imaging), the model is prone to overfitting; second, the data distribution between different modalities varies greatly, and direct fusion is prone to introduce noise; third, the diagnostic results lack interpretability and are difficult to gain the trust of clinicians; finally, these systems usually only focus on a single diagnosis and lack consideration of long-term follow-up and management of patients.

[0005] In view of the above problems, there is an urgent need for an intelligent diagnostic system that can effectively integrate multimodal breast imaging information, has adaptive learning capabilities, and can provide personalized diagnosis and treatment plans. Summary of the invention

[0006] The present invention aims to solve the technical problems existing in the existing multimodal imaging diagnosis system for breast cancer, such as insufficient feature fusion, insufficient adaptive ability, lack of interpretability, and neglect of long-term follow-up. By innovatively designing multimodal feature extraction and fusion strategies, introducing a dynamic weight adjustment mechanism, and combining reinforcement learning and knowledge graph technology, the present invention proposes a comprehensive, intelligent, and sustainably optimized multimodal imaging collaborative diagnosis system and method for breast cancer.

[0007] The present invention proposes a breast cancer multimodal imaging collaborative diagnosis system, comprising:

[0008] Image acquisition module for:

[0009] Collect the patient's mammography, ultrasound and magnetic resonance imaging data;

[0010] Preprocessing the image data;

[0011] A feature extraction module is connected to the image acquisition module for:

[0012] Receiving the pre-processed image data sent by the image acquisition module;

[0013] Extracting multimodal image features based on the preprocessed image data;

[0014] A data processing module is connected to the feature extraction module for:

[0015] Receiving the multimodal image features sent by the feature extraction module;

[0016] Based on the multimodal imaging features, generating an initial diagnosis result;

[0017] A data analysis module is connected to the data processing module for:

[0018] Receiving the initial diagnosis result sent by the data processing module;

[0019] Analyzing the initial diagnosis result to generate a diagnosis analysis result;

[0020] A judgment module, which is in communication with the data analysis module, is used to:

[0021] Receiving the diagnostic analysis result sent by the data analysis module;

[0022] Determining treatment recommendations based on the diagnostic analysis results;

[0023] The automatic inspection module is connected to the judgment module for:

[0024] triggering a reexamination based on the treatment recommendations;

[0025] A data output module is connected to the judgment module and the automatic inspection module for:

[0026] Output final diagnosis results and treatment recommendations.

[0027] Preferably, the feature extraction module comprises:

[0028] Mammography feature extraction unit, used to extract mammography image features;

[0029] An ultrasonic feature extraction unit, used for extracting breast ultrasonic image features;

[0030] A magnetic resonance feature extraction unit, used for extracting breast magnetic resonance image features;

[0031] A multimodal feature fusion unit is communicatively connected with the molybdenum target feature extraction unit, the ultrasound feature extraction unit and the magnetic resonance feature extraction unit, and is used for fusing the breast molybdenum target image feature, the breast ultrasound image feature and the breast magnetic resonance image feature.

[0032] Preferably, the feature extraction module further comprises:

[0033] Feature weight transfer learning unit, used for:

[0034] Pre-training the feature extraction network using mammography and ultrasound image data;

[0035] Fine-tuning the feature extraction network using breast magnetic resonance imaging data;

[0036] The optimized feature extraction network is used to extract multimodal image features.

[0037] Preferably, the data processing module comprises:

[0038] A fully connected network unit, used for mapping the multimodal image features;

[0039] a classifier unit, communicatively connected to the fully connected network unit, and configured to calculate a classification probability based on the mapped features;

[0040] A dynamic weight adjustment unit is connected to the classifier unit for adjusting the weight parameters of the feature extraction module in real time according to the classification probability.

[0041] Preferably, the data analysis module comprises:

[0042] A reinforcement learning model unit, used for learning and optimizing the initial diagnosis result;

[0043] A classification decision network unit, which is in communication connection with the reinforcement learning model unit and is used to generate a classification decision according to the learning result of the reinforcement learning model unit;

[0044] An adaptive weight factor unit is communicatively connected with the reinforcement learning model unit and the classification decision network unit, and is used to dynamically adjust the weight according to the importance of different modes.

[0045] As a preference, it also includes:

[0046] A data storage module, which is in communication with the image acquisition module, the data processing module and the data analysis module, and is used to store patient information, image data, diagnosis results and historical medical records;

[0047] The intelligent evaluation module is communicatively connected with the data storage module and the data output module, and is used to perform intelligent evaluation based on the patient information and the diagnosis result to generate personalized treatment suggestions.

[0048] Preferably, the image acquisition module comprises:

[0049] A mammography image acquisition unit, used for acquiring mammography images;

[0050] An ultrasonic image acquisition unit, used for acquiring breast ultrasonic images;

[0051] A magnetic resonance image acquisition unit, used for acquiring breast magnetic resonance images;

[0052] The image preprocessing unit is communicatively connected with the mammography image acquisition unit, the ultrasound image acquisition unit and the magnetic resonance image acquisition unit, and is used for preprocessing the acquired images, including image enhancement and annotation.

[0053] Preferably, the automatic inspection module comprises:

[0054] Mammography automatic inspection unit, used to trigger re-inspection of mammography images;

[0055] An automatic ultrasound examination unit, used to trigger a re-examination of ultrasound images;

[0056] The inspection strategy adjustment unit is communicatively connected with the molybdenum target automatic inspection unit and the ultrasonic automatic inspection unit, and is used to adjust the inspection strategy according to the diagnosis result.

[0057] As a preference, it also includes:

[0058] An intelligent follow-up module, which is in communication with the data output module and the data storage module, is used to:

[0059] Generate a follow-up plan based on patient diagnosis and treatment information;

[0060] using reinforcement learning methods to optimize the follow-up plan;

[0061] The patients' disease progression and survival were tracked.

[0062] The diagnostic method based on the breast cancer multimodal imaging collaborative diagnosis system comprises the following steps:

[0063] S1, collecting the patient's breast mammography, ultrasound and magnetic resonance imaging data, and preprocessing the imaging data;

[0064] S2. Using a feature extraction module to extract features from the preprocessed image data to obtain multimodal image features;

[0065] S3, the data processing module generates an initial diagnosis result based on the multimodal image features;

[0066] S4, the data analysis module analyzes the initial diagnosis result to generate a diagnosis analysis result;

[0067] S5, the judgment module determines the treatment recommendation according to the diagnosis and analysis results;

[0068] S6. The automatic inspection module triggers another inspection as needed;

[0069] S7. The data output module outputs the final diagnosis results and treatment recommendations.

[0070] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0071] First, in terms of the overall system architecture, the present invention realizes the intelligence of the entire process from image acquisition to the output of diagnostic results through modular design. The functional modules work closely together to form a closed-loop diagnosis-follow-up-optimization system. This design not only improves the accuracy and efficiency of diagnosis, but also provides technical support for long-term patient management.

[0072] Secondly, in terms of multimodal fusion, the present invention innovatively adopts a feature weight transfer learning strategy. By pre-training with molybdenum target and ultrasound data and then fine-tuning with magnetic resonance data, the problem of inconsistent distribution of data in different modalities is effectively solved. This method fully utilizes the advantages of each modality and significantly improves the effect of feature extraction, especially for magnetic resonance images with a small sample size.

[0073] In the data processing and analysis phase, the present invention introduces a dynamic weight adjustment mechanism and a decision optimization algorithm based on reinforcement learning. These innovative designs enable the system to have adaptive learning capabilities and continuously optimize diagnostic strategies based on new data. Especially when facing complex or rare cases, the system can gradually improve the diagnostic accuracy through continuous learning and adjustment.

[0074] In terms of the interpretability of diagnostic results, the present invention combines the attention mechanism and knowledge graph technology. By visualizing key feature areas and using medical knowledge graphs for reasoning, the system can provide a clear explanation for the diagnostic results. This greatly enhances the credibility of the system and helps to gain recognition and adoption from clinicians.

[0075] In terms of long-term follow-up management, the intelligent follow-up module of the present invention adopts a follow-up plan generation algorithm based on reinforcement learning. This enables the system to dynamically adjust the follow-up strategy according to the individual characteristics of the patient and the progression of the disease, thereby achieving personalized long-term health management.

[0076] Finally, the automatic inspection module of the present invention optimizes the inspection strategy through the multi-armed tiger algorithm, which improves the accuracy of diagnosis while effectively balancing the utilization efficiency of medical resources. This design is not only conducive to timely detection of potential risks, but also reduces unnecessary excessive inspections and relieves the burden on patients.

[0077] In summary, the multimodal imaging collaborative diagnosis system and method for breast cancer proposed in the present invention has achieved a comprehensive improvement in diagnostic accuracy, system adaptability, result interpretability, and long-term management capabilities through the organic combination of multiple innovative technologies. This system is expected to play an important role in clinical practice, providing strong technical support for the early diagnosis, precision treatment, and long-term management of breast cancer, and ultimately significantly improving patient prognosis and quality of life. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a system logic block diagram of the present invention.

[0079] Figure 2 It is a logic block diagram of the image acquisition module of the present invention.

[0080] Figure 3 It is a logic block diagram of the feature extraction module of the present invention.

[0081] Figure 4 It is a logic block diagram of the data processing module of the present invention.

[0082] Figure 5 It is a logic block diagram of the data analysis module of the present invention.

[0083] Figure 6 It is a logic block diagram of the automatic inspection module of the present invention.

[0084] Figure 7 It is a logic block diagram of the intelligent follow-up module of the present invention.

[0085] Figure 8 It is a logic block diagram of the data storage, block intelligent evaluation module and intelligent follow-up module of the present invention. DETAILED DESCRIPTION

[0086] See also Figure 1-8 The present invention proposes a multimodal imaging collaborative diagnosis system for breast cancer, which includes an image acquisition module 1, a feature extraction module 2, a data processing module 3, a data analysis module 4, a judgment module 5, an automatic inspection module 6 and a data output module 7.

[0087] The image acquisition module 1 is used to collect the image data of the patient's breast mammography, ultrasound and magnetic resonance in three modalities, and pre-process the image data. In a preferred embodiment of the present invention, the image acquisition module 1 uses a fully digital mammography DR system to collect mammography images, uses an ultrasound scanner to collect ultrasound images, and obtains magnetic resonance images through an MRI magnetic resonance imaging system. The preprocessing process includes image enhancement and annotation, wherein image enhancement can use techniques such as histogram equalization to improve image contrast, and annotation mainly performs special markings on high brightness and missing areas to provide important information for subsequent processing.

[0088] The feature extraction module 2 is communicatively connected with the image acquisition module 1, and is used to receive the preprocessed image data sent by the image acquisition module 1, and extract multimodal image features based on the preprocessed image data. The present invention adopts an innovative feature weight transfer learning strategy to effectively solve the problem of inconsistent distribution of multimodal data. Specifically, the feature extraction module 2 first pre-trains the feature extraction network using breast mammography and ultrasound image data, then fine-tunes the feature extraction network using breast magnetic resonance image data, and finally uses the optimized feature extraction network to extract multimodal image features.

[0089] In another embodiment of the present invention, the feature extraction module 2 includes a molybdenum target feature extraction unit 21, an ultrasound feature extraction unit 22, a magnetic resonance feature extraction unit 23 and a multimodal feature fusion unit 24. These units are responsible for extracting image features of different modalities, and realizing feature fusion through the multimodal feature fusion unit 24. During the fusion process, the present invention adopts a multi-head attention mechanism to realize adaptive weight fusion of different modal features by calculating query vectors, key vectors and value vectors. The fusion formula is as follows:

[0090]

[0091] Among them, Q, K, and V represent query, key, and value matrices respectively, and d k is the dimension of the key vector. This method can effectively capture the complex relationship between different modalities and improve the fusion effect.

[0092] The data processing module 3 is in communication with the feature extraction module 2, and is used to receive the multimodal image features sent by the feature extraction module 2 and generate an initial diagnosis result based on the multimodal image features. In one embodiment of the present invention, the data processing module 3 uses a fully connected neural network to map the features, and then uses a Softmax classifier to calculate the classification probability. The classification probability calculation formula is as follows:

[0093]

[0094] Among them, y i represents the i-th category, x is the input feature, z i is the score of the corresponding category. An innovative point of the present invention is the introduction of a dynamic weight adjustment mechanism. The data processing module 3 adjusts the weight parameters of the feature extraction network in real time according to the classification results, so that the system has adaptive learning capabilities. The weight adjustment adopts the back propagation algorithm, and the update formula is:

[0095]

[0096] Among them, w ij represents the weight, η is the learning rate, and E is the error function. The learning rate η is usually set between 0.001 and 0.1. The present invention adopts a dynamic adjustment strategy, with the initial value set to 0.01, and gradually decreases as the training progresses.

[0097] Through the above design, the breast cancer multimodal imaging collaborative diagnosis system of the present invention can effectively fuse image information of different modalities and improve the accuracy and reliability of diagnosis. The adaptive learning ability of the system enables it to continuously optimize performance and adapt to individual differences of different patients. In clinical practice, the system is expected to significantly improve the efficiency and accuracy of breast cancer diagnosis and provide powerful decision-making support for doctors. The feature extraction module 2 of the present invention includes a molybdenum target feature extraction unit 21, an ultrasound feature extraction unit 22, a magnetic resonance feature extraction unit 23 and a multimodal feature fusion unit 24. This modular design enables the system to perform specialized feature extraction for image data of different modalities, thereby making full use of the advantages of various imaging technologies.

[0098] The molybdenum target feature extraction unit 21 is mainly used to extract the features of mammary gland mammography images. In a preferred embodiment of the present invention, the unit adopts an improved convolutional neural network (CNN) structure, specifically including 5 convolutional layers and 3 fully connected layers. The convolution kernel sizes are 7x7, 5x5, 3x3, 3x3, and 3x3, respectively, and each convolution layer is followed by a maximum pooling operation. This design can effectively extract features such as microcalcifications and masses in mammography images, providing an important basis for subsequent diagnosis.

[0099] The ultrasound feature extraction unit 22 is responsible for extracting breast ultrasound image features. Considering the characteristics of ultrasound images, the present invention introduces an attention mechanism in this unit. Specifically, a dual attention module combining spatial attention and channel attention can better capture the texture and edge information in ultrasound images. The mathematical expression of the attention module is as follows:

[0100]

[0101] Among them, F represents the input feature map, M c and M sdenote channel attention and spatial attention mapping respectively, Represents element-wise multiplication.

[0102] The magnetic resonance feature extraction unit 23 is used to extract breast magnetic resonance image features. An innovative point of the present invention is that the unit adopts a 3D convolutional neural network structure, which can make full use of the three-dimensional information of magnetic resonance images. The network contains 4 3D convolutional layers, the convolution kernel size is 3X3X3, the step size is 1, and the LeakyReLU activation function and 3D maximum pooling are used after each convolution. This design can effectively extract three-dimensional features such as volume, shape and texture in magnetic resonance images. The multimodal feature fusion unit 24 is one of the core components of the present invention and is responsible for fusing features from different modalities.

[0103] In the preferred embodiment of the present invention, a Transformer-based multimodal fusion method is adopted. Specifically, each modal feature is first mapped to the same feature space through linear projection, and then a multi-head self-attention mechanism is used for feature interaction and fusion. The mathematical expression of the fusion process is as follows:

[0104] Z=MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0105] Where Q, K, V represent query, key, and value matrices respectively, h is the number of attention heads, and W O is the output projection matrix. In the present invention, the number of attention heads h is set to 8, which is an empirical value obtained through a large number of experiments, and can achieve a good balance between computational efficiency and expression ability. The feature weight transfer learning unit is another innovation of the present invention, which is used to solve the problem of inconsistent distribution of multimodal data. The unit first pre-trains the feature extraction network using breast mammography and ultrasound imaging data, and then fine-tunes it using breast magnetic resonance imaging data. This strategy can effectively utilize the commonalities of data of different modalities while retaining the particularity of each modality. Specifically, the cross entropy loss function is used in the pre-training stage.

[0106]

[0107] Where N is the number of categories, y i is the true label, is the predicted probability.

[0108] In the fine-tuning stage, the present invention introduces knowledge distillation technology and uses the pre-trained model as a teacher network to guide the feature extraction of magnetic resonance data. The distillation loss function is as follows:

[0109]

[0110] Wherein, T is a temperature parameter, which is usually set between 2 and 4, and 3 is used in the present invention; t and z s They represent the output logits of the teacher network and the student network respectively, KL represents the KL divergence, L CE is the cross entropy loss, and α is the balance parameter, which is set to 0.5.

[0111] Through this feature weight transfer learning strategy, the system of the present invention can better adapt to the data characteristics of different modalities, improve the effect of feature extraction and the accuracy of diagnosis. In practical applications, this strategy significantly reduces the demand for large-scale annotated magnetic resonance data and reduces the deployment cost of the system.

[0112] The data processing module 3 of the present invention includes a fully connected network unit 31, a classifier unit 32 and a dynamic weight adjustment unit 33. The fully connected network unit 31 is used to map multimodal image features, and adopts a three-layer fully connected network structure, with BatchNormalization and ReLU activation functions in each layer, which effectively improves the expression ability of features.

[0113] The classifier unit 32 is in communication connection with the fully connected network unit 31, and uses the Softmax classifier to calculate the classification probability according to the mapped features. The calculation formula of the classification probability is as described above. In one embodiment of the present invention, in order to improve the reliability of classification, the temperature scaling technology is introduced, and the modified Softmax function is as follows:

[0114]

[0115] Wherein, T is a temperature parameter, which is usually greater than 1. The present invention determines that the optimal value of T is 1.5 through cross-validation, which helps to obtain a more calibrated probability output.

[0116] The dynamic weight adjustment unit 33 is another innovative point of the present invention, which is used to adjust the weight parameters of the feature extraction module in real time according to the classification probability. This unit uses the gradient descent method to update the weight, but introduces an adaptive learning rate strategy. Specifically, the Adam optimizer is used, and its parameter update rules are as follows:

[0117] m t =β1m t-1 +(1-β1)g t

[0118]

[0119] Among them, m t and v tare the first-order and second-order moment estimates of the gradient, β1 and β2 are the decay rates, usually set to 0.9 and 0.999, η is the learning rate, and ∈ is a small constant for numerical stability, usually set to 10 -8 .

[0120] This dynamic weight adjustment mechanism enables the system of the present invention to continuously optimize itself, adapt to the individual differences of different patients, and improve the accuracy and robustness of diagnosis. In practical applications, this mechanism significantly improves the generalization ability of the system, enabling it to better handle new cases that have not appeared in the training set.

[0121] Through the above design, the breast cancer multimodal imaging collaborative diagnosis system of the present invention realizes efficient and accurate feature extraction and data processing. The system can make full use of the complementary information of different modal images, and provide reliable decision support for clinical diagnosis through advanced deep learning technology and innovative algorithm strategies. This not only improves the accuracy of diagnosis, but also helps doctors better understand and interpret the diagnosis results, and provides an important basis for the formulation of personalized treatment plans. The data analysis module 4 of the present invention includes a reinforcement learning model unit 41, a classification decision network unit 42 and an adaptive weight factor unit 43. This modular design enables the system to deeply analyze the initial diagnosis results and continuously optimize the decision-making strategy through continuous learning.

[0122] The reinforcement learning model unit 41 is an innovative point of the present invention, which is used to learn and optimize the initial diagnosis results. In a preferred embodiment of the present invention, a deep Q network (DQN) is used as the core algorithm of reinforcement learning. DQN combines Q learning and deep neural network, and can effectively process high-dimensional state space. Its update formula is as follows:

[0123]

[0124] Among them, s t and a t Represent the current state and action respectively, r t is the immediate reward, γ is the discount factor, and α is the learning rate. In the present invention, γ is set to 0.95, the initial value of α is 0.001, and it is dynamically adjusted with training. This setting can achieve a good balance between long-term benefits and immediate rewards, and improve the accuracy and stability of diagnostic decisions. The classification decision network unit 42 is communicated with the reinforcement learning model unit 41, and is used to generate classification decisions based on the learning results of the reinforcement learning model unit. The present invention introduces an attention mechanism in this unit to better capture the correlation between different features. Specifically, a multi-head self-attention mechanism is adopted, and its mathematical expression is as follows:

[0125]

[0126] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0127] Where Q, K, and V represent query, key, and value matrices, respectively. k is the dimension of the key vector, and h is the number of attention heads. In the present invention, h is set to 8, which is an empirical value obtained through a large number of experiments and can achieve a good balance between computational efficiency and expression ability.

[0128] The adaptive weight factor unit 43 is another innovative point of the present invention, which is connected to the reinforcement learning model unit 41 and the classification decision network unit 42 for dynamically adjusting the weights according to the importance of different modalities. The weight adjustment adopts a gradient-based method, and the update formula is as follows:

[0129]

[0130] Among them, w i represents the weight of the i-th modality, η is the learning rate, and L is the loss function. This invention introduces a new loss function that takes into account both the classification accuracy and the complementarity between modalities:

[0131]

[0132] Among them, L CE is the cross entropy loss, sim(f i ,f j ) represents the similarity between the jth and jth modal features, and λ is a balance parameter, which is determined to be 0.1 through cross-validation. This design encourages different modalities to provide complementary information and improves the effect of multimodal fusion.

[0133] The present invention also includes a data storage module 8, which is in communication with the image acquisition module 1, the data processing module 3 and the data analysis module 4, and is used to store patient information, image data, diagnosis results and historical medical records. In one embodiment of the present invention, the data storage module 8 adopts a distributed storage architecture, including a relational database and a non-relational database. A relational database (such as PostgreSQL) is used to store structured patient information and diagnosis results, while a non-relational database (such as MongoDB) is used to store a large amount of image data. This hybrid storage strategy not only ensures the consistency and reliability of the data, but also improves the system's ability to process massive image data.

[0134] The intelligent evaluation module 9 is connected to the data storage module 8 and the data output module 7 for intelligent evaluation based on the patient information and the diagnosis results to generate personalized treatment recommendations. The present invention introduces a reasoning mechanism based on the knowledge graph in the intelligent evaluation module 9. First, a breast cancer diagnosis and treatment knowledge graph is constructed based on medical literature and expert knowledge, and then a graph neural network (GNN) is used to reason on the knowledge graph. The update formula of GNN is as follows:

[0135]

[0136] in, represents the features of node v at the kth layer, N(v) represents the set of neighbor nodes of v, W and b are learnable parameters, and σ is the activation function. Through multi-layer GNN, the system can make full use of the structured information in the knowledge graph to generate more personalized and explainable treatment recommendations for patients.

[0137] The image acquisition module 1 of the present invention includes a breast mammography image acquisition unit 11, an ultrasound image acquisition unit 12, a magnetic resonance image acquisition unit 13 and an image preprocessing unit 14. This modular design enables the system to comprehensively acquire breast images of different modalities to provide a rich data source for subsequent analysis. The breast mammography image acquisition unit 11 adopts a fully digital breast X-ray photography DR system, which has the characteristics of high resolution and low radiation dose. In a preferred embodiment of the present invention, the unit adopts dual-energy imaging technology to acquire two images through high and low energy levels of X-rays, and then obtains a high-contrast mammography image through algorithm synthesis. The mathematical expression of the synthesis algorithm is as follows

[0138] I synth =w1I low +w2I high

[0139] Among them, I low and I high Represent low energy and high energy images respectively, w1 and w2 are weight coefficients, which are automatically optimized by machine learning algorithms. This technology can better display pathological features such as microcalcification and masses, and improve the sensitivity of diagnosis. The ultrasound image acquisition unit 12 uses a high-frequency linear array probe with a frequency range of 7-18MHz. The present invention introduces elastic imaging technology in this unit, which can simultaneously obtain B-ultrasound images and tissue elasticity maps. The generation of elasticity maps is based on tissue strain rate, and the calculation formula is as follows:

[0140]

[0141] Among them, ε represents the strain rate, ΔL is the deformation, and L is the original length. The strain rate is mapped to the image through color coding to provide doctors with additional diagnostic information.

[0142] The magnetic resonance imaging acquisition unit 13 adopts a 3T superconducting magnetic resonance imaging system equipped with a dedicated breast coil. The present invention adds dynamic enhanced scanning (DCE-MRI) and diffusion weighted imaging (DWI) sequences based on conventional T1WI and T2WI sequences. DCE-MRI observes the blood perfusion characteristics of the lesion by injecting contrast agents; DWI reflects the degree of diffusion restriction of the tissue, and the calculation formula of its apparent diffusion coefficient (ADC) is:

[0143]

[0144] Among them, b1 and b2 are different b values, and S1 and S2 are the corresponding signal intensities. These advanced imaging technologies provide important basis for the precise positioning and differential diagnosis of breast lesions. The image preprocessing unit 14 is communicatively connected with the breast mammography image acquisition unit 11, the ultrasound image acquisition unit 12 and the magnetic resonance image acquisition unit 13, and is used to preprocess the acquired images, including image enhancement and annotation. In terms of image enhancement, the present invention adopts super-resolution reconstruction technology based on deep learning. Specifically, a generative adversarial network (GAN) is used for image super-resolution reconstruction, and its objective function is:

[0145]

[0146] Among them, G is the generator network, D is the discriminator network, x is the high-resolution image, and z is the low-resolution image. Through this method, the system can recover more details from low-resolution images and improve the accuracy of subsequent analysis.

[0147] In terms of image annotation, the present invention introduces semi-automatic annotation technology. First, a pre-trained deep learning model is used to perform preliminary segmentation and annotation of the image, and then professional doctors review and correct it. This human-computer collaboration method not only improves the annotation efficiency, but also ensures the annotation quality.

[0148] Through the above design, the breast cancer multimodal imaging collaborative diagnosis system of the present invention realizes comprehensive and high-quality image data acquisition and preprocessing. The system makes full use of various advanced medical imaging technologies to provide a rich and reliable data basis for subsequent feature extraction and diagnostic analysis. This not only improves the accuracy and comprehensiveness of the diagnosis, but also provides important support for the formulation of personalized treatment plans. The automatic inspection module 6 of the present invention includes a molybdenum target automatic inspection unit 61, an ultrasonic automatic inspection unit 62 and an inspection strategy adjustment unit 63. This modular design enables the system to automatically trigger necessary re-inspections based on the preliminary diagnosis results, thereby improving the accuracy and reliability of the diagnosis.

[0149] The automatic molybdenum target inspection unit 61 is used to trigger a re-inspection of the molybdenum target image. In a preferred embodiment of the present invention, the unit adopts an abnormality detection algorithm based on deep learning. Specifically, a variational autoencoder (VAE) is used to model the image features of normal breast tissue, and then the potential abnormal area is identified by calculating the reconstruction error. The loss function of VAE includes two parts: reconstruction error and KL divergence:

[0150] L=E q(z|x) [logp(x|z)]-D KL (q(z|x)||p(z))

[0151] Where x is the input image, z is the latent variable, q(z|x) is the encoder, p(x|z) is the decoder, and p(z) is the prior distribution. When the reconstruction error exceeds the preset threshold (determined to be 0.15 by ROC curve analysis), the system automatically triggers a re-examination. This method can effectively identify small lesions that may be missed in the initial examination.

[0152] The ultrasonic automatic examination unit 62 is used to trigger a re-examination of the ultrasonic image. The present invention introduces an automatic triggering mechanism based on time series analysis in this unit. The system identifies possible lesion progression by comparing the current ultrasonic image with the patient's historical examination data. Specifically, a long short-term memory network (LSTM) is used to model the ultrasonic image sequence. When the lesion risk predicted by the model exceeds a threshold (the empirical value is set to 0.7), the system will recommend a re-ultrasound examination. This time-series-based analysis method can capture the dynamic changes of lesions and improve the sensitivity of diagnosis.

[0153] The inspection strategy adjustment unit 63 is connected to the automatic inspection unit 61 and the automatic ultrasonic inspection unit 62 for adjusting the inspection strategy according to the diagnosis result. The present invention uses a multi-armed bandit algorithm (Multi-Armed Bandit, MAB) in this unit to optimize the inspection strategy. Each inspection method is regarded as an "arm", and the system finds the optimal inspection combination through continuous trial and learning. The upper confidence bound (UCB) algorithm is used to balance exploration and utilization, and its selection strategy is:

[0154]

[0155] in, is the average return of arm a, t is the total number of attempts, n a is the number of times arm a is selected. This adaptive examination strategy can dynamically adjust the examination plan according to the characteristics and examination history of different patients and improve the diagnostic efficiency.

[0156] The present invention also includes an intelligent follow-up module 10, which is communicatively connected to the data output module 7 and the data storage module 8. The design of the intelligent follow-up module 10 is intended to achieve long-term tracking and management of patients, and to provide important support for early detection and prognosis evaluation of breast cancer. In one embodiment of the present invention, the intelligent follow-up module 10 adopts a follow-up plan generation algorithm based on reinforcement learning. The system models the follow-up process as a Markov decision process (MDP), where the state includes the patient's clinical characteristics, imaging characteristics, and biomarkers, etc., and the actions include different follow-up intervals and examination items. A deep Q network (DQN) is used to learn the optimal follow-up strategy, and its Q value update formula is:

[0157]

[0158] Where s and a represent the state and action respectively, r is the reward, α is the learning rate, and γ is the discount factor. The reward function design takes into account multiple factors such as detection rate, patient burden, and medical resource utilization. With this approach, the system is able to generate a personalized, dynamically adjusted follow-up plan for each patient.

[0159] In addition, the intelligent follow-up module 10 also integrates natural language processing (NLP) technology to analyze the unstructured medical text generated during the follow-up process. A BERT-based named entity recognition (NER) model is used to extract key clinical information. Its pre-training objective function includes two parts: masked language model (MLM) and next sentence prediction (NSP):

[0160] L=L MLM +L NSP

[0161] This design can automatically extract important information from texts such as doctors' notes and pathology reports, enrich patients' longitudinal data, and provide strong support for long-term follow-up and prognosis studies.

[0162] The multimodal imaging collaborative diagnosis method for breast cancer of the present invention comprises the following steps:

[0163] S1. Collect the patient's breast mammography, ultrasound and magnetic resonance imaging data in three modalities, and preprocess the image data. In this step, the present invention uses advanced image acquisition technology and preprocessing algorithms. For example, for mammography images, dual-energy imaging technology is used to improve contrast; for ultrasound images, elastic imaging technology is introduced; for magnetic resonance images, dynamic enhancement scanning and diffusion weighted imaging sequences are added. During the preprocessing process, GAN-based super-resolution reconstruction technology is used to improve image quality.

[0164] S2. Use the feature extraction module to extract features from the preprocessed image data to obtain multimodal image features. The present invention innovatively adopts a feature weight transfer learning strategy in this step. First, the network is pre-trained using molybdenum target and ultrasound data, and then fine-tuned using magnetic resonance data. This method effectively solves the problem of inconsistent distribution of multimodal data. The feature extraction network adopts an improved ResNe t structure and introduces an attention mechanism:

[0165] S3, the data processing module generates an initial diagnosis result based on the multimodal image features. In this step, the present invention uses a multi-layer perceptron (MLP) to map the features, and then calculates the probability of each category through a Softmax classifier. In order to improve the reliability of classification, the temperature scaling technology is introduced:

[0166] S4, the data analysis module analyzes the initial diagnosis result and generates a diagnosis analysis result. In this step, the present invention introduces a reinforcement learning model and uses a deep Q network (DQN) to optimize the diagnosis strategy.

[0167] S5. The judgment module determines the treatment recommendation based on the diagnostic analysis results. In this step, the system combines the knowledge graph and the graph neural network (GNN) for reasoning to generate personalized treatment recommendations. The update formula of GNN is:

[0168] S6. The automatic inspection module triggers another inspection as needed. The present invention uses a multi-armed slot machine algorithm to optimize the inspection strategy, and the selection formula is:

[0169] S7, the data output module outputs the final diagnosis results and treatment recommendations. In this step, the system generates a structured diagnosis report and generates an easy-to-understand diagnosis description through natural language generation (NLG) technology.

[0170] Through the above steps, the multimodal imaging collaborative diagnosis method for breast cancer of the present invention realizes the intelligentization of the whole process from image acquisition to the output of diagnostic results. This method makes full use of the complementary information of multimodal images and improves the accuracy and personalization level of diagnosis through advanced machine learning algorithms. At the same time, automatic inspection and intelligent follow-up mechanisms ensure the reliability of diagnosis and long-term management of patients. This comprehensive and intelligent diagnostic method is expected to significantly improve the early detection rate and treatment effect of breast cancer, bringing important value to clinical practice.

[0171] Through the above method, the system of the present invention can realize the security analysis of sensitive data in a dynamic and intelligent manner, taking into account both privacy protection and performance optimization, and has strong practicality and innovation.

Claims

1. A breast cancer multimodal imaging collaborative diagnosis system, characterized in that: include: Image acquisition module for: Collect the patient's mammography, ultrasound and magnetic resonance imaging data; Preprocessing the image data; A feature extraction module is connected to the image acquisition module for: Receiving the pre-processed image data sent by the image acquisition module; Extracting multimodal image features based on the preprocessed image data; A data processing module is connected to the feature extraction module for: Receiving the multimodal image features sent by the feature extraction module; Based on the multimodal imaging features, generating an initial diagnosis result; A data analysis module is connected to the data processing module for: Receiving the initial diagnosis result sent by the data processing module; Analyzing the initial diagnosis result to generate a diagnosis analysis result; The judgment module is connected to the data analysis module for: Receiving the diagnostic analysis result sent by the data analysis module; Determining treatment recommendations based on the diagnostic analysis results; The automatic inspection module is connected to the judgment module for: triggering a reexamination based on the treatment recommendations; A data output module is connected to the judgment module and the automatic inspection module for: Output final diagnosis results and treatment recommendations.

2. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: The feature extraction module comprises: Mammography feature extraction unit, used to extract mammography image features; An ultrasonic feature extraction unit, used for extracting breast ultrasonic image features; A magnetic resonance feature extraction unit, used for extracting breast magnetic resonance image features; A multimodal feature fusion unit is communicatively connected with the molybdenum target feature extraction unit, the ultrasound feature extraction unit and the magnetic resonance feature extraction unit, and is used for fusing the breast molybdenum target image feature, the breast ultrasound image feature and the breast magnetic resonance image feature.

3. The multimodal imaging collaborative diagnosis system for breast cancer according to claim 2, characterized in that: The feature extraction module also includes: Feature weight transfer learning unit, used for: Pre-training the feature extraction network using mammography and ultrasound image data; Fine-tuning the feature extraction network using breast magnetic resonance imaging data; The optimized feature extraction network is used to extract multimodal image features.

4. The multimodal imaging collaborative diagnosis system for breast cancer according to claim 1, characterized in that: The data processing module comprises: A fully connected network unit, used for mapping the multimodal image features; a classifier unit, communicatively connected to the fully connected network unit, and configured to calculate a classification probability based on the mapped features; A dynamic weight adjustment unit is connected to the classifier unit for adjusting the weight parameters of the feature extraction module in real time according to the classification probability.

5. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: The data analysis module includes: A reinforcement learning model unit, used for learning and optimizing the initial diagnosis result; A classification decision network unit, which is in communication connection with the reinforcement learning model unit and is used to generate a classification decision according to the learning result of the reinforcement learning model unit; An adaptive weight factor unit is communicatively connected with the reinforcement learning model unit and the classification decision network unit, and is used to dynamically adjust the weight according to the importance of different modes.

6. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: Also includes: A data storage module, which is in communication with the image acquisition module, the data processing module and the data analysis module, and is used to store patient information, image data, diagnosis results and historical medical records; The intelligent evaluation module is communicatively connected with the data storage module and the data output module, and is used to perform intelligent evaluation based on the patient information and the diagnosis result to generate personalized treatment suggestions.

7. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: The image acquisition module comprises: A mammography image acquisition unit, used for acquiring mammography images; An ultrasonic image acquisition unit, used for acquiring breast ultrasonic images; A magnetic resonance image acquisition unit, used for acquiring breast magnetic resonance images; The image preprocessing unit is communicatively connected with the mammography image acquisition unit, the ultrasound image acquisition unit and the magnetic resonance image acquisition unit, and is used for preprocessing the acquired images, including image enhancement and annotation.

8. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: The automatic inspection module comprises: Mammography automatic inspection unit, used to trigger re-inspection of mammography images; An automatic ultrasound examination unit, used to trigger a re-examination of ultrasound images; The inspection strategy adjustment unit is communicatively connected with the molybdenum target automatic inspection unit and the ultrasonic automatic inspection unit, and is used to adjust the inspection strategy according to the diagnosis result.

9. The breast cancer multimodal imaging collaborative diagnosis system according to claim 1, characterized in that: Also includes: An intelligent follow-up module, which is in communication with the data output module and the data storage module, is used to: Generate a follow-up plan based on patient diagnosis and treatment information; using reinforcement learning methods to optimize the follow-up plan; The patients' disease progression and survival were tracked.

10. A diagnostic method based on the breast cancer multimodal imaging collaborative diagnosis system according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1, collecting the patient's breast mammography, ultrasound and magnetic resonance imaging data, and preprocessing the imaging data; S2. Using a feature extraction module to extract features from the preprocessed image data to obtain multimodal image features; S3, a data processing module generates an initial diagnosis result based on the multimodal image features; S4, the data analysis module analyzes the initial diagnosis result to generate a diagnosis analysis result; S5, the judgment module determines the treatment recommendation according to the diagnosis and analysis results; S6. The automatic inspection module triggers another inspection as needed; S7. The data output module outputs the final diagnosis results and treatment recommendations.

Images