An explainable breast cancer ai prediction method and system based on counterfactual reasoning

Abstract

The application provides an explainable breast cancer AI prediction method and system based on counterfactual reasoning, which comprises the following steps: inputting image data and medical record text data in a training set into a pre-trained medical visual language model (VLM), performing comparative learning fine-tuning, and using the fine-tuned VLM to perform multi-modal feature extraction and cross-modal feature alignment, and outputting a text-image feature alignment mapping table obtained; training a counterfactual classification model integrated with the VLM, in each training round, based on the current model state and the text-image feature alignment mapping table, and a pre-defined medical feature linkage constraint rule, dynamically generating medical credible counterfactual samples, and integrating them into the model training process to realize causal consistency constraint; using the trained counterfactual classification model to predict breast image data to be predicted, outputting an AI prediction result, and outputting an incremental visual semantic explainability path. The application can perform explainable breast cancer AI prediction.

Description

Technical Field

[0001] This invention belongs to the field of AI prediction technology, and specifically refers to an interpretable AI prediction method and system for breast cancer based on counterfactual reasoning. Background Technology

[0002] Existing AI prediction models for disease-aided diagnosis are mostly "black box" structures, unable to provide doctors with understandable and verifiable diagnostic evidence, thus violating the basic principles of "evidence-based medicine".

[0003] To break the black-box structure of models, improving their interpretability becomes particularly important. However, most current research on model interpretability focuses on post-training interpretation. Therefore, there is an urgent need for a new method that interprets the model during training and makes AI predictions. Summary of the Invention

[0004] To address the technical problems existing in the prior art, the present invention provides an interpretable AI prediction method and system for breast cancer based on counterfactual reasoning, the technical solution of which is as follows: On the one hand, an interpretable AI prediction method for breast cancer based on counterfactual reasoning is provided, which includes: S1. Collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesion based on the medical record text data, and divide it into training set and validation set. S2. Input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM) for comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. S3. Train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state and the text-image feature alignment mapping table, as well as predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. S4. Use the trained counterfactual classification model to predict the breast image data to be predicted, output the AI ​​prediction results, and output the progressive visual semantic interpretability path.

[0005] On the other hand, an interpretable AI prediction system for breast cancer based on counterfactual reasoning is provided, the system comprising: The preprocessing module is used to collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesions based on the medical record text data, and divide them into training set and validation set. The alignment extraction module is used to input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM), perform comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. The training module is used to train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module, and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state, the text-image feature alignment mapping table, and predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. The prediction output module is used to make predictions on the breast image data to be predicted using the trained counterfactual classification model, output the AI ​​prediction results, and output the progressive visual semantic interpretability path.

[0006] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described interpretable breast cancer AI prediction method based on counterfactual reasoning.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described interpretable breast cancer AI prediction method based on counterfactual reasoning.

[0008] The beneficial effects of the technical solution provided by this invention include at least the following: This invention enables interpretable AI prediction of breast cancer, specifically: (1) The present invention dynamically generates counterfactual images during training: counterfactual images are generated during model training and used in the model optimization process to achieve the unity of interpretation and decision-making; (2) Medical credibility of the counterfactual images of the present invention: The operation of feature causality assessment and linkage constraint is based on the key lesion feature regions of real benign / malignant samples. The construction and generation process continuously considers the counterfactual characteristics of the generated images to ensure that the generated results conform to the lesion morphology rules. (3) The present invention has high efficiency in generating counterfactual images: it uses the influence function to guide sample selection, focuses on opposing samples that have a substantial impact on the current prediction, and enables the model to learn the key image regions and features of label reversal; (4) The present invention has strong interpretability: it intuitively shows how feature modification drives changes in image regions through multi-image gradient paths, thereby leading to diagnostic reversal. It shows the basis of model diagnostic reasoning in a chain-like path, while providing text-level explanations and quantitative indicators of changes, effectively providing visual semantic level prediction interpretability.

[0009] (5) The present invention has strong applicability: because it meets the basic principles of "evidence-based medicine" and conforms to the diagnostic reasoning steps of ultrasound doctors, it has strong acceptability and field applicability. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of an interpretable AI prediction method for breast cancer based on counterfactual reasoning, provided by an embodiment of the present invention. Figure 2 This is an overall block diagram of an interpretable AI prediction method for breast cancer based on counterfactual reasoning, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of dynamically generated credible medical counterfactual samples provided in an embodiment of the present invention; Figure 4 This is a block diagram of an interpretable AI prediction system for breast cancer based on counterfactual reasoning, provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0013] This invention provides an interpretable AI prediction method for breast cancer based on counterfactual reasoning. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 This is an overall block diagram of an interpretable AI prediction method for breast cancer based on counterfactual reasoning, provided by an embodiment of the present invention. The processing flow may include the following steps:

[0014] S1. Collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesion based on the medical record text data, and divide it into training set and validation set. This invention collects static ultrasound image data and medical record text data from patients at hospitals. Then, data cleaning is performed, using pathological gold standards as the inclusion and exclusion criteria. Only patient image data and their corresponding medical record text data that have been pathologically verified as benign or malignant are retained, while patient data based solely on imaging diagnosis is excluded. For patients with multiple lesions in their image data, only a single dominant lesion (the largest or malignant lesion) is retained to ensure consistent labeling. Simultaneously, images showing blood flow signals from the lesion are discarded. The entire data collection and cleaning process aims to maintain a 1:1 ratio of benign to malignant image data. In addition to the benign / malignant labeling of the image data, the average of the Region of Interest (ROI) labeled by three ultrasound physicians is obtained as the location label of the lesion.

[0015] Based on the obtained medical record text data, and referring to the Breast Imaging Reporting and Data System (BI-RADS) grading report standards, and through consultation with sonographers, causal variables, namely key imaging features, in breast cancer were screened. Ultimately, six lesion features—border, shape, calcification, aspect ratio, echo pattern, and posterior acoustic shadowing—were selected as key causal features.

[0016] Finally, the dataset was constructed. The benign and malignant ultrasound images of patients and their corresponding medical records collected in the early stage were divided into training set, validation set and test set in a ratio of 6:2:2. During the data partitioning process, it was ensured that the data of the same patient were assigned to the same set.

[0017] S2. Input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM) for comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. Optionally, an open-source medical vision-language model (VLM), such as MedCLIP, can be selected as the basic architecture. Based on the existing training parameters, comparative learning fine-tuning is performed using collected breast ultrasound images and medical record text datasets (translated into English). Specifically, a two-stage fine-tuning method is adopted. The first stage performs global comparative learning on all image data (scaled to 224×224 size), and the second stage performs local comparative learning on images cropped from the region of interest (ROI) to enhance the semantic features of the lesion region. During the fine-tuning process, the MedCLIP text encoder is frozen, and only the image encoder (ViT-Base / 16) and image / text projection heads are fine-tuned.

[0018] Optionally, the VLM, for a certain image data After passing through the image encoder (ViT-Base / 16), the global features are output. and spatial feature map For text feature extraction, the input image data... Corresponding medical record sentences (e.g., “The lesion has an irregular margin and posterior shadowing”), after passing through a text encoder, outputs sentence-level features. And word-level features (embedding of each word) ; The cross-modal feature alignment includes two aspects: global feature alignment and local feature alignment. Regarding global feature alignment: Calculate global image-text similarity (cosine similarity). And use the InfoNCE loss function to measure the global contrastive loss. : in, The projection matrix of the VLM is... Represents the text in the current batch. For temperature hyperparameters (e.g.) It can take the value 0.07). Regarding local feature alignment: First, calculate keywords (like ) and spatial features The similarity is used to obtain the attention heatmap. : Regenerate keyword-region mask, for Thresholding segmentation: Set the top 10% of pixel values ​​to 1 and the rest to 0 to obtain the boundary mask region corresponding to a certain key feature. : Recalculate the region-text matching loss: extract the average visual features of the masked regions. And calculate its matching loss with the keyword embedding. : Finally, verify the alignment results and optimize them: If If the IoU between the region with a value of 1 and the image region corresponding to the key causal feature is greater than 0.5, then the alignment is successful (e.g., "margin: irregular" corresponds to the lesion boundary region in the image (ROI outer 10 pixel band)); otherwise, the weights are increased during the fine-tuning stage to force the model to learn the correct mapping. The total loss for cross-modal feature alignment: in, A value of 0.5 is used to balance global feature alignment and local feature alignment.

[0019] After the model is fine-tuned, a text-image feature alignment mapping table can be output simultaneously. The size of this mapping table is... ,in, The number of features is specified. Each row includes the feature name, the image region corresponding to the feature, and the IoU value (optional) between the feature keyword and the corresponding image region.

[0020] S3. Train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state and the text-image feature alignment mapping table, as well as predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. Optionally, such as Figure 3 As shown, the training module is specifically used for: S31. Define the counterfactual problem of the prediction results of the inversion model, analyze the changes in the lesion area in the image data, and formulate counterfactual intervention targets based on the counterfactual problem; The counterfactual question is, for example, "If the model's prediction changes from malignant to benign, how will the lesion area in the imaging data change?"

[0021] S311. Conduct characteristic causality assessment; From each high-confidence malicious sample in the current training batch of Select a high-causal keyword: the boundary feature "irregular" (here, "irregular" is used as an example, but other features can also be used; this embodiment of the invention is not limited, and all are within the protection scope of this embodiment of the invention), and calculate its expected necessity (EN): by analyzing whether the prediction result will necessarily change after eliminating a certain feature, we can infer the necessity of the feature for maintaining the prediction result. The calculation method is as follows: in, To eliminate the "irregular" feature attribute, the predicted probability is obtained by replacing this key feature with the neutral word "unknown" and recoding it. If this proves that the feature is highly necessary for the model to judge it as a malignant lesion, then the feature can be selected as the counterfactual intervention target. S312, Perform linkage feature detection; The predefined medical feature linkage rule base (predefined by ultrasound physicians) is queried to obtain the intervention target "irregular margin" which may be linked to "tissue infiltration". The VLM is then used to calculate the attention heatmap of the linkage features. ,like , If the preset threshold is used, then a linkage feature is determined to exist; S313. After obtaining the linkage characteristics, modify the region expansion; Obtain the main feature region mask from the text-image feature alignment mapping table. (e.g., "irregular" → boundary region). If there are linked features, then obtain its region mask. (e.g., "infiltration" → surrounding tissue area), merging yields the expanded modified area: S32. Retrieval of benign reference samples under linkage constraints; S321. First, construct counterfactual text; Replace "irregular" with "smooth" or "regular" to change irregular boundaries into smooth / regular boundaries, resulting in new text. If there are linked features, the linked words must be replaced at the same time, replacing "infiltration" with "normal tissue"; S322. Among all benign samples in the current batch, select candidate samples that satisfy the linkage constraint. For each candidate image sample The intensity of its linkage characteristics (e.g., using the VLM) is detected. Only samples with "benign main features and normal linkage features" are retained, thus obtaining a candidate set of all benign samples; S323. Using the influence function IF method, select the... The benign sample with the highest opposition strength, i.e., the lowest IF. : in, This represents the candidate set of all benign samples within the current batch. Represents the cross-entropy loss CE. and Representing the first Learning rate and model parameters for each epoch. This represents the gradient operation on the model parameters; S33, Integration of Counterfactual Image Generation and Training; from Extract the extended modification region Corresponding image patch ,replace The corresponding area in: And the generated counterfactual samples Incorporate this into the loss calculation process for the current batch, and calculate the counterfactual consistency loss: Where CE represents cross-entropy; Then, calculate the IF response of the counterfactual sample to the original malicious sample: If the model learns the correct causal features It should be a significantly negative value, meaning that the counterfactual sample strongly suppresses malicious prediction; otherwise, if If the intervention is ineffective, then it is necessary to adjust the model parameters or choose a stronger intervention, while also imposing a penalty term on the model. The total training loss is calculated as follows: in, The classification loss of the counterfactual classification model is... The value is 0.5. and The values ​​are 1.0 and 0.5 respectively. This ensures that the model does not forget the ability to align modal features during training based on counterfactual data. The method of adding counterfactual samples as regularization terms to the total loss ensures the effectiveness of the causal chain by guaranteeing that the third term forces the IF response to be large enough.

[0022] S4. Use the trained counterfactual classification model to perform AI prediction on the breast image data to be predicted, output the AI ​​prediction results, and output the progressive visual semantic interpretability path.

[0023] Optionally, the progressive visual semantic interpretability path consists of three parts: The first part involves the model outputting a counterfactual sequence map that progresses from the original image to the images modified during training, and highlighting the relevant data. The regions form a visual explanatory path; The second part is a numerical interpretation path based on the expected necessity value, IF response value, and confidence change curve as quantification criteria. Part Three: Textual Interpretation Path.

[0024] In this embodiment of the invention, during model training, for each validation set sample, the counterfactual sequence generated in each training epoch is saved. and the corresponding extended modification area .

[0025] During inference, for each new sample, the feature alignment mapping table (for locating image regions), linkage rule base (for locating extended regions), and dynamic IF retrieval (for selecting benign reference samples) learned during training are reused to output a progressive visual semantic interpretability path, consisting of three parts: The first part is that the model outputs a counterfactual sequence map that progresses from the original image to the gradually modified counterfactual sequence map from training, and highlights it. The first part is a visual interpretation path, formed by the region; the second part is a numerical interpretation path using expected necessity value, IF response value, and confidence change curve as quantitative criteria; the third part is a textual interpretation path, such as "A patient's imaging data was judged as malignant by the model due to irregular lesion boundaries and surrounding tissue infiltration (EN=0.52). If both are changed to benign, the confidence of malignancy drops from 95% to 8% (ΔIF = -0.41)".

[0026] like Figure 4As shown, this embodiment of the invention also provides an interpretable breast cancer AI prediction system based on counterfactual reasoning, the system comprising: The collection and preprocessing module 410 is used to collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesion based on the medical record text data, and divide it into training set and validation set. The alignment module 420 is used to input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM), perform comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. Training module 430 is used to train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state and the text-image feature alignment mapping table, as well as predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. The prediction output module 440 is used to predict the breast image data to be predicted using the trained counterfactual classification model, output the AI ​​prediction result, and output the progressive visual semantic interpretability path.

[0027] Optionally, the annotation includes: benign and malignant labeling of image data, and obtaining the average value of the region of interest (ROI) annotated by multiple ultrasound physicians as the location label of the lesion; The key causal features include six lesion characteristics: boundary, shape, calcification, aspect ratio, echo pattern, and posterior acoustic shadowing.

[0028] Optionally, the VLM, for a certain image data After passing through the image encoder, global features are output. and spatial feature map For text feature extraction, the input image data... Corresponding medical record sentences After passing through a text encoder, sentence-level features are output. and word-level features ; The cross-modal feature alignment includes two aspects: global feature alignment and local feature alignment. Regarding global feature alignment: Calculate global image-text similarity And use the InfoNCE loss function to measure the global contrastive loss. : in, The projection matrix of the VLM is... Represents the text in the current batch. Temperature overparameter; Regarding local feature alignment: First, calculate keywords Spatial features The similarity is used to obtain the attention heatmap. : Regenerate keyword-region mask, for Thresholding segmentation: Set the top 10% of pixel values ​​to 1 and the rest to 0 to obtain the boundary mask region corresponding to a certain key feature. : Recalculate the region-text matching loss: extract the average visual features of the masked regions. And calculate its matching loss with the keyword embedding. : Finally, verify the alignment results and optimize them: If If the IoU between the region with a value of 1 and the image region corresponding to the key causal feature is greater than 0.5, then the alignment is successful; otherwise, the weights are increased during the fine-tuning stage to force the model to learn the correct mapping. The total loss for cross-modal feature alignment: in, A value of 0.5 is used to balance global feature alignment and local feature alignment. Same as above.

[0029] Optionally, the counterfactual module is specifically used for: S31. Define the counterfactual problem of the prediction results of the inversion model, analyze the changes in the lesion area in the image data, and formulate counterfactual intervention targets based on the counterfactual problem; S311. Conduct characteristic causality assessment; From each high-confidence malicious sample in the current training batch of We select a high-causality keyword: the boundary feature "irregular", and calculate its expected necessity EN: by analyzing whether the prediction result will necessarily change after eliminating a certain feature, we infer the necessity of that feature for maintaining the prediction result. The calculation method is as follows: in, To eliminate the "irregular" feature attribute, the predicted probability is obtained by replacing this key feature with the neutral word "unknown" and recoding it. If this proves that the feature is highly necessary for the model to judge it as a malignant lesion, then the feature can be selected as the counterfactual intervention target. S312, Perform linkage feature detection; The predefined medical feature linkage rule base is queried to obtain the intervention target "irregular margin" which may be linked to "tissue infiltration". The VLM is then used to calculate the attention heatmap of the linkage features. ,like , If the preset threshold is used, then a linkage feature is determined to exist; S313. After obtaining the linkage characteristics, modify the region expansion; Obtain the main feature region mask from the text-image feature alignment mapping table. If there are linkage features, then obtain their region mask. The merged area results in an expanded modification region: S32. Retrieval of benign reference samples under linkage constraints; S321. First, construct counterfactual text; Replace "irregular" with "smooth" or "regular" to change irregular boundaries into smooth / regular boundaries, resulting in new text. If there are linked features, the linked words must be replaced at the same time, replacing "infiltration" with "normal tissue"; S322. Among all benign samples in the current batch, select candidate samples that satisfy the linkage constraint. For each candidate image sample The intensity of the linkage feature is detected by the VLM, and only samples with "benign main feature and normal linkage feature" are retained, thereby obtaining a candidate set of all benign samples. S323. Using the influence function IF method, select the... The benign sample with the highest opposition strength, i.e., the lowest IF. : in, This represents the candidate set of all benign samples within the current batch. Represents the cross-entropy loss CE. and Representing the first Learning rate and model parameters for each epoch. This represents the gradient operation on the model parameters; S33, Integration of Counterfactual Image Generation and Training; from Extract the extended modification region Corresponding image patch ,replace The corresponding area in: And the generated counterfactual samples Incorporate this into the loss calculation process for the current batch, and calculate the counterfactual consistency loss: Where CE represents cross-entropy; Then, calculate the IF response of the counterfactual sample to the original malicious sample: If the model learns the correct causal features It should be a significantly negative value, meaning that the counterfactual sample strongly suppresses malicious prediction; otherwise, if If the intervention is ineffective, then it is necessary to adjust the model parameters or choose a stronger intervention, while also imposing a penalty term on the model. The total training loss is calculated as follows: in, The classification loss of the counterfactual classification model is... The value is 0.5. and The values ​​are 1.0 and 0.5 respectively. This ensures that the model does not forget the ability to align modal features during training based on counterfactual data. The method of adding counterfactual samples as regularization terms to the total loss ensures the effectiveness of the causal chain by guaranteeing that the third term forces the IF response to be large enough.

[0030] Optionally, the progressive visual semantic interpretability path consists of three parts: The first part involves the model outputting a counterfactual sequence map that progresses from the original image to the images modified during training, and highlighting the relevant data. The regions form a visual explanatory path; The second part is a numerical interpretation path based on the expected necessity value, IF response value, and confidence change curve as quantification criteria. Part Three: Textual Interpretation Path.

[0031] The present invention provides an interpretable breast cancer AI prediction system based on counterfactual reasoning, the functional structure of which corresponds to the interpretable breast cancer AI prediction method based on counterfactual reasoning provided in the present invention, and will not be described again here.

[0032] Figure 5 This is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 501 and one or more memories 502. The memory 502 stores at least one instruction, which is loaded and executed by the processor 501 to implement the steps of the above-described interpretable breast cancer AI prediction method based on counterfactual reasoning.

[0033] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned counterfactual reasoning-based interpretable breast cancer AI prediction method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0034] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interpretable AI prediction method for breast cancer based on counterfactual reasoning, characterized in that, The method includes: S1. Collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesion based on the medical record text data, and divide it into training set and validation set. S2. Input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM), perform comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. S3. Train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state and the text-image feature alignment mapping table, as well as predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. S4. Use the trained counterfactual classification model to predict the breast image data to be predicted, output the AI ​​prediction results, and output the progressive visual semantic interpretability path. The VLM, for a certain image data After passing through the image encoder, global features are output. and spatial feature map For text feature extraction, the input image data... Corresponding medical record sentences After passing through a text encoder, sentence-level features are output. and word-level features ; The cross-modal feature alignment includes two aspects: global feature alignment and local feature alignment. Regarding global feature alignment: Calculate global image-text similarity And use the InfoNCE loss function to measure the global contrastive loss. : in, Let be the projection matrix of the VLM. Represents the text in the current batch. Temperature overparameter; Regarding local feature alignment: First, calculate keywords Spatial features The similarity is used to obtain the attention heatmap. : Regenerate keyword-region mask, for Thresholding segmentation: Set the top 10% of pixel values ​​to 1 and the rest to 0 to obtain the boundary mask region corresponding to a certain key feature. : Recalculate the region-text matching loss: extract the average visual features of the masked regions. And calculate its matching loss with the keyword embedding. : Finally, verify the alignment results and optimize them: If If the IoU between the region with a value of 1 and the image region corresponding to the key causal feature is > 0.5, then the alignment is successful; otherwise, the weights are increased during the fine-tuning stage to force the model to learn the correct mapping. The total loss for cross-modal feature alignment: in, A value of 0.5 is used to balance global feature alignment and local feature alignment.

2. The method according to claim 1, characterized in that, The annotation includes: benign and malignant labeling of image data, and obtaining the average value of the region of interest (ROI) annotated by multiple ultrasound doctors as the location label of the lesion; The key causal features include six lesion characteristics: boundary, shape, calcification, aspect ratio, echo pattern, and posterior acoustic shadowing.

3. The method according to claim 1, characterized in that, The counterfactual module is specifically used for: S31. Define the counterfactual problem of the prediction results of the inversion model, analyze the changes in the lesion area in the image data, and formulate counterfactual intervention targets based on the counterfactual problem; S311. Conduct characteristic causality assessment; From each high-confidence malicious sample in the current training batch of We select a high-causality keyword: the boundary feature "irregular", and calculate its expected necessity EN: by analyzing whether the prediction result will necessarily change after eliminating a certain feature, we infer the necessity of that feature for maintaining the prediction result. The calculation method is as follows: in, To eliminate the "irregular" feature attribute, the predicted probability is obtained by replacing this key feature with the neutral word "unknown" and recoding it. If this proves that the feature is highly necessary for the model to judge it as a malignant lesion, then the feature can be selected as the counterfactual intervention target. S312, Perform linkage feature detection; The predefined medical feature linkage rule base is queried to obtain the intervention target "irregular margin" linked to "tissueinfiltration". The VLM is then used to calculate the attention heatmap of the linkage features. ,like , If the preset threshold is used, then a linkage feature is determined to exist; S313. After obtaining the linkage characteristics, modify the region expansion; Obtain the main feature region mask from the text-image feature alignment mapping table. If there are linkage features, then obtain their region mask. The merged area results in an expanded modification region: S32. Retrieval of benign reference samples under linkage constraints; S321. First, construct counterfactual text; Replace "irregular" with "smooth" or "regular" to change irregular boundaries into smooth / regular boundaries, resulting in new text. If there are linked features, the linked words must be replaced at the same time, replacing "infiltration" with "normaltissue"; S322. Among all benign samples in the current batch, select candidate samples that satisfy the linkage constraint. For each candidate image sample The intensity of the linkage feature is detected by the VLM, and only samples with "benign main feature and normal linkage feature" are retained, thereby obtaining a candidate set of all benign samples. S323. Using the influence function IF method, select the... The benign sample with the highest opposition strength, i.e., the lowest IF. : in, This represents the candidate set of all benign samples within the current batch. Represents the cross-entropy loss CE. and Representing the first Learning rate and model parameters for each epoch. This represents the gradient operation on the model parameters; S33, Integration of Counterfactual Image Generation and Training; from Extract the extended modification region Corresponding image patch ,replace The corresponding area in: And the generated counterfactual samples Incorporate this into the loss calculation process for the current batch, and calculate the counterfactual consistency loss: Where CE represents cross-entropy; Then, calculate the IF response of the counterfactual sample to the original malicious sample: If the model learns the correct causal features It should be a significantly negative value, meaning that the counterfactual sample strongly suppresses malicious prediction; otherwise, if If the intervention is ineffective, then it is necessary to adjust the model parameters or choose a stronger intervention, while also imposing a penalty term on the model. The total training loss is calculated as follows: in, The classification loss of the counterfactual classification model is... The value is 0.

5. and The values ​​are 1.0 and 0.5 respectively, to ensure that the model does not forget its ability to align modal features during training based on counterfactual data.

4. The method according to claim 3, characterized in that, The progressive visual semantic interpretability path consists of three parts: The first part involves the model outputting a counterfactual sequence map that progresses from the original image to the images modified during training, and highlighting the relevant data. The regions form a visual explanatory path; The second part is a numerical interpretation path based on the expected necessity value, IF response value, and confidence change curve as quantification criteria. Part Three: Textual Interpretation Path.

5. An interpretable AI prediction system for breast cancer based on counterfactual reasoning, characterized in that, The system includes: The preprocessing module is used to collect and preprocess the patient's static ultrasound image data and medical record text data, annotate the image data, determine the key causal features of the lesions based on the medical record text data, and divide them into training set and validation set. The alignment extraction module is used to input the image data and medical record text data in the training set into the pre-trained medical visual language model (VLM), perform comparative learning and fine-tuning, and use the fine-tuned VLM to perform multimodal feature extraction and cross-modal feature alignment, and output the obtained text-image feature alignment mapping table. The training module is used to train the counterfactual classification model that integrates the VLM. The counterfactual classification model includes the VLM, the counterfactual module, and the classification head. In each training round, the counterfactual module dynamically generates medically credible counterfactual samples based on the current model state, the text-image feature alignment mapping table, and predefined medical feature linkage constraint rules, and integrates them into the model training process to achieve causal consistency constraints. The prediction output module is used to predict the breast image data to be predicted using the trained counterfactual classification model, output the AI ​​prediction results, and output the progressive visual semantic interpretability path. The VLM, for a certain image data After passing through the image encoder, global features are output. and spatial feature map For text feature extraction, the input image data... Corresponding medical record sentences After passing through a text encoder, sentence-level features are output. and word-level features ; The cross-modal feature alignment includes two aspects: global feature alignment and local feature alignment. Regarding global feature alignment: Calculate global image-text similarity And use the InfoNCE loss function to measure the global contrastive loss. : in, Let be the projection matrix of the VLM. Represents the text in the current batch. Temperature overparameter; Regarding local feature alignment: First, calculate keywords Spatial features The similarity is used to obtain the attention heatmap. : Regenerate keyword-region mask, for Thresholding segmentation: Set the top 10% of pixel values ​​to 1 and the rest to 0 to obtain the boundary mask region corresponding to a certain key feature. : Recalculate the region-text matching loss: extract the average visual features of the masked regions. And calculate its matching loss with the keyword embedding. : Finally, verify the alignment results and optimize them: If If the IoU between the region with a value of 1 and the image region corresponding to the key causal feature is > 0.5, then the alignment is successful; otherwise, the weights are increased during the fine-tuning stage to force the model to learn the correct mapping. The total loss for cross-modal feature alignment: in, A value of 0.5 is used to balance global feature alignment and local feature alignment.

6. The system according to claim 5, characterized in that, The annotation includes: benign and malignant labeling of image data, and obtaining the average value of the region of interest (ROI) annotated by multiple ultrasound doctors as the location label of the lesion; The key causal features include six lesion characteristics: boundary, shape, calcification, aspect ratio, echo pattern, and posterior acoustic shadowing.

7. The system according to claim 5, characterized in that, The counterfactual module is specifically used for: S31. Define the counterfactual problem of the prediction results of the inversion model, analyze the changes in the lesion area in the image data, and formulate counterfactual intervention targets based on the counterfactual problem; S311. Conduct characteristic causality assessment; From each high-confidence malicious sample in the current training batch of We select a high-causality keyword: the boundary feature "irregular", and calculate its expected necessity EN: by analyzing whether the prediction result will necessarily change after eliminating a certain feature, we infer the necessity of that feature to maintain the prediction result. The calculation method is as follows: in, To eliminate the "irregular" feature attribute, the predicted probability is obtained by replacing this key feature with the neutral word "unknown" and recoding it. If this proves that the feature is highly necessary for the model to judge it as a malignant lesion, then the feature can be selected as the counterfactual intervention target. S312, Perform linkage feature detection; The predefined medical feature linkage rule base is queried to obtain the intervention target "irregular margin" linked to "tissueinfiltration". The VLM is then used to calculate the attention heatmap of the linkage features. ,like , If the preset threshold is used, then a linkage feature is determined to exist; S313. After obtaining the linkage characteristics, modify the region expansion; Obtain the main feature region mask from the text-image feature alignment mapping table. If there are linkage features, then obtain their region mask. The merged area results in an expanded modification region: S32. Retrieval of benign reference samples under linkage constraints; S321. First, construct counterfactual text; Replace "irregular" with "smooth" or "regular" to change irregular boundaries into smooth / regular boundaries, resulting in new text. If there are linked features, the linked words must be replaced at the same time, replacing "infiltration" with "normaltissue"; S322. Among all benign samples in the current batch, select candidate samples that satisfy the linkage constraint. For each candidate image sample The intensity of the linkage feature is detected by the VLM, and only samples with "benign main feature and normal linkage feature" are retained, thereby obtaining a candidate set of all benign samples. S323. Using the influence function IF method, select the... The benign sample with the highest opposition strength, i.e., the lowest IF. : in, This represents the candidate set of all benign samples within the current batch. Represents the cross-entropy loss CE. and Representing the first Learning rate and model parameters for each epoch. This represents the gradient operation on the model parameters; S33, Integration of Counterfactual Image Generation and Training; from Extract the extended modification region Corresponding image patch ,replace The corresponding area in: And the generated counterfactual samples Incorporate this into the loss calculation process for the current batch, and calculate the counterfactual consistency loss: Where CE represents cross-entropy; Then, calculate the IF response of the counterfactual sample to the original malicious sample: If the model learns the correct causal features It should be a significantly negative value, meaning that the counterfactual sample strongly suppresses malicious prediction; otherwise, if If the intervention is ineffective, then it is necessary to adjust the model parameters or choose a stronger intervention, while also imposing a penalty term on the model. The total training loss is calculated as follows: in, The classification loss of the counterfactual classification model is... The value is 0.

5. and The values ​​are 1.0 and 0.5 respectively, to ensure that the model does not forget its ability to align modal features during training based on counterfactual data.

8. The system according to claim 7, characterized in that, The progressive visual semantic interpretability path consists of three parts: The first part involves the model outputting a counterfactual sequence map that progresses from the original image to the images modified during training, and highlighting the relevant data. The regions form a visual explanatory path; The second part is a numerical interpretation path based on the expected necessity value, IF response value, and confidence change curve as quantification criteria. Part Three: Textual Interpretation Path.

Images