An artificial intelligence-based medical image recognition and analysis system

Abstract

The application discloses a medical image recognition and analysis system based on artificial intelligence, relates to the technical field of medical artificial intelligence, and comprises a feature extraction layer, an evidence anchoring layer, a causal decoupling fusion module, a Shapley attribution decomposition module and an evidence package generation unit. The evidence anchoring layer generates a pixel-level lesion segmentation mask and structured medical entity evidence before outputting a diagnosis conclusion and performs cross-modal consistency cross-validation, and after verification, it is allowed to enter the subsequent process. The causal decoupling fusion module decouples real pathological characteristics and device domain mixed factors based on a structural causal model and a gradient inversion layer, improves the model cross-institutional generalization capability, the Shapley attribution decomposition module reversely decomposes the diagnosis result into the quantitative contribution degree of each feature group, generates a structured decision contribution degree list, and the evidence package generation unit integrates visual evidence, text evidence, diagnosis prediction results and the decision contribution degree list to generate a responsibility traceable evidence package.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a medical image recognition and analysis system based on artificial intelligence. Background Technology

[0002] With the development of deep learning technology, artificial intelligence has made significant progress in the field of medical image recognition. Existing technologies typically use convolutional neural networks or visual transformation networks as the backbone network for feature extraction, performing end-to-end feature extraction on medical images such as computed tomography images, magnetic resonance images, and X-ray images, and directly outputting diagnostic conclusions, such as benign or malignant discrimination results or disease grading results, in conjunction with classification heads. Some advanced systems further introduce multimodal data such as pathological data, genetic data, and electronic medical record data, and form a comprehensive diagnostic model through feature splicing or attention fusion mechanisms. In terms of interpretability, the industry generally adopts ex post-interpretation methods such as class activation mapping tools and locally interpretable model-independent interpretation tools to visually annotate the image regions on which the model's decisions depend.

[0003] The closest existing technical solution to this invention is an end-to-end multimodal fusion diagnostic system. Its technical process is as follows: First, the system extracts features from image data and text medical record data respectively. For image data, a convolutional network or visual transformation network backbone is usually used to extract global feature vectors, and for text data, a pre-trained language model is used to extract semantic vectors. Second, the two types of feature vectors are fused by concatenation, weighted summation, or cross-modal attention mechanism to obtain a unified multimodal fusion feature. Third, the fused feature is directly input into a classifier or regression head to generate the final diagnostic conclusion in one step. Finally, to meet certain interpretability requirements, the system uses tools such as class activation mapping to generate a heatmap of the image input after outputting the conclusion as a post-hoc explanation. Structurally, this solution belongs to a processing pipeline where feature extraction, fusion, direct output of conclusion, and post-hoc interpretation are independent of each other.

[0004] The existing technical solutions have the following drawbacks: First, in cross-institutional and cross-device deployment scenarios, the model is prone to learning pseudo-features of the device domain that are unrelated to pathological features but have statistical correlation with the labels, resulting in insufficient generalization ability of the model. Second, multimodal features are mixed into a single high-dimensional vector during the fusion process, and no longer have a one-to-one correspondence with the original modalities. The ex post-interpretation tool can only generate approximate estimates on the original input space and cannot penetrate the fusion layer to restore the degree of contribution of specific evidence to the prediction results. Third, existing generative models do not require verifiable intermediate evidence to be output first and then diagnostic conclusions during the inference stage. The model may still give seemingly reasonable but unfounded diagnostic statements when the evidence is insufficient, which poses a risk of fabricating lesion details. Therefore, we propose an artificial intelligence-based medical image recognition and analysis system. Summary of the Invention

[0005] The purpose of this invention is to provide an artificial intelligence-based medical image recognition and analysis system to solve the technical problems in existing end-to-end multimodal fusion diagnostic systems, such as insufficient cross-institutional generalization ability caused by device domain pseudo-features, inability of fusion features to be penetrated and attributed by ex post-hoc interpretation tools, and lack of mandatory evidence verification before the output of diagnostic conclusions, which leads to the fabrication of lesion details.

[0006] To address the aforementioned technical problems, this invention provides an artificial intelligence-based medical image recognition and analysis system, comprising a feature extraction layer, an evidence anchoring layer, a causal decoupling and fusion module, a Shapley attribution decomposition module, and an evidence package generation unit.

[0007] The feature extraction layer is used to extract features from the input medical image data and structured electronic medical record text data respectively, to obtain image features and text features.

[0008] The evidence anchoring layer is connected to the feature extraction layer and includes a lesion segmentation and detection submodule, a medical entity recognition submodule, and an evidence cross-validation unit. The lesion segmentation and detection submodule is used to generate pixel-level segmentation masks and bounding boxes of lesions based on the image features as visual evidence. The medical entity recognition submodule is used to extract a set of medical entities based on the text features as text evidence. The evidence cross-validation unit is used to calculate the semantic consistency score between the visual evidence and the text evidence, and determine whether the consistency score reaches a preset threshold. When the preset threshold is reached, the visual evidence and text evidence that have passed cross-validation are output. When the preset threshold is not reached, an evidence insufficiency warning is triggered and the subsequent diagnostic process is terminated, thereby forcing the system to generate verifiable structured evidence before the diagnostic conclusion is output.

[0009] The causal decoupling fusion module is connected to the evidence anchoring layer and includes a shared feature extractor, a diagnostic task classification head, a gradient inversion layer, and a domain adversarial classifier. The shared feature extractor is used to fuse visual and textual evidence that have passed cross-validation and output fused features. The fused features are respectively input to the diagnostic task classification head and input to the domain adversarial classifier via the gradient inversion layer. The diagnostic task classification head uses the real diagnostic label as a supervision signal and outputs the diagnostic prediction result. The domain adversarial classifier uses the device domain label of the sample as a supervision signal and outputs the domain prediction label. The gradient inversion layer is used to multiply the gradient returned by the domain adversarial classifier by a negative coefficient and then pass it to the shared feature extractor during backpropagation, so that the shared feature extractor learns the domain-invariant fused features, thereby removing the interference of device domain confounding factors on the diagnostic results.

[0010] The Shapley attribution decomposition module is connected to the causal decoupling fusion module and is used to divide the diagnostic prediction results into multiple feature groups according to the evidence source modality, calculate the Shapley attribution value of each feature group to the diagnostic prediction results, and generate a structured list of decision contribution.

[0011] The evidence package generation unit is used to integrate the visual evidence, the textual evidence, the diagnostic prediction results, and the decision contribution list to generate a traceable evidence package, which is then fed back to the clinical diagnosis interface via an interface.

[0012] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0013] 1. This invention sets up a mandatory evidence anchoring layer before the output of diagnostic conclusions, requiring the system to generate pixel-level visual evidence and structured text evidence and complete cross-validation before proceeding to the next process. This algorithmically blocks the path of outputting diagnostic conclusions when the evidence is insufficient, reducing the risk of fabricating lesion details.

[0014] 2. This invention overcomes the technical bias of existing technologies that separate the three stages of evidence generation, feature fusion, and post-interpretation, which cannot form a causal closed loop, by linking three mechanisms in a chain: mandatory cross-validation of pre-diagnosis evidence anchoring, domain adversarial decoupling guided by structural causal models, and Shapley attribution decomposition for fused modal feature groups. The output of the previous stage is used as the mandatory input premise of the next stage, rather than being stacked in parallel and independently. This allows the attribution results to directly penetrate to the original modal evidence that has passed cross-validation, achieving the effect of full-link traceability of evidence that cannot be achieved by applying domain adversarial training or SHAP method alone.

[0015] 3. This invention integrates the visualized evidence materials output by the evidence anchoring layer, the domain-invariant diagnostic results output by the causal decoupling module, and the contribution list output by the Shapley attribution decomposition module to form an end-to-end closed-loop traceable evidence package, providing a specific technical means to meet the traceability requirements of medical device regulatory approval. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall vertical flow of the system in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the vertical subdivision process within the evidence anchoring layer in an embodiment of the present invention;

[0018] Figure 3 This is a simplified vertical flowchart illustrating the causal decoupling, attribution, and evidence package process in an embodiment of the present invention. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0020] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0021] The medical image recognition and analysis system provided by this invention comprises a feature extraction layer, an evidence anchoring layer (101), a causal decoupling and fusion module (102), a Shapley attribution decomposition module (103), and an evidence package generation unit (104). The leftmost part of the system is a multimodal data input layer, including a medical image input module and a structured electronic medical record text input module. The medical image input module is used to receive raw medical image data such as computed tomography images, magnetic resonance images, and X-ray images. The structured electronic medical record text input module is used to receive patient complaints, past medical history, and laboratory test results. In the feature extraction layer, the image branch uses a hierarchical visual transformation network based on a sliding window (105) as the visual backbone network to downsample the medical image data step by step and output multi-scale feature representations at different resolutions. The text branch uses a medical language model (106) to semantically encode the structured electronic medical record text data and output semantic feature vectors.

[0022] In the evidence anchoring layer 101, the lesion segmentation and detection submodule, based on the multi-scale feature map, generates a pixel-level segmentation mask for candidate lesions through parallel processing by the segmentation head and the detection head, denoted as the mask. And the keypoint bounding box, denoted as the bounding box. The segmentation mask The bounding box is used to mark the outline of the lesion in the image. The rectangular region containing the lesion and its confidence level are used for labeling. Simultaneously, the medical entity recognition submodule uses a medical named entity recognition model to extract a set of medical entities from the structured electronic medical record text data, denoted as the entity set. The entity set The elements in the document include symptom entities, sign entities, past medical history entities, and test indicator entities, with each entity accompanied by a corresponding semantic category label.

[0023] Subsequently, the evidence cross-validation unit maps visual evidence and textual evidence to the same joint semantic embedding space through a projection function, and calculates the semantic consistency score between the two. The specific calculation method is as follows:

[0024] ;

[0025] in, The semantic consistency score represents the relationship between visual and textual evidence. Indicates the segmentation mask With bounding box The visual evidence vector obtained after mapping to the joint embedding space This represents a set of medical entities. The text evidence vector obtained after mapping to the joint embedding space, symbol The 2-norm of a vector, the sign in the numerator. This represents the inner product operation of two vectors;

[0026] The cross-validation unit further determines the consistency score. Is it greater than or equal to a preset threshold? :like Less than the threshold If the visual and textual evidence fails to corroborate each other, the system triggers an insufficient evidence warning, refuses to proceed with the subsequent diagnostic decision-making process, and generates a manual review request, returning it to the clinician's diagnostic interface. Greater than or equal to the threshold The system determines that the evidence has passed cross-validation. The cross-validated visual and textual evidence, along with their respective original feature representations, are output to the causal decoupling and fusion module 102. At the same time, the visual and textual evidence are retained as visual evidence materials in the final diagnostic report. Through the above-mentioned mandatory evidence generation and cross-validation mechanism, the system blocks the path of directly outputting diagnostic conclusions when the evidence is insufficient at the algorithm level, thereby suppressing the risk of fictitious lesion details that may appear in generative models.

[0027] In the causal decoupling and fusion module 102, a structural causal model is first constructed, which decomposes the observed diagnostic prediction results into the combined effect of two parts: the actual pathological features and the device domain confounding factors. Its structural expression is as follows:

[0028] ;

[0029] in, This represents the diagnostic prediction result output by the model. Representing the true pathological characteristics, these are causal variables that determine the diagnostic outcome. This indicates confounding factors in the device domain, including non-pathological differences introduced by parameters such as scanner model, contrast agent dosage, and slice thickness. This represents exogenous noise terms that are independent of both pathological features and the device domain. This represents the structural causal function implemented by the diagnostic decision module;

[0030] A diagnostic task classification head and a domain adversarial classifier are connected in parallel after the shared feature extractor: the diagnostic task classification head is trained using the real diagnostic labels as supervision signals and outputs diagnostic prediction results; the domain adversarial classifier is trained using the device domain labels of the samples as supervision signals and outputs domain prediction labels. A gradient inversion layer is inserted between the domain adversarial classifier and the shared feature extractor: during forward propagation, the gradient inversion layer does not change the feature values ​​and directly passes the features to the domain adversarial classifier; during backward propagation, the gradient inversion layer multiplies the gradient returned by the domain adversarial classifier by a negative coefficient before passing it to the shared feature extractor. Specifically:

[0031] ;

[0032] in, Representation domain adversarial loss relative to shared feature extractor parameters The original gradient, This represents the gradient actually used to update the parameters of the shared feature extractor after processing by the gradient inversion layer. This represents the gradient reversal strength coefficient, used to control the degree of influence of domain adversarial training on the feature extractor;

[0033] Further, a joint optimization objective function is constructed, and the diagnostic task loss and domain adversarial loss are jointly trained, specifically expressed as follows:

[0034] ;

[0035] in, This represents the total loss from joint training. This represents the prediction loss for the diagnostic task. For true diagnostic labeling, The diagnostic results predicted by the model. Representation domain adversarial classification loss, The actual device domain label of the sample. For the domain labels predicted by the domain adversarial classifier, The meaning is the same as above. Through the above joint training process, the shared feature extractor is forced to learn the feature representation of the sample source domain, which makes it difficult for the domain adversarial classifier to correctly distinguish the sample source domain, thereby decoupling the real pathological features from the equipment domain confounding factors, and finally outputting domain-invariant fused diagnostic features, improving the model's generalization ability in cross-institution and cross-equipment deployment scenarios.

[0036] In Shapley's attribution decomposition module 103, the fusion features of the input diagnostic decision module are divided according to source modality and clinical semantics. Each feature group constitutes the complete set of feature groups. Each feature group corresponds to a type of interpretable clinical evidence, such as the imaging morphology feature group, the gene mutation biomarker group, and the medical history and lifestyle feature group. For the complete set of feature groups... Any group that does not contain the target feature subset of Construct a subset-only structure The input, consisting of the middle feature group and the remaining feature groups filled with baseline values, is fed into the diagnostic decision module to obtain the model output value of that subset of the alliance. As the marginal contribution function value of this subset, each feature group is calculated according to the Shapley value definition. The marginal contribution to the final diagnostic result is specifically expressed as follows:

[0037] ;

[0038] in, Representing feature groups Shapley attribution value, Represents the set consisting of all feature groups. Indicates that it does not contain feature groups any subset, Representing a subset The number of feature groups contained therein Represents the complete set of feature groups The total number, Indicates the feature group Join a subset The output value of the model consortium after that, Representing a subset The model consortium output value when used as input alone, with the summation symbol indicating summation over feature groups. All possible front subsets Weighted summation is performed according to the weighting coefficients;

[0039] Considering the high computational complexity of directly enumerating all subsets, this implementation method uses... A weighted regression method with a penalty term is used to approximate the Shapley attribution value in order to improve computational efficiency and suppress the interference of redundant modal noise on the attribution results. The specific optimization objective is as follows:

[0040] ;

[0041] in, Indicates that the Shapley kernel function is a subset The assigned sampling weights, This represents the baseline output value of the model when there are no feature groups as input. The meaning is the same as described above. This represents the penalty coefficient, used to control the sparsity of the attribution results, suppressing redundant feature groups with small contributions from entering the final report. (Symbol: ...) Indicates to Take the absolute value, sort the Shapley attribution values ​​of each feature group according to their contribution, and normalize them into percentage form to generate a structured list of decision contribution, which may include structured data items such as the contribution ratio of image features, the contribution ratio of genetic markers, and the contribution ratio of past history.

[0042] Finally, the evidence package generation unit 104 packages and integrates the pixel-level mask and bounding box annotation map output by the evidence anchoring layer, the domain-invariant diagnostic results output by the causal decoupling fusion module, and the decision contribution list output by the Shapley attribution decomposition module to generate a responsibility-traceable evidence package, and feeds it back to the radiologist's diagnostic interface through the interface for clinicians to verify the diagnostic conclusions and their supporting evidence.

[0043] In an alternative implementation, the causal decoupling fusion module may not adopt the adversarial training method of domain adversarial classifier and gradient reversal layer, but instead adopt the distribution alignment loss based on the maximum mean difference. By adding a constraint term to the training objective, the distribution distance of samples from different device domains in the feature space can be directly reduced, which can also achieve the learning of domain invariant features and achieve the technical effect of suppressing shortcut learning and improving cross-mechanism generalization ability.

[0044] In another alternative implementation, the Shapley attribution decomposition module may not use the Shapley value decomposition method, but instead use the integrated gradient method. This method estimates the contribution of each feature group to the prediction result by integrating and accumulating the gradient of the model output along the path from the baseline input to the actual input. This method has relatively low computational complexity and can also generate a structured list of decision contributions as part of the accountability evidence package.

[0045] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A medical image recognition and analysis system based on artificial intelligence, characterized in that, include: The feature extraction layer extracts features from the input medical image data and structured electronic medical record text data to obtain image features and text features, respectively. The evidence anchoring layer, connected to the feature extraction layer, includes a lesion segmentation and detection submodule, a medical entity recognition submodule, and an evidence cross-validation unit. The lesion segmentation and detection submodule generates pixel-level segmentation masks and bounding boxes for lesions based on the image features, serving as visual evidence. The medical entity recognition submodule extracts a set of medical entities based on the text features, serving as text evidence. The evidence cross-validation unit calculates the semantic consistency score between the visual evidence and the text evidence, and determines whether the consistency score reaches a preset threshold. When the consistency score reaches the preset threshold, it outputs the visual and text evidence that have passed cross-validation; when the consistency score does not reach the preset threshold, it triggers an insufficient evidence warning and terminates the subsequent diagnostic process. The causal decoupling and fusion module, connected to the evidence anchoring layer, includes a shared feature extractor, a diagnostic task classification head, a gradient inversion layer, and a domain adversarial classifier. A feature extractor is used to fuse visual and textual evidence obtained through cross-validation and output fused features. These fused features are input to the diagnostic task classification head and, via the gradient inversion layer, to the domain adversarial classifier. The diagnostic task classification head uses the real diagnostic label as a supervision signal and outputs a diagnostic prediction result. The domain adversarial classifier uses the device domain label of the sample as a supervision signal and outputs a domain prediction label. The gradient inversion layer is used to multiply the gradient returned by the domain adversarial classifier by a negative coefficient during backpropagation and then pass it to the shared feature extractor, so that the shared feature extractor outputs fused features with unchanged domain. A Shapley attribution decomposition module, connected to the causal decoupling fusion module, is used to divide the diagnostic prediction result into multiple feature groups according to the evidence source modality, calculate the Shapley attribution value of each feature group for the diagnostic prediction result, and generate a structured decision contribution list. An evidence package generation unit is used to integrate the visual evidence, the textual evidence, the diagnostic prediction result, and the decision contribution list to generate a responsibility-traceable evidence package.

2. The medical image recognition and analysis system based on artificial intelligence according to claim 1, characterized in that: The evidence cross-validation unit maps the visual evidence and the text evidence to the same joint semantic embedding space through a projection function, and calculates the cosine similarity between the visual evidence vector and the text evidence vector after mapping, which is used as the consistency score.

3. The medical image recognition and analysis system based on artificial intelligence according to claim 1, characterized in that: The feature extraction layer includes a visual backbone network and a medical language model. The visual backbone network is a hierarchical visual transformation network based on a sliding window, used to downsample the medical image data step by step and output multi-scale feature maps. The medical language model is used to perform semantic encoding on the structured electronic medical record text data and output semantic feature vectors.

4. The medical image recognition and analysis system based on artificial intelligence according to claim 1, characterized in that: The causal decoupling and fusion module is also used to represent the diagnostic prediction result as the result of the combined effect of real pathological features, device domain confounding factors and exogenous noise terms based on the structural causal model, and to use the structural causal model to guide the adversarial training process of the domain adversarial classifier and the gradient inversion layer.

5. The medical image recognition and analysis system based on artificial intelligence according to claim 1, characterized in that: The Shapley attribution decomposition module uses a weighted regression method with an L1 penalty term to solve for the Shapley attribution value, and then normalizes the obtained Shapley attribution values ​​into percentage form after sorting them by contribution.

6. The medical image recognition and analysis system based on artificial intelligence according to claim 1, characterized in that: It also includes an interface for insufficient evidence warning and manual review, which is connected to the evidence cross-validation unit and is used to generate a manual review request and send it to the clinician's diagnostic interface when the consistency score does not reach a preset threshold.

7. A medical image recognition and analysis method based on artificial intelligence, characterized in that, include: The process involves acquiring medical image data to be analyzed and corresponding structured electronic medical record text data. Feature extraction is performed on the medical image data to obtain image features, and feature extraction is performed on the structured electronic medical record text data to obtain text features. Based on the image features, pixel-level segmentation masks and bounding boxes of lesions are generated as visual evidence. A set of medical entities is extracted based on the text features as text evidence. A semantic consistency score is calculated between the visual evidence and the text evidence. It is determined whether the consistency score reaches a preset threshold. If the consistency score does not reach the preset threshold, an insufficient evidence warning is triggered, and the subsequent diagnostic process is terminated. If the consistency score reaches the preset threshold, the visual evidence and the text evidence are fused to obtain fused features. The fused features are input into the diagnostic task classification head to obtain diagnostic prediction results as reference information for clinicians. Simultaneously, the fused features are input into a domain adversarial classifier via a gradient inversion layer. The domain adversarial classifier is trained using the device domain label of the sample as a supervision signal. During backpropagation, the gradient inversion layer multiplies the gradient returned by the domain adversarial classifier by a negative coefficient and then sends it back to the shared feature extractor, so that the shared feature extractor outputs fused features with unchanged output domain. The diagnostic prediction results are divided into multiple feature groups according to the evidence source modality. The Shapley attribution value of each feature group to the diagnostic prediction results is calculated to generate a structured decision contribution list. The visual evidence, the textual evidence, the diagnostic prediction results, and the decision contribution list are integrated to generate a responsibility traceability evidence package.

8. The medical image recognition and analysis method based on artificial intelligence according to claim 7, characterized in that: Calculating the semantic consistency score between the visual evidence and the text evidence includes: mapping the visual evidence and the text evidence to the same joint semantic embedding space through projection functions, calculating the cosine similarity between the two vectors obtained after mapping, and using it as the consistency score.

9. The medical image recognition and analysis method based on artificial intelligence according to claim 7, characterized in that: After obtaining the fusion features, the method further includes: constructing a structural causal model to represent the diagnostic prediction results as the result of the combined effects of real pathological features, device domain confounding factors, and exogenous noise terms, and using the structural causal model to guide the adversarial training process of the domain adversarial classifier and the gradient inversion layer.

10. The method for medical image recognition and analysis based on artificial intelligence according to claim 7, characterized in that: Calculating the Shapley attribution value for each feature group in relation to the diagnostic prediction result includes: solving the Shapley attribution value using a weighted regression method with an L1 penalty term, and then normalizing the obtained Shapley attribution values ​​into percentage form after sorting them by contribution.

Images