A task-aware multi-modal medical image decision arbitration and semantic verification generation system and method

Abstract

The application discloses a task-aware multi-modal medical image decision arbitration and semantic verification generation system and method, and relates to the technical field of medical image artificial intelligence auxiliary diagnosis. The system comprises: a multi-modal clinical task awareness module for acquiring data and identifying a clinical task scene; a cross-modal consistency arbitration module for performing logical arbitration according to a medical knowledge graph when conclusions of multiple expert sub-models conflict, and outputting consistent diagnostic features; a dynamic meta-operator generation module for calling a meta-operator of a corresponding granularity according to the task scene, and constructing a graphical execution chain based on the consistent features to generate an initial diagnosis report; and a semantic consistency reverse verification module for projecting report text semantics to an original image pixel space in reverse, and verifying the matching degree. The application ensures the authenticity of the report through reverse verification, and improves the reliability, safety and clinical applicability of diagnosis.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted diagnosis technology for medical imaging, and more specifically, to a task-aware multimodal medical image decision arbitration and semantic verification generation system and method. Background Technology

[0002] Existing multimodal medical imaging diagnostic technologies often employ a uniform processing logic, making it difficult to adapt to the needs of clinical tasks with varying degrees of urgency and depth. Furthermore, they lack effective decision-making arbitration mechanisms when dealing with cross-modal data conflicts. More critically, AI-generated diagnostic reports often exhibit an "illusion" phenomenon where semantic descriptions are disconnected from the pixel features of the original images, lacking reliable means of verifying authenticity and posing potential safety risks to clinical diagnosis and treatment. Summary of the Invention

[0003] The main objective of this application is to provide a task-aware multimodal medical image decision arbitration and semantic verification generation system and method, which aims to at least solve one of the above-mentioned technical problems.

[0004] The first aspect of this application provides a task-aware multimodal medical image decision arbitration and semantic verification generation system, including: The multimodal clinical task perception module is used to acquire the examinee's multimodal medical data and identify the current clinical task scenario label; The cross-modal consistency arbitration module is used to extract features from the multimodal medical data using multiple parallel deep learning expert sub-models, and to perform logical arbitration based on the medical knowledge graph to output consistent diagnostic features when there are conflicts in the preliminary conclusions output by different expert sub-models. The dynamic meta-operator generation module is used to extract meta-operators of corresponding granularity from a preset operator library according to the clinical task scenario tags, and to construct a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. The semantic consistency reverse verification module is used to back-project the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data in order to verify the matching degree between the report description content and the image features and output the verification result.

[0005] In some embodiments of this application, the multimodal clinical task perception module performs natural language parsing on the application form information in the electronic medical record to divide the clinical task scenario label into emergency screening scenario, routine diagnosis scenario, or preoperative assessment scenario.

[0006] In some embodiments of this application, the cross-modal consistency arbitration module includes: The conflict detection unit is used to determine whether the deviation of image features of different modalities from the disease classification probability distribution exceeds a preset deviation threshold. The logic arbitration unit is used to retrieve the causal determination rules in the medical knowledge graph when the deviation exceeds the preset deviation threshold, and assign the highest decision weight to the expert sub-model that has the highest matching degree with the causal determination rule.

[0007] In some embodiments of this application, the dynamic meta-operator generation module includes a granularity adjustment unit. The granularity adjustment unit is used to control the graphical execution chain to call only the core positive / negative judgment operator when the clinical task scenario label is the emergency screening scenario; and to control the graphical execution chain to call all operators, including anatomical measurement and grading assessment, when the clinical task scenario label is the preoperative assessment scenario.

[0008] In some embodiments of this application, the semantic consistency reverse verification module includes a spatial heatmap generation unit, which is used to calculate the cross-modal attention weights of medical keywords in the initial diagnostic report on the original image pixels and generate the corresponding spatial heatmap distribution.

[0009] In some embodiments of this application, the semantic consistency reverse verification module includes an authenticity determination unit, which is used to determine that the initial diagnostic report contains semantic illusions when the center coordinates of the spatial thermal distribution fall outside the range of the anatomical structure described in the initial diagnostic report.

[0010] In some embodiments of this application, the system further includes an interactive logic evolution module, which is used to capture user commands to manually adjust the graphical execution chain and to optimize the default assembly strategy of the dynamic meta-operator generation module using reinforcement learning algorithms.

[0011] In some embodiments of this application, the system further includes a missing modality compensation module, which is used to synthesize a virtual feature map of the missing modality based on the existing modality data when some modality data in the multimodal medical data is missing, using a generative adversarial network.

[0012] In some embodiments of this application, the system further includes a privacy desensitization module, which is used to automatically identify and mask the examinee's identity and privacy information before the initial diagnostic report is output.

[0013] Another aspect of this application provides a task-aware multimodal medical image decision arbitration and semantic verification generation method, applicable to any of the aforementioned task-aware multimodal medical image decision arbitration and semantic verification generation systems. The method includes the following steps: Step 1: Acquire the examinee's multimodal medical data and identify the current clinical task scenario label through the multimodal clinical task perception module; Step 2: The cross-modal consistency arbitration module uses multiple parallel deep learning expert sub-models to extract features from the multimodal medical data. When there are conflicts in the preliminary conclusions output by different expert sub-models, logical arbitration is performed based on the medical knowledge graph to output consistent diagnostic features. Step 3: The dynamic meta-operator generation module extracts meta-operators of corresponding granularity from the preset operator library according to the clinical task scenario tags, and constructs a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. Step 4: The semantic consistency reverse verification module back-projects the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data to verify the matching degree between the report description content and the image features and outputs the verification results.

[0014] The embodiments of this application include at least the following beneficial effects: By introducing a task-aware processing mechanism, the diagnostic process can be flexibly adjusted according to different clinical task scenarios, effectively solving the problem of the single processing logic of traditional technologies. Simultaneously, its cross-modal consistency arbitration mechanism can effectively resolve conflicts between multimodal data, improving the reliability of diagnostic conclusions. Furthermore, through semantic consistency reverse verification, the system can promptly detect and correct potential "illusion" phenomena in AI diagnostic reports, significantly enhancing the authenticity and security of diagnostic reports, thereby reducing potential risks in clinical diagnosis and treatment.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the relevant drawings of the embodiments of this application are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the architecture of a task-aware multimodal medical image decision arbitration and semantic verification generation system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a task-aware multimodal medical image decision arbitration and semantic verification generation method provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] For ease of understanding, the following explains some key terms in this embodiment: Multimodal medical data typically refers to data sets derived from different medical imaging equipment or examination methods, such as imaging data from CT, MRI, X-rays, and ultrasound, as well as non-imaging data such as pathology reports and laboratory test results. These data reflect the physiological and pathological information of the examinee from different dimensions.

[0020] Clinical task scenario tags are classification identifiers used for the type of current medical diagnostic or assessment task, reflecting the urgency, required diagnostic depth, or focus of the task. For example, a clinical task scenario tag could indicate whether the task is an emergency screening, routine diagnosis, or preoperative assessment.

[0021] Deep learning expert sub-models refer to deep learning models trained for specific medical imaging modalities or specific disease diagnosis tasks. Each sub-model is designed to extract features from the modality data it is good at and provide preliminary judgments.

[0022] A medical knowledge graph is a structured medical knowledge base that contains medical concepts such as diseases, symptoms, anatomical structures, treatment plans, causal relationships, and their interrelationships. It is used to provide logical reasoning and decision support.

[0023] Meta-operators are the basic functional units that make up the diagnostic process. For example, a meta-operator can be a lesion detection algorithm, an organ segmentation algorithm, or a disease classifier. These operators are stored in an operator library and can be combined as needed.

[0024] A graphical execution chain refers to a visual processing flow consisting of multiple meta-operators connected in a specific logical order. This graphical execution chain is used to automate the entire process from data input to diagnostic report generation.

[0025] The initial diagnostic report is a preliminary diagnostic text automatically generated by the system based on the processing results. This initial diagnostic report includes a description of the image features, possible diagnostic conclusions, and relevant quantitative indicators.

[0026] Textual semantic features refer to vector representations extracted from the text content of diagnostic reports that characterize their meaning and theme. These textual semantic features are used for cross-modal matching and verification with image features.

[0027] The raw pixel space refers to the unprocessed image data level of medical imaging data. Projecting textual semantic features back into this raw pixel space helps to visually verify the correspondence between report descriptions and image details.

[0028] A PACS (Picture Archiving and Communication System) is a digital information system used by hospitals to acquire, store, transmit, access, and process medical images. It can replace traditional film and enable hospital-wide sharing and remote application of image resources.

[0029] Existing multimodal medical imaging diagnostic technologies often employ a uniform processing logic, making it difficult to adapt to the needs of clinical tasks with varying degrees of urgency and depth. Furthermore, they lack effective decision-making arbitration mechanisms when dealing with cross-modal data conflicts. More critically, AI-generated diagnostic reports often exhibit an "illusion" phenomenon where semantic descriptions are disconnected from the pixel features of the original images, lacking reliable means of verifying authenticity and posing potential safety risks to clinical diagnosis and treatment.

[0030] In this regard, a first aspect of the embodiments of this application provides a task-aware multimodal medical image decision arbitration and semantic verification generation system. See also Figure 1 The system includes: a multimodal clinical task perception module, used to acquire multimodal medical data of examinees and identify the current clinical task scenario label; a cross-modal consistency arbitration module, used to extract features from multimodal medical data using multiple parallel deep learning expert sub-models, and to perform logical arbitration based on a medical knowledge graph to output consistent diagnostic features when there are conflicts in the preliminary conclusions output by different expert sub-models; a dynamic meta-operator generation module, used to extract meta-operators of corresponding granularity from a preset operator library based on the clinical task scenario label, and to construct a graphical execution chain based on the consistent diagnostic features to generate an initial diagnostic report; and a semantic consistency reverse verification module, used to back-project the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data to verify the matching degree between the report description content and the image features and output the verification result.

[0031] First, the system is configured with a multimodal clinical task awareness module. This module is used to acquire the subject's multimodal medical data, such as CT, MRI, and ultrasound imaging data, as well as related clinical information, through manual upload, import from a PACS system, or interface connection to a hospital information system. After acquiring the data, the multimodal clinical task awareness module is used to identify the current clinical task scenario label. For example, clinical task scenario labels can be determined through manual input, selection from a preset list, or keyword matching to indicate whether the current task requires rapid screening or detailed diagnosis.

[0032] Secondly, the system is configured with a cross-modal consistency arbitration module. This module is used to extract features from multimodal medical data using multiple parallel deep learning expert sub-models. Each expert sub-model is trained to process data of a specific modality or identify a specific type of lesion. When there is a conflict between the initial conclusions output by different expert sub-models—for example, one model suggests a malignant lesion while another suggests a benign lesion—the cross-modal consistency arbitration module is used to perform logical arbitration based on a medical knowledge graph. This arbitration process may involve querying information such as relevant diseases, symptoms, and imaging manifestations in the knowledge graph to find evidence supporting or refuting a certain conclusion, thereby outputting a consistent diagnostic feature.

[0033] Furthermore, the system is configured with a dynamic meta-operator generation module. This module is used to extract meta-operators of corresponding granularity from a pre-defined operator library based on clinical task scenario labels. For example, if the task scenario is identified as requiring rapid judgment, only a small number of core operators may be extracted; if the task scenario requires detailed analysis, more detailed operators may be extracted. Subsequently, the dynamic meta-operator generation module constructs a graphical execution chain based on consistent diagnostic features. The construction of this graphical execution chain can be based on a predefined template or through rule matching, connecting the selected meta-operators in a logical order to generate an initial diagnostic report.

[0034] Finally, the system is configured with a semantic consistency back-validation module. This module is used to back-project the textual semantic features of the initial diagnostic report onto the original pixel space of the multimodal medical data. This back-projection can be achieved through feature mapping or based on a pre-trained cross-modal alignment model. By comparing the textual description of the report with the original image features, the semantic consistency back-validation module is used to verify the degree of matching between the report description and the image features. For example, if the report describes a lesion in a certain area, the system will check whether there are corresponding abnormal manifestations in that area on the image and output the validation result to assess the authenticity of the report.

[0035] The system proposed in this application, by introducing a task-aware processing mechanism, can flexibly adjust the diagnostic process according to different clinical task scenarios, effectively solving the problem of the single processing logic of traditional technologies. Simultaneously, its cross-modal consistency arbitration mechanism can effectively resolve conflicts between multimodal data, improving the reliability of diagnostic conclusions. Furthermore, through semantic consistency reverse verification, the system can promptly detect and correct potential "semantic illusions" in AI diagnostic reports, significantly enhancing the authenticity and security of diagnostic reports, thereby reducing potential risks in clinical diagnosis and treatment.

[0036] In some embodiments of this application, the multimodal clinical task perception module performs natural language parsing on the application form information in the electronic medical record to classify clinical task scenario tags into emergency screening scenarios, routine diagnosis scenarios, or preoperative assessment scenarios.

[0037] Specifically, the multimodal clinical task perception module is a front-end component of the system. Its core function is to understand the current medical context and provide contextual information for subsequent image analysis and report generation. In this embodiment, the multimodal clinical task perception module identifies clinical task scenario labels by processing application form information in electronic medical records. Natural language parsing of application form information in electronic medical records refers to using natural language processing (NLP) technology to analyze and understand unstructured application form text data in the electronic medical record system. These application forms typically contain information such as the patient's chief complaint, clinical diagnosis, examination purpose, and lesions of concern. Through NLP technology, the system can automatically extract key medical entities, events, and relationships between them from these texts, such as identifying key phrases and semantic information like "acute abdominal pain," "follow-up of pulmonary nodules," and "preoperative localization of liver tumors." This parsing process may include steps such as lexical analysis, syntactic analysis, named entity recognition, relation extraction, and text classification, thereby transforming the original unstructured text into a structured, machine-understandable feature representation.

[0038] Based on this, the system categorizes clinical task scenario tags into emergency screening scenarios, routine diagnosis scenarios, or preoperative assessment scenarios. This categorization is based on key information extracted from natural language parsing. For example, when a request form contains keywords such as "emergency," "urgent," or "exclude acute lesions," the system can classify it as an emergency screening scenario, which typically requires rapid response and preliminary assessment. When a request form contains keywords such as "follow-up," "re-examination," or "routine examination," the system can classify it as a routine diagnosis scenario, which allows for more comprehensive and detailed analysis. And when a request form contains keywords such as "preoperative," "surgical assessment," or "localization," the system can classify it as a preoperative assessment scenario, which requires providing accurate anatomical measurements and lesion grading information. This scenario classification can be achieved through a pre-set rule base, machine learning-based classification models (such as support vector machines or deep neural networks), or a reasoning mechanism combined with medical knowledge graphs, ensuring that the parsed text features are accurately mapped to the corresponding clinical task scenario tags.

[0039] Through the aforementioned technical solutions, this system can automatically and accurately identify the current clinical task scenario from unstructured electronic medical record request information using natural language processing technology. This automated scenario awareness capability avoids the subjectivity and inefficiency that may arise from manual judgment, ensuring that subsequent image analysis and report generation processes can be dynamically adjusted according to actual clinical needs. For example, in an emergency screening scenario, the system can prioritize the use of fast, highly sensitive algorithms to quickly rule out or identify emergencies; while in a preoperative assessment scenario, it can activate more refined anatomical measurement and lesion grading algorithms to provide detailed surgical guidance information. This refined task scenario division enables the system to provide more targeted, efficient, and clinically relevant diagnostic support, significantly improving the intelligence and personalization of multimodal medical imaging decision-making, thereby optimizing the efficiency and accuracy of the overall diagnostic process.

[0040] In some embodiments of this application, the cross-modal consistency arbitration module includes: a conflict detection unit, used to determine whether the deviation of image features of different modalities in the disease classification probability distribution exceeds a preset deviation threshold; and a logic arbitration unit, used to retrieve causal determination rules in the medical knowledge graph when the deviation exceeds the preset deviation threshold, and assign the highest decision weight to the expert sub-model with the highest matching degree to the causal determination rule.

[0041] The conflict detection unit determines whether the deviation of image features from different modalities in the disease classification probability distribution exceeds a preset deviation threshold. Specifically, the conflict detection unit receives preliminary diagnostic conclusions from different deep learning expert sub-models analyzing multimodal medical data of the same subject. These conclusions are typically presented as probability distributions of disease classification. The conflict detection unit quantifies the inconsistency in diagnostic tendencies among different expert sub-models by calculating dissimilarity indices between these probability distributions, such as using statistical methods like chi-square distance, Kullback-Leibler divergence, or Jensen-Shannon divergence. Subsequently, the calculated deviation is compared with a preset deviation threshold to determine if there are significant conflicts requiring arbitration. For example, if one expert sub-model classifies a malignant tumor with an 80% probability based on CT images, while another expert sub-model classifies a benign lesion with a 70% probability based on MRI images, the conflict detection unit calculates the deviation between these two diagnostic tendencies.

[0042] The logical arbitration unit retrieves causal determination rules from the medical knowledge graph when the deviation exceeds a preset deviation threshold, and assigns the highest decision weight to the expert sub-model with the highest matching degree to the causal determination rule. The logical arbitration unit is activated when the conflict detection unit identifies a significant diagnostic conflict. It accesses a pre-built medical knowledge graph containing rich medical concepts, pathophysiological mechanisms of diseases, diagnostic advantages and limitations of different imaging modalities, and causal relationships between various diagnostic indicators. The logical arbitration unit intelligently retrieves the most relevant causal determination rules based on the current clinical task scenario, the disease type involved in the conflict, and the combination of modalities. For example, in the diagnosis of pulmonary nodules, if the expert sub-model conclusions of CT and PET images conflict, the knowledge graph might contain rules such as "PET has high sensitivity for metabolically active malignant tumors" or "CT is more advantageous in assessing the morphological features of nodules." The logical arbitration unit evaluates the matching degree of these rules with the current conflict and assigns a higher decision weight to the expert sub-model supported by the rule with the highest matching degree. This weighting mechanism allows the system to prioritize the conclusions of expert sub-models that are more authoritative or diagnostically valuable in specific contexts during arbitration.

[0043] Through the above technical solution, this application can accurately identify diagnostic conflicts between different expert sub-models in multimodal medical data analysis, and on this basis, use the deep causal judgment rules of medical knowledge graphs for intelligent arbitration. The conflict detection unit avoids simple yes-or-no judgments by quantifying the degree of deviation in probability distributions, making conflict identification more refined. The logical arbitration unit further utilizes the professional knowledge of medical knowledge graphs to adjust the decision weights of expert sub-models in a basis-based and biased manner when conflicts occur, rather than blindly or averagely arbitrarily arbitrating. This enables the system to more effectively integrate the advantages of different modalities when facing complex and uncertain multimodal medical data, prioritizing the conclusions of expert sub-models that are more diagnostically valuable in specific medical contexts, thereby significantly improving the accuracy, reliability, and medical rationality of the final output consistent diagnostic features, and effectively avoiding the risk of misdiagnosis caused by inconsistencies in information between modalities.

[0044] In some embodiments of this application, the dynamic meta-operator generation module includes a granularity adjustment unit. The granularity adjustment unit is used to control the graphical execution chain to call only the core positive / negative judgment operator when the clinical task scenario is labeled as an emergency screening scenario; and to control the graphical execution chain to call all operators, including anatomical measurement and grading assessment, when the clinical task scenario is labeled as a preoperative assessment scenario.

[0045] Specifically, the granularity adjustment unit is a functional module whose core function is to dynamically adjust or control the set of meta-operators invoked during the diagnostic report generation process and / or the depth of their execution based on specific clinical task scenario labels. It ensures that the output granularity of the report matches current clinical needs. The granularity adjustment unit can be implemented through a configuration management module that stores the mapping relationship between different clinical task scenario labels (such as emergency screening scenarios and preoperative assessment scenarios) and corresponding meta-operator sets (such as core positive / negative determination operators and full set operators). When a specific clinical task scenario label is received, the granularity adjustment unit queries this mapping table and issues instructions to the dynamic meta-operator generation module, instructing it to load and activate the corresponding meta-operator set. Furthermore, it can control the granularity of the report by adjusting the structure of the graphical execution chain, such as enabling or disabling specific branches, nodes, or sub-chains.

[0046] When the clinical task scenario is labeled as an emergency screening scenario, the granularity adjustment unit controls the graphical execution chain to only call the core positive / negative assessment operators. This is to ensure that in emergency situations, the system can quickly focus on the most critical diagnostic information, namely the presence or abnormality of lesions, avoiding unnecessary detailed analysis, thereby improving diagnostic efficiency and response speed. When the granularity adjustment unit recognizes the "emergency screening scenario" label, it instructs the dynamic meta-operator generation module to select and connect only those meta-operators specifically used for rapid, high-level lesion presence assessment when constructing the graphical execution chain. These core positive / negative assessment operators may include a fast classifier based on a deep learning model to determine the presence of key abnormalities such as tumors, inflammation, and bleeding, and output a simple "positive / negative" or "abnormal / normal" conclusion, ignoring time-consuming operations such as precise measurement, grading, or detailed description of lesions.

[0047] When the clinical task scenario is labeled as a preoperative assessment scenario, the granularity adjustment unit controls the graphical execution chain to invoke all operators, including anatomical measurements and grading assessments. This is to ensure that, during the preoperative phase requiring detailed planning and decision-making, the system can provide comprehensive and accurate diagnostic information, including the precise location, size, morphology of the lesion, its relationship with surrounding tissues, and the stage and grade of the disease, providing a sufficient basis for surgical planning. When the granularity adjustment unit recognizes the "preoperative assessment scenario" label, it instructs the dynamic meta-operator generation module to activate and integrate all available meta-operators when constructing the graphical execution chain, including but not limited to: operators for accurately measuring the three-dimensional dimensions of the lesion, operators for assessing the morphological characteristics of the lesion, operators for analyzing the relationship between the lesion and key anatomical structures such as blood vessels / nerves, and operators for staging and grading the disease according to international standards (such as TNM staging). These all operators work together to generate a comprehensive diagnostic report containing all necessary details.

[0048] Through the aforementioned technical solutions, the system can intelligently adjust the diagnostic report generation strategy according to different clinical task scenario labels (such as emergency screening scenarios and preoperative assessment scenarios). In the emergency screening scenario, only the core positive / negative test operator is invoked, greatly shortening the report generation time, improving emergency response efficiency, and avoiding information overload. In the preoperative assessment scenario, all operators, including anatomical measurements and grading assessments, are invoked, ensuring the comprehensiveness and accuracy of the diagnostic report and providing clinicians with all the key information needed to formulate a detailed surgical plan. This dynamic granularity adjustment mechanism, combined with the clinical task scenario labels identified by the multimodal clinical task perception module and the ability of the dynamic meta-operator generation module to construct a graphical execution chain, enables the entire system to adapt to diverse clinical needs more flexibly and efficiently. It avoids the traditional "one-size-fits-all" report generation mode, significantly improving the clinical usability and relevance of diagnostic reports, thereby optimizing the medical decision-making process and improving the quality of medical care.

[0049] In some embodiments of this application, the semantic consistency reverse verification module includes a spatial heatmap generation unit, which is used to calculate the cross-modal attention weights of medical keywords in the initial diagnostic report on the original image pixels and generate the corresponding spatial heatmap distribution.

[0050] Specifically, the spatial heatmap generation unit is configured to generate spatial heat map distributions. This unit can be an independent computational module integrating natural language processing, cross-modal feature extraction, and visualization rendering. Its core function is to establish a perceptible visual connection between abstract textual semantics and specific image pixels, providing an intuitive basis for subsequent verification and decision-making. To calculate the cross-modal attention weights of medical keywords in the initial diagnostic report on the original image pixels, the system first performs natural language processing on the initial diagnostic report to identify and extract diagnostically significant medical keywords, such as "nodule," "calcification," and "lesion." Then, using a deep learning model, such as a Transformer-based cross-modal attention network, the semantic embeddings of these medical keywords interact with the visual features of the original image pixels. This network generates a series of attention weights by calculating the similarity or correlation between the keywords and each pixel. These weights reflect the degree of attention or importance of specific keywords in the report to specific regions in the image. For example, when the report mentions "right upper lobe nodule," the attention weights will focus on the corresponding region of the right upper lobe in the image. After calculating the cross-modal attention weights of medical keywords on the original image pixels, the spatial heatmap generation unit maps these weights onto the spatial coordinates of the original image. Specifically, by normalizing the attention weights and using a preset color gradient (e.g., from cool to warm tones, representing from low to high weight), the weight values ​​are visualized as a color atlas overlaid on the original image. This color atlas is presented in a semi-transparent form, allowing users to observe the anatomical structures of the original image and clearly identify the image regions and intensities of the report keywords, thus forming an intuitive spatial heatmap distribution.

[0051] Through the aforementioned technical solution, this application can intuitively display the specific areas of interest and intensity of medical keywords in the initial diagnostic report on the original images. This visualization capability greatly enhances the interpretability and credibility of the semantic consistency reverse verification module, enabling doctors or the system to clearly determine whether the report content accurately reflects the image features, thereby effectively avoiding semantic illusions or descriptive biases and improving the accuracy and reliability of the diagnostic report. Compared to performing only abstract matching degree verification, spatial heatmaps provide fine-grained spatial correspondences, helping to quickly locate specific areas where the report and images are inconsistent, providing clear guidance for subsequent manual review or model optimization.

[0052] In some embodiments of this application, the semantic consistency reverse verification module includes an authenticity determination unit, which is used to determine that the initial diagnostic report contains semantic illusions when the center coordinates of the spatial thermal distribution fall outside the range of the anatomical structure described in the initial diagnostic report.

[0053] Specifically, the authenticity determination unit is a key component of the semantic consistency reverse verification module. Its core function is to evaluate the authenticity or accuracy of the semantic correspondence between the initial diagnostic report text description and image features. The authenticity determination unit aims to avoid "semantic illusions" in the report that do not match the actual images, thereby significantly improving the reliability of the diagnostic report. The authenticity determination unit can be configured to receive the center coordinates of the spatial thermal distribution output by the spatial heatmap generation unit, and the anatomical structure range extracted from the initial diagnostic report and mapped to the image space as input, and perform logical judgments based on these inputs.

[0054] The center coordinates of the spatial thermal distribution refer to the geometric center of the region in the image space where the cross-modal attention weights of medical keywords in the initial diagnostic report on the original image pixels are most prominent or concentrated, calculated by the spatial thermal map generation unit. These center coordinates accurately locate the region of interest that the model actually focuses on in the image. Methods for determining these center coordinates may include, but are not limited to: calculating a weighted average of the spatial thermal distribution, where the coordinates of each pixel are multiplied by its corresponding thermal value as the weight; identifying the region with the most concentrated thermal distribution through cluster analysis and taking its geometric center; or directly finding the pixel with the highest thermal value as the center.

[0055] The anatomical structure range described in the initial diagnostic report refers to the descriptive information about a specific anatomical structure identified from the text content of the initial diagnostic report using Natural Language Processing (NLP) technology, and precisely mapped to a specific region in the original pixel space of multimodal medical data. For example, when the report mentions "left upper lobe nodule," the system uses medical image segmentation and registration techniques to determine the corresponding region on the image as a specific coordinate range or region of interest (ROI). This anatomical structure range provides an objective benchmark for the accuracy assessment unit, used to compare with the center coordinates of the spatial thermal distribution to determine the accuracy of the report description.

[0056] Once the authenticity assessment unit receives the center coordinates of the spatial thermal distribution and the anatomical structure range described in the initial diagnostic report, it performs a comparative analysis. If the center coordinates of the spatial thermal distribution fall outside the anatomical structure range described in the initial diagnostic report—that is, the image region actually of interest to the model does not overlap with or deviates significantly from the anatomical structure region claimed in the report—the authenticity assessment unit will determine that the initial diagnostic report contains semantic illusion. This determination can be made by calculating the intersection-over-union (IoU) ratio, Euclidean distance, or by setting a preset distance threshold between the two regions. Once semantic illusion is determined, the system can issue an alert, prompting manual review, thereby avoiding the potential risk of misdiagnosis due to discrepancies between the report description and the actual image.

[0057] Through the above technical solution, this application can effectively solve the semantic illusion problem that may exist in the initial diagnostic report. When the attention focus of medical keywords calculated by the spatial heatmap generation unit on the original image pixels (i.e., the center coordinates of the spatial heat distribution) is inconsistent with the range of anatomical structures described in the initial diagnostic report text, the authenticity determination unit can promptly identify and determine that the report contains semantic illusion. This allows the system to not only verify the matching degree between the report description and image features, but also further ensure the authenticity and accuracy of the report's semantics, avoiding the potential risk of misdiagnosis due to discrepancies between the report description and the actual image. This mechanism significantly improves the reliability of diagnostic reports, providing clinicians with more reliable auxiliary diagnostic information.

[0058] In some embodiments of this application, the system further includes an interactive logic evolution module, which is used to capture user-manually adjusted graphical execution chain operation instructions and optimize the default assembly strategy of the dynamic meta-operator generation module using reinforcement learning algorithms.

[0059] Specifically, the interactive logic evolution module is a key component of the system, its core function being to enable the system's self-learning and adaptive evolution. This module continuously monitors and records all user actions when reviewing and modifying the graphical execution chain constructed by the dynamic meta-operator generation module. These actions may include, but are not limited to: adding new meta-operators, deleting unnecessary meta-operators, adjusting the execution order of meta-operators, and modifying the internal parameters or logical connections of meta-operators. By capturing this detailed user interaction data, the interactive logic evolution module obtains valuable expert feedback, which directly reflects the user's expectations and directions for improvement regarding the current graphical execution chain.

[0060] After capturing the user's manual adjustment instructions for the graphical execution chain, the interactive logic evolution module optimizes the default assembly strategy of the dynamic meta-operator generation module using a reinforcement learning algorithm. The reinforcement learning algorithm plays a role in both learning and decision-making here. It treats the process of the dynamic meta-operator generation module constructing the graphical execution chain under specific clinical task scenario labels and consistent diagnostic features as a decision-making process. The user's manual adjustment behavior is transformed into reward or punishment signals in reinforcement learning. For example, if the user makes significant modifications to the system-generated graphical execution chain, it can be considered a negative reward, prompting the algorithm to adjust its strategy to avoid similar problems in the future; conversely, if the user accepts the system-generated chain or makes only minor adjustments, it can be considered a positive reward, encouraging the algorithm to continue using or reinforce the current strategy. By continuously interacting with the environment (i.e., the generation process of the dynamic meta-operator generation module and user feedback), the reinforcement learning algorithm learns and iteratively updates its internal policy model, enabling the dynamic meta-operator generation module to generate graphical execution chains that better align with expert experience and actual clinical needs.

[0061] Through the aforementioned technical solutions, the system proposed in this application can transform from passively generating reports to actively learning and adapting. The interactive logic evolution module captures user manual adjustments to the graphical execution chain, integrating the experience and wisdom of human experts into the system's decision-making process. Utilizing reinforcement learning algorithms, the system can continuously optimize the default assembly strategy of the dynamic meta-operator generation module, enabling it to generate more accurate, efficient, and user-expected graphical execution chains and initial diagnostic reports when facing new clinical task scenarios or complex cases. This not only significantly reduces the frequency of manual intervention and correction, improving the level and quality of automated diagnostic report generation, but also endows the system with the ability to self-evolve and continuously optimize, thereby better adapting to ever-changing clinical needs and updates in medical knowledge, and enhancing the intelligence and practicality of the entire multimodal medical image decision arbitration and semantic verification generation system.

[0062] In some embodiments of this application, the system further includes a missing modality compensation module, which is used to synthesize a virtual feature map of the missing modality based on the existing modality data when some modality data in multimodal medical data is missing, using a generative adversarial network.

[0063] This missing modality compensation module aims to address the problem of incomplete multimodal medical data, ensuring that the system can still make effective decisions and generate reports even with missing data. Its core function is to identify and process missing modal data, filling information gaps through intelligent synthesis technology. Specifically, when some modalities in multimodal medical data are missing—for example, if the original plan was to acquire CT, MRI, and ultrasound data, but only CT and ultrasound data were acquired, leaving MRI data missing—the missing modality compensation module can use a generative adversarial network (GAN) to synthesize virtual feature maps of the missing modalities based on the existing modal data. A GAN is a deep learning model consisting of a generator network and a discriminator network. The generator network is responsible for learning and generating virtual feature maps with a similar distribution to the real missing modal data based on the input existing modal data. The discriminator network is responsible for distinguishing the virtual feature maps generated by the generator from the real modal data; through adversarial training, the generator is prompted to generate increasingly realistic virtual feature maps. Specifically, the generator can receive existing modality data as conditional input. For example, when MRI data is missing, the generator can use CT and ultrasound data as input to learn and generate virtual MRI feature maps that are highly consistent with real MRI data in feature space. These virtual feature maps are not original pixel-level images, but high-dimensional representations after feature extraction, which can capture key diagnostic information of the missing modality.

[0064] By introducing a missing modality compensation module, this application effectively addresses the challenges posed by the incompleteness of multimodal medical data. When some modal data is missing, this missing modality compensation module can utilize generative adversarial networks to intelligently synthesize virtual feature maps of the missing modalities based on existing modal data. These synthesized virtual feature maps effectively fill information gaps, ensuring that subsequent cross-modal consistency arbitration, dynamic meta-operator generation, and semantic consistency reverse verification modules can still obtain relatively complete and consistent input information, thereby guaranteeing the robustness of the system and the accuracy of decision-making under incomplete data conditions. This not only improves the applicability and reliability of the system, avoiding diagnostic interruptions or errors due to missing data, but also ensures that the system can continuously provide high-quality diagnostic support even under non-ideal data conditions, significantly improving the efficiency and safety of clinical decision-making.

[0065] In some embodiments of this application, the system further includes a privacy desensitization module, which is used to automatically identify and mask the examinee's identity and privacy information before the initial diagnostic report is output.

[0066] Specifically, the privacy anonymization module is designed to process initial diagnostic reports containing sensitive patient information to ensure that all identifiable privacy information is properly handled before the report is output or shared. Patient identity privacy information refers to any data that can identify a specific patient, alone or in combination with other information. This typically includes, but is not limited to, the patient's name, identification number, social security number, telephone number, email address, home address, date of birth, medical record number, biometric information, and any other personal information that directly or indirectly identifies the patient. This information may exist in text form in medical imaging diagnostic reports, such as in the report header, patient information section, or report body description.

[0067] In its implementation, the privacy anonymization module employs various techniques to identify and mask identity and privacy information. For identification, it utilizes Natural Language Processing (NLP) technology, employing Named Entity Recognition (NER) models to automatically detect sensitive entities such as names, ID numbers, medical record numbers, addresses, contact information, and dates of birth in the report text. It can also identify sensitive data in specific formats using pre-defined regular expressions or pattern matching algorithms. For masking, the privacy anonymization module can replace, delete, generalize, or encrypt the identified identity and privacy information according to pre-defined strategies. For example, it can replace the patient's name with an anonymous code, generalize a specific date to a year, or partially mask the ID number. The privacy anonymization module executes after the initial diagnostic report is generated but before its official output, ensuring the accuracy of the report content and the timeliness of privacy protection.

[0068] Through the aforementioned technical solution, after the system generates the initial diagnostic report, the privacy desensitization module can automatically identify and mask the examinee's identity and privacy information before the report is output. This effectively solves the problem of patient privacy protection in medical data processing and avoids the risk of sensitive information leakage. By desensitizing the report, not only are the legitimate rights and interests of patients protected and the compliance of the medical system improved, but the diagnostic report can also be stored, transmitted, and shared more widely while protecting privacy, promoting the rational use of medical data and the conduct of medical research. The introduction of the privacy desensitization module enables the entire task-aware multimodal medical image decision arbitration and semantic verification generation system to fully meet the requirements of data privacy protection while providing efficient and accurate diagnostic services, enhancing the system's practicality and social acceptance.

[0069] Another aspect of this application provides a task-aware multimodal medical image decision arbitration and semantic verification generation method, applicable to any of the aforementioned task-aware multimodal medical image decision arbitration and semantic verification generation systems. See also Figure 2 The method includes the following steps: Step 1: Acquire the examinee's multimodal medical data and identify the current clinical task scenario label through the multimodal clinical task perception module; Step 2: The cross-modal consistency arbitration module uses multiple parallel deep learning expert sub-models to extract features from multimodal medical data. When there are conflicts in the preliminary conclusions output by different expert sub-models, logical arbitration is performed based on the medical knowledge graph to output consistent diagnostic features. Step 3: The dynamic meta-operator generation module extracts meta-operators of corresponding granularity from the preset operator library based on the clinical task scenario tags, and constructs a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. Step 4: The semantic consistency reverse verification module back-projects the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data to verify the matching degree between the report description and the image features and outputs the verification results.

[0070] First, the multimodal clinical task awareness module acquires the patient's multimodal medical data and identifies the current clinical task scenario label. This step is the starting point of the entire diagnostic process and aims to collect various medical imaging data (e.g., CT, MRI, ultrasound, X-ray, etc.) and related clinical information from the patient. The multimodal clinical task awareness module analyzes this data, for example, by performing natural language parsing on request form information in the electronic medical record, to identify the current clinical task scenario label. These scenario labels can be categorized into emergency screening scenarios, routine diagnostic scenarios, or preoperative assessment scenarios, providing crucial contextual information for subsequent diagnostic procedures and ensuring that the system can adjust its behavior patterns according to actual needs.

[0071] Secondly, the cross-modal consistency arbitration module utilizes multiple parallel deep learning expert sub-models to extract features from multimodal medical data. When the preliminary conclusions output by different expert sub-models conflict, logical arbitration is performed based on the medical knowledge graph to output consistent diagnostic features. In this step, the system launches multiple specially trained deep learning expert sub-models for parallel processing of the acquired multimodal medical data. Each expert sub-model may focus on image analysis of a specific modality or the identification of a specific disease. When these expert sub-models give contradictory preliminary conclusions on the same lesion or diagnosis, the cross-modal consistency arbitration module intervenes. The cross-modal consistency arbitration module first uses a conflict detection unit to determine whether the deviation of image features from different modalities in the disease classification probability distribution exceeds a preset deviation threshold. Once a conflict is detected, the logical arbitration unit retrieves a pre-constructed medical knowledge graph, which contains rich medical facts, causal relationships, and diagnostic rules. Through matching degree analysis, the system assigns the highest decision weight to the expert sub-model with the highest matching degree to the causal judgment rule, thereby performing logical arbitration and ultimately outputting an integrated and corrected consistent diagnostic feature to improve the accuracy and reliability of the diagnosis.

[0072] Next, the dynamic meta-operator generation module extracts meta-operators of corresponding granularity from a pre-set operator library based on the clinical task scenario label, and constructs a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. After obtaining the consistency diagnostic features, this step is responsible for generating a preliminary diagnostic report based on the specific clinical task scenario. The dynamic meta-operator generation module extracts meta-operators with corresponding granularity from a pre-set operator library containing multiple functional modules (i.e., meta-operators) based on the clinical task scenario label identified in the first step. For example, when the clinical task scenario label is an emergency screening scenario, the granularity adjustment unit controls the graphical execution chain to only call the core positive / negative judgment operator to quickly provide a preliminary judgment; while when the clinical task scenario label is a preoperative assessment scenario, it calls all operators, including anatomical measurements and grading assessments, to provide detailed assessment information. These extracted meta-operators are then organized into a graphical execution chain, which takes the consistency diagnostic features as input and processes them according to pre-set logic, ultimately generating a structured initial diagnostic report.

[0073] Finally, the semantic consistency reverse verification module back-projects the textual semantic features from the initial diagnostic report onto the original pixel space of the multimodal medical data to verify the matching degree between the report description and image features and outputs the verification results. This step is crucial for ensuring the quality of the diagnostic report. The semantic consistency reverse verification module extracts textual semantic features from the initial diagnostic report, such as medical keywords and anatomical location descriptions, and calculates the cross-modal attention weights of these medical keywords on the original image pixels through the spatial heatmap generation unit, generating the corresponding spatial heatmap distribution. Subsequently, the authenticity determination unit analyzes this spatial heatmap distribution. For example, if the center coordinates of the spatial heatmap distribution fall outside the range of the anatomical structure described in the initial diagnostic report, it is determined that the initial diagnostic report contains semantic illusions. Through this reverse verification mechanism, the system can check whether the textual description in the report matches the actual visual features of the original medical image, thereby effectively identifying and correcting possible semantic biases or "illusions" in the report, ensuring the accuracy and credibility of the diagnostic report.

[0074] Through the above methods, this application provides a systematic workflow. First, a multimodal clinical task perception module acquires the examinee's multimodal medical data and identifies the current clinical task scenario label, ensuring the targetedness and efficiency of subsequent processing. Second, a cross-modal consistency arbitration module utilizes multiple parallel deep learning expert sub-models to extract features from the multimodal medical data. When there are conflicts in the preliminary conclusions output by different expert sub-models, logical arbitration is performed based on a medical knowledge graph to output consistent diagnostic features, effectively resolving potential conflicts between multiple expert models and significantly improving the reliability and consistency of diagnostic features. Furthermore, a dynamic meta-operator generation module extracts meta-operators of corresponding granularity from a preset operator library based on the clinical task scenario label, and constructs a graphical execution chain based on the consistent diagnostic features to generate an initial diagnostic report. This allows the diagnostic report to flexibly adjust its granularity and level of detail according to different clinical task scenarios, meeting diverse clinical needs. Finally, the semantic consistency reverse verification module back-projects the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data to verify the matching degree between the report description and the image features and outputs the verification results. As a key quality control link, it can actively detect and correct any semantic deviations or "illusions" that may exist in the report, ensuring a high degree of matching between the diagnostic report content and the original image features, thereby greatly enhancing the decision-making accuracy, reliability and clinical practical value of the intelligent diagnostic system.

[0075] In some embodiments of this application, it is further proposed that in step two above, the deviation of different modal image features in the disease classification probability distribution is monitored in real time, and when the deviation exceeds a preset deviation threshold, the causal judgment rules of medical knowledge graph are introduced to redistribute the decision weights of the expert sub-model.

[0076] Specifically, real-time monitoring of the deviation of different modalities of image features from disease classification probability distributions refers to the system continuously evaluating the consistency or difference between the predicted probability distributions of medical image data from different modalities (e.g., CT, MRI, ultrasound) after processing by their respective expert sub-models when classifying a specific disease. This monitoring can calculate the distance between the disease classification probability vectors output by different modal expert sub-models, for example, using metrics such as KL divergence, JS divergence, or Euclidean distance, or compare their prediction confidence differences for key disease categories. Real-time monitoring ensures that the system can dynamically and promptly capture potential inconsistencies between modalities.

[0077] When the deviation exceeds a preset deviation threshold, it means that the differences or inconsistencies in the disease classification probability distributions of the different modalities of image features detected above have reached a pre-set limit. At this point, the system will trigger a more in-depth arbitration mechanism. This preset deviation threshold can be set based on historical data, expert experience, or specific clinical needs. For example, when the difference in classification probability between two modalities for the same disease exceeds a certain percentage, or when the distance between their probability distributions exceeds a certain statistical significance level, it is considered that the deviation is too large and further intervention is required.

[0078] Building upon this foundation, the system incorporates causal determination rules from a medical knowledge graph. This means that upon detecting a significant deviation, the system no longer relies solely on simple majority voting or preset weights, but actively queries the medical knowledge graph for rules regarding the causal relationships between disease occurrence and development, imaging manifestations, and pathophysiology. These causal determination rules can include statements such as "If imaging feature A is present, it is usually accompanied by pathological change B, and modality C is more sensitive to this." By introducing these rules, the system can understand the potential causes of intermodal conflicts at a deeper level of medical logic, thus providing a solid medical basis for subsequent decision-making weight adjustments.

[0079] Finally, based on the analysis results of the causal determination rules introduced from the medical knowledge graph, the system reallocates the decision weights of the expert sub-models. This means that the system dynamically adjusts the influence of each expert sub-model participating in the arbitration in the final decision. For example, if the causal determination rules indicate that a certain modality (and its corresponding expert sub-model) has higher reliability or specificity for diagnosing a specific lesion in the current clinical context, then the decision weight of the expert sub-model corresponding to that modality will be increased, while the weights of other modalities may be decreased. This reallocation is based on medical logic and aims to improve the accuracy and clinical relevance of the arbitration results.

[0080] Through the above technical solution, this application introduces a more refined and dynamic conflict handling mechanism in the multimodal medical imaging decision arbitration process. When the system detects significant deviations in the disease classification probability distribution of different modal imaging features in real time, i.e., large disagreements occur between expert sub-models, it no longer relies solely on preset arbitration logic, but proactively introduces causal judgment rules from the medical knowledge graph for in-depth analysis. This causal judgment based on medical logic helps the system understand the underlying causes of conflicts and intelligently redistributes the decision weights of each expert sub-model accordingly. For example, under certain specific pathological manifestations, if the medical knowledge graph indicates that the imaging manifestations of a certain modality have higher diagnostic specificity or sensitivity, the system will correspondingly increase the weight of the expert sub-model corresponding to that modality, thereby enabling it to play a greater role in the final decision. This effectively avoids misjudgments that may be caused by simple majority voting or fixed weight allocation in complex or ambiguous cases, significantly improving the accuracy, robustness, and reliability of clinical decision-making of the cross-modal consistency arbitration module when handling intermodal conflicts, enabling the system to more intelligently and accurately integrate multimodal information and output more convincing consistent diagnostic features.

[0081] In some embodiments of this application, step three is further proposed to include: calculating the cross-modal attention weights of keywords in the initial diagnostic report on the original image pixels and generating a spatial thermal distribution; when the center coordinates of the spatial thermal distribution do not match the anatomical location corresponding to the text semantic features, outputting a semantic illusion warning and prompting manual review.

[0082] Specifically, to effectively detect semantic illusions in initial diagnostic reports, this application first calculates the cross-modal attention weights of keywords in the initial diagnostic report on pixels in the original image, and generates a spatial heatmap based on this. This process aims to quantify the association strength between key medical terms in the diagnostic report (e.g., "nodule," "tumor," "inflammation") and specific pixel regions in the original medical images (e.g., CT, MRI images). By employing a deep learning-based cross-modal attention mechanism, such as using Transformer or Attention-GAN models, the text embeddings of the diagnostic report (pre-trained models such as BERT and Word2Vec) are fused with the visual feature embeddings of the medical images (models such as CNN and ViT). During the fusion process, the system calculates the attention score between the text keywords and each pixel in the image feature map; a higher score indicates greater attention from the keyword to that pixel region. Subsequently, the calculated attention scores are mapped back to the spatial coordinates of the original image, forming a two-dimensional or three-dimensional "heatmap." Darker colors or higher brightness on the heatmap indicate a stronger association between the region and the specific keywords in the report. This allows the textual descriptions in the diagnostic report to have a visual spatial location on the image. Keyword extraction can be performed from the initial diagnostic report using natural language processing (NLP) techniques, such as TF-IDF, TextRank, or entity recognition methods based on medical dictionaries, to identify core words describing lesions, anatomical structures, or disease states.

[0083] Building upon this, this application further outputs a semantic illusion warning and prompts manual review when the center coordinates of the spatial thermal distribution do not match the anatomical location corresponding to the text semantic features. This step is the core verification mechanism used to detect whether there is a "semantic illusion" in the report, i.e., the report text is grammatically correct and seems reasonable, but the entity or event it describes does not have a corresponding spatial location on the image, or the location is incorrect. In specific implementation, the center coordinates of the spatial thermal distribution can be determined by calculating the geometric center or centroid of the region with the highest attention weight in the heatmap. At the same time, obtaining the anatomical location corresponding to the text semantic features requires the support of a medical knowledge base or anatomical ontology. For example, if the report mentions "right upper lobe nodule," the system needs to be able to query the knowledge base to find the approximate spatial range of "right upper lobe" on a standard anatomical model or patient image. This can be achieved through pre-labeled anatomical region masks or a deep learning-based anatomical structure recognition model. Subsequently, the center coordinates of the thermal distribution are compared with the anatomical location described in the text. The comparison method can be to calculate the Euclidean distance, IoU (Intersection over Union), or determine whether the thermal center falls within a predefined anatomical region. A preset deviation threshold can be set. For example, if the distance between the thermal center and the target anatomical area exceeds a certain pixel distance, or if the thermal center falls completely outside the target anatomical area, it is considered a mismatch. When a mismatch is determined, the system will generate an alert message, clearly indicating that there may be semantic illusion in the report, and providing relevant evidence (such as the overlay of the thermal map and the anatomical area). At the same time, it will prompt doctors or professionals to manually review the report to confirm whether there is an actual error, thereby avoiding potential risks that may arise from the automated diagnostic system.

[0084] Through the above technical solution, this application can calculate the cross-modal attention weights of keywords in the initial diagnostic report on the original image pixels and generate a spatial thermal distribution, thereby intuitively revealing the spatial correspondence between the report text description and the image. When the center coordinates of this spatial thermal distribution do not match the anatomical location corresponding to the semantic features of the text, the system can promptly output a semantic illusion warning and prompt manual review. This effectively solves the problem that traditional verification methods may fail to detect discrepancies between the report description and the actual lesion location in the image, significantly improving the accuracy and reliability of the diagnostic report, avoiding the risk of misdiagnosis caused by "semantic illusion," and providing clinicians with safer and more reliable auxiliary diagnostic information.

[0085] The following example will provide a more detailed explanation of the above technical solution: In a specific clinical scenario, a patient visits a hospital due to chest discomfort, and the doctor orders multimodal medical imaging examinations including chest CT, MRI, and PET-CT. Upon receiving this imaging data, the system first uses a multimodal clinical task perception module to perform natural language parsing on the request form information in the patient's electronic medical record. The system identifies the current clinical task scenario as a "routine diagnostic scenario" because the request form does not explicitly indicate an emergency or preoperative assessment need. The multimodal clinical task perception module then acquires the patient's chest CT, MRI, and PET-CT multimodal medical data.

[0086] Next, the cross-modal consistency arbitration module begins operation. This module utilizes multiple parallel deep learning expert sub-models to extract features and perform preliminary analysis on lung lesions in CT, MRI, and PET-CT image data. For example, one expert sub-model might determine that the lesion is benign calcification based on CT images, while another expert sub-model based on PET-CT images might suggest that the lesion is metabolically active and potentially malignant. At this point, the conflict detection unit determines that the deviation of the image features from different modalities in the disease classification probability distribution exceeds a preset deviation threshold, indicating a conflict in the preliminary conclusions. The logical arbitration unit is then activated, which retrieves causal determination rules for the diagnosis of lung nodules from the medical knowledge graph, such as "if CT shows dense calcification inside the lesion and the PET-CT standardized uptake value (SUVmax) is low, then the probability of benignity is high." The system assigns the highest decision weight to the expert sub-model (e.g., the CT expert sub-model) that best matches this causal determination rule, thereby outputting a consistent diagnostic feature, i.e., a judgment leaning towards benign calcified nodules. This solves the problem of the lack of an effective decision-making arbitration mechanism for cross-modal data conflicts in existing technologies.

[0087] Subsequently, the dynamic meta-operator generation module extracts meta-operators of corresponding granularity from a pre-set operator library based on the clinical task scenario label of "routine diagnostic scenario". Since it is a routine diagnosis, the granularity adjustment unit controls the graphical execution chain to call a series of medium-granularity operators, including lesion detection, morphological measurement, density analysis, edge feature assessment, and benign / malignant probability prediction. These meta-operators construct a graphical execution chain based on the aforementioned consistent diagnostic features; for example, lesion segmentation is performed first, followed by morphological measurement, and finally, benign / malignant probability assessment. After this execution chain runs, the system generates an initial diagnostic report, which describes "a calcified nodule, approximately 1.0 cm in size, with clear borders, considered a benign lesion, seen in the upper lobe of the right lung".

[0088] Finally, the semantic consistency reverse verification module verifies the initial diagnostic report. The spatial heatmap generation unit of this module calculates the cross-modal attention weights of medical keywords in the report (such as "calcified nodule" and "right upper lobe") on the original CT image pixels and generates the corresponding spatial heatmap distribution. For example, the system generates a heatmap showing the area of ​​greatest interest for the term "calcified nodule" on the CT image. The authenticity determination unit checks whether the center coordinates of this spatial heatmap distribution accurately fall within the actual calcified nodule area in the original image. If the center coordinates of the heatmap deviate from the actual nodule location or point to an irrelevant area in the image, the authenticity determination unit will determine that the initial diagnostic report contains semantic illusions and output a verification result, indicating a problem with the matching degree between the report description and image features, requiring manual review. This effectively solves the problem in existing technologies where AI-generated diagnostic reports often exhibit "illusion" phenomena where semantic descriptions are disconnected from the pixel features of the original image, lacking reliable authenticity verification methods.

[0089] During this process, if user A, acting as a radiologist, discovers an inaccurate result from a measurement operator while reviewing the report and manually adjusts the measurement parameters in the graphical execution chain or replaces the measurement operator, the interactive logic evolution module will capture this operation instruction and utilize reinforcement learning algorithms to optimize the default assembly strategy of the dynamic meta-operator generation module. This will enable the system to more intelligently select and assemble operators in similar "routine diagnostic scenarios" in the future. Furthermore, if the patient's PET-CT data is missing for any reason, the missing modality compensation module can use a generative adversarial network to synthesize a virtual PET feature map of the missing modality based on existing CT and MRI data for analysis by the expert sub-model. Before the initial diagnostic report is output, the privacy desensitization module automatically identifies and masks the patient's identity and privacy information to ensure data security.

[0090] Through the above process, this system can flexibly adapt its processing logic according to different clinical task needs, effectively arbitrate cross-modal data conflicts, and reliably verify the semantic authenticity of the generated diagnostic reports, thereby providing more accurate and safer medical imaging diagnostic assistance.

[0091] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A task-aware multimodal medical image decision arbitration and semantic verification generation system, characterized in that, include: The multimodal clinical task perception module is used to acquire the examinee's multimodal medical data and identify the current clinical task scenario label; The cross-modal consistency arbitration module is used to extract features from the multimodal medical data using multiple parallel deep learning expert sub-models, and to perform logical arbitration based on the medical knowledge graph to output consistent diagnostic features when there are conflicts in the preliminary conclusions output by different expert sub-models. The dynamic meta-operator generation module is used to extract meta-operators of corresponding granularity from a preset operator library according to the clinical task scenario tags, and to construct a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. The semantic consistency reverse verification module is used to back-project the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data in order to verify the matching degree between the report description content and the image features and output the verification result.

2. The system according to claim 1, characterized in that, The multimodal clinical task perception module categorizes the clinical task scenario tags into emergency screening scenarios, routine diagnosis scenarios, or preoperative assessment scenarios by performing natural language parsing on the application form information in the electronic medical record.

3. The system according to claim 1, characterized in that, The cross-modal consistency arbitration module includes: The conflict detection unit is used to determine whether the deviation of image features of different modalities from the disease classification probability distribution exceeds a preset deviation threshold. The logic arbitration unit is used to retrieve the causal determination rules in the medical knowledge graph when the deviation exceeds the preset deviation threshold, and assign the highest decision weight to the expert sub-model that has the highest matching degree with the causal determination rule.

4. The system according to claim 2, characterized in that, The dynamic meta-operator generation module includes a granularity adjustment unit. The granularity adjustment unit is used to control the graphical execution chain to call only the core positive / negative judgment operator when the clinical task scenario label is the emergency screening scenario; and to control the graphical execution chain to call all operators, including anatomical measurement and grading assessment, when the clinical task scenario label is the preoperative assessment scenario.

5. The system according to claim 1, characterized in that, The semantic consistency reverse verification module includes a spatial heatmap generation unit, which is used to calculate the cross-modal attention weights of medical keywords in the initial diagnostic report on the original image pixels and generate the corresponding spatial heatmap distribution.

6. The system according to claim 5, characterized in that, The semantic consistency reverse verification module includes an authenticity determination unit, which is used to determine that the initial diagnostic report contains semantic illusions when the center coordinates of the spatial thermal distribution fall outside the range of the anatomical structure described in the initial diagnostic report.

7. The system according to claim 1, characterized in that, The system also includes an interactive logic evolution module, which is used to capture user commands to manually adjust the graphical execution chain and to optimize the default assembly strategy of the dynamic meta-operator generation module using reinforcement learning algorithms.

8. The system according to claim 1, characterized in that, The system also includes a missing modality compensation module, which is used to synthesize a virtual feature map of the missing modality based on the existing modality data when some modality data is missing in the multimodal medical data using a generative adversarial network.

9. The system according to claim 1, characterized in that, The system also includes a privacy desensitization module, which is used to automatically identify and mask the examinee's identity and privacy information before the initial diagnostic report is output.

10. A task-aware multimodal medical image decision arbitration and semantic verification generation method, applied to the system of any one of claims 1 to 9, characterized in that, Includes the following steps: Step 1: Acquire the examinee's multimodal medical data and identify the current clinical task scenario label through the multimodal clinical task perception module; Step 2: The cross-modal consistency arbitration module uses multiple parallel deep learning expert sub-models to extract features from the multimodal medical data. When there are conflicts in the preliminary conclusions output by different expert sub-models, logical arbitration is performed based on the medical knowledge graph to output consistent diagnostic features. Step 3: The dynamic meta-operator generation module extracts meta-operators of corresponding granularity from the preset operator library according to the clinical task scenario tags, and constructs a graphical execution chain based on the consistency diagnostic features to generate an initial diagnostic report. Step 4: The semantic consistency reverse verification module back-projects the textual semantic features in the initial diagnostic report to the original pixel space of the multimodal medical data to verify the matching degree between the report description content and the image features and outputs the verification results.

Images