A 3D medical image intelligent question and answer method and device based on uncertainty perception and instruction guidance feature compression

Abstract

The application discloses a kind of based on uncertainty perception and instruction guide feature compression 3D medical image intelligent question and answer method and device, it is related to medical image artificial intelligence analysis technical field, including obtaining 3D medical image and user question and answer instruction and carries out standardization pretreatment;Local details and global context features are respectively extracted and cross-scale fusion is carried out by double-flow 3D visual encoder;Fusion features are mapped into normal inverse gamma distribution parameters based on evidence deep learning, quantify uncertainty and generate uncertainty conditional word element;The semantics of question and answer instruction is used to independently conditionally scale modulation to multiple learnable query vectors, and compressed visual word element is generated after cross attention calculation;Each word element sequence is sequentially spliced and input into the large language model of LoRA adapter, and the overall confidence score and the text answer are jointly output.The application realizes the clinical auxiliary diagnosis with confidence estimation, and significantly improves the safety and reliability of system deployment.

Description

Technical Field

[0001] This invention relates to the field of medical image artificial intelligence analysis technology, and in particular to a 3D medical image intelligent question answering method and device based on uncertainty perception and instruction-guided feature compression. Background Technology

[0002] With the rapid development of medical imaging technology, computed tomography (CT) has become an indispensable tool in clinical diagnosis, widely used in various fields such as tumor detection and cardiovascular disease diagnosis. Statistics show that more than 300 million CT scans are performed globally each year, and this number continues to grow. However, accurate interpretation of CT images faces significant challenges: accurate interpretation requires radiologists to possess in-depth anatomical knowledge and long-term professional training; a single CT scan may generate hundreds of slides, placing a heavy workload on doctors and easily leading to fatigue; different doctors may interpret the same image subjectively; and manual image interpretation in emergency situations is time-consuming.

[0003] In recent years, multimodal large language models (such as the M3D series of works) that combine 3D vision Transformers with large language models have been initially applied to question-answering tasks processing CT volumetric data. These methods typically employ a single 3DViT encoder to extract global features, map them to the language space through simple linear or MLP projection layers, and then use fine-tuning techniques such as LoRA to adapt the large language model. However, existing techniques still have the following significant limitations: Lack of uncertainty modeling: Existing methods only output text results and do not provide confidence estimates. When there is significant uncertainty in the model's judgment of a case, providing a definitive answer would pose a serious risk of misdiagnosis, thus hindering the reliable deployment of AI-assisted diagnosis in clinical practice.

[0004] Visual feature extraction lacks multi-scale coordination: using a single-scale encoder makes it difficult to simultaneously take into account fine-grained local lesion information (such as micronodules) and macroscopic anatomical context (such as organ spatial relationships), resulting in insufficient perception of small lesions.

[0005] Feature compression lacks task orientation: Visual feature mapping usually adopts a fixed compression strategy, which retains a large amount of redundant visual information that is irrelevant to diagnosis. This not only wastes the context window of a large language model, but also dilutes the attention weight of key diagnostic features. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a 3D medical image intelligent question answering method and device based on uncertainty perception and instruction-guided feature compression.

[0007] To solve the above technical problems, the technical solution of the present invention is as follows: A 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression includes: Obtain the 3D medical images to be analyzed and their corresponding user Q&A commands; Multi-scale parallel feature extraction is performed on 3D medical images to obtain local detail features with fine-grained information and global context features with macroscopic anatomical information. The local detail features and global context features are then fused across scales to generate a multi-scale fused feature sequence. Based on evidence-based deep learning, multi-scale fused feature sequences are mapped to distribution parameters. Cognitive uncertainty and accidental uncertainty of corresponding visual features are calculated based on the distribution parameters, and uncertainty embedding vectors are generated. The mean parameter in the distribution parameters is modulated using the uncertainty embedding vectors to generate uncertainty conditionalized lexical units. Extract semantic information from user question-and-answer instructions, use the semantic information to independently conditionally scale and modulate multiple preset learnable query vectors, and use the modulated query vectors as queries, and use multi-scale fused feature sequences as keys and values ​​to perform cross-attention calculation to generate instruction-guided compressed visual lexical units. The compressed visual lexical units, uncertain conditionalized lexical units, and user question-and-answer instructions are concatenated and input into a pre-trained large language model for decoding and inference, generating question-and-answer text results and corresponding confidence scores.

[0008] Secondly, the present invention also provides a 3D medical image intelligent question-answering system based on uncertainty perception and instruction-guided feature compression, comprising: The data acquisition module is used to acquire the 3D medical images to be analyzed and their corresponding user question-and-answer commands; The multi-scale coding module is used to perform multi-scale parallel feature extraction on 3D medical images, respectively acquiring local detail features with fine-grained information and global context features with macroscopic anatomical information, and fusing the local detail features and global context features across scales to generate a multi-scale fused feature sequence. The uncertainty perception module is used to map multi-scale fused feature sequences into distribution parameters based on evidence-based deep learning, calculate the cognitive uncertainty and accidental uncertainty of the corresponding visual features based on the distribution parameters and generate uncertainty embedding vectors, and use the uncertainty embedding vectors to modulate the mean parameter in the distribution parameters to generate uncertainty conditionalized lexical units. The adaptive compression module is used to extract the semantic information of user question-and-answer instructions, use the semantic information to independently conditionally scale and modulate multiple preset learnable query vectors, and use the modulated query vector as the query, and use the multi-scale fused feature sequence as the key and value to perform cross-attention calculation to generate instruction-guided compressed visual lexical units. The model inference module is used to concatenate compressed visual lexical units, uncertain conditionalized lexical units, and user question-and-answer instructions, and then input them into the large language model for decoding and inference to generate question-and-answer text results and corresponding confidence scores.

[0009] Thirdly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-mentioned question-and-answer method.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described question-and-answer method.

[0011] The beneficial effects of this invention are: (1) This invention abandons the traditional deterministic linear mapping at the projection layer and innovatively introduces a projection structure based on evidence learning, which maps multi-scale fused feature sequences to normal inverse gamma distribution parameters and generates uncertainty conditionalized lexical units based on this. This enables the system to quantify the reliability of each diagnostic conclusion while outputting text answers, providing multi-level uncertainty reports for clinical decision-making and significantly reducing the risk of black-box misdiagnosis.

[0012] (2) After generating the confidence score, the present invention sets intelligent routing discrimination logic with a first threshold and a second threshold. The system can automatically identify low-confidence cases and recommend manual review, avoiding the system from giving misleading definitive answers in uncertain situations, and greatly improving the credibility and security of the system.

[0013] (3) By introducing different pattern marker lexical units (conservative pattern instruction or exploratory pattern instruction) into the splicing sequence, this invention can flexibly control the output of a large language model with high certainty or a comprehensive analysis report containing speculative diagnosis without retraining the model, perfectly matching the differentiated needs of high-risk diagnosis scenarios and scientific research and teaching scenarios.

[0014] (4) This invention utilizes 3D patch blocks of the first size to extract local details, and uses 3D patch blocks of the second size with a large volume to extract global context, and performs semantic fusion through a cross-scale attention mechanism. This dual-stream parallel design enables the system to simultaneously capture features of small lesions (such as lung nodules of a few millimeters) and macroscopic anatomical relationships, achieving a substantial breakthrough in detection accuracy.

[0015] (5) This invention extracts the semantic vector of the user's question-and-answer instruction to generate an attention modulation factor and dynamically scales the learnable query vector. This mechanism enables visual attention to be dynamically allocated according to the specific question-and-answer task, and the limited visual lexical units carry diagnostic information related to the current question to the maximum extent, eliminating the occupation of the context window by redundant features.

[0016] (6) This invention adopts a multi-task collaborative loss function, and in particular, introduces a “guided compression allocation loss”. This design converts the uncertainty of the visual region into target weights, and guides the model in reverse through cross-entropy loss, so that the query vector is forced to retain the visual feature details of the high uncertainty region (difficult lesions), thus realizing deep feedback coupling between visual perception and language reasoning. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the intelligent question-answering method for 3D medical images based on uncertainty perception and instruction-guided feature compression provided by the present invention. Figure 2 This is a structural diagram of a dual-stream multi-scale 3D vision encoder. Figure 3 A structural diagram of the uncertainty perception projection and reasoning module; Figure 4 Flowchart for uncertain routing and controllable response modes; Figure 5 Diagram of an instruction-guided adaptive feature compressor structure; Figure 6 Diagram showing the structure of the collaborative training strategy and loss function; Figure 7 This is a schematic diagram of the framework of the 3D medical image intelligent question-answering system based on uncertainty perception and instruction-guided feature compression provided by the present invention. Figure 8 The overall architecture diagram of the 3D medical image intelligent question-answering system based on uncertainty perception and instruction-guided feature compression provided by the present invention; Figure 9 A schematic diagram of the computer device provided by the present invention. Detailed Implementation

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0020] Example 1: See Figures 1-6 This embodiment provides a 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression. The method specifically includes the following steps: Step S101: Obtain the 3D medical image to be analyzed and its corresponding user question and answer instructions.

[0021] Specifically, for acquiring 3D medical images, the input data is the raw CT volume data. To eliminate sampling differences between different devices, the Spacingd transform from the MONAI library was used to uniformly resample the voxel spacing to 1mm×1mm×1mm, and Resized was used to uniformly adjust the physical dimensions of the volume. = (128, 128, 128). Meanwhile, the CT value (Hounsfield Unit) typically has a wide range. This invention, based on the sampling range of the pre-trained model, strictly clips the value range to [-1000, 400] to accurately capture image information of key diagnostic tissues such as air, soft tissue, and bone. Subsequently, the pixel values ​​are scaled to the [0, 1] interval as model input according to the following formula: .

[0022] To obtain user response instructions, the system receives natural language text input by doctors or users through a human-computer interaction interface (or API interface) as user question-and-answer instructions. In this embodiment, user question-and-answer instructions can take various forms according to clinical needs, including but not limited to: closed-ended diagnostic inquiries (e.g., "Are there any small pulmonary nodules in this CT image?", "Does the patient's cardiovascular system have calcification?") or open-ended disease analysis requests (e.g., "Please describe in detail the abnormal lesions in the liver region of this 3D image and provide a preliminary pathological inference"). After obtaining the natural language text, the system saves it as a text sequence. This text sequence will be used in subsequent steps to extract semantic vectors to guide the adaptive compression of visual features and will serve as the initial prompt text for decoding and inference by the large language model.

[0023] Step S102: Perform multi-scale parallel feature extraction on 3D medical images, obtain local detail features with fine-grained information and global context features with macroscopic anatomical information respectively, and fuse the local detail features and global context features across scales to generate a multi-scale fused feature sequence.

[0024] Specifically, existing methods typically use a single 3D ViT encoder to process the entire CT volume, making it difficult to simultaneously capture fine-grained local lesion features and global anatomical context. See also Figure 2This application designs a dual-stream parallel coding architecture, which focuses on the extraction of visual information at different scales.

[0025] (1) Local Detail Stream: Focuses on extracting fine-grained local features to enhance the perception of small lesions.

[0026] Use smaller 3D patch blocks (e.g., 4×4×4) The volume is divided into a dense sequence of local patches. This is mapped using linear projection. =384-dimensional vector, superimposed with 3D position encoding, input into a lightweight 3D Transformer encoder (number of layers) =6, number of attention heads =8). Extracting local feature sequences. Number of patches = 32×32×32 = 32768.

[0027] (2) Global Context Stream: focuses on extracting macroscopic anatomical features.

[0028] Large-size 3D patch segmentation (e.g., 16×16×16), the volume is divided into a sparse global patch sequence. Mapped to... After the 3D vectors are superimposed with positional encodings, they are input into another set of 3D Transformer encoders (layer number). =6, number of attention heads =6). Extract global feature sequences. Number of patches And satisfy .

[0029] (3) Multi-scale feature fusion.

[0030] See Figure 2 It integrates local and global features through a learnable fusion mechanism: Dimension alignment: Unifying the features of two streams to the same dimension through linear projection. : .

[0031] Cross-scale attention fusion: A cross-scale attention module (8-head attention, 2048 feedforward dimensions) is designed, using global features as queries and local features as keys / values, to achieve semantic guidance of local details from the global context. .

[0032] Fusion Output: The final result is a multi-scale fused feature sequence. At this point, the feature sequence dimension is 33280×768 (random sampling of local flows can be performed during training to control the amount of computation), and it contains both fine-grained local information and global anatomical context.

[0033] Step S103: Based on evidence-based deep learning, the multi-scale fused feature sequence is mapped to distribution parameters. The cognitive uncertainty and accidental uncertainty of the corresponding visual features are calculated according to the distribution parameters, and uncertainty embedding vectors are generated. The mean parameter in the distribution parameters is modulated using the uncertainty embedding vectors to generate uncertainty conditionalized lexical units.

[0034] Specifically, see Figure 3 Uncertainty modeling is introduced into the process of mapping visual features to the language space, enabling the system to output diagnostic confidence while generating answers.

[0035] Unlike traditional deterministic MLP projection, this application designs a projection layer based on Evidential Deep Learning, see [link to relevant documentation]. Figure 3 Each visual token is mapped to parameters of a Normal-Inverse-Gamma (NIG) distribution. The projection layer consists of a shared backbone network (two-layer MLP). (GELU activation) and four parallel output heads (all dimensions are 1). )composition: ,in: The mean parameter corresponds to the traditional visual token embedding. The precision parameter represents the degree of confidence in the mean estimate; The shape parameter controls the degree of concentration of the distribution; This is a scaling parameter, which is related to variance estimation.

[0036] Based on the above parameters, the cognitive uncertainty of each visual token is calculated as follows: The uncertainty of chance is: .

[0037] In order to inject uncertain information into the inference process of LLM, such as Figure 3 As shown, this application designs an uncertain token generation mechanism: Uncertainty Embedding: The uncertainty scalar of each visual token is extended into an uncertainty embedding vector through sinusoidal position encoding.

[0038] Uncertainty Conditioning: Element-level Modulation of Uncertainty Embedding and Visual Token Embedding: ,in, Represents the i-th uncertain conditionalized term; The i-th mean parameter of the output of the normal inverse gamma distribution is used as the initial visual feature representation. This represents the uncertainty embedding vector corresponding to the i-th visual feature; and These are the trainable weight matrix and bias term of the modulation network, respectively; This represents the Sigmoid activation function; This indicates an element-wise multiplication operation.

[0039] Embeddings corresponding to visual tokens with high uncertainty are suppressed, naturally reducing the LLM's reliance on these regions during inference. This is achieved through the generation of uncertainty-conditionalized lexical units, enabling adaptive feature suppression of high-uncertainty regions.

[0040] Step S104: Extract the semantic information of the user's question-and-answer instructions, use the semantic information to scale and modulate the preset learnable query vector, and use the modulated query vector as the query, and use the multi-scale fusion feature sequence as the key and value to perform cross-attention calculation to generate instruction-guided compressed visual lexical units. Specifically, traditional methods directly input all visual tokens (potentially hundreds) into the LLM, wasting computational resources and diluting key information. For example... Figure 5 As shown, this application designs an adaptive compression mechanism guided by LLM semantic feedback, which dynamically selects and compresses visual features according to the semantic requirements of the question-answering task.

[0041] First, initialize a learnable query vector with a query count of M=64 and a dimension of d=768. Using a normal distribution Initialization, where, .

[0042] Secondly, the input question text is processed through the word embedding layer of the semantic large model to obtain the question embedding. The semantic vector of the problem is extracted using mean pooling. The semantic vector of the question is transformed linearly to generate an attention modulation factor, which is then used to scale and modulate the query vector according to the following formula: ,in, This represents the modulated query vector; This represents a predefined, learnable query vector; The projection matrix that maps the semantic space to the visual feature space. For bias terms; This represents the Sigmoid activation function; This indicates an element-wise multiplication operation, which causes the modulated query vector to carry the attention weights corresponding to the user's question-and-answer instructions.

[0043] In particular, the projection matrix Bias term Semantic vectors After transformation, a semantic guiding factor of dimension M×d is generated, which enables independent conditional modulation of each query vector and its features to obtain a task-aware query vector. .

[0044] Finally, using the modulated query vector For the query, feature sequences are fused at multiple scales. Using the key and value, M compressed visual lexical units are extracted through a cross-attention mechanism: .

[0045] Step S105: Concatenate the compressed visual lexical units, the uncertain conditionalized lexical units, and the user's question-and-answer instructions, and input them into the large language model for decoding and inference to generate the question-and-answer text results and the confidence scores corresponding to the question-and-answer text results.

[0046] Specifically, regarding the LLM and LoRA configuration: Qwen 1.5-1.8B is used as the base language model, and LoRA technology is employed for efficient parameter fine-tuning. The rank r is set to 16, and the scaling parameters are adjusted accordingly. =16. Freeze the pre-trained model weights. The forward propagation formula is: ,in, , , The target injection module is configured in the attention layer of the Transformer block, i.e. Freeze all parameters except LoRA, projection layer, and compressor.

[0047] Regarding the construction of the input sequence: the compressed visual token sequence Conditioning with uncertainty Concatenate with text token sequence: ,in, Mark the [CONSERVATIVE] or [EXPLORATION] mode.

[0048] Regarding the multi-task collaborative training objective: Create the AdamW optimizer (initial learning rate) The joint optimization formula for the total loss function is: (This is incomplete and requires further context.) , in, Indicates the total loss; This represents the loss for answer generation based on text. This represents the expected calibration error loss used to match the confidence score with the prediction accuracy; This represents the compression diversity regularization loss used to constrain different learnable query vectors to focus on different visual regions; This indicates the loss from the guided compression allocation; These are the preset balancing hyperparameters, and the balancing hyperparameters are set. It is 0.1. It is 0.01. It is 0.05.

[0049] The loss for generating the answer is: ,in It is the t-th token of the target answer. It includes projection layer parameters and LoRA parameters.

[0050] Uncertainty calibration loss (ECE) is: ,in For the b-th confidence interval, and These represent the accuracy and average confidence level for that interval, respectively.

[0051] Compression Diversity Regularization Loss: To prevent conditional query vectors from converging to the same visual region, regularization constraints are applied to the task-aware query vectors participating in cross-attention calculation. ,in, For cosine similarity, and These represent task-aware query vectors respectively. Independent vectors in the query vector. This loss applies to the conditionalized query vector. The corresponding attention distribution encourages focusing on complementary areas.

[0052] Uncertainty-Driven Compressed Distribution Loss: Leveraging the Total Uncertainty of Visual Tokens This is used to assist in compression allocation. Since the compressor input is a multi-scale fused feature... Uncertainty information does not directly participate in the forward compression calculation, but rather guides the allocation of losses through uncertainty. During training, the compressor's attention distribution is inversely constrained to preserve more visual details in regions of high uncertainty. The target compression weight distribution is constructed based on the uncertainty of the visual tokens. ,in, Let the compressor actually assign attention weights to the i-th visual token. For regions with high uncertainty (potentially containing intractable lesions), assign higher target compression weights to guide more query vectors to focus on that region and retain more details. Further define the uncertainty-guided assignment loss: ,in, Weights are assigned to the actual components. During the training phase, the gradient of the diagnostic text loss generated by the LLM is backpropagated through the compressor, and the visual encoder is collaboratively constrained to focus on regions with high uncertainty and diagnostic relevance.

[0053] It should be noted that during the inference phase, a confidence-aware intelligent routing and controllable answering mode are implemented. Specifically, in the LLM decoding phase, a confidence prediction head is attached to the final hidden state, and the overall confidence score of the generated answer is calculated, outputting a scalar confidence score. ,in The hidden state of the last token in the LLM. The normalized confidence score is used to generate a visual token-level uncertainty heatmap.

[0054] Based on this confidence score, the system executes a three-level intelligent routing logic (with a preset threshold). and ): High confidence path ( When the confidence level is higher than the high threshold, the diagnostic answer is output directly, which is suitable for cases with clear features and a high degree of confidence in the model.

[0055] Medium confidence path ( When the confidence level is in the middle range, output the answer and attach a label for the uncertain area to prompt the doctor to pay special attention to the image area that the model is uncertain about.

[0056] Low confidence path ( When the confidence level is below the low threshold, the system automatically marks the case as "requiring manual review" and generates an uncertainty analysis report (including uncertain visual areas, possible alternative diagnoses, etc.), which is then routed to the manual review process.

[0057] Meanwhile, the system supports a controllable response mode: if the [CONSERVATIVE] flag is injected, the system only outputs a deterministic reference analysis (suitable for high-risk scenarios); if the [EXPLORATION] flag is injected, it outputs an analytical response that includes speculative differential diagnosis and a detailed uncertainty decomposition report.

[0058] Example 2: This example provides a multimodal visual question-answering system for uncertainty perception in 3D medical imaging. See also... Figures 7-8The system includes: a data acquisition module 201, a multi-scale encoding module 202, an uncertainty perception module 203, an adaptive compression module 204, and a model inference module 205.

[0059] Specifically, the data acquisition module 201 is configured with a data loading interface for reading CT volume data and receiving corresponding natural language user commands. The module integrates code blocks for performing spacingd voxel resampling and resized spatial dimensions, and has a built-in specific HU value clipping and normalization mapping logic unit for executing the range clipping formula and outputting a normalized tensor in the range [0, 1].

[0060] The multi-scale encoding module 202 serves as the system's visual perception front-end, instantiated at the code level as a Vision Transformer3D dual-stream encoder based on M3D-CLIP pre-trained weights and with frozen parameters. Internally, it includes: Local detail extraction unit: Configured with 4×4×4 block parameters, it performs dense patch partitioning and extracts 32768×384-dimensional local detail features through the built-in 6-layer, 6-head Transformer network. Global Context Stream Extraction Unit: Configured with 16×16×16 block parameters, it extracts 512×384-dimensional global context features through another set of parallel 6-layer, 6-head Transformer networks; Cross-scale fusion unit: It includes a dimension-aligned linear layer (mapped to 768 dimensions) and an 8-head cross-scale attention network. It performs feature fusion using global features as the query and finally outputs a multi-scale fused feature sequence with a dimension of 33280×768 to the subsequent modules.

[0061] The uncertainty perception module 203 is the core diagnostic confidence evaluator of the system, and it incorporates an MLP Projection network based on evidence deep learning. Its internal operating logic consists of two dimensions... It consists of a shared backbone network and four parallel linear output heads. Each outputs the mean of a corresponding inverse normal gamma (NIG) distribution. Precision ,shape and scale Parameters. The module is internally configured with tensor computation nodes to calculate cognitive and accidental uncertainties based on the aforementioned parameters, and generates a 2048-dimensional uncertainty embedding vector through a sinusoidal position encoder. Subsequently, the module utilizes a sigmoid modulation network (…). , Perform element-wise multiplication to generate uncertain conditionalized lexicals with high uncertainty region suppression properties.

[0062] The adaptive compression module 204 is used to dynamically select and compress visual features according to the question-answering task instructions, eliminating redundant calculations. Internally, it includes: Independent conditional modulation subunit: The system initializes and stores 64 modulators that obey the condition. Learnable query vectors with a distribution and a dimension of 768 The question embedding is obtained by calling the word embedding layer of the base large language model, and the question semantic vector is extracted through mean pooling. Specifically, a built-in dimension is... projection matrix and their corresponding biases By mapping semantic vectors to semantic guidance factors of dimension M×d, we can accurately achieve independent conditional scaling and modulation for each query vector, thereby generating task-aware query vectors. ; Compressed attention computation unit: configured with an 8-head cross-attention mechanism with a feedforward dimension of 2048, to achieve... To perform calculations for the query, the original 30,000+ visual tokens were compressed into 64×768 compressed visual units. sequence.

[0063] The model inference module 205 serves as the system's comprehensive decision-making center, equipped with a parameter-efficient fine-tuning language model (Qwen1.5-1.8B) that incorporates a target-specific adapter. Internally, it includes: LoRA and Multi-Task Loss Engine: Injecting a Low-Rank Decomposition Matrix (r=16, ...) into the Transformer's Attention Layer =16). In training mode, the engine is responsible for performing sequence concatenation (including Mode_Token) and calculating the collaborative loss. It includes conditional query vectors Constrained compression diversity regularization loss And guided compression allocation loss based on uncertain target distribution. .

[0064] Confidence routing prediction unit: In the inference state, it generates answer-level confidence scores using a confidence prediction head (containing a two-layer MLP and sigmoid activation) attached to the final hidden state of the large model. Built-in includes a first threshold and a second threshold. The three-level comparator logic initially output the answer directly through the interface; when... Output the answer and call the heatmap generation function to add annotations; when If the direct output is interrupted, an abnormal alarm for manual review will be issued and a speculative analysis report will be output.

[0065] Example 3: See Figure 9 This embodiment provides a computer device, the components of which may include, but are not limited to: one or more processors or processing units, system memory, and buses connecting different system components (including system memory and processing units).

[0066] A bus refers to one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0067] Computer systems / servers typically include a variety of computer system-readable media. These media can be any available media that can be accessed by the computer system / server, including volatile and non-volatile media, and removable and non-removable media.

[0068] System memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computer device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system may be used to read and write non-removable, non-volatile magnetic media. Disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to a bus via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0069] A program / utility having a set (at least one) of program modules can be stored, for example, in memory. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this invention.

[0070] Computer devices can also communicate with one or more external devices (such as keyboards, pointing devices, monitors, etc.). This communication can be done through input / output (I / O) interfaces. Furthermore, computer devices can communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapters.

[0071] The processing unit executes the functions and / or methods described in the embodiments of the present invention by running programs stored in the system memory.

[0072] The aforementioned computer program can be stored in a computer storage medium, that is, the computer storage medium is encoded with a computer program, which, when executed by one or more computers, causes one or more computers to perform the method flow and / or device operation shown in the above embodiments of the present invention.

[0073] With the development of time and technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media; they can also be downloaded directly from the network. Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example,—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0074] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0075] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0076] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0077] In addition to the above embodiments, the present invention may have other implementation methods; all technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression, characterized in that: include: Obtain the 3D medical images to be analyzed and their corresponding user Q&A commands; Multi-scale parallel feature extraction is performed on the 3D medical image to obtain local detail features with fine-grained information and global context features with macroscopic anatomical information, respectively. The local detail features and the global context features are then fused across scales to generate a multi-scale fused feature sequence. Based on evidence-based deep learning, the multi-scale fused feature sequence is mapped to distribution parameters. The cognitive uncertainty and accidental uncertainty of the corresponding visual features are calculated according to the distribution parameters, and uncertainty embedding vectors are generated. The mean parameter in the distribution parameters is modulated using the uncertainty embedding vectors to generate uncertainty conditionalized lexical units. Extract the semantic information of the user question-and-answer instruction, use the semantic information to independently conditionally scale and modulate multiple preset learnable query vectors, and use the modulated query vector as the query, and use the multi-scale fusion feature sequence as the key and value to perform cross-attention calculation to generate instruction-guided compressed visual lexical units. The compressed visual lexical units, the uncertain conditionalized lexical units, and the user question-and-answer instructions are concatenated and input into a pre-trained large language model for decoding and inference, generating question-and-answer text results and confidence scores corresponding to the question-and-answer text results.

2. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: Multi-scale parallel feature extraction is performed on the 3D medical image to obtain local detail features with fine-grained information and global context features with macroscopic anatomical information. The local detail features and global context features are then fused across scales to generate a multi-scale fused feature sequence, including: The 3D medical image is processed using 3D Patch of the first size, and the local detail features are extracted using the first 3DTransformer encoder. The same 3D medical image is processed using 3D Patch blocks of the second size, and the global context features are extracted by a second 3DTransformer encoder, wherein the volume of the second size is larger than the volume of the first size. After unifying the local detail features and the global context features to the same dimension, the global context features are used as the query vector, and the local detail features are used as the key vector and value vector. Feature fusion is performed through a cross-scale cross-attention mechanism, and residuals are superimposed to output the multi-scale fused feature sequence.

3. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: The method of mapping the multi-scale fused feature sequence to distribution parameters based on evidence-based deep learning, calculating the cognitive uncertainty and accidental uncertainty of the corresponding visual features according to the distribution parameters and generating uncertainty embedding vectors, and using the uncertainty embedding vectors to modulate the mean parameter in the distribution parameters to generate uncertainty conditionalized lexical units includes: The evidence-learning projection layer maps the multi-scale fused feature sequence to parameters of a normal inverse gamma distribution, including a mean parameter. Precision parameters Shape parameters and scale parameters ; Based on the accuracy parameters Shape parameters and scale parameters The cognitive uncertainty and accidental uncertainty of the corresponding visual features are calculated respectively, and the two are summed to obtain the total uncertainty scalar. The total uncertainty scalar is extended into the uncertainty embedding vector through sinusoidal position encoding; After applying Sigmoid activation to the uncertainty embedding vector, and then combining it with the mean parameter... Feature modulation is performed to generate the uncertain conditionalized lexical units. The specific modulation process satisfies the following formula: ,in, Represents the i-th uncertain conditionalized term; The i-th mean parameter of the output of the normal inverse gamma distribution is used as the initial visual feature representation. This represents the uncertainty embedding vector corresponding to the i-th visual feature; and These are the trainable weight matrix and bias term of the modulation network, respectively; This represents the Sigmoid activation function; This indicates an element-wise multiplication operation.

4. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: Extracting the semantic information of the user's question-and-answer instructions, and using the semantic information to scale and modulate a preset learnable query vector, includes: The user question-answering command is input into the word embedding layer to obtain the question embedding sequence, and the question embedding sequence is pooled to extract the question semantic vector. ; The semantic vector of the problem Attention modulation factors are generated using a linear transformation network; The preset learnable query vector is scaled and modulated using the attention modulation factor, and the specific modulation process satisfies the following formula: ,in, This represents the modulated query vector; This represents the preset learnable query vector; This represents the semantic vector of the question extracted from the user's question-and-answer instructions; The projection matrix that maps the semantic space to the visual feature space. For bias terms; This represents the Sigmoid activation function; This indicates an element-wise multiplication operation, which transforms the semantic vector of the problem into multiple independent attention modulation factors.

5. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: The compressed visual lexical units, the uncertain conditionalized lexical units, and the user question-and-answer instructions are concatenated and input into a large language model for decoding and inference, generating question-and-answer text results including: Obtain a preset response mode instruction, which includes a conservative mode instruction or an exploratory mode instruction; Generate corresponding pattern marker lexical units based on the response pattern instruction; The compressed visual lexical units, the uncertain conditionalized lexical units, the pattern marker lexical units, and the user question-answering instructions are sequentially concatenated and input into the large language model for decoding and inference. When the pattern marker lemmas correspond to the conservative mode instruction, the large language model is constrained to generate reference analysis text carrying uncertainty level annotations; when the pattern marker lemmas correspond to the exploratory mode instruction, the large language model is constrained to generate analytical responses containing speculative diagnoses and uncertainty decomposition reports.

6. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: The pre-trained large language model is obtained through parameter updating and model training using a low-rank adaptive method. During the training phase, a multi-task collaborative loss function is used for joint optimization. The calculation formula for the multi-task collaborative loss function is as follows: ,in, Indicates the total loss; This represents the loss for answer generation based on text. This represents the expected calibration error loss used to match the confidence score with the prediction accuracy. This represents the compression diversity regularization loss used to constrain different learnable query vectors to focus on different visual regions; This indicates the loss from the guided compression allocation; These are the preset balance hyperparameters; The guided compression allocation loss The calculation process includes: Extracting the total uncertainty of the i-th visual region Combined with temperature parameters Calculate the target weights for attention compression assignment ; The attention prediction weights actually assigned to the i-th region are calculated using the cross-entropy function. With the target weight The loss between them is calculated using the following formula: ,in, This represents the summation of the uncertainty index terms over all visual regions.

7. The 3D medical image intelligent question answering method based on uncertainty perception and instruction-guided feature compression according to claim 1, characterized in that: After generating the question-and-answer text results and the corresponding confidence scores, the method further includes: intelligent routing based on the confidence scores, which includes: A first threshold and a second threshold are preset, and the first threshold is greater than the second threshold; When the confidence score is determined to be greater than or equal to a preset first threshold, the question and answer text result is directly output. When the confidence score is determined to be less than the first threshold and greater than or equal to the preset second threshold, the question-and-answer text result is output, and visual annotation information of the uncertainty area is added to the question-and-answer text result; When the confidence score is determined to be less than the second threshold, an alarm flag indicating that manual review is required is generated, and an analysis report based on uncertainty is output.

8. A 3D medical image intelligent question-answering system based on uncertainty perception and instruction-guided feature compression, characterized in that: include: The data acquisition module is used to acquire the 3D medical images to be analyzed and their corresponding user question-and-answer commands; A multi-scale encoding module is used to perform multi-scale parallel feature extraction on the 3D medical image, respectively acquiring local detail features with fine-grained information and global context features with macroscopic anatomical information, and fusing the local detail features and the global context features across scales to generate a multi-scale fused feature sequence. The uncertainty perception module is used to map the multi-scale fused feature sequence into distribution parameters based on evidence deep learning, calculate the cognitive uncertainty and accidental uncertainty of the corresponding visual features according to the distribution parameters and generate uncertainty embedding vectors, and use the uncertainty embedding vectors to perform feature modulation on the mean parameter in the distribution parameters to generate uncertainty conditionalized lexical units. An adaptive compression module is used to extract the semantic information of the user's question-and-answer instructions, use the semantic information to independently conditionally scale and modulate multiple preset learnable query vectors, and use the modulated query vector as the query, and use the multi-scale fusion feature sequence as the key and value to perform cross-attention calculation to generate instruction-guided compressed visual lexical units. The model inference module is used to concatenate the compressed visual lexical units, the uncertain conditionalized lexical units, and the user question-and-answer instructions, and then input them into the large language model for decoding and inference to generate question-and-answer text results and confidence scores corresponding to the question-and-answer text results.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images