Characteristic distillation multi-modal large model-based autonomous diagnosis system

Abstract

The invention discloses an autonomous diagnosis system based on a characteristic distillation multi-modal large model, which relates to the field of medical equipment and comprises an input module, a preprocessing module, a first analysis module, a local knowledge base, a second analysis module, an output module and a display module. Through fusion of multi-modal data and application of a deep learning model, the accuracy and precision of pulmonary nodule diagnosis are remarkably improved, and the probability of misdiagnosis and missed diagnosis is reduced; automation of the diagnosis process is realized, the diagnosis time is greatly shortened, a high-quality diagnosis result can be quickly provided, and the working efficiency of a hospital is improved; in combination with information of a local knowledge base, personalized treatment suggestions can be provided according to specific conditions of patients, and pertinence and effectiveness of medical services are improved; compared with the prior art, manual intervention is reduced, consumption of human resources is reduced, unnecessary repeated examination and treatment are reduced by improving diagnosis efficiency and accuracy, and medical resources and cost are saved.

Description

Technical Field

[0001] The present invention relates to the field of medical equipment, and in particular to an autonomous diagnosis system based on a feature distillation multimodal large model. Background Art

[0002] Early detection and diagnosis of lung nodules are crucial for the prevention and treatment of lung cancer. However, due to the wide variety and different morphologies of lung nodules, traditional imaging diagnostic methods have the problems of low accuracy and high misdiagnosis rate. In recent years, artificial intelligence technology, especially the application of multimodal large models in medical image analysis, has provided new ideas for improving diagnostic accuracy. In the modern medical environment, hospitals are faced with the task of screening and diagnosing a large number of lung nodule cases. With the advancement of imaging technology, the widespread use of imaging equipment such as CT, MRI and X-ray has made the detection of early lung nodules more possible. However, the current diagnostic process relies on manual analysis of images by radiologists, which is not only time-consuming and laborious, but also easy to cause visual fatigue of doctors due to the large amount of image data, thereby increasing the risk of misdiagnosis and missed diagnosis. In addition, differences in doctors' experience and expertise also make the diagnostic results somewhat subjective and uncertain. In order to improve diagnostic efficiency and accuracy, hospitals are in urgent need of a system that can assist doctors in automated and accurate diagnosis. Summary of the invention

[0003] The purpose of the present invention is to design an autonomous diagnosis system based on feature distillation multimodal large model in order to solve the above problems.

[0004] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0005] The autonomous diagnosis system based on feature distillation multimodal large model includes:

[0006] Input module: The input module is used to input medical imaging data of multiple modalities;

[0007] Preprocessing module: The preprocessing module is used to preprocess medical imaging data of multiple modalities;

[0008] The first analysis module; the first analysis module is used to analyze medical imaging data of multiple modalities to obtain analysis results; the first analysis module uses a multimodal large model to analyze and extract feature information, and the multimodal large model includes a modality fusion module, a feature purification module and a feature analysis module from input to output. The modality fusion module is used to extract and fuse the features of each modality, the feature purification module is used to gradually add noise to the fused features and remove redundant components in the features, and the feature analysis module purifies the features at different scales and splices and integrates the features of each scale;

[0009] Local knowledge base; the local knowledge base stores pulmonary nodule information;

[0010] A second analysis module; the second analysis module retrieves matching information matching the analysis result in the local knowledge base according to the analysis result;

[0011] Output module: The output module generates a diagnosis report and treatment recommendations based on the matching information and analysis results;

[0012] Display module: The display module is used for multimodal medical imaging data, displaying diagnostic reports and treatment recommendations.

[0013] The beneficial effects of the present invention are as follows: through the fusion of multimodal data and the application of deep learning models, the accuracy and precision of lung nodule diagnosis are significantly improved, and the probability of misdiagnosis and missed diagnosis is reduced; the automation of the diagnostic process is realized, the diagnosis time is greatly shortened, and high-quality diagnostic results can be provided quickly, thereby improving the work efficiency of the hospital; combined with the information in the local knowledge base, personalized treatment suggestions can be provided according to the patient's specific situation, thereby improving the pertinence and effectiveness of medical services; manual intervention is reduced, the consumption of human resources is reduced, and by improving diagnostic efficiency and accuracy, unnecessary repeated examinations and treatments are reduced, saving medical resources and costs; by reducing unnecessary repeated examinations and treatments, the generation of medical waste is reduced, and pollution to the environment is reduced, thereby having environmental benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a framework diagram of the autonomous diagnosis system of the multi-modal large model based on feature distillation of the present invention;

[0015] Figure 2 It is a data processing flow chart of the feature distillation multi-modal large model autonomous diagnosis system of the present invention;

[0016] Figure 3 It is a schematic diagram of the first analysis module in the feature distillation-based multimodal large model autonomous diagnosis system of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0020] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0021] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0022] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0024] The autonomous diagnosis system based on feature distillation multimodal large model includes:

[0025] Input module: The input module is used to input medical imaging data of various modalities. CT scanners, MRI equipment, X-ray machines and other imaging equipment collect medical imaging data of the patient's lungs and input them into the diagnostic system through the input module.

[0026] Preprocessing module: The preprocessing module is used to preprocess medical imaging data of various modalities; the preprocessing module includes:

[0027] Denoising module: The denoising module uses image processing algorithms to denoise medical image data, remove noise and artifacts in the image, and improve image quality;

[0028] Standardization module: The standardization module is used to standardize the denoised medical image data so that the medical image data of different modalities have consistent resolution and grayscale range, which is convenient for subsequent analysis;

[0029] Encoding module: The encoding module is used to perform position encoding on the standardized medical image data, so that the subsequent large model can better learn the detailed information and spatial relationship in the image, and can focus on the information of pathology (such as lung nodules). Then, the encoded medical image data is spliced ​​with the input medical image data. The position encoding is expressed as: , Where i,j represents the row index and column index of the image, x ij represents the corresponding pixel value, d represents the encoding dimension, w represents the width of the input image, and k represents any positive integer;

[0030] Feature extraction module: The feature extraction module extracts the features of lung nodules in the spliced ​​medical image data through a feature extraction algorithm, providing input data for the analysis of the multimodal large model.

[0031] Pseudocode example:

[0032] def preprocess_image(image, modality):

[0033] # Image preprocessing, including normalization, denoising and feature extraction

[0034] preprocessed_image = standardize(image)

[0035] preprocessed_image = denoise(preprocessed_image)

[0036] pe = position_encoding(preprocessed_image)

[0037] preprocessed_image = cat(pe, preprocessed)

[0038] preprocessed_image = extract_features(preprocessed_image,modality)

[0039] return preprocessed_image

[0040] The first analysis module is used to analyze medical imaging data of multiple modalities to obtain analysis results. The first analysis module uses a multi-modal large model to analyze and extract feature information, such as Figure 3 As shown in the figure, the multimodal large model includes a modal fusion module, a feature purification module and a feature analysis module from input to output. The modal fusion module is used to extract and fuse the features of each modality. The feature purification module is used to gradually add noise to the fused features and remove redundant components in the features. The feature analysis module purifies the features at different scales and splices and integrates the features of each scale.

[0041] The modality fusion module uses a multi-branch deep learning network (convolutional neural network, transforms). Medical imaging data of one modality is input into one branch. The network depth of each branch is consistent, and the layers between different branches are densely connected, which can be expressed as: ;in, represents the sth branch network The output of the layer, [...] represents the vector concatenation operation, represents the sth branch network The mapping function of the layer.

[0042] The feature purification module first gradually adds noise to the fused features and finally learns the reverse process to gradually remove the redundant components in the features. For a given original feature x0, the forward process is expressed as: ;in represents Gaussian distribution, β t represents the variance hyperparameter at time t. The reverse learning purification process is expressed as: ,in, represents the function learned by the neural network, θ represents the parameters of the neural network, σ t Represents the variance of the Gaussian distribution.

[0043] The feature analysis module includes a basic model and multiple attention classification layers. The basic model is GPT4. The output of the basic model is used as the input of multiple attention classification layers. Each attention classification layer includes a multi-scale attention network and a classifier from input to output. The basic model is a symmetrical structure of U-shaped encoding and decoding. The number of layers of decoding and encoding is four, and each layer adopts a res-block structure (1x1 convolution, 3x3 convolution, 1x1 convolution plus residual connection). Time information t is injected into each layer of encoding and decoding. After position encoding and three-layer MLP processing, the time information t is spliced ​​with the output of each layer in the input next layer, which is expressed as: ;in, Representative The output of the layer, Represents a splicing operation, Indicates The mapping function of the layer, Represents the mapping function of a multi-layer neural network, Represents position encoding, expressed as: .

[0044] Multimodal Data Encoding:

[0045] Encode the preprocessed image data into a feature vector.

[0046] Using a specific encoding method, the features of CT, MRI and X-ray images are fused into a multimodal feature vector.

[0047] Python

[0048] Copy code

[0049] def encode_multimodal_data(ct_image, mri_image, xray_image):

[0050] # Encode CT, MRI and X-ray image data as feature vectors

[0051] ct_features = preprocess_image(ct_image, 'CT')

[0052] mri_features = preprocess_image(mri_image, 'MRI')

[0053] xray_features = preprocess_image(xray_image, 'X-ray')

[0054] # Fusion of multimodal features

[0055] multimodal_features = torch.cat((ct_features, mri_features, xray_features), dim=0)

[0056] return multimodal_features

[0057] GPT-DiffAttention model analysis:

[0058] 1) Input the encoded multimodal feature vector into the GPT-attention base model to output the preprocessed feature vector.

[0059] 2) The GPT-DiffAttention model extracts the features of lung nodules through its powerful generation and understanding capabilities, and performs classification and pathological analysis.

[0060] GPT-DiffAttention model pseudocode implementation example (python):

[0061] import torch

[0062] import torch.nn as nn

[0063] import torch.nn.functional as F

[0064] class ModalFusionModule(nn.Module):

[0065] def __init__(self, num_branches, depth):

[0066] super(ModalFusionModule, self).__init__()

[0067] self.branches = nn.ModuleList([self._build_branch(depth) for_ in range(num_branches)])

[0068] def _build_branch(self, depth):

[0069] layers = []

[0070] for _ in range(depth):

[0071] layers.append(nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, padding=1))

[0072] layers.append(nn.ReLU(inplace=True))

[0073] return nn.Sequential(*layers)

[0074] def forward(self, x):

[0075] outputs = [branch(x[i]) for i, branch enumerate(self.branches)]

[0076] # Dense connections across branches

[0077] combined = torch.cat(outputs, dim=1)

[0078] return combined

[0079] class FeatureRefinementModule(nn.Module):

[0080] def __init__(self):

[0081] super(FeatureRefinementModule, self).__init__()

[0082] def forward(self, x):

[0083] # Assume forward process adds noise and backward processremoves it

[0084] noisy_features = self.add_noise(x)

[0085] refined_features = self.refine_features(noisy_features)

[0086] return refined_features

[0087] def add_noise(self, x):

[0088] noise = torch.randn_like(x) * 0.1

[0089] return x + noise

[0090] def refine_features(self, x):

[0091] return self.denoise(x)

[0092] def denoise(self, x):

[0093] # Simple denoising example (for illustration purposes)

[0094] return F.relu(x)

[0095] class GPTDiffAttentionModel(nn.Module):

[0096] def __init__(self, base_model, num_scales):

[0097] super(GPTDiffAttentionModel, self).__init__()

[0098] self.base_model = base_model

[0099] self.scales = nn.ModuleList([self._build_scale() for _ inrange(num_scales)])

[0100] self.classifier = nn.Linear(256, 10) # Example classifier

[0101] def _build_scale(self):

[0102] layers = []

[0103] for _ in range(4): # Assume 4 layers for U-Net likestructure

[0104] layers.append(self._build_res_block())

[0105] return nn.Sequential(*layers)

[0106] def _build_res_block(self):

[0107] return nn.Sequential(

[0108] nn.Conv2d(64, 64, kernel_size=1),

[0109] nn.Conv2d(64, 64, kernel_size=3, padding=1),

[0110] nn.Conv2d(64, 64, kernel_size=1),

[0111] nn.ReLU(inplace=True) )

[0113] def forward(self, x, time_info):

[0114] # Base model processing

[0115] base_output = self.base_model(x)

[0116] scale_outputs = [scale(base_output) for scale in self.scales]

[0117] # Integrate time information

[0118] time_embedded = self.embed_time(time_info)

[0119] combined = torch.cat([s + time_embedded for s in scale_outputs], dim=1)

[0120] # Classification

[0121] output = self.classifier(combined.view(combined.size(0), -1))

[0122] return output

[0123] def embed_time(self, t):

[0124] # Example time embedding

[0125] return t.expand(-1, 64, -1, -1)

[0126] Local knowledge base; the local knowledge base stores pulmonary nodule information;

[0127] A second analysis module; the second analysis module retrieves matching information matching the analysis result in the local knowledge base according to the analysis result;

[0128] Knowledge base search:

[0129] - Obtain the analysis results of the multimodal large model and extract characteristic information including nodule type, location, size, morphology, density, etc.

[0130] - Input: Analysis results of multimodal large models

[0131] - Output: extracted feature information

[0132] ```Python

[0133] def extract_features(model_result):

[0134] # Extract feature information of lung nodules from the multimodal large model results

[0135] features = {

[0136] 'type': model_result.get('nodule_type'),

[0137] 'location': model_result.get('location'),

[0138] 'size': model_result.get('size'),

[0139] 'shape': model_result.get('shape'),

[0140] 'density': model_result.get('density')

[0141] }

[0142] return features

[0143] ```

[0144] 2) Feature matching:

[0145] - Retrieve lung nodule types and related information matching the extracted features in the local knowledge base.

[0146] - Input: extracted feature information

[0147] - Output: matching knowledge base information

[0148] ```Python

[0149] def match_features_with_knowledge_base(features, knowledge_base):

[0150] # Retrieve information matching the feature in the local knowledge base

[0151] matched_info = knowledge_base.search(features)

[0152] return matched_info

[0153] ```

[0154] Output module: The output module generates a diagnosis report and treatment recommendations based on the matching information and analysis results;

[0155] Information Integration:

[0156] - Integrate the analysis results of the multimodal large model with the matching information in the local knowledge base.

[0157] - Input: extracted feature information, matched knowledge base information

[0158] - Output: integrated diagnostic information

[0159] ```Python

[0160] def integrate_information(model_result, matched_info):

[0161] # Integrate analysis results and knowledge base information

[0162] diagnostic_report = {

[0163] 'nodule_type': model_result.get('nodule_type'),

[0164] 'location': model_result.get('location'),

[0165] 'size': model_result.get('size'),

[0166] 'shape': model_result.get('shape'),

[0167] 'density': model_result.get('density'),

[0168] 'knowledge_base_info': matched_info

[0169] }

[0170] return diagnostic_report

[0171] ```

[0172] Personalized diagnosis:

[0173] - Provide personalized treatment recommendations based on local clinical data and knowledge base information.

[0174] - Input: integrated diagnostic information

[0175] - Output: Personalized treatment recommendations

[0176] ```Python

[0177] def provide_personalized_diagnosis(diagnostic_report, clinical_data):

[0178] # Provide personalized diagnosis and treatment recommendations based on integrated information and local clinical data

[0179] personalized_suggestions = clinical_data.get_treatment_suggestions(diagnostic_report)

[0180] return personalized_suggestions

[0181] Display module: The display module is used for multimodal medical imaging data, displaying diagnostic reports and treatment recommendations.

[0182] By comparing and integrating the analysis results of the multimodal large model with the information in the local knowledge base, a detailed diagnostic report and personalized treatment recommendations are generated, thereby improving the accuracy of diagnosis and the level of personalized service; a detailed diagnostic report is generated, including information such as the type, location, size, and corresponding treatment recommendations of lung nodules; auxiliary decision support is provided to doctors to help them formulate treatment plans; doctors can input new patient data and update the information in the local knowledge base as needed to ensure the real-time and accuracy of the diagnostic system; the system regularly obtains new research results and diagnostic standards from external databases and the latest medical literature, and updates the multimodal large model and the local knowledge base; the parameters and structure of the multimodal large model are updated based on the latest external data to ensure the advancement and accuracy of the diagnostic model.

[0183] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. Based on feature distillation multimodal large model autonomous diagnosis system, characterized by: include: Input module; The input module is used to input medical imaging data of multiple modalities; Preprocessing module; The preprocessing module is used to preprocess medical imaging data of multiple modalities; The first analysis module; the first analysis module is used to analyze medical imaging data of multiple modalities to obtain analysis results; the first analysis module uses a multimodal large model to analyze and extract feature information, and the multimodal large model includes a modality fusion module, a feature purification module and a feature analysis module from input to output. The modality fusion module is used to extract and fuse the features of each modality, the feature purification module is used to gradually add noise to the fused features and remove redundant components in the features, and the feature analysis module purifies the features at different scales and splices and integrates the features of each scale; Local knowledge base; the local knowledge base stores pulmonary nodule information; The second analysis module; The second analysis module retrieves matching information matching the analysis result in the local knowledge base according to the analysis result; Output module; The output module generates a diagnosis report and treatment recommendations based on the matching information and analysis results; Display module: The display module is used for multimodal medical imaging data, displaying diagnostic reports and treatment recommendations.

2. The multi-modal large model autonomous diagnosis system based on feature distillation according to claim 1 is characterized in that: The modality fusion module uses a multi-branch deep learning network. Medical imaging data of one modality is input into one branch, and the layers between different branches are densely connected, which can be expressed as: ;in, represents the sth branch network The output of the layer, [...] represents the vector concatenation operation, represents the sth branch network The mapping function of the layer.

3. The multi-modal large model autonomous diagnosis system based on feature distillation according to claim 1 is characterized in that: The feature purification module first gradually adds noise to the fused features and finally learns the reverse process to gradually remove the redundant components in the features. For a given original feature x0, the forward process is expressed as: in represents Gaussian distribution, β t represents the variance hyperparameter at the tth moment; the reverse learning purification process is expressed as: ,in, represents the function learned by the neural network, θ represents the parameters of the neural network, σ t Represents the variance of the Gaussian distribution.

4. The multi-modal large model autonomous diagnosis system based on feature distillation according to claim 1 is characterized in that: The feature analysis module includes a basic model and multiple attention classification layers. The output of the basic model is used as the input of multiple attention classification layers. Each attention classification layer includes a multi-scale attention network and a classifier from input to output. The basic model is a symmetrical structure of U-shaped encoding and decoding. The number of layers of decoding and encoding are four respectively. Each layer adopts a res-block structure. Time information t is injected into each layer of encoding and decoding. The time information t is spliced ​​with the output of each layer after position encoding and three-layer MLP processing in the input next layer, which is expressed as: ;in, Representative The output of the layer, Represents a splicing operation, Indicates The mapping function of the layer, Represents the mapping function of a multi-layer neural network, Represents position encoding, expressed as: .

5. The multi-modal large model autonomous diagnosis system based on feature distillation according to claim 1 is characterized in that: The preprocessing modules include: Denoising module: The denoising module uses image processing algorithms to denoise medical image data and remove noise and artifacts in the image; Standardization module: The standardization module is used to standardize the denoised medical image data so that the medical image data of different modalities have consistent resolution and grayscale range; Coding module: The coding module is used to perform position coding on the standardized medical image data and to concatenate the coded medical image data with the input medical image data. The position coding is expressed as: , Where i, j represent the row index and column index of the image, x ij Represents the corresponding pixel value, d represents the encoding dimension, w represents the width of the input image, and k represents any positive integer; Feature extraction module: The feature extraction module extracts the features of lung nodules in the spliced ​​medical image data through a feature extraction algorithm.

Images