A multimodal semi-supervised image analysis and structured report generation method and application

Abstract

The application belongs to the field of medical image analysis, and aims at the problem that the generation efficiency of standardized report for patients with chronic obstructive pulmonary disease is low and highly dependent on manual experience, and proposes a multi-modal semi-supervised image analysis and structured report generation method and application, which comprises the following steps: acquiring lung medical image data and performing pretreatment; constructing an improved variational autoencoder to obtain a latent variable representation of the lung medical image data; using domain adaptation adversarial learning to align the data in the unlabeled data set to the distribution of the data in the labeled data set in the latent variable and segmentation result; generating a preliminary report text using a text decoder according to the hidden variable output by the encoder of the variational autoencoder; performing knowledge distillation based on the pre-trained Teacher network and the text decoder to modify the medical terms and abbreviations in the preliminary report text; and outputting a structured report. The application can accurately segment lung lesions, quantify disease severity and automatically generate a structured report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, in particular to a multi-modal semi-supervised image analysis and structured report generation method. BACKGROUND

[0002] Chronic obstructive pulmonary disease (COPD) is one of the leading causes of death and disability worldwide. Due to its diverse and progressive pathological characteristics, patients often need to undergo repeated lung imaging examinations (CT, X-ray, etc.) to monitor lesion development and evaluate treatment effectiveness. However, traditional image analysis methods often have insufficient efficiency in dealing with complex lung structures, identifying multiple lesions, and writing standardized reports, and are highly dependent on human experience, which can easily lead to inconsistent diagnoses and descriptions. In recent years, with the rapid development of deep learning, multi-modal fusion, and natural language processing technologies, how to fully utilize a small amount of labeled data and a large amount of unlabeled data in a semi-supervised scenario to build an end-to-end solution that can accurately segment lung lesions, quantify disease severity, and automatically generate structured reports has gradually become the focus of clinical needs and academic research.

[0003] Currently, a lesion segmentation model based on U-Net has been disclosed in the prior art, which uses the segmentation model to segment the lesion, and then uses a multi-modal fusion framework based on CNN-Transformer and a report generation system based on a pre-trained language model to generate a report. However, it has the following defects:

[0004] First, semi-supervised learning is inefficient: most methods rely on fully supervised training, requiring a large amount of labeled data (such as pixel-level lesion masks), and semi-supervised strategies (such as consistency regularization) are easily disturbed by noise in the lung multi-lesion scenario, resulting in blurred segmentation boundaries (such as the junction area between emphysema and normal lung tissue).

[0005] Second, multi-task collaboration is insufficient: segmentation, quantification, and text generation modules are often designed independently, lacking cross-task feature sharing mechanisms. For example, the text decoder cannot directly associate with the heat map features of the emphysema distribution, leading to a deviation between the generated report description and the image features.

[0006] Third, lack of explainability: existing models are often "black boxes" with no visual explanation of the decision-making basis (such as attention region heat maps), making it difficult for doctors to verify whether the model is focusing on the true lesion, especially the reliability of low confidence predictions (such as early small airway lesions).

[0007] Fourth, poor clinical adaptability: the generated text report is mostly a fixed template filling, which cannot dynamically combine patient historical data (such as LAA% time series changes) or personalized terms (such as SNOMED CT coding), making it difficult to meet the international standard requirements of structured reports. SUMMARY

[0008] The present application is directed to the shortcomings of the prior art described above, and the present application proposes a multi-modal semi-supervised image analysis and structured report generation method for COPD patients, which is used to provide more standardized and intelligent support for clinical diagnosis and patient follow-up management.

[0009] In a first aspect, as shown in the accompanying drawings, the present application provides a multi-modal semi-supervised image analysis and structured report generation method, which comprises the following steps: Figure 3

[0010] Step 1: Obtain the lung medical images and text reports of patients with chronic obstructive pulmonary disease, preprocess the lung medical image data and establish an image data set, and divide the image data set into an annotated data set and an unannotated data set;

[0011] Step 2: Construct an improved variational autoencoder and train it on the annotated data set and the unannotated data set to obtain the latent variables and segmentation masks of the lung medical images;

[0012] Step 3: Fuse the text report with the latent variables and segmentation masks, and use the text decoder to generate a preliminary report text;

[0013] Step 4: Obtain a pre-trained Teacher network on a medical knowledge library, and use the text decoder as a Student network to perform knowledge distillation with the Teacher network, modify the medical terminology and abbreviations in the preliminary report text, and output the final structured report.

[0014] Further, in step 2, the improved variational autoencoder is obtained by introducing a Dirichlet process prior in the encoder of a traditional variational autoencoder, adding a segmentation branch in the decoder, and inserting a Flow model for multi-level coupling between the encoder and the decoder.

[0015] Further, step 2 specifically comprises the following steps:

[0016] Step 2.1: Introduce a Dirichlet process prior in the encoder of a traditional variational autoencoder, and obtain the prior distribution of the latent variables according to the Dirichlet process prior;

[0017] Step 2.2: Add a segmentation branch parallel to the image branch in the decoder of the traditional variational autoencoder;

[0018] Wherein, the segmentation branch is a segmentation mask channel, and the image branch is an image reconstruction channel;

[0019] ​Step 2.3: Adding a regression head at the end of the segmentation branch to process sample data containing numerical quantitative indicator labels;

[0020] Step 2.4: Inserting a Flow model between the encoder and the decoder for multi-level coupling, and establishing the final improved variational autoencoder;

[0021] wherein the final loss function of the improved variational autoencoder is:

[0022]

[0023] wherein, represents the latent variable after the Flow reversible transformation; represents the determinant of each layer of reversible transformation; represents the log-likelihood of the transformed latent variable under the Gaussian prior; represents the overall loss function containing the regression loss function; represents the segmentation loss function; represents the labeled data set; represents the unlabeled data set; x represents the image in the image data set; represents the posterior distribution; represents the weight coefficient used to balance the relative importance of different loss terms;

[0024] Step 2.5: Obtaining the reconstructed image and segmentation mask of each sample in the image data set from the latent variable , and using the improved adversarial discriminator to align the data in the unlabeled data set to the data distribution of the labeled data set;

[0025] Step 2.6: Based on the final loss function, training the improved variational autoencoder on the labeled data set and the aligned distribution of the unlabeled data set, and outputting the reconstructed image and segmentation mask through the image reconstruction channel and the segmentation mask channel;

[0026] Step 2.7: Performing sobel edge constraint on the reconstructed image and segmentation mask output by step 2.6 to make the lesion edge smoother.

[0027] Further, the specific steps of step 2.5 include:

[0028] Step 2.5.1: Adding a domain classifier at the end of the traditional adversarial discriminator to obtain an improved adversarial discriminator;

[0029] Step 2.5.2: Based on the real image x and its latent variable in the image data set, constructing a real combination ; ​

[0030] wherein, denotes an encoder of the improved variational autoencoder;

[0031] Step 2.5.3: obtaining the latent variable from the reconstructed image and inputting the reconstructed image back to the improved variational autoencoder to regenerate the latent variable , constructing a fake combination ;

[0032] wherein, is a decoder of the improved variational autoencoder;

[0033] Step 2.5.4: inputting the real combination and the fake combination into the adversarial discriminator together, and predicting the domain label of the image in the real combination and the fake combination through the domain classifier to determine the source of the image;

[0034] Step 2.5.5: defining the adversarial loss of the discriminator:

[0035] ;

[0036] wherein, denotes the score of the discriminator on the real image ; denotes the score of the discriminator on the fake image output by the decoder; is a gradient penalty term; is the weight of the gradient penalty term;

[0037] Step 2.5.6: aligning the unlabeled data of different sources in the unlabeled data set to the distribution of the labeled data in the labeled data set by minimizing the adversarial loss.

[0038] Further, the specific steps of step 3 include:

[0039] Step 3.1: connecting a fusion head at the end of the segmentation branch, and the fusion head fuses the latent variable and the segmentation mask with the text report through a multi-head cross-attention mechanism to obtain fused context features c;

[0040] Step 3.2: generating a preliminary report text based on the obtained context features c using a text decoder.

[0041] Further, the specific steps of step 3.2 include:

[0042] Step 3.2.1: taking each word in the text report as a word vector, and then converting the word vector into an embedding and inputting it into the text decoder;

[0043] Step 3.2.2: The text decoder performs a check on the first text at each time step. t Perform a cross-attention operation on the word vector to obtain the... t The first decoding state of each word vector Then the first decoding state With context features Perform cross-attention operation again to obtain the second decoding state. ;

[0044] Step 3.2.3: Based on the second decoding state Obtain the word probability distribution and perform softmax processing on the word probability distribution to predict the word probability distribution. t Predicted words from word vectors;

[0045] Step 3.2.4: Output a preliminary report text consisting of all predicted words.

[0046] Furthermore, the loss function during knowledge distillation training is:

[0047] ;

[0048] in, Represents the distillation loss function; , These represent the embedding vectors of corresponding words or concepts in the Student network and Teacher network, respectively. This indicates taking the negative value of the Euclidean distance or cosine similarity. It represents each medical term in the medical vocabulary set; A collection of terms representing the medical field.

[0049] Secondly, this aspect provides a multimodal semi-supervised image analysis and structured report generation system, which is implemented based on a multimodal semi-supervised image analysis and structured report generation method, including:

[0050] The data acquisition module is used to acquire lung medical images and text reports of patients with chronic obstructive pulmonary disease, preprocess the lung medical image data and establish an image dataset, and divide the image dataset into labeled dataset and unlabeled dataset.

[0051] The model building module is used to build an improved variational autoencoder and train it on labeled and unlabeled datasets to obtain latent variables and segmentation masks for lung medical images.

[0052] The structured report generation module is used to generate preliminary report text, correct medical terms and abbreviations, and output the final structured report.

[0053] Preferably, the preliminary report generation module comprises:

[0054] a report generation unit for fusing the text report with the latent variable and the segmentation mask, and generating a preliminary report text by using the text decoder;

[0055] a report modification unit for obtaining a pre-trained Teacher network on a medical knowledge base, and performing knowledge distillation between the text decoder as a Student network and the Teacher network to modify medical terms and abbreviations in the preliminary report text;

[0056] a report generation module for outputting a final structured report.

[0057] In a third aspect, the present application provides an electronic device comprising at least one processor; and a memory connected in communication with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the multi-modal semi-supervised image analysis and structured report generation method.

[0058] Therefore, the present application has the following beneficial effects by using the above multi-modal semi-supervised image analysis and structured report generation method:

[0059] Firstly, the present application adds a segmentation branch to the decoder of the original VAE model, so that the encoder can segment the lesion area while decoding to generate a reconstructed image;

[0060] Secondly, the present application couples the obtained latent variable through multi-level inter-layer Flow on the encoder to obtain a latent variable with better disentangling ability, so as to approximate the real image distribution during decoding, and learn more COPD variation patterns with the support of a large amount of unlabeled data;

[0061] Thirdly, the present application introduces an adversarial discriminator and a pseudo-label strategy to further align the unlabeled data to the distribution of the labeled data in the latent variable and the segmentation result, and realize domain adaptation in a multi-hospital or multi-device scenario;

[0062] Fourthly, the present application performs knowledge distillation between a Teacher model pre-trained on a large amount of medical text and the text decoder in the present application, so as to ensure the correct use of medical terms and abbreviations in the report text.

[0063] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 FIG. 1 is a schematic diagram of a patient structured report output in an embodiment of the present application.

[0065] Figure 2 The technical framework diagram for generating a structured report of the present application.

[0066] Figure 3 The flowchart of the method of the present application.

[0067] Figure 4 The original CT image and its segmentation results; wherein, Figure 4 a- Figure 4 b are respectively the coronal CT image, the segmentation result of the model on the coronal CT lung; Figure 4 c- Figure 4 d are respectively the transverse CT image, the segmentation result of the model on the transverse CT lung.

[0068] Figure 5 The flowchart of using the PatchGAN architecture to distinguish different regions of the image. DETAILED DESCRIPTION

[0069] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0070] 1. Data collection and preprocessing:

[0071] (1) Data collection:

[0072] The data obtained by the present application includes two types of lung medical images of patients and text reports, wherein the lung medical images of patients refer to the lung CT images or X-ray images of COPD patients, and the text report refers to the description or diagnosis conclusion written by a clinician, which is a kind of phenotype information;

[0073] Part of the data in all lung medical images of COPD patients obtained is manually labeled, and a domain label is added to each image during labeling , to obtain a labeled data set , and the remaining lung medical image data constitutes an unlabeled data set :

[0074] ① The labeled data set :

[0075] ;

[0076] Among them, x i represents the ia lung medical image, indicates label information, which is pre-set; l indicates labeled, L indicates the total number of samples of the labeled dataset;

[0077] Specifically, the label information includes the following:

[0078] 1) labeled lesion mask corresponding to the position of the lesion labeled in the lung medical image, the lesion being emphysema, chronic bronchitis, etc.

[0079] 2) COPD classification label such as the Global Initiative for Chronic Obstructive Pulmonary Disease (GOLD) guidelines, COPD classification is I to IV level, or other staging standards;

[0080] 3) numerical quantitative indicators, such as LAA%, Pi10, etc.

[0081] 4) text report label information, including doctors' observations of images, diagnoses, and descriptions of lesions, etc.

[0082] ② unlabeled dataset:

[0083] ;

[0084] wherein, indicates the j lung medical image, u indicates unlabeled, U indicates the total number of samples of the unlabeled dataset, which is usually larger than the labeled dataset ;

[0085] (2) preprocessing step

[0086] Before dividing the labeled dataset and the unlabeled dataset, the collected lung medical images are preprocessed as follows ①-⑤.

[0087] ① Resampling: resample the CT voxels (or light pixels) in the acquired lung medical images to a consistent resolution, such as , to reduce the interference of size differences on the model;

[0088] ② Normalization and denoising: truncate in the interval of HU value (i.e. CT grayscale) and then linearly normalize the HU value to or ; Noise suppression was achieved in the normalized lung medical images using algorithms such as wavelet denoising and nonlocal mean filtering.

[0089] ③ Preliminary lung region segmentation (coarse segmentation): For CT images, thresholding combined with morphological operations, or a lightweight U-Net, is used to segment the lung field from the image background first, reducing the computational load and interference of subsequent models; for The light images are obtained by taking front and back views (i.e., anteroposterior and lateral views) to obtain images from different directions. The morphological differences between the two are then used to segment the lung field from the image background to obtain a coarse segmentation result of the lung area.

[0090] ④ Data augmentation (optional): Augment the data using methods such as rotation, translation, scaling, elastic deformation, and random cropping to avoid overfitting;

[0091] ⑤ Numerical Quantitative Indicator Calculation: Quantitative indicators are calculated using external software Pulmonary Workstation. These indicators include LAA% (low density area percentage), Pi10 (small airway wall thickness), MLD (mean lung density), TLC (lung volume or total lung capacity), WA% (wall area percentage), FEV1, and the FEV1 / FVC ratio. These indicators will be used as labels to assist in segmentation and report generation.

[0092] 2. Construct an improved variational autoencoder:

[0093] like Figure 2 As shown, before segmenting images and generating reports, it is necessary to fully utilize unlabeled images in the unlabeled dataset. Variational autoencoders (VAEs) can construct latent variables, allowing all lung medical images (regardless of whether they are labeled) to be embedded in the same latent variable space. However, since COPD lesions often exhibit a multimodal distribution (emphysema, small airway stenosis, etc., show significant morphological differences), "modal collapse" or difficult-to-represent multimodal features can easily occur. Therefore, this invention constructs an improved variational autoencoder based on the traditional VAE. Its improvements include:

[0094] (1) Introduce Dirichlet procedure priors into the encoder:

[0095] Let the encoder in VAE be denoted as , Used to receive input images Output the posterior distribution Meanwhile, the decoder is denoted as , Used from encoder The output latent variables are used to obtain the reconstructed image. In standard VAEs, the encoder maps the input image to a probability distribution (usually Gaussian) in the latent space, i.e. the latent variables are usually drawn from a single prior distribution, such as a Gaussian prior The decoder samples from the latent distribution output by the encoder and decodes the sampled latent vector to generate a new data sample.

[0096] The present application introduces a Dirichlet process prior in the encoder of a traditional variational autoencoder, which enables the encoder to output latent variables with adaptive multimodal distributions. The Dirichlet process prior can be represented as:

[0097] ;

[0098] where, represents the prior distribution of the latent variable based on the Dirichlet process prior, DP represents the Dirichlet process, represents the concentration parameter, represents the initial concentration parameter, represents the base distribution (BaseDistribution). When new categories or mixture components are added to the Dirichlet process, each new component is initially sampled according to the distribution;

[0099] In the above formula, is used to control the frequency of new components appearing, i.e. how many different modes (components) will appear in the mixture distribution. When is larger, the Dirichlet process tends to include more types of mixture components, and vice versa is smaller, the distribution will be more concentrated on a small number of components.

[0100] When training the improved variational autoencoder, the or Rényi divergence measure is used, where, represents the distribution difference measure generalized by or Rényi divergence, which is used to measure the difference between the encoder posterior distribution and the prior distribution ; and for images in the labeled data set , a supervised loss is imposed, while unlabeled samples only rely on reconstruction and prior divergence. Therefore, the overall loss function of the improved variational autoencoder is defined as:

[0101] ;

[0102] where, β denotes a regulation coefficient; λ denotes an importance coefficient of the supervision loss; denotes the conditional distribution of the decoder; denotes the random variable take the expectation, i.e. in the posterior distribution , the expectation of a function about z is taken; denotes the conditional distribution of the decoder; denotes the learnable parameters of the VAE encoder; denotes the learnable parameters of the VAE decoder; denotes the concentration parameter at the level of the current Dirichlet process prior; take the expectation, i.e. in the posterior distribution , the expectation of a function about z is taken; denotes the i-th image in the labeled dataset; i denotes the i-th image in the unlabeled dataset; j

[0103] the reconstruction error in equation (1) is used to encourage the decoder to recover the original image as much as possible on the latent variable , so as to improve the representation ability; the latent variable distribution alignment is used to push the posterior output by the encoder to the DP prior ; the supervision loss is used for the supervision of the segmentation, typing or quantification index of the labeled sample data in the labeled dataset.

[0104] (2) increase the segmentation branch in the decoder:

[0105] Since generating a structured report requires segmentation of the lesion area for analysis, in order to make the decoder of the improved variational autoencoder complete lesion segmentation while decoding, the segmentation result is as shown in Figure 4 a- Figure 4 d.

[0106] The present application increases the segmentation branch parallel to the image branch in the decoder of the traditional variational autoencoder as a segmentation mask channel, and takes the image branch of the traditional variational autoencoder as an image reconstruction channel, so that the output of the decoder is divided into two parts: one part is the reconstructed image output by the image reconstruction channel, and the other part is the segmentation mask predicted by the model output by the segmentation mask channel .

[0107] Define the segmentation loss function:

[0108] ​​ (2);

[0109] wherein, denotes the lesion mask annotated sample with label, denotes the loss function, BCE denotes binary cross entropy, denotes the segmentation loss function, denotes the segmentation mask predicted by the model;

[0110] The segmentation loss function is added to the supervised loss of formula (1) as part of the supervised loss.

[0111] A regression head is additionally added at the end of the segmentation branch of the encoder to process the input annotated image containing numerical quantitative indicators, so that the encoder can predict these numerical indicators while segmenting the lesions, and a regression loss function is defined:

[0112] (3);

[0113] wherein, is the i th real value of the input, is the th predicted value output by the encoder.

[0114] The regression loss function is also added to the supervised loss of formula (1) as part of the supervised loss;

[0115] In addition, in order to preserve the consistency of the lung structure, a sobel edge constraint is performed on the feature map output by the decoder (i.e. the segmentation mask and the reconstructed image), so that the lesion edge is smoother. Thus, a semi-supervised VAE framework that can both image reconstruction and image segmentation is obtained, wherein the unannotated sample mainly contributes to the reconstruction and prior divergence loss, the annotated sample can also be supervised for segmentation, and further regression of related indicators can be performed.

[0116] (3) Enhance latent variable disentanglement:

[0117] Since COPD lesions can present multiple shapes and sizes, in order to make the latent variable have better disentanglement ability for this diversity, the present application inserts a Flow model (such as RealNVP, Glow) between the encoder and the decoder for multi-level coupling, thereby obtaining a new latent variable further improve the representation ability of the latent variable for multiple lesion shapes;

[0118] By calculating the log-likelihood of the latent variable , the reversible transformation of the latent variable is obtained:

[0119] (4) ;

[0120] wherein, denotes the determinant of the Jacobian of each layer, which guarantees the invertibility of the transformation and can be calculated in backpropagation.

[0121] Since the goal of training the Flow model is to maximize the log probability of the data under the target distribution, i.e., to minimize the negative log-likelihood (invertible transformation), the invertible transformation term is added as a loss to the overall loss function of formula (1) , so as to obtain the final loss function of the improved variational autoencoder:

[0122] (5) ;

[0123] wherein, denotes the latent variable after the Flow invertible transformation, denotes the determinant of each layer of the invertible transformation, denotes the log-likelihood of the transformed latent variable under the Gaussian prior; is a summation symbol, indicating that the calculation and summation are performed on all samples in the data set , which means that the calculation of the Flow log-likelihood term is for each sample; is an expectation symbol, indicating that the posterior distribution x of the latent variable z is taken under the condition that the input is given, and in actual calculation, this expectation is usually approximated by sampling several z from the posterior distribution and then taking the average; γ represents a weight coefficient, used to balance the relative importance of different loss terms; full represents that the loss function is complete.

[0124] Based on this, all samples (labeled images and unlabeled images) can be included in the encoder training of the improved variational autoencoder, and the improved variational autoencoder is trained on the labeled data set and the unlabeled data set, and the loss function during training adopts formula (5), and the latent variable can be obtained through training.

[0125] In summary, the present application constrains the latent variable space by the Dirichlet process prior and can be coupled by the interlayer coupling layer for multi-step unwinding, capturing different lesion shape distributions. The segmentation branch and the reconstruction branch are changed to obtain latent codes from instead of , so as to obtain more flexible expression capability. For unlabeled samples, the Flow can also be aligned to the prior, and the real image distribution is approximated during decoding, and more COPD variation patterns are learned under the support of a large amount of unlabeled data.

[0126] Furthermore, in order to align unlabeled data to the distribution of labeled data and achieve domain adaptation in multi-center, multi-device scenarios, this invention introduces an adversarial discriminator and a pseudo-label strategy.

[0127] First, based on the improved variational autoencoder, latent variables can be obtained for each image (regardless of whether it is labeled) through the encoder. The decoder can output a reconstructed image. and segmentation mask .

[0128] First, based on the improved variational autoencoder, latent variables can be obtained for each image (regardless of whether it is labeled) through the encoder. Based on real footage x and its latent variables Constructing real combinations ;

[0129] The forged combination is generated through a "decode-re-encode" loop, that is, first through the decoder... From latent variables Obtain the reconstructed image Then reconstruct the image Input encoder New latent variables are obtained. This leads to the formation of a forged combination. ;

[0130] Next, the real and fake combinations are fed into the adversarial discriminator. The discriminator tries to distinguish which combination is the real "image + latent variable" and which is the "image + latent variable extracted by the encoder" faked by the decoder. The improved variational autoencoder tries to deceive the discriminator to make the fake combination look real enough.

[0131] At the same time, adversarial loss is defined using WGAN-GP. :

[0132] (6);

[0133] in, This indicates that the discriminator recognizes the real image. Rating; This indicates a score for the artifacts output by the decoder; It is a gradient penalty term; The weights of the gradient penalty term.

[0134] By minimizing this adversarial loss Unlabeled images can be approximated to the distribution of labeled data in the latent variable and reconstruction space, thus enabling unlabeled samples to learn more "realistic" lesion structures and latent variable representations.

[0135] Secondly, when data comes from different sources—that is, when data is collected, the acquired lung medical images come from multiple medical centers (hospitals, clinics, etc.) or use different types or models of imaging equipment (such as different brands of CT scanners or X-ray machines)—the distribution of data may vary significantly due to differences in protocols, equipment performance, or operator skills between centers. This heterogeneity can lead to inconsistent model performance across different centers or devices, affecting its generalization ability and the reliability of clinical applications. Therefore, this invention also performs domain adaptation to ensure that adversarial alignment is not affected by device differences. Specifically, a domain classifier branch is added to the discriminator. The discriminator adopts a PatchGAN architecture, and the domain classifier branch uses a sliding window approach, based on the domain label... Different regions of the labeled image are distinguished, such as Figure 5 As shown.

[0136] PatchGAN is used to discriminate input images, extracting image features and feeding these features into two independent branches: one branch is a real / fake discrimination branch, used for real / fake classification, outputting a binary classification result for each image; the other branch is a newly added domain classifier branch, which introduces a gradient inversion layer to invert gradients during backpropagation, forcing the backbone network to learn domain-independent feature representations, thereby reducing the impact of distribution differences from different centers and devices on the model. The domain classifier branch outputs the probability distribution of the domain to which the sample belongs through a series of fully connected layers. When the discriminator attempts to distinguish whether an image is real or fake, it also attempts to predict which domain the sample comes from, thus achieving data distribution alignment across devices and hospitals. Furthermore, in multi-hospital (multi-center) scenarios, adversarial learning is used to simultaneously complete real / fake discrimination and domain confusion, making the features extracted by the model from unlabeled data more discriminative and effective, and closer to the feature patterns contained in labeled data. After the distribution is aligned, the improved variational autoencoder is trained based on a large amount of unlabeled data combined with labeled data, thereby obtaining better training results.

[0137] 3. Preliminary report generation:

[0138] To automatically generate structured medical reports, this invention is based on the latent variables output by a trained improved variational autoencoder. and the output features of the decoder's segmented branches (i.e., the sum of the reconstructed image and the segmentation mask), a fusion head is added at the end of the segmentation branch. Image information (i.e., output features) This data is then fused with the patient's phenotypic information (such as age, sex, and lung function FEV1 / FVC), and this phenotypic information is the aforementioned text report. ;

[0139] The specific process of integration is as follows: First, and Each vector is flattened into a fixed-dimensional vector, and then the phenotypic information is numerically embedded to obtain vector P. Finally, the fixed-dimensional vector and vector P are concatenated along the feature dimensions to form a joint embedding. Finally, the joint embedding is input into the fusion head. middle.

[0140] To automatically learn and assign appropriate image feature weights to different clinical fields, achieving deeper modality fusion, this invention provides a fusion head. Based on the multi-head cross-attention mechanism To achieve integration with phenotypic information, specifically, first integrate latent variables... The output feature h is used to generate the result as a key / value pair through two sets of linear projections; then, the fields in each phenotypic information are... Using a set of linear projections, the projection results are used as the Query. The similarity between the phenotypic field (Query) and the image token (Key / Value) is calculated using a multi-head attention formula. The phenotypic field and the image token are then weighted and fused to output the fused contextual feature c (i.e., the latent variable). The multi-head attention formula is as follows:

[0141] (7);

[0142] in, Indicates Query; These represent the Key and Value, respectively. t The sequence length of the query (the number of table fields); s The sequence length of the key / value pair (the number of image feature vectors or feature sequences). d k For the projection dimension of Key / Value, This is a scaling factor used to prevent the softmax distribution from becoming too steep due to excessively large inner product values;

[0143] The multi-head cross-attention mechanism enables different clinical fields to retrieve the most relevant image features, thereby automatically learning to allocate more visual attention to key fields (such as FEV1 / FVC or smoking history), forming a deeper and more flexible modality fusion.

[0144] Based on the obtained contextual features c, the text decoder generates report text using an autoregressive approach, specifically including:

[0145] Firstly, all the words in the acquired text report r i are composed into a word vector sequence , where represents the T th word vector, and a start symbol is added in front of the first word vector . Each word vector is converted into an embedding , and the embedding is taken as the input of the text decoder;

[0146] Secondly, the text decoder performs a cross- self-attention operation on each word vector in turn at each time step to obtain the decoding state of the current word vector, and then performs a cross- self-attention operation again on the decoding state of the word vector and the context feature output by the fusion head, where is taken as Query, and is taken as Key / Value after linear mapping, to finally obtain the updated decoding state of the word vector, so that the text decoder can query the image and phenotype information at each time step;

[0147] Thirdly, the updated decoding state of the word vector is input into a fully connected layer, and a word probability distribution is output, which is processed by softmax to predict the th word , until all the words corresponding to the word vectors are predicted, and an end symbol is added after the last word vector after the last word is predicted.

[0148] Finally, when the end symbol is generated, all the predicted words are composed into a preliminary report text and output.

[0149] In summary, the text decoder starts from the start symbol , selects and feeds it back to the next step by step, until the end symbol is generated. This mechanism ensures that the text decoder can access the context feature output by the fusion head at any time through the cross-attention mechanism when describing the lesion, so as to generate a more complete, accurate and clinically logical structured report by combining the multi-scale context of the image and the clinical phenotype information.

[0150] 4. Structured report generation:

[0151] To ensure the correct use of medical terms and abbreviations in structured report text, the present application first acquires a Teacher network (such as PubMedBERT, BioGPT) pre-trained on a vast medical knowledge base (including a vast amount of medical text), and uses the above text decoder as a Student network, and performs knowledge distillation with the trained Teacher network to obtain a trained Student network, and then uses the Student network to modify new medical terms and abbreviations in the preliminary text report, and outputs the final structured report.

[0152] Specifically, for key medical terms, the embedding of the Teacher network and the Student network are aligned to reduce the problem of rare words or OOV (Out-of-Vocabulary); if a completely new abbreviation appears, it can be found by BFS (Breadth-First Search) in the Unified Medical Language System (UMLS), SNOMED CT (international standard medical terminology) knowledge graph to find similar concepts and insert them into the word vector matrix in the Student network. The word vector matrix is a matrix composed of all predicted words in the preliminary report text output by the text decoder.

[0153] The distillation loss is defined by the following formula :

[0154] (8);

[0155] wherein, represents the embedding vector of the corresponding word or concept in the Student network and the Teacher network, respectively, represents the negative number of Euclidean distance or cosine similarity, represents each medical term in the medical vocabulary set, represents the vocabulary set in the medical field.

[0156] The distillation loss defined according to formula (8) Training the Student network can avoid using meaningless abbreviations or misspelled words as much as possible when generating reports through knowledge distillation, and is closer to clinical language.

[0157] 5. Explainable analysis:

[0158] Due to the high explainability requirement of medical scenarios, Grad-CAM heatmaps can be further calculated for both the segmentation branch and the text decoder after outputting the preliminary report text and before outputting the final structured report. Specifically, it includes:

[0159] For the segmentation branch of the improved variational autoencoder, the last convolutional layer in the segmentation branch is taken as the target layer. After forward propagation to obtain the segmentation mask, the gradients of the target lesion class with respect to each feature map output by the last convolutional layer are calculated. Global average pooling is performed on these gradients to obtain the weight of each feature map. Then, the obtained weight of each feature map is multiplied by the corresponding feature map and summed, and ReLU activation is applied to generate the Grad-CAM heat map of the segmentation mask, showing the pixel regions that the network focuses on;

[0160] For the text decoder, its key attention layer (such as the last self-attention layer) is selected. When generating each predicted word, the gradient of the predicted word with respect to the feature map of the layer is calculated. Similarly, the gradients are globally averaged to obtain the weight, which is multiplied by the feature map and summed, and then ReLU is applied to generate the Grad-CAM heat map of the corresponding predicted word, which is used to show the most focused word vector position of the network.

[0161] If these high attention areas and real lesion annotations are significantly different, use XAI penalty term Force the network to be more consistent in key areas;

[0162] Specifically, the heat map generated by Grad-CAM is denoted as , and the real lesion mask or prior attention area is denoted as Then use the simplified XAI constraint in the form of mean square error :

[0163] ;

[0164] where T is the total number of generated words, is the spatial range, is the total number of pixels in the corresponding spatial range, for example, the total number of pixels in the range of the segmentation output , is the Grad-CAM heat map activation value (normalized) of the t th word at position p , is the annotation value of the t th word corresponding to the real lesion mask or prior area at position p (0 indicates a non-key area and 1 indicates a key area), p is the position of the pixel or feature map traversed;

[0165] By minimizing , the network can be forced to pay more attention to the key areas that are consistent with the real lesions or prior attention areas, reducing deviation and improving interpretability.

[0166] On the other hand, when the case data to be analyzed is a low confidence case, a prompt such as "further examination needed" needs to be inserted in the structured report to remind the doctor to focus on the review. Specifically, if the segmentation score of a lesion is low (i.e. a low confidence case), the following formula is used to calculate:

[0167]

[0168] wherein represents the consistency probability of the lesion prediction under multiple sampling or multi-view fusion, represents the number of samplings, represents the entropy of the prediction distribution;

[0169] If the calculated entropy value is less than 0.3, it means that the uncertainty is low (high confidence), otherwise, if the entropy value is greater than or equal to 0.7, it means that the uncertainty is high (low confidence), and the low confidence cases are marked as "manual secondary review" through regularization or gating. This interpretability, combined with adversarial correction, can ensure safety and clinical acceptability in medical scenarios.

[0170] Finally, the method generates a structured report for a patient encountered in a clinical image workstation. The basic process is as follows: given a new patient scan image , the improved encoder of the variational autoencoder outputs the latent variable , and the segmentation branch of the decoder generates the emphysema / chronic bronchitis segmentation mask , and can simultaneously regress the LAA%, Pi10 and other indicators. The fusion head fuses the latent variable and and the reconstructed image to obtain the context feature , and the text decoder outputs the preliminary report text , and the lesion attention heat map or uncertainty prompt is generated based on the interpretability analysis. The doctor can view the results on the PACS or EHR interface, and if there is an error, the lesion boundary can be corrected or the report can be modified, and the system records these incremental annotations for online fine-tuning. The final confirmed segmentation and report are exported in DICOM SR or HL7 FHIR (ImagingStudy / DiagnosticReport) format, and are mapped to LOINC, SNOMED CT, ICD-10 and other standard terms. This can facilitate clinical follow-up (such as viewing the change of LAA% in the same patient three months ago) and interconnection with medical insurance or research systems. ​

[0171] Embodiment:

[0172] Patient Zhang, male, repeated cough, sputum with shortness of breath after 5 years, symptoms worsened in the past half year. Past history: smoking history 25 years (average 15 per day), has quit smoking for 2 years; no history of other respiratory diseases. The doctor issued an examination sheet, performed imaging examination: high-resolution chest CT (HRCT), pulmonary function test (PFT): FEV1 / FVC ratio determination, arterial blood gas analysis: assess oxygenation status, focus on indicators: LAA%, Pi10. Diagnosis: emphysema distribution: mainly in both upper lobes, left upper lobe accounts for , right upper lobe accounts for . Small airway lesions: , suggesting significant airway wall thickening. Uncertainty prompts: right lower lobe local area predicted entropy value , system marked as "need thin layer scanning review", finally generate a structured report as shown in Figure 1 .

[0173] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application rather than limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements also cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A multimodal semi-supervised image analysis and structured report generation method, characterized in that, The method comprises the following steps: Step 1: obtaining lung medical images and text reports of patients with chronic obstructive pulmonary disease, preprocessing the lung medical image data and establishing an image data set, and dividing the image data set into an annotated data set and an unannoted data set; Step 2: constructing an improved variational autoencoder and training it on the annotated data set and the unannoted data set to obtain the latent variable and the segmentation mask of the lung medical image; Step 3: fusing the text report with the latent variable and the segmentation mask, and generating a preliminary report text by using a text decoder; Step 4: obtaining a pre-trained Teacher network on a medical knowledge massive library, and performing knowledge distillation on the text decoder as a Student network and the Teacher network to modify the medical terms and abbreviations in the preliminary report text and output a final structured report; In step 2, the improved variational autoencoder is obtained by introducing a Dirichlet process prior in the encoder of the traditional variational autoencoder, adding a segmentation branch in the decoder, and inserting a Flow model between the encoder and the decoder for multi-level coupling.

2. A multi-modal semi-supervised image analysis and structured report generation method as claimed in claim 1, wherein, Step 2 specifically comprises the following steps: Step 2.1: introducing a Dirichlet process prior in the encoder of the traditional variational autoencoder to obtain the prior distribution of the latent variable according to the Dirichlet process prior; Step 2.2: adding a segmentation branch parallel to the image branch in the decoder of the traditional variational autoencoder; The segmentation branch is a segmentation mask channel, and the image branch is an image reconstruction channel; Step 2.3: adding a regression head at the end of the segmentation branch for processing sample data containing numerical quantitative index labels; Step 2.4: inserting a Flow model between the encoder and the decoder for multi-level coupling to establish the final improved variational autoencoder; The final loss function of the improved variational autoencoder is: ； wherein, represents the latent variable after the Flow reversible transformation; represents the determinant of the reversible transformation of each layer; represents the log-likelihood of the transformed latent variable under the Gaussian prior; represents the overall loss function including the regression loss function; represents the segmentation loss function; represents the annotated dataset; represents the unannotated dataset; x represents the image in the image dataset; represents the posterior distribution; represents the weight coefficient to balance the relative importance of different loss terms; Step 2.5: Reconstructing images and segmentation masks for each sample in the image dataset from the latent variables Step 2.5: Reconstructing images and segmentation masks for each sample in the image dataset from the latent variables Step 2.6: based on the final loss function, training the improved variational autoencoder on the annotated data set and the unannoted data set after aligning the distribution, and outputting the reconstructed image and the segmentation mask through the image reconstruction channel and the segmentation mask channel; Step 2.7: performing sobel edge constraint on the reconstructed image and the segmentation mask output by step 2.6 to make the lesion edge smoother.

3. A multi-modal semi-supervised image analysis and structured report generation method as claimed in claim 2, wherein, The specific steps of step 2.5 include: Step 2.5.1: adding a domain classifier at the end of the traditional adversarial discriminator to obtain an improved adversarial discriminator; Step 2.5.2: Realistic images based on the image dataset x and their latent variables , construct realistic combinations ; wherein, denotes an encoder of the improved variational autoencoder; Step 2.5.3: From the latent variables reconstruct the image and input the reconstructed image back into the improved variational autoencoder to regenerate the latent variables , construct the fake combination ; wherein, to improve the decoder of a variational autoencoder; Step 2.5.4: inputting the real combination and the fake combination into the adversarial discriminator, and predicting the domain label of the image in the real combination and the fake combination through the domain classifier to determine the source of the image; Step 2.5.5: defining the adversarial loss of the discriminator: ； wherein, represents the score of the discriminator on real images ; represents the score of the discriminator on fake images output by the decoder; is a gradient penalty term; is a weight for the gradient penalty term; Step 2.5.6: aligning the unannotated data of different sources in the unannotated data set to the distribution of the annotated data in the annotated data set by minimizing the adversarial loss.

4. A multi-modal semi-supervised image analysis and structured report generation method as claimed in claim 3, wherein, The specific steps of step 3 include: Step 3.1: connecting a fusion head at the end of the segmentation branch, and the fusion head fuses the latent variable and the segmentation mask with the text report through a multi-head cross-attention mechanism to obtain fused context features c; Step 3.2: generating a preliminary report text based on the obtained context feature c using a text decoder.

5. A multi-modal semi-supervised image analysis and structured report generation method as claimed in claim 4, wherein, The specific steps of step 3.2 include: Step 3.2.1: taking each word in the text report as a word vector, converting the word vector to an embedding and inputting it to the text decoder; Step 3.2.2: The text decoder performs a check on the first text at each time step. t Perform a cross-attention operation on the word vector to obtain the... t The first decoding state of each word vector Then the first decoding state With context features Perform cross-attention operation again to obtain the second decoding state. ; Step 3.2.3: According to the second decoding state obtaining a word probability distribution and performing softmax processing on the word probability distribution to predict a predicted word of the t word vector; Step 3.2.4: outputting a preliminary report text composed of all predicted words.

6. A multi-modal semi-supervised image analysis and structured report generation method as claimed in claim 5, wherein, The loss function during knowledge distillation training is: ； wherein, represents a distillation loss function; , respectively represent embedding vectors of corresponding words or concepts in the Student network and the Teacher network; represents taking the negative of the Euclidean distance or cosine similarity, represents each medical term in the medical vocabulary set; represents a vocabulary set in the medical field.

7. A multimodal semi-supervised image analysis and structured report generation system, characterized in that, Including: The data acquisition module is used to acquire the lung medical images and text reports of patients with chronic obstructive pulmonary disease, pre-process the lung medical image data and establish an image data set, and divide the image data set into an annotated data set and an unannotated data set. The model construction module is used to construct an improved variational autoencoder and train it on the annotated data set and the unannotated data set to obtain the latent variables and segmentation masks of the lung medical images; the improved variational autoencoder is obtained by introducing a Dirichlet process prior in the encoder of a traditional variational autoencoder, adding a segmentation branch in the decoder, and inserting a Flow model for multi-level coupling between the encoder and the decoder; The structured report generation module is used to generate a preliminary report text and correct medical terms and abbreviations therein, and output a final structured report. The structured report generation module includes: The report generation unit is used to fuse the text report with the latent variables and the segmentation masks, and generate a preliminary report text using a text decoder; The report modification unit is used to obtain a pre-trained Teacher network on a medical knowledge base, and perform knowledge distillation between the text decoder as a Student network and the Teacher network to modify medical terms and abbreviations in the preliminary report text; The report generation module is used to output a final structured report.

8. An electronic device, comprising: The device includes at least one processor; and a memory connected in communication with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the multi-modal semi-supervised image analysis and structured report generation method of any one of claims 1-6.

Images