Radiology clinical medical image report generation method and system
Through weakly supervised comparative learning and cross-modal alignment network, the data deviation and multimodal alignment problems in radiological report generation are solved, and the detection ability of abnormal features in medical images and the alignment effect of multimodal information is improved.
Patent Information
- Application Number
- CN202411050224.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
AI Technical Summary
There are data bias and multimodal alignment problems in the generation of radiological clinical medical imaging reports, which leads to models that tend to focus on normal discoveries and ignore abnormalities when capturing image features, and it is difficult to align features between different modal information.
采用基于弱监督对比学习和跨模态对齐网络的方法,通过重校准视觉抽取器和跨模态门控记忆模块,识别医学影像中的异常特征并对齐图像和文本信息,生成准确一致的医学报告。
It enhances the detection and positioning ability of abnormal features in medical images, alleviates data deviation and multimodal alignment problems, and realizes effective information alignment and interaction between image and text information.
Smart Images

Figure CN119028509A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method and system for generating a radiology clinical medical image report based on weakly supervised contrastive learning and a cross-modal alignment network. Background Art
[0002] The continuous development and in-depth application of health and medical big data have made it a new driving force for the development of the country's life economy. Continuous breakthroughs in technologies such as artificial intelligence and computer hardware have provided effective technical support for the information processing and mining of health and medical big data, thereby promoting the deep integration of artificial intelligence in the medical field. With the invasion of large-scale infectious diseases and the acceleration of the aging era of the population, a large amount of medical imaging data is generated every moment in the medical and health field. These medical imaging data include both medical images and corresponding diagnostic reports.
[0003] The task of automatically generating radiological clinical medical image reports has become a systematic study in medical language processing and clinical decision support tasks. The task aims to use deep learning technology to automatically learn and analyze radiological medical images and generate consistent medical diagnosis reports. Traditional report generation tasks rely on radiologists to manually observe images and then analyze and write reports based on their own experience. Due to the surge in the number of patients, this process will be time-consuming, labor-intensive and prone to errors. At the same time, since the lesions in the images are relatively small and may be blurred, this places great demands on the knowledge and experience of radiologists.
[0004] With the continuous development of natural language processing and computer hardware technology, deep neural network models have been widely used in the task of automatic generation of radiology reports. However, this automation process is still challenging, mainly in the following two aspects:
[0005] (1) Data bias problem: Data bias problems mainly include visual data bias and text data bias. Visual data bias is due to the fact that normal medical images often account for a large proportion in the data set, while the number of medical images containing abnormal parts only accounts for a small part. At the same time, in specific medical images, lesions or abnormal lesions usually occupy a small part of the image, and some are even blurred. This causes the model to focus more on normal findings and ignore the identification of abnormal findings when capturing image features. Similarly, since there are more descriptions of normal reports in the data set, it is easier for the model to generate descriptions of normal findings.
[0006] (2) Multimodal alignment problem: Radiology report generation involves two modalities of information: images and text. Different modal information has different modal structures and semantic expressions, which makes it difficult to align features between different modalities, especially abnormal findings in images and corresponding reports.
[0007] Therefore, data bias and multimodality alignment issues are key problems in the task of automatic radiology report generation. Summary of the invention
[0008] In view of this, the present invention proposes an efficient and fast method and system for generating radiology clinical medical image reports based on weakly supervised contrastive learning and cross-modal alignment networks, so as to solve the problem of data bias caused by unbalanced data distribution in medical images, which reduces the learning ability of key feature information and generates data, as well as the problem of cross-modal information fusion and alignment between image and text information.
[0009] The technical solution adopted by the present invention is as follows:
[0010] On the one hand, the present invention provides a radiology clinical medical image reporting method based on weakly supervised contrastive learning and cross-modal alignment network, comprising:
[0011] S1: Preprocess the medical images and texts in the electronic medical record dataset and vectorize the data;
[0012] S2: The image embedding represented by the vectorization in S1 is input into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector; and the text is word-vectorized by word2vec to obtain the text feature;
[0013] S3: The visual feature vector in S2 is fed into the recalibrated visual extractor, which focuses on the abnormal areas of the medical image through the recalibrated model and identifies fine-grained features related to the abnormal features; the aggregated visual feature vector in S2 is fed into the weakly supervised contrast loss module to project similar visual features into the same representation space;
[0014] S4: The text features of step S2 and the recalibrated visual features obtained in step S3 are input into a cross-modal gated memory module to generate accurate and consistent radiological clinical medical images by aligning feature information between different modalities.
[0015] Furthermore, the text in the electronic medical record dataset is preprocessed, including: deleting stop words, doing word segmentation and building a vocabulary.
[0016] Furthermore, the convolutional neural network is a pre-trained ResNet-101.
[0017] Furthermore, the image embedding represented by the vectorization in S1 is input into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector, including:
[0018] Given a batch of radiology medical images As input data, N I It represents the number of radiological medical images;
[0019] Extracting visual feature vectors from radiology medical images using ResNet-101 pre-trained on ImageNet as encoder where v i It is the visual feature extracted by the last convolutional layer of ResNet-101;
[0020] The visual feature vector is averaged and pooled to obtain the aggregated visual feature vector.
[0021] Furthermore, the recalibrated visual extractor mainly includes a residual block and a dual attention mechanism;
[0022] The visual feature vector in S2 is fed into the recalibrated visual extractor. The recalibrated model focuses on the abnormal areas of the medical image and identifies fine-grained features related to the abnormal features, including:
[0023] The extracted visual feature vector V is input into the residual block and the following operations are performed:
[0024]
[0025] Among them, k i ∈R H×W×C ,δ(·) and f r (·) represent the ReLU activation function and residual operation respectively;
[0026] Dual attention receives the visual feature vector K extracted from the residual block and obtains two visual feature information:
[0027]
[0028] M C =σ(F c2 ((F c1 (X))))
[0029] M S =Conv 3×3 (K)
[0030] Among them, M C ∈R H×W×C , M S ∈R H×W×C They represent the visual maps obtained by channel attention and spatial attention mechanisms, respectively. GP (·), F c (·) and σ(·) represent the global pooling operation, the fully connected layer, and the sigmoid function, respectively. Conv 3×3 (·) represents a 3×3 convolution operation;
[0031] Aggregate the visual feature information obtained by the two attention mechanisms:
[0032]
[0033] in, and They represent element-by-element addition and matrix multiplication operations respectively. The set of recalibrated visual feature vectors is expressed as Each visual feature vector The dimensions are H×W×C.
[0034] Furthermore, the aggregated visual feature vector in S2 is fed into the weakly supervised contrast loss module to project similar visual features into the same representation space, including:
[0035] Aggregate the visual feature vector As input, perform data augmentation operation A(·);
[0036] Using CNN Encoder Map the enhanced information to a low-dimensional embedding space to obtain an embedding vector, so that similar images are closer in the embedding space;
[0037] An auxiliary mapping head φ(·) is introduced to find similar samples in the embedding space and generate weak labels as supervision signals. The embedding vector is extracted from the batch samples through the auxiliary mapping head φ(·).
[0038] During model training, p i Input to the weakly supervised contrast loss for calculation, maximizing the loss between positive samples and minimizing the loss between negative samples:
[0039]
[0040] Among them l i ≠l j is an indicator used to distinguish the contrast categories of positive and negative samples; sim(·) represents the cosine similarity between two input vectors, and τ represents the temperature parameter used to balance the contrast relationship between positive and negative samples.
[0041] Furthermore, the text features of step S2 and the recalibrated visual features obtained in step S3 are input into a cross-modal gated memory module to generate accurate and consistent radiological clinical medical images by aligning feature information between different modalities, including:
[0042] The memory matrix M is randomly initialized t ={m1,m2,…,m t ,…,m N}, N represents the number of memory matrices;
[0043] Using attention mechanism to embed word vectors in radiology medical reports and the visual feature vector obtained by the recalibrated visual extractor Interact and align to obtain the memory matrix M for text representation g and the memory matrix M for visual feature representation i , the calculation process is as follows:
[0044] q g =y g W q , k t =m t W k ,
[0045]
[0046] Among them, W q ,W k and M v They represent trainable weight parameters respectively;
[0047] U is introduced to represent the dynamic update process:
[0048] M=U(M t ;M g ,M i )
[0049] During the model training and iteration process, a shared memory matrix M containing rich visual and text feature information is generated, and the information in M is continuously adjusted and updated during the training process.
[0050] Furthermore, it also includes:
[0051] The key text embedding information in the radiology report is obtained dynamically using the state gate, and the information gate is used to reintegrate the multimodal information and the original text information stored in the shared memory matrix; the calculation process of the state gate is as follows:
[0052] G s =σ(W1·y g +b1)⊙y g
[0053] G o =σ(W2·y g +b2)⊙y g
[0054]
[0055] Among them, W1, W2, Wc refers to the trainable weight parameter, b1, b2 refer to the bias parameter, σ is the sigmoid activation function, G s ,G i They represent the final outputs of the state gate and the information gate respectively;
[0056] The output G obtained through the information gate i The output G obtained by the state gate s Add together to obtain the memory matrix G that is finally output by the cross-modal gated memory module c .
[0057] In another aspect, the present invention further provides a radiology clinical medical image reporting system based on weakly supervised contrastive learning and cross-modal alignment network, comprising:
[0058] The data preprocessing module is used to preprocess the medical images and texts in the electronic medical record dataset and vectorize the data; embed the vectorized image into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector; and vectorize the text through word2vec to obtain the text features;
[0059] The recalibrated visual extractor module is used to send the visual feature vector obtained by the data preprocessing module to the recalibrated visual extractor, focus on the abnormal area of the medical image through the recalibrated model, and identify fine-grained features related to the abnormal features;
[0060] The weakly supervised contrastive learning module is used to feed the aggregated visual feature vector obtained by the data preprocessing module into the weakly supervised contrastive loss module to project similar visual features into the same representation space;
[0061] The cross-modal gated memory module is used to input the text features obtained by the data preprocessing module and the recalibrated visual features obtained by the recalibrated visual extractor module into the cross-modal gated memory module, and generate accurate and consistent radiological clinical medical images by aligning the feature information between different modalities.
[0062] Compared with the prior art, the present invention has the following beneficial effects:
[0063] The present invention introduces weakly supervised contrastive learning to maximize the difference between positive samples and negative samples, thereby assisting the medical report generation model to learn useful medical image representation information and enhance the detection and location capabilities of abnormal features. Furthermore, the present invention designs a recalibrated visual extractor module that integrates residual blocks and dual attention mechanisms to identify fine-grained visual features related to abnormal lesions in radiological medical images, thereby alleviating the problem of visual data bias. At the same time, the present invention also proposes a cross-modal gated memory module, which cleverly integrates a gated memory matrix to selectively memorize abnormal areas in images and text description information for abnormal findings, and continuously updates and adjusts the information. A cross-modal gating unit is designed in the module to regulate the transmission between different modal information, making up for the problem of insufficient interaction between cross-modal information, and realizing the alignment and interaction of effective information between cross-modalities. Therefore, the present invention alleviates the data bias and multi-modal alignment problems existing in most models in the current radiology report generation task. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0065] Figure 1 The present invention is a flowchart of a method for generating a radiology clinical medical image report based on weakly supervised contrastive learning and a cross-modal alignment network in an embodiment of the present invention.
[0066] Figure 2 Schematic diagram of the RCAN neural network in an embodiment of the present invention. DETAILED DESCRIPTION
[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0069] like Figure 1 As shown, a method for generating a radiology clinical medical image report based on weakly supervised contrastive learning and a cross-modal alignment network in an embodiment of the present invention specifically includes the following steps:
[0070] S1: Preprocess the radiology medical images and text files in the electronic medical record dataset and vectorize the data in the files.
[0071] S2: The vectorized image embedding in step S1 is input into the pre-trained ResNet-101 to extract rich visual features; and the text is word-vectorized using word2vec.
[0072] First, given a batch of radiological medical images As input data, N I Represents the number of radiological medical images. ResNet-101 pre-trained on ImageNet is used as an encoder to extract visual feature information from radiological medical images and extract a series of visual feature vectors. As input, where v i It is extracted by the last convolutional layer of ResNet-101. The calculation process is as follows:
[0073]
[0074] Among them, v i ∈R H×W×C is the extracted visual feature vector of the medical image, f RIF Refers to the visual feature extraction function.
[0075] Next, in order to retain more local feature information, the visual feature vector V is averaged and pooled to obtain the aggregated visual feature vector The calculation process is expressed as follows:
[0076]
[0077] Among them, b i ∈R HW×C , the aggregated visual feature vector b i The dimension is HW×C, and AvgPool(·) refers to the average pooling operation.
[0078] S3: The visual features in S2 It is sent to the recalibrated visual extractor to enhance the RCAN model's attention to abnormal areas in medical images and identify fine-grained features related to abnormal features; the visual feature vector obtained in S2 is sent to the weakly supervised contrast loss module after an average pooling operation to project similar visual features into the same representation space.
[0079] The RCAN model (Residual Channel Attention Networks) is an advanced deep learning architecture for image super-resolution reconstruction. RCAN effectively improves image quality and achieves advanced results in super-resolution tasks through its extremely deep network structure and channel attention mechanism. The network architecture of RCAN consists of four parts: shallow feature extraction, deep feature extraction (using RIR), upsampling module, and reconstruction part. Through these components, RCAN can effectively extract and utilize shallow and deep features in low-resolution images, and ultimately generate high-quality super-resolution images.
[0080] Specifically, the recalibrated visual extractor introduces a residual block to help better adjust and recalibrate the visual feature vector. The skip connection operation of the residual block allows the feature representation to be dynamically adjusted between channels, thereby improving the quality of the feature representation. The extracted visual feature vector V is input into the residual block and the following operations are performed:
[0081]
[0082] Among them, k i ∈R H×W×C ,δ(·) and f r (·) represent the ReLU activation function and residual operation respectively.
[0083] In this paper, a dual attention mechanism is designed to cleverly integrate the channel attention mechanism (CA) and the spatial attention mechanism (SA), aiming to deeply explore the correlation between the relevant information between channels and within channels in medical visual features. Through the synergy of these two attention mechanisms, the recalibrated visual extractor module can provide richer visual feature representations and improve the model's ability to recognize fine-grained visual features of medical impact.
[0084] Subsequently, the channel attention receives the visual feature vector K extracted from the residual block, which contains rich visual information. This mechanism captures global feature information by averaging the pixels in each channel.
[0085]
[0086] M C =σ(F c2 ((F c1 (X))))#(4)
[0087] M S =Conv 3×3 (K)#(5)
[0088] Among them, M C ∈R H×W×C ,M S ∈R H×W×C They represent the visual maps obtained by channel attention and spatial attention mechanisms, respectively. GP (·), F c (·) and σ(·) represent the global pooling operation, the fully connected layer, and the sigmoid function, respectively. Conv 3×3 (·) represents a 3×3 convolution operation.
[0089] In order to recalibrate the model's ability to focus on abnormal features, the present invention aggregates the visual feature information in the two attention mechanisms, and the formula is as follows:
[0090]
[0091] in, and They represent element-by-element addition and matrix multiplication operations respectively. The set of recalibrated visual feature vectors is expressed as Each visual feature vector The dimensions are H×W×C.
[0092] In weakly supervised contrastive learning, following the working principle of SimCLR (Similarity Contrast for Learning Representations, a self-supervised learning method for visual representation learning. Through the contrastive learning framework, SimCLR can use unlabeled image data to train deep neural networks to obtain high-quality image feature embedding without the need for a large amount of labeled data required by traditional supervised learning), we first transform the aggregated visual feature vector obtained from formula (2) into As input, perform data augmentation operation A(·), which obtains diverse feature enhancement representation by performing different transformations on the image, so that the model can capture richer visual information of medical images. Its calculation formula is as follows:
[0093] b 1 =A(b i ,θ1)#(7)
[0094] b 2 =A(b j ,θ2)#(8)
[0095] Among them, θ1 and θ2 are seeds for random transformation, ensuring that each enhancement is random.
[0096] Then, use the CNN (Convolutional Neural Networks, a type of feedforward neural network with a deep structure that represents input information through convolution calculation) encoder The enhanced information is mapped to a low-dimensional embedding space to obtain an embedding vector, making similar images closer in the embedding space. The process is expressed by the following formula:
[0097]
[0098] Among them, h 1 and h 2 They respectively represent the embedded vectors after being processed by the CNN encoder.
[0099] Afterwards, the auxiliary mapping head φ(·) is introduced in the present invention to find similar samples in the embedding space and generate weak labels as supervisory signals. It is extracted from the batch samples by the auxiliary mapping head φ(·) and can be expressed as:
[0100]
[0101] During model training, p i The input is calculated into the weakly supervised contrast loss. This process maximizes the loss between positive samples and minimizes the loss between negative samples, so that similar image representations are close to each other, thereby improving the model's recognition of the similarities and differences of medical images, allowing the model to recognize the semantic information in medical images. The formula is as follows:
[0102]
[0103] Among them l i ≠l jis an indicator used to distinguish the contrast categories of positive and negative samples. sim(·) represents the cosine similarity between the two input vectors, and τ represents the temperature parameter used to balance the contrast relationship between positive and negative samples.
[0104] S4: The text features obtained in step (2) and the recalibrated visual features obtained in step (3) are input into the cross-modal memory module to generate an accurate and consistent medical report by aligning the feature information between different modalities.
[0105] Specifically, the memory matrix M is randomly initialized t ={m1,m2,…,m t ,…,m N}, N represents the number of memory matrices, using the attention mechanism to embed the word vector of the radiology medical report and the visual feature vector obtained by the recalibrated visual extractor After interaction and alignment, the memory matrix M for text representation can be obtained respectively. g and the memory matrix M for visual feature representation i , the specific calculation process is as follows:
[0106] q g =y g W q , k t =m t W k ,
[0107]
[0108]
[0109] Among them, W q ,W k and W v They represent the trainable weight parameters respectively.
[0110] Because updating the memory matrix is a recursive process, U is introduced to represent this dynamic update process, which is expressed using formula (16):
[0111] M=U(M t ;M g ,M i )#(16)
[0112] Finally, during the model training and iteration process, a shared memory matrix M containing rich visual and text feature information is generated, and is continuously updated and adjusted during the training process, which helps to better capture the correlation between medical imaging and medical text report modality information.
[0113] In addition, in order to retain the effective interactive feature information between different modalities, a cross-modal gating mechanism is adopted in this module. By introducing a cross-modal gating unit to regulate the transmission between different modal information, the present invention uses a state gate to dynamically obtain the key text embedding information in the radiology report, and uses an information gate to reintegrate the multimodal information and original text information stored in the shared memory matrix. The calculation process of the state gate is as follows:
[0114] G s =σ(W1·y g +b1)⊙y g #(17)
[0115] G o =σ(W2·y g +b2)⊙y g #(18)
[0116]
[0117] Among them, W1, W2, W c refers to the trainable weight parameter, b1, b2 refer to the bias parameter, σ(·) is the sigmoid activation function, G s ,G i They represent the final outputs of the state gate and the information gate respectively.
[0118] Finally, the output G obtained through the information gate i The output G obtained by the state gate s Add together to obtain the memory matrix G that is finally output by the cross-modal gated memory module c :
[0119] G c =G s +G i #(20)
[0120] In the above embodiment, a recalibrated visual extractor is used to improve the model's extraction of fine-grained visual features of abnormal lesion areas. A weakly supervised contrast learning strategy is used to effectively identify and enhance the similarities between medical images, and obtain feature representations of key information in medical images. A cross-modal gated memory module is used to retain effective interactive features between different modal information, thereby accurately capturing the association between text and image information, and achieving in-depth understanding and alignment between different modal information.
[0121] The method in the above embodiment solves the problem of reduced learning ability of key feature information due to unbalanced data distribution in medical images, as well as the problem of cross-modal information fusion and alignment between image and text information. The present invention not only improves the learning ability of key features in medical images, but also enhances the interaction and fusion ability between multimodal information, and realizes the interaction and alignment between different modal information.
[0122] Corresponding to the above-mentioned method for generating a radiology clinical medical image report based on weakly supervised contrastive learning and a cross-modal alignment network, an embodiment of the present invention further provides a radiology clinical medical image report system based on weakly supervised contrastive learning and a cross-modal alignment network, comprising:
[0123] The data preprocessing module is used to preprocess the medical images and texts in the electronic medical record dataset and vectorize the data; embed the vectorized image into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector; and vectorize the text through word2vec to obtain the text features;
[0124] The recalibrated visual extractor module is responsible for adjusting and recalibrating the extracted visual feature vectors of medical images through residual blocks and dual attention mechanisms, enhancing the model's ability to identify abnormal features and thus alleviating the problem of visual data bias. Specifically, the visual feature vectors obtained by the data preprocessing module are fed into the recalibrated visual extractor, which focuses on the abnormal areas of the medical images through the recalibrated model and identifies fine-grained features related to the abnormal features;
[0125] The weakly supervised contrast learning module is responsible for bringing similar image features closer together and dissimilar features farther apart. This strategy effectively identifies and strengthens the similarities between medical images and obtains the feature representation of key information in medical images. Specifically, the aggregated visual feature vector obtained by the data preprocessing module is fed into the weakly supervised contrast loss module to project similar visual features into the same representation space.
[0126] The cross-modal gated memory module designs a memory matrix to store text and image information, and continuously updates and adjusts it during the training process. It then introduces a gating mechanism to effectively filter out noise information and retain effective interactive features between different modal information, thereby accurately capturing the association between text and image information and achieving in-depth understanding and alignment between different modal information. Specifically, the cross-modal gated memory module inputs the text features obtained by the data preprocessing module and the recalibrated visual features obtained by the recalibrated visual extractor module into the cross-modal gated memory module, and generates accurate and consistent radiological clinical medical images by aligning the feature information between different modalities.
[0127] As for the system in the embodiment of the present invention, since it corresponds to the method in the above embodiment, the description is relatively simple. For the relevant similarities, please refer to the description of the method part in the above embodiment, which will not be described in detail here.
[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A radiology clinical medical image reporting method based on weakly supervised contrastive learning and cross-modal alignment network, characterized in that: include: S1: Preprocess the medical images and texts in the electronic medical record dataset and vectorize the data; S2: The image embedding represented by the vectorization in S1 is input into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector; and the text is word-vectorized by word2vec to obtain the text feature; S3: The visual feature vector in S2 is fed into the recalibrated visual extractor, which focuses on the abnormal areas of the medical image through the recalibrated model and identifies fine-grained features related to the abnormal features; the aggregated visual feature vector in S2 is fed into the weakly supervised contrast loss module to project similar visual features into the same representation space; S4: The text features of step S2 and the recalibrated visual features obtained in step S3 are input into a cross-modal gated memory module to generate accurate and consistent radiological clinical medical images by aligning feature information between different modalities.
2. The method according to claim 1, characterized in that: Preprocess the text in the electronic medical record dataset, including: removing stop words, doing word segmentation, and building a vocabulary.
3. The method according to claim 1, characterized in that The convolutional neural network is a pre-trained ResNet-101.
4. The method according to claim 3, characterized in that: The image embedding represented by the vectorization in S1 is input into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector, including: Given a batch of radiology medical images As input data, N I It represents the number of radiological medical images; Extracting visual feature vectors from radiology medical images using ResNet-101 pre-trained on ImageNet as encoder where v i It is the visual feature extracted by the last convolutional layer of ResNet-101; The visual feature vector is averaged and pooled to obtain the aggregated visual feature vector.
5. The method according to claim 4, characterized in that The recalibrated visual extractor mainly includes a residual block and a dual attention mechanism; The visual feature vector in S2 is fed into the recalibrated visual extractor. The recalibrated model focuses on the abnormal areas of the medical image and identifies fine-grained features related to the abnormal features, including: The extracted visual feature vector V is input into the residual block and the following operations are performed: Among them, k i ∈R H×W×C ,δ(·) and f r (·) represent the ReLU activation function and residual operation respectively; Dual attention receives the visual feature vector K extracted from the residual block and obtains two visual feature information: M C =σ(F c2 9(F c1 (X)))) M S =Conv 3×3 (K) Among them, M C ∈R H×W×C , M S ∈R H×W×C They represent the visual maps obtained by channel attention and spatial attention mechanisms, respectively. GP (·), F c (·) and σ(·) represent the global pooling operation, the fully connected layer, and the sigmoid function, respectively. Conv 3×3 (·) represents a 3×3 convolution operation; Aggregate the visual feature information obtained by the two attention mechanisms: in, and They represent element-by-element addition and matrix multiplication operations respectively. The set of recalibrated visual feature vectors is expressed as Each visual feature vector The dimensions are H×W×C.
6. The method according to claim 4, characterized in that The aggregated visual feature vector in S2 is fed into the weakly supervised contrast loss module to project similar visual features into the same representation space, including: Aggregate the visual feature vector As input, perform data augmentation operation A(·); Using CNN Encoder Map the enhanced information to a low-dimensional embedding space to obtain an embedding vector, so that similar images are closer in the embedding space; An auxiliary mapping head φ(·) is introduced to find similar samples in the embedding space and generate weak labels as supervision signals. The embedding vector is extracted from the batch samples through the auxiliary mapping head φ(·). During model training, p i Input to the weakly supervised contrast loss for calculation, maximizing the loss between positive samples and minimizing the loss between negative samples: Among them l i ≠l j is an indicator used to distinguish the contrast categories of positive and negative samples; sim(·) represents the cosine similarity between two input vectors, and τ represents the temperature parameter used to balance the contrast relationship between positive and negative samples.
7. The method according to claim 1, characterized in that The text features of step S2 and the recalibrated visual features obtained in step S3 are input into the cross-modal gated memory module to generate accurate and consistent radiological clinical medical images by aligning the feature information between different modalities, including: The memory matrix M is randomly initialized t ={m1,m2,…,m t ,…,m N }, N represents the number of memory matrices; Using attention mechanism to embed word vectors in radiology medical reports and the visual feature vector obtained by the recalibrated visual extractor Interact and align to obtain the memory matrix M for text representation g and the memory matrix M for visual feature representation i , the calculation process is as follows: Among them, W q ,W k and W v They represent trainable weight parameters respectively; U is introduced to represent the dynamic update process: M=U(M t ;M g ,M i ) During the model training and iteration process, a shared memory matrix M containing rich visual and text feature information is generated, and the information in M is continuously adjusted and updated during the training process.
8. The method according to claim 7, characterized in that Also includes: The key text embedding information in the radiology report is obtained dynamically using the state gate, and the information gate is used to reintegrate the multimodal information and the original text information stored in the shared memory matrix; the calculation process of the state gate is as follows: G s =σ(W1·y g +b1)⊙y g G o =σ(W2·y g +b2)⊙y g Among them, W1, W2, W c refers to the trainable weight parameter, b1, b2 refer to the bias parameter, σ is the sigmoid activation function, G s ,G i They represent the final outputs of the state gate and the information gate respectively; The output G obtained through the information gate i The output G obtained by the state gate s Add together to obtain the memory matrix G that is finally output by the cross-modal gated memory module c .
9. A radiology clinical medical image reporting system based on weakly supervised contrastive learning and cross-modal alignment network, characterized in that: include: The data preprocessing module is used to preprocess the medical images and texts in the electronic medical record dataset and vectorize the data; The vectorized image embedding is input into the convolutional neural network to extract the visual feature vector and aggregate the visual feature vector; and the text is word-vectorized through word2vec to obtain the text features; The recalibrated visual extractor module is used to send the visual feature vector obtained by the data preprocessing module to the recalibrated visual extractor, focus on the abnormal area of the medical image through the recalibrated model, and identify fine-grained features related to the abnormal features; The weakly supervised contrastive learning module is used to feed the aggregated visual feature vector obtained by the data preprocessing module into the weakly supervised contrastive loss module to project similar visual features into the same representation space; The cross-modal gated memory module is used to input the text features obtained by the data preprocessing module and the recalibrated visual features obtained by the recalibrated visual extractor module into the cross-modal gated memory module, and generate accurate and consistent radiological clinical medical images by aligning the feature information between different modalities.
Citation Information
Cited By
Urinary surgery question and answer method based on clinical data and nuclear magnetic resonance imaging analysis
CN119513272A
Medical report generation method and system based on cross-modal interactive memory network
CN119626440A
Three-dimensional image data processing method, data processing method, equipment and storage medium
CN119723004A
Three-dimensional image data processing method, data processing method, device and storage medium
CN119723004B
Chest radiograph report generation method and system based on cross-modal alignment and significant semantic region
CN120748604A
