Radiology report generation method and system based on global dependency learning and multi-modal alignment network

Abstract

The invention discloses a radiology report generation method and system based on global dependency learning, and belongs to the technical field of natural language processing. According to the method, a global dependency learning module is designed, and the module integrates rotation position coding to enhance original Mama, so that long-sequence visual feature dependency is effectively modeled, and the model calculation efficiency is improved. According to the method, the problems that the visual long-term dependence modeling and calculation efficiency in the radiology image are difficult to effectively balance, and the heterogeneity of the characteristics of different modal information of the image and the text is difficult to effectively align and fuse are solved; the method not only can effectively capture the long-term dependency relationship on the key visual features in the radiology image, but also can improve the model calculation efficiency, and can enhance the alignment and fusion capability among the multi-modal heterogeneous information.

Description

Technical Field

[0001] This invention belongs to the technical field of natural language processing, and relates to a radiology report generation technology, particularly to a radiology report generation method and system based on global dependency learning and multimodal alignment networks. Background Technology

[0002] Medical imaging data, especially radiological images, plays a crucial role in clinical diagnosis, but its rapid growth also poses new challenges to traditional diagnostic models. Radiological reports are a key link between imaging examinations and clinical treatment, and their accuracy and standardization directly impact medical decisions. However, traditional manual report generation methods have several limitations: First, physicians need to analyze images case by case, resulting in a heavy workload and limited efficiency; second, diagnostic results are easily influenced by physician experience and subjective judgment, and reports from different physicians may differ; furthermore, multimodal image analysis of complex cases often requires interdisciplinary knowledge, posing a challenge to institutions with insufficient medical resources.

[0003] In recent years, artificial intelligence technology has made significant progress in the field of medical image analysis, particularly excelling in tasks such as lesion detection and image segmentation. However, automatically converting image data into radiology reports that meet clinical requirements still faces many technical challenges: (1) It is difficult to effectively balance visual modeling and computational efficiency: Although the method of using pre-trained CNN can effectively extract local image features, its inherent receptive field limitation makes it difficult to model key global contextual information and long-range dependencies in medical images. Visual Transformer (ViT) breaks through this limitation through self-attention mechanism and can capture semantic associations across regions. However, the self-attention mechanism used by this architecture requires calculating the association between all positions in the input sequence, which leads to the computational complexity increasing quadratically with the sequence length. Therefore, the trade-off between visual context modeling and computational efficiency remains an unsolved challenge.

[0004] (2) Cross-modal information alignment problem: Automatic generation of radiology reports involves data processing of two heterogeneous modalities: visual and linguistic. Achieving efficient fusion of image features and text semantics is the core challenge of this task. First, there is a representation gap between the pixel-level features of medical images and the semantic-level features of natural language descriptions, requiring the establishment of a cross-modal mapping bridge. Second, there is a non-linear correspondence between anatomical features in images and professional terms in reports, and this complex association pattern is difficult to effectively fuse through simple feature splicing. Finally, clinical reports often require comprehensive judgment of global information from multiple images, and existing methods still have shortcomings in cross-modal feature alignment. This heterogeneity between modalities not only affects the effectiveness of feature interaction but also restricts the clinical accuracy and completeness of the generated reports.

[0005] Therefore, balancing visual modeling with computational efficiency and cross-modal information alignment is a key issue in the task of automatically generating radiological reports. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an efficient and rapid method and system for generating radiological reports based on global dependency learning and multimodal alignment networks, aiming to generate corresponding textual descriptive reports based on radiological images.

[0007] This invention designs a global dependency learning module that integrates rotational position encoding to enhance the original Mamba (Mamba is an efficient sequence modeling architecture based on the Selective State Space Model (SSM) that can capture long-range dependencies with linear complexity), effectively modeling long-sequence visual feature dependencies and improving model computational efficiency. Simultaneously, this invention develops a cross-modal alignment module, which mainly designs a bidirectional adapter to facilitate multi-level fusion of visual and textual information, achieving heterogeneous semantic-level feature alignment and refinement.

[0008] The technical solution adopted in this invention is as follows: On the one hand, this invention provides a radiology report generation method based on global dependency learning, comprising: S1: Preprocess the medical images and text files in the electronic medical record dataset, and then embed the preprocessed data into vectorized form. S2: Construct a radiology report model; the radiology report model includes: a visual extractor, a global dependency learning module, and a report generator; the visual extractor inputs the vectorized medical image representation into a pre-trained convolutional neural network to extract abstract visual feature representations; and the text report is word-vectorized using word2vec; the global dependency learning module obtains global visual feature representations by integrating Mamba and rotational position encoding to model complex spatial relationships while dynamically capturing relative position information; the report generator performs cross-modal alignment between the word-vectorized text report feature representation and the global visual feature representation to obtain the radiology report; S3. Generate a radiology report using the radiology report model.

[0009] Furthermore, the pre-trained convolutional neural network is ResNet-101.

[0010] Furthermore, abstract visual feature representations are extracted, including: via visual encoder Input medical images Transform into high-dimensional feature representation ; Visual features Two different visual representations are generated: the overall features obtained through global average pooling. and the serialized token features ; Use word2vec to obtain the word embeddings of each word in the text of the report through an embedding layer.

[0011] Furthermore, the global dependency learning module includes: Adjust the serialized token using dynamic position bias. Spatial information; Adjusted features Input convolution operation; Integrating rotational position encoding to enhance Mamba: ; ; ; ; in, and This represents the intermediate features projected into the latent space. , These are the input sequences The first in The and the first A visual embedding vector, and Represent two orthogonal matrices. , There are two weight matrices; This indicates the feature sequence obtained through RoPE, and This represents the output of the improved Mamba; SiLU activation function is represented by , DWConv by , depthwise separable convolution by , Linear by , Norm by , and SS2D by , which represents two-dimensional selective scanning. ; splicing and Obtain global visual features : ; in, Represents the obtained global visual features, Concat( ) indicates a splicing operation.

[0012] Furthermore, in the two-dimensional selective scanning process, the visual features are first divided into non-overlapping blocks, and then scanned along four different paths to generate four independent sequences. Each sequence is then processed by the selective scanning state space sequence model to extract spatial information from different directions while effectively preserving key contextual features. Finally, the four sequences are merged to form a unified 2D visual feature representation.

[0013] Furthermore, cross-modal alignment is performed on the word vectorized text report feature representation and the global visual feature representation, including: A bidirectional adapter module is embedded in the encoder-decoder framework to achieve bidirectional interaction between the visual and textual modalities through a dynamic feature cueing mechanism. The bidirectional adapter module includes: a visual-to-text feature enhancement process and a text-to-visual feature compensation process. ; ; ; in, and These represent visual-to-text feature cues and text-to-visual feature cues, respectively, and the updated visual features. It will simultaneously contain original image features and textual semantic information; Each visual feature was extracted using the CheXpert radiology disease label extractor. Generate pseudo tags Automatically label the presence of various common diseases in medical images: ; in, It is the number of disease categories. These are weight parameters. It is a tag generation function.

[0014] The visual feature sequence incorporating pseudo-labels is projected through a projection layer. and the hidden state of the decoder Projected onto a shared feature space: ; A Transformer-based model is used to generate radiology reports.

[0015] Furthermore, a Transformer-based model is used to generate radiology reports, including: At time step Visual features Mapping to intermediate feature representation: ; The word embeddings and visual intermediate feature representations are fed into the decoder to generate the output for the current time step. ;in, It is a time step Predicted words, It is the number of intermediate features; Iteratively repeat the above process until the report is fully generated, as shown below. .

[0016] In another aspect, the present invention also provides a radiological report generation system based on global dependency learning and multimodal alignment networks, comprising: Data preprocessing module: preprocesses medical images and text files in the electronic medical record dataset, and embeds the preprocessed data into vectorized form; Model building module: Constructs a radiology report model; the radiology report model includes: a visual extractor, a global dependency learning module, and a report generator; the visual extractor inputs the vectorized medical image representation into a pre-trained convolutional neural network to extract abstract visual feature representations; and performs word vectorization on the text report using word2vec; the global dependency learning module uses the obtained visual features, and dynamically captures relative position information while modeling complex spatial relationships by integrating Mamba and rotational position encoding, to obtain a global visual feature representation; the report generator performs cross-modal alignment between the word vectorized text report feature representation and the global visual feature representation to obtain the radiology report; Report generation module: Generates radiology reports using the radiology report model.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention designs a global dependency learning module that effectively models long-range dependencies in medical image sequences using a State-Space Model (SSM) backbone network and captures key lesion features. Simultaneously, this module integrates Rotation Position Encoding (RoPE) to enhance the base Mamba model, improving its efficiency and flexibility in capturing relative positional information in visual feature sequences. Furthermore, this invention designs a multimodal alignment module that utilizes a novel cross-modal bidirectional adapter to facilitate multi-level fusion of visual and textual information, addressing the modal heterogeneity problem and achieving semantic-level feature alignment and refinement. Therefore, this invention alleviates two challenges present in most current radiology report generation models: how to balance visual modeling and computational efficiency with cross-modal information alignment. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a radiological report generation method based on global dependency learning and multimodal alignment network in an embodiment of the present invention. Figure 2 This is a schematic diagram of the radiology report model in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] This invention provides a radiology report generation method based on global dependency learning and a multimodal alignment network. Radiology images are input into a pre-trained ResNet-101 convolutional neural network to capture preliminary visual feature representations. Text reports are preprocessed and vectorized for embedding. The global dependency learning module effectively models long-range dependencies in medical image sequences using a State Space Model (SSM) backbone network and captures key lesion features. Simultaneously, the global dependency learning module integrates Rotation Position Encoding (RoPE) to enhance the base Mamba model, improving its efficiency and flexibility in capturing relative positional information in visual feature sequences. Furthermore, to align feature information between different modalities, this invention also designs a multimodal alignment module. This module utilizes a novel bidirectional adapter to promote multi-level fusion of visual and textual information, mitigating the heterogeneity of data from different modalities and thus better aligning and fusing visual and textual information.

[0023] like Figure 1 As shown in the figure, a radiology report generation method based on global dependency learning in an embodiment of the present invention includes the following specific steps: S1: Preprocess the medical images and text files in the electronic medical record dataset, and then embed the preprocessed data into vectorized form.

[0024] S2: Constructing a radiology report model; like Figure 2 As shown, the radiology report model includes: a visual extractor, a global dependency learning module, and a report generator.

[0025] (1) Visual extractor: inputs the vectorized image representation from step S1 into ResNet-101 to extract the abstract visual feature representation; and performs word vectorization on the text report using word2vec.

[0026] First, in the visual feature extraction stage, the visual encoder is used... Input medical images Convert to high-dimensional feature representation:

[0027] in, , Represents the spatial dimension of the feature map. This represents the number of feature channels.

[0028] Then, visual features Two different visual representations are generated: the overall features obtained through global average pooling. and the serialized token features The calculation process can be expressed as follows:

[0029]

[0030] in, , , and These represent global average pooling, layer normalization, projection, and flattening operations, respectively.

[0031] Finally, word2vec is used to obtain the word embeddings of each word in the report through an embedding layer:

[0032] (2) Global dependency learning module, which combines the two visual features obtained in S2 , The data is fed into the global dependency learning module to alleviate the limitations of the local receptive field of CNN, so as to effectively model long-distance dependencies in medical image sequences and capture key lesion features.

[0033] The global dependency learning module uses Mamba (state space selection model) and rotational position encoding to dynamically model the relative positional information between visual features. This module aggregates features that are spatially distant but semantically related, thereby strengthening the global context representation.

[0034] Specifically, the global dependency learning module integrates Rotary Position Encoding (RoPE) to enhance the base Mamba model, improving its efficiency and flexibility in capturing relative positional information from visual feature sequences. Inspired by Dynamic Position Bias (DPB), DPB is first used to adjust the serialized token. Spatial information:

[0035] in, It is a visual feature sequence adjusted by DPB.

[0036] Next, in order to enhance the correlation between different locations in the medical image, the adjusted features... Input convolution operation:

[0037] in, It is the feature representation obtained after the convolution operation.

[0038] To further capture global long-term dependencies and contextual relationships in visual feature sequences, this invention enhances Mamba by integrating Rotation Position Encoding (RoPE):

[0039]

[0040]

[0041]

[0042] in, and This represents the intermediate features projected into the hidden space. , These are the input sequences The first in The and the first A visual embedding vector, and Represent two orthogonal matrices. , There are two weight matrices. This indicates the feature sequence obtained through RoPE, and This represents the output of the improved Mamba. represents the SiLU activation function, DWConv represents depthwise separable convolution, Linear represents a linear layer, Norm represents a normalized layer, and SS2D represents a 2D selective scan. .

[0043] Ultimately, through splicing and Obtain the output of GDLM:

[0044] in, Let represent the global visual features obtained by GDLM, and Concat represent the stitching operation. In Equation (11), SS2D is a key component of the Mamba framework. In this process, the visual features are first divided into non-overlapping patches, and then scanned along four different paths to generate four independent sequences. Each sequence is then processed by a selectively scanned state space sequence model (S6) to extract spatial information from different directions while effectively preserving key contextual features. Finally, the four sequences are merged to form a unified 2D visual feature representation.

[0045] (3) The report generator introduces a bi-directional adapter to enhance the interaction and fusion between different modal features. Specifically, a bi-directional adapter module (Bi-Adapter, BAT) is embedded in the encoder-decoder framework to achieve bi-directional interaction between visual and text modalities through a dynamic feature cueing mechanism. This module includes two core processing stages: the enhancement process from visual to text features and the compensation process from text to visual features. The process is as follows:

[0046]

[0047]

[0048] in, and These represent visual-to-text feature cues and text-to-visual feature cues, respectively, and the updated visual features. It will contain both original image features and textual semantic information.

[0049] Then, CheXpert is used for each visual feature. Generate pseudo tags The system automatically labels the presence of 14 common diseases in medical images. Pseudo-labels act as a "bridge" connecting the visual feature space and the text semantic space, forcing the model to learn the mapping relationship between visual features and standard medical terminology. The calculation process is as follows:

[0050] in, It is the number of disease categories. These are weight parameters. It is a tag generation function.

[0051] Then, the visual feature sequence fused with pseudo-labels is projected through a projection layer. and the hidden state of the decoder Projected onto a shared feature space:

[0052] Next, a Transformer-based model is used to generate the radiology report. Specifically, at the time step... Visual features Mapping to intermediate feature representation:

[0053] Then, the text word embeddings and visual intermediate feature representations obtained in S2 are fed into the decoder to generate the output for the current time step:

[0054] in, It is a time step Predicted words, This represents the number of intermediate features. The above process will be repeated iteratively until the report is fully generated, denoted as... .

[0055] S3: Generate a radiology report using the radiology report model constructed in S2.

[0056] The above embodiments design a global dependency learning module that effectively models long-range dependencies in medical image sequences using a State-Space Model (SSM) backbone network and captures key lesion features. Simultaneously, this module integrates Rotation Position Encoding (RoPE) to enhance the base Mamba model, improving its efficiency and flexibility in capturing relative positional information in visual feature sequences. Furthermore, this invention also designs a multimodal alignment module, which utilizes a novel cross-modal bidirectional adapter to promote multi-level fusion of visual and textual information, solving the modal heterogeneity problem and achieving semantic-level feature alignment and refinement.

[0057] Corresponding to the radiology report generation method in the above embodiments, the present invention also provides a radiology report generation system based on global dependency learning and multimodal alignment networks, comprising: Data preprocessing module: preprocesses medical images and text files in the electronic medical record dataset, and embeds the preprocessed data into vectorized form; Model building module: Constructs a radiology report model; the radiology report model includes: a visual extractor, a global dependency learning module, and a report generator; the visual extractor inputs the vectorized medical image representation into a pre-trained convolutional neural network to extract abstract visual feature representations; and performs word vectorization on the text report using word2vec; the global dependency learning module uses the obtained visual features, and dynamically captures relative position information while modeling complex spatial relationships by integrating Mamba and rotational position encoding, to obtain a global visual feature representation; the report generator performs cross-modal alignment between the word vectorized text report feature representation and the global visual feature representation to obtain the radiology report; Report generation module: Generates radiology reports using the radiology report model.

[0058] The radiology report generation method of this embodiment is relatively simple to describe because it corresponds to the radiology report generation method in the above embodiment. For related similarities, please refer to the description of the radiology report generation method in the above embodiment, which will not be described in detail here.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A radiology report generation method based on global dependency learning, characterized by, The method comprises the following steps: S1: preprocessing medical images and text files in an electronic medical record dataset, and vectorizing and embedding the preprocessed data; S2: constructing a radiology report model; the radiology report model comprises a visual extractor, a global dependency learning module, and a report generator; the visual extractor inputs the vectorized medical image representation into a pre-trained convolutional neural network to extract abstract visual feature representation; and the text report is word vectorized by word2vec; the global dependency learning module obtains global visual feature representation by integrating Mamba and rotation position coding to model complex spatial relationships while dynamically capturing relative position information; the report generator performs cross-modal alignment on the word vectorized text report feature representation and the global visual feature representation to obtain a radiology report; S3: generating a radiology report using the radiology report model.

2. The method of claim 1, wherein, The pre-trained convolutional neural network is ResNet-101.

3. The method of claim 2, wherein, The visual extractor comprises: By a visual encoder Transforming an input medical image into a high-dimensional feature representation ; Visual features Two different visual representations are generated: overall features obtained by global average pooling , and token features after sequential processing ; The text in the report is obtained by a word embedding layer using word2vec to obtain the word embedding of each word.

4. The method of claim 3, wherein, The global dependency learning module comprises: Using dynamic position bias adjustment to serialize tokens spatial information; adjusted features input convolution operation; Integrating rotation position coding to enhance Mamba: ； ； ； ； wherein, and denote the intermediate features projected to the latent space, , are the input sequence and the and the visual embedding vectors, and denote two orthogonal matrices, , are two weight matrices; denotes the sequence of features obtained by RoPE, and denotes the output of the improved Mamba; denotes the SiLU activation function, DWConv denotes the depthwise separable convolution, Linear denotes the linear layer, Norm denotes the normalization layer, SS2D denotes the two-dimensional selective scanning, ; stitching and obtaining global visual features : ； wherein, represents the obtained global visual features, Concat( ) represents a concatenation operation.

5. The method of claim 4, wherein, In the two-dimensional selective scanning process, the visual features are first divided into non-overlapping blocks, then scanned along four different paths to generate four independent sequences; each sequence is then processed by a selective scanning state space sequence model to extract spatial information from different directions while effectively preserving key context features; finally, the four sequences are combined to form a unified 2D visual feature representation.

6. The method of claim 1, wherein, The report generator comprises: A bidirectional adapter module is embedded in the encoder-decoder framework to realize bidirectional interaction between visual and text modalities through a dynamic feature prompting mechanism; the bidirectional adapter module comprises an enhancement process of visual-to-text features and a compensation process of text-to-visual features: ； ； ； wherein, and respectively represent visual-to-text feature hints and text-to-visual feature hints, updated visual features will contain both original image features and text semantic information; using the radiology disease label extractor CheXpert for each visual feature generating pseudo-labels automatically annotating medical images for the presence of a plurality of common diseases: ； wherein, is the number of disease categories, is a weight parameter, is a label generating function. by the projection layer and the hidden state of the decoder projected to a shared feature space: ; A Transformer-based model is used to generate a radiology report.

7. The method of claim 6, wherein, A Transformer-based model is used to generate a radiology report, comprising: At time step visual features are mapped to intermediate feature representations: ； The word embeddings and visual intermediate feature representations are fed into the decoder to generate the output for the current time step. ;in, It is a time step Predicted words, It is the number of intermediate features; The above process is iteratively repeated until the report is fully generated, denoted as .

8. A radiology report generation system based on global dependency learning and multi-modal alignment network, characterized in that, The method comprises the following steps: A data preprocessing module is used to preprocess medical images and text files in an electronic medical record dataset, and vectorize and embed the preprocessed data; A model construction module is used to construct a radiology report model; The radiology report model comprises a visual extractor, a global dependency learning module, and a report generator; the visual extractor inputs the vectorized medical image representation into a pre-trained convolutional neural network to extract abstract visual feature representation; and the text report is word vectorized by word2vec; the global dependency learning module obtains global visual feature representation by integrating Mamba and rotation position coding to model complex spatial relationships while dynamically capturing relative position information; the report generator performs cross-modal alignment on the word vectorized text report feature representation and the global visual feature representation to obtain a radiology report; report generation module: generating a radiology report using the radiology report model.

Images