Radiology report generation method fusing clinical semantic modulation and hyperbolic prototype classification

Abstract

The invention provides a radiology report generation method fusing clinical semantic modulation and hyperbolic prototype classification, and the method comprises the steps: obtaining a chest X-ray image and clinical context knowledge associated with the chest X-ray image, extracting the visual features of the image through a visual encoder, extracting clinical semantic embedding through a medical language encoder, and generating a radiology report. Related report features are retrieved from the reference report database; constructing and utilizing a clinical semantic modulation module to generate modulation visual representation; constructing and utilizing a hyperbolic prototype classification module to generate a diagnosis perception prompt; and inputting the modulated visual features and diagnosis perception prompts into a decoder to complete autoregression generation of the radiology report. According to the method, cross-modal fusion of clinical knowledge and visual features can be fully utilized, the exponential expansion embedding capacity of the hyperbolic space is utilized to improve the disease detection capability under the class imbalance condition, and the clinical accuracy of radiology report generation is improved.

Description

A Radiology Report Generation Method Integrating Clinical Semantic Modulation and Hyperbolic Prototype Classification Technical Field

[0001] This invention relates to the fields of medical image processing and natural language generation, and in particular to a method for generating radiology reports that integrates clinical semantic modulation and hyperbolic prototype classification. Background Technology

[0002] Radiology reports are crucial for clinical diagnosis and treatment planning; however, manually drafting these reports is time-consuming and labor-intensive, heavily reliant on the radiologist's professional experience, and can easily lead to diagnostic delays and inconsistencies in medical decisions. Automated Radiology Report Generation (RGG) technology has emerged as an effective solution to reduce the workload of radiologists and improve diagnostic efficiency.

[0003] Existing RRG methods typically employ an encoder-decoder architecture to extract visual features from medical images and generate text reports using language models. Recent research has further incorporated techniques such as retrieval enhancement, knowledge graphs, and large language models to improve generation quality. However, existing work suffers from several drawbacks: Firstly, the utilization of clinical contextual knowledge (such as examination indications and patient history) is fundamentally flawed. Current mainstream paradigms often treat clinical contextual knowledge merely as input-level textual prompts or simply pieced-together contextual information, over-relying on the implicit alignment capabilities of large language model decoders. This not only incurs significant computational overhead but, more importantly, fails to fundamentally alter the behavior of the visual encoder, resulting in visual feature extraction being independent of prior clinical knowledge. Consequently, the generated visual representations lack diagnostic relevance to specific cases and struggle to capture fine-grained visual evidence highly relevant to current clinical indications. Another type of method completely ignores this crucial prior knowledge.

[0004] On the other hand, the inherent characteristics of disease distribution present significant challenges. Unlike general image description tasks that treat all visual concepts equally, radiological report generation must prioritize anomalies, which are often subtle and exhibit severe class imbalances compared to normal observations. Medical datasets exhibit a severe long-tail distribution, with a few common diseases dominating while a large number of rare diseases are severely underrepresented. In RRGs, disease classifiers are often used to guide report generation, but traditional classifiers operating in Euclidean space tend to favor high-frequency classes, and their limited embedding capacity makes it difficult to provide sufficient separation for rare classes, resulting in predictions that overlook subtle rare lesions. Existing methods typically address this imbalance through loss reweighting or retrieval augmentation, but these operate at the output or decision level and fail to fundamentally improve the discriminability of rare disease representations. Summary of the Invention

[0005] In view of the above, the main objective of this invention is to propose a radiology report generation method that integrates clinical semantic modulation and hyperbolic prototype classification to solve the aforementioned technical problems.

[0006] This invention proposes a radiology report generation method integrating clinical semantic modulation and hyperbolic prototype classification. The method includes the following steps: Step 1: Acquire chest X-ray images and clinical context knowledge; preprocess the chest X-ray images to obtain preprocessed images; Step 2: Extract features from the preprocessed images using a visual encoder to obtain block-level visual features and global image representation; extract features from clinical context knowledge using a medical language encoder to obtain clinical semantic embedding; retrieve reference report features related to the global image representation from a pre-set reference report database to obtain a reference report feature set; Step 3: ... 3. Construct a clinical semantic modulation module based on a dual-pathway architecture and gating fusion mechanism using spatial semantic injection and channel feature recalibration pathways; input block-level visual features and clinical semantic embeddings into the clinical semantic modulation module for processing to obtain modulated visual representations; Step 4. Construct a hyperbolic prototype classification module based on the Poincaré sphere and Softmax function; input global image representations and reference report feature sets into the hyperbolic prototype classification module for processing to generate diagnostic perception prompts; Step 5. Input the modulated visual representations and diagnostic perception prompts into a BERT-based decoder for processing to achieve autoregressive generation of radiology reports.

[0007] Compared with existing technologies, the beneficial effects of this invention are as follows: 1. This invention designs a Clinical Semantic Modulation (CSM) module, which explicitly modulates visual features at the feature level through a dual-pathway architecture of spatial semantic injection and channel feature recalibration, rather than relying solely on the implicit alignment of the decoder. The spatial pathway injects clinical semantics into each visual block, achieving fine-grained alignment between clinical intent and visual patterns; the channel pathway recalibrates feature channels by learning to amplify or suppress specific activation patterns. A learnable gating mechanism adaptively fuses the two pathways to generate visual representations rich in clinical information, enhancing the case-specific diagnostic relevance of visual features; 2. This invention designs a Hyperbolic Prototype Classification (HPC) module, which transfers disease classification from Euclidean space to hyperbolic space (Poincaré sphere model), utilizing the exponentially growing volume characteristics of hyperbolic space to provide greater separation capacity for rare disease categories. 1. This invention replaces the traditional linear classifier with a prototype-based classification strategy. By comparing the distance between the input embedding and the prototype embedding through metric learning, it exhibits stronger robustness to class imbalance and effectively improves the detection capability for rare diseases with insufficient representation. 2. This invention comprehensively utilizes a visual encoder, a medical language encoder, and a report retrieval mechanism to extract visual features of images, semantic embeddings of clinical context knowledge, and textual features of relevant historical reports, fully exploring the complementary semantic information between multimodal information to provide rich and comprehensive feature representations for subsequent clinical semantic modulation and disease classification. 3. This invention employs a diagnostic perception prompt mechanism to discretize the disease prediction results of the hyperbolic prototype classification module into classification state labels, constructing an ordered sequence of diagnostic perception prompts as explicit input guidance for the report decoder. This enables the report generation process to perceive the diagnostic status of each disease, thereby generating radiological reports that are consistent with the diagnostic results and have stronger clinical relevance. 4. Through the collaborative work of the clinical semantic modulation module and the hyperbolic prototype classification module, this invention significantly enhances clinical efficacy indicators while improving the quality of language generation, achieving state-of-the-art performance on both the MIMIC-CXR and IU X-Ray datasets. Long-tail analysis shows that the present invention improves the average F1 score by 6.2% for rare diseases, while maintaining competitive performance for common categories, effectively alleviating the class imbalance problem.

[0008] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0009] Figure 1 is a flowchart of the steps of a radiology report generation method that integrates clinical semantic modulation and hyperbolic prototype classification proposed in this invention; Figure 2 is an overall structural diagram of a radiology report generation method that integrates clinical semantic modulation and hyperbolic prototype classification proposed in this invention; Figure 3 is a schematic diagram of the structure of the clinical semantic modulation module in this invention; Figure 4 is a schematic diagram of the structure of the hyperbolic prototype classification module in this invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0012] Please refer to Figure 1. This embodiment provides a radiology report generation method that integrates clinical semantic modulation and hyperbolic prototype classification. The method includes the following steps: Step 1: Obtain chest X-ray images and clinical context knowledge, preprocess the chest X-ray images, and obtain preprocessed images.

[0013] Please refer to Figure 2. In step 1, the chest X-ray image and clinical context knowledge are acquired, and the chest X-ray image is preprocessed to obtain a preprocessed image. The chest X-ray image is a two-dimensional chest X-ray image. The clinical context knowledge includes examination indications, patient history, comparative examination information, and examination technology information. The preprocessing includes adjusting the chest X-ray image to a fixed resolution of 224×224.

[0014] Step 2: Use a visual encoder to extract features from the preprocessed image to obtain block-level visual features and global image representation; use a medical language encoder to extract features from clinical context knowledge to obtain clinical semantic embedding; retrieve reference report features related to the global image representation from a pre-set reference report database to obtain a reference report feature set.

[0015] In step 2, a visual encoder is used to extract features from the preprocessed image to obtain block-level visual features and a global image representation; a medical language encoder is used to extract features from clinical context knowledge to obtain clinical semantic embedding; reference report features related to the global image representation are retrieved from a pre-set reference report database to obtain a reference report feature set. Specifically, this includes the following sub-steps: using a visual encoder (using ResNet-101) to extract features from the preprocessed image to obtain block-level visual features, with the following relationship existing in the process: ;in, Represents block-level visual features. Represents a visual encoder. This represents the preprocessed image; the block-level visual features are subjected to global average pooling along the spatial dimension to obtain a global image representation, and the following relationship exists in the corresponding process: ;in, Represents the global image representation. This indicates that global average pooling has been applied; a medical language encoder (using CXR-BERT) is used to extract features from clinical context knowledge to obtain clinical semantic embeddings. The following relationship exists in the corresponding process: ;in, This indicates clinical semantic embedding. This represents a medical language encoder. It represents clinical context knowledge; based on a pre-defined reference report database, using global image representation as the query vector, it calculates the cosine similarity between the global image representation and the visual features corresponding to all reports in the database (usually global image representations stored during training), and selects the reports with the highest similarity. This report; using the MedKLIP encoder to filter the top... Each report is encoded into text features to obtain a reference report feature set. ;in, Indicates the first Features of each reference report An index representing the characteristics of the reference report. This indicates the total number of features in the reference report.

[0016] It should be noted that, to ensure compatibility with subsequent modules, clinical semantics are typically embedded. Projected onto a 512-dimensional feature space.

[0017] Step 3: Construct a clinical semantic modulation module based on a dual-pathway architecture and gating fusion mechanism of spatial semantic injection pathway and channel feature recalibration pathway; input block-level visual features and clinical semantic embeddings into the clinical semantic modulation module for processing to obtain modulated visual representations.

[0018] Please refer to Figure 3. In step 3, a clinical semantic modulation module is constructed based on a dual-pathway architecture and a gating fusion mechanism using the spatial semantic injection pathway and the channel feature recalibration pathway. The block-level visual features and clinical semantic embeddings are input into the clinical semantic modulation module for processing to obtain modulated visual representations. Specifically, this includes the following sub-steps: In the spatial semantic injection pathway, clinical semantic embeddings are injected into the block-level visual features through an attention mechanism to obtain the features output by the attention mechanism. The corresponding relationship in this process is as follows: ;in, Represents the query matrix. Represents the key matrix. Represents a value matrix, , and Both represent learnable projection matrices. This indicates that the input features are expanded to match the dimensions of the block-level visual features. Features representing the output of the attention mechanism This indicates that the data has been processed using the Softmax function. Indicates matrix transpose. The dimension of the key vector is represented; the features output by the attention mechanism are sequentially processed through regularization, residual connections, and layer normalization to obtain the output of the spatial semantic injection path. The following relationship exists in the corresponding process: ;in, This represents the output of the spatial semantic injection pathway. This indicates that the process has undergone layer normalization. This indicates that regularization has been applied to prevent overfitting. In the channel feature recalibration pathway, a multilayer perceptron is used to process the clinical semantic embedding to generate scaling and offset parameters. The following relationship exists in this process: ;in, This represents the scaling parameter. Indicates the offset parameter. This indicates processing by a multilayer perceptron; it should be noted that in Figure 3, the multilayer perceptron... As a calibration generator.

[0019] Layer normalization is performed on the block-level visual features to obtain normalized block-level visual features. Scaling and offset parameters are used as affine transformation coefficients to perform an affine transformation on the normalized block-level visual features to obtain the output of the channel feature recalibration path. The following relationship exists in this process: ;in, This indicates the output of the channel characteristic recalibration path. This represents element-wise multiplication; the output of the spatial semantic injection pathway is concatenated with the output of the channel feature recalibration pathway to obtain the concatenated features. The following relationship exists in the corresponding process: ;in, Indicates the features after splicing. This indicates that the features have been concatenated. The concatenated features are then processed sequentially through a learnable projection matrix and a sigmoid activation function to obtain a gated tensor. The following relationship exists in this process: ;in, Represents the gate tensor. This indicates that the process has been performed using the Sigmoid activation function. Let represent the learnable projection matrix. After mapping the concatenated features using the learnable projection matrix, we then perform element-wise multiplication with the gate tensor. Finally, we perform residual connections and layer normalization sequentially to obtain the modulated visual representation. The following relationship exists in this process: ;in, This represents the modulation of visual representation. This represents the learnable projection matrix.

[0020] Step 4: Construct a hyperbolic prototype classification module based on the Poincaré sphere and the Softmax function. Input the global image representation and reference report feature set into the hyperbolic prototype classification module for processing to generate diagnostic perception prompts.

[0021] Please refer to Figure 4. In step 4, a hyperbolic prototype classification module is constructed based on the Poincaré sphere and the Softmax function. The global image representation and the reference report feature set are input into the hyperbolic prototype classification module for processing to generate diagnostic perception prompts. Specifically, this includes the following sub-steps: performing mean pooling on the reference report feature set to obtain the aggregated report features. The following relationship exists in the corresponding process: ;in, The aggregated report features are represented; the aggregated report features are concatenated with the global image representation to obtain the initial fused features, and the following relationship exists in the corresponding process: ;in, The initial fusion features are represented by ; it should be noted that in Figure 4, C represents feature splicing.

[0022] The initial fused features are processed sequentially through a first-layer MLP, a GELU activation function, and a second-layer MLP to obtain case embedding features in Euclidean space. The following relationship exists in the corresponding process: ;in, This represents the case embedding features in Euclidean space. and Both represent weight matrices. and Both represent bias vectors. This indicates that the data has been processed using the GELU activation function. The case embedding features in Euclidean space are linearly projected onto the target hyperbolic cone, and then subjected to L2 normalization and scaling to obtain the normalized case embedding vector in Euclidean space. The following relationship exists in this process: ;in, Let represent the normalized case embedding vector in Euclidean space, and ; Indicates the target hyperbolic taper. Indicates a fixed scaling factor. Represents the projection matrix. This indicates taking the L2 norm; the normalized case embedding vector in Euclidean space is mapped to a curve with an exponential mapping at the origin. Poincaré The hyperbolic embedding of the current case is obtained above, and the following relationship exists in the corresponding process: ;in, Represents the hyperbolic embedding of the current case. Represents the hyperbolic tangent function. Represents the curvature parameter. Let the Poincaré sphere be represented; construct a learnable hyperbolic prototype: initialize a set of learnable prototype embeddings on the Poincaré sphere for each disease-state pair; denoted as For the number of diseases, The number of states, including four types: blank, positive, negative, and uncertain; maintain the prototype tensor. ;in, Indicates disease In state The prototype embedding is implemented as a manifold parameter and optimized by Riemann gradient descent, which converts the Euclidean gradient into the Riemann gradient and projects the updated embedding back onto the manifold, ensuring that all prototypes remain within the effective region of the Poincaré sphere throughout the training process.

[0023] Given case embeddings and prototype embedding Using the Poincaré distance Classify; embedded in the Poincaré sphere With Embedded The distance between them is defined as: ;in, Indicates the embedded in the Poincaré ball With Embedded The distance between them This represents the Möbius strip. Represents the inverse hyperbolic tangent function; where the Möbius method is defined as: ;in, Represents the standard Euclidean inner product. Represents the standard Euclidean norm; for each disease and each state Calculate case embeddings and prototype embedding The square of the hyperbolic distance between them is transformed by scaling the negative distance with temperature parameters to obtain the disease. In state The classification scores correspond to the following relationship: ;in, Indicates disease In state Classification score, Represents the temperature parameter, and ; Indicates case embedding and prototype embedding Hyperbolic distance between; for disease In state The probability is calculated using the Softmax function on the classification scores of all states, and the following relationship exists in the corresponding process: ;in, Indicates disease In state The probability, Represents an exponential function. Indicates disease In state The classification score; in reasoning, for each disease The state with the highest probability is selected as the prediction result, and the following relationship exists in the corresponding process: ;in, Indicates the prediction result. This indicates taking the maximum value; the prediction result will be... Mapped to specific disease markers The mapping rule is: All disease markers are assembled in a fixed order to obtain diagnostic perception prompts. The following relationship exists in the correspondence process: ;in, This indicates a diagnostic perception prompt. , and All of these indicate special disease markers. It should be noted that in Figure 4, the disease markers... , , , Marking specific diseases The result.

[0024] Step 5: Input the modulated visual representation and diagnostic perception cues into the BERT-based decoder for processing to achieve autoregressive generation of radiology reports.

[0025] In step 5, the modulated visual representation and diagnostic perception cues are input into a BERT-based decoder for processing to achieve autoregressive generation of radiology reports. This includes the following sub-steps: The modulated visual representation is used as the encoder's hidden state and attended to through a cross-attention layer, enabling the decoder to dynamically focus on the visual region most relevant to the currently generated content at each step of report generation; the labeled diagnostic perception cues are pre-placed in the input sequence to guide the autoregressive generation of the report, allowing the decoder to perceive the diagnostic status of each disease during report generation; the report generation is modeled as a conditional distribution, following a standard autoregressive formula. ;in, Indicates in a given image and clinical information Generate target report marker sequence under the condition of The joint conditional probability distribution, This represents a series of multiplication operations. Indicates the length of the target report tag sequence. This represents the target report tag sequence, and ; Indicates time step Tags generated from a fixed vocabulary This represents the previously generated labels; furthermore, during training, the sum of state-level cross-entropy losses for all diseases is minimized, corresponding to the following relationship: ;in, Indicates disease classification loss, Indicates disease Unique hot-coded tags, This indicates taking the logarithm; the report generator is optimized by minimizing the negative log-likelihood loss (i.e., autoregressive language modeling loss), and the following relationship exists in the corresponding process: ;in, This represents the report generation loss; the overall training loss of the model is comprised of the report generation loss. and disease classification loss Together, they optimize the two losses during training.

[0026] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0027] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0028] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0029] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for generating radiology reports that integrates clinical semantic modulation and hyperbolic prototype classification, characterized in that, The method includes the following steps: Step 1, acquiring chest X-ray images and clinical context knowledge, preprocessing the chest X-ray images to obtain preprocessed images; Step 2, using a visual encoder to extract features from the preprocessed images to obtain block-level visual features and global image representation; using a medical language encoder to extract features from the clinical context knowledge to obtain clinical semantic embedding. Step 3: Retrieve reference report features related to the global image representation from the preset reference report database to obtain the reference report feature set; Step 4: Construct a clinical semantic modulation module based on a dual-pathway architecture and gating fusion mechanism of spatial semantic injection pathway and channel feature recalibration pathway; Input block-level visual features and clinical semantic embedding into the clinical semantic modulation module for processing to obtain modulated visual representation; Step 5: Construct a hyperbolic prototype classification module based on Poincaré sphere and Softmax function; Input the global image representation and reference report feature set into the hyperbolic prototype classification module for processing to generate diagnostic perception prompts; Step 6: Input the modulated visual representation and diagnostic perception prompts into a BERT-based decoder for processing to achieve autoregressive generation of radiology reports.

2. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 1, characterized in that, In step 1, a chest X-ray image and clinical context knowledge are acquired, and the chest X-ray image is preprocessed to obtain a preprocessed image. The chest X-ray image is a two-dimensional chest X-ray image. The clinical context knowledge includes examination indications, patient history, comparative examination information, and examination technique information. The preprocessing includes adjusting the chest X-ray image to a fixed resolution of 224×224.

3. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 2, characterized in that, In step 2, a visual encoder is used to extract features from the preprocessed image to obtain block-level visual features and global image representation; a medical language encoder is used to extract features from clinical context knowledge to obtain clinical semantic embedding; and reference report features related to global image representation are retrieved from a preset reference report database to obtain a reference report feature set. Specifically, this includes the following sub-steps: using a visual encoder to extract features from the preprocessed image to obtain block-level visual features; The block-level visual features are subjected to global average pooling along the spatial dimension to obtain a global image representation. A medical language encoder is used to extract features from clinical context knowledge to obtain a clinical semantic embedding. Based on a pre-defined reference report database, using the global image representation as the query vector, the cosine similarity between the global image representation and the visual features corresponding to all reports in the database is calculated, and the reports with the highest similarity are selected. This report; Using the MedKLIP encoder to filter the top Each report is encoded into text features to obtain a reference report feature set. ;in, Indicates the first Features of each reference report An index representing the characteristics of the reference report. This indicates the total number of features in the reference report.

4. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 3, characterized in that, In the step of extracting features from the preprocessed image using a visual encoder to obtain block-level visual features, the following relationship exists: ;in, Represents block-level visual features. Represents a visual encoder. The image represents the preprocessed image. In the step of performing global average pooling on the block-level visual features along the spatial dimension to obtain a global image representation, the following relationship exists: ;in, Represents the global image representation. This indicates that global average pooling has been applied. In the step of extracting features from clinical context knowledge using a medical language encoder to obtain clinical semantic embeddings, the following relationship exists: ;in, This indicates clinical semantic embedding. This represents a medical language encoder. This indicates knowledge of clinical context.

5. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 4, characterized in that, In step 3, a clinical semantic modulation module is constructed based on a dual-pathway architecture and a gating fusion mechanism of spatial semantic injection pathway and channel feature recalibration pathway. The block-level visual features and clinical semantic embeddings are input into the clinical semantic modulation module for processing to obtain modulated visual representations. Specifically, the following sub-steps are included: In the spatial semantic injection pathway, clinical semantic embeddings are injected into the block-level visual features through an attention mechanism to obtain the features output by the attention mechanism. The features output by the attention mechanism are sequentially processed by regularization, residual connection and layer normalization to obtain the output of the spatial semantic injection path; In the channel feature recalibration pathway, a multilayer perceptron is used to process the clinical semantic embedding to generate scaling and offset parameters; The block-level visual features are subjected to layer normalization to obtain the normalized block-level visual features. The scaling and offset parameters are used as affine transformation coefficients to perform affine transformation on the normalized block-level visual features in order to obtain the output of the channel feature recalibration path. The output of the spatial semantic injection pathway is concatenated with the output of the channel feature recalibration pathway to obtain the concatenated features. The concatenated features are processed sequentially through a learnable projection matrix and a sigmoid activation function to obtain a gated tensor. The concatenated features are then mapped using the learnable projection matrix and multiplied element-wise with the gated tensor. Finally, residual connections and layer normalization are performed sequentially to obtain a modulated visual representation.

6. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping according to claim 5, characterized in that, In the step of injecting clinical semantics into block-level visual features through an attention mechanism in the spatial semantic injection pathway to obtain the features output by the attention mechanism, the following relationship exists: ;in, Represents the query matrix. Represents the key matrix. Represents a value matrix, 、 and Both represent learnable projection matrices. This indicates that the input features are expanded to match the dimensions of the block-level visual features. Features representing the output of the attention mechanism This indicates that the data has been processed using the Softmax function. Indicates matrix transpose. The dimension of the key vector is represented. In the steps of processing the features output by the attention mechanism through regularization, residual connections, and layer normalization to obtain the output of the spatial semantic injection pathway, the following relationship exists: ;in, This represents the output of the spatial semantic injection pathway. This indicates that the process has undergone layer normalization. This indicates that regularization has been applied.

7. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 6, characterized in that, In the step of recalibrating the channel features and processing the clinical semantic embedding using a multilayer perceptron to generate scaling and offset parameters, the following relationship exists: ;in, This represents the scaling parameter. Indicates the offset parameter. The process involves multilayer perceptron processing; layer normalization of the block-level visual features to obtain normalized block-level visual features; and using scaling and offset parameters as affine transformation coefficients to perform an affine transformation on the normalized block-level visual features to obtain the output of the channel feature recalibration path. The following relationship exists in this process: ;in, This indicates the output of the channel characteristic recalibration path. This indicates element-wise multiplication.

8. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping according to claim 7, characterized in that, In the step of concatenating the output of the spatial semantic injection pathway with the output of the channel feature recalibration pathway to obtain the concatenated features, the following relationship exists: ;in, Indicates the features after splicing. This indicates that the features have been concatenated. In the step of processing the concatenated features sequentially through a learnable projection matrix and a Sigmoid activation function to obtain the gated tensor, the following relationship exists: ;in, Represents the gate tensor. This indicates that the process has been performed using the Sigmoid activation function. Let represent the learnable projection matrix. After mapping the stitched features using the learnable projection matrix, it is then multiplied element-wise with the gate tensor. Finally, residual connections and layer normalization are performed sequentially to obtain the modulated visual representation. In this process, the following relationship exists: ;in, This represents the modulation of visual representation. This represents the learnable projection matrix.

9. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototyping as described in claim 8, characterized in that, In step 4, a hyperbolic prototype classification module is constructed based on the Poincaré sphere and the Softmax function. The global image representation and the reference report feature set are input into the hyperbolic prototype classification module for processing to generate diagnostic perception prompts. Specifically, this includes the following sub-steps: performing mean pooling on the reference report feature set to obtain aggregated report features. The following relationship exists in the corresponding process: ;in, The aggregated report features are represented; the aggregated report features are concatenated with the global image representation to obtain the initial fused features, and the following relationship exists in the corresponding process: ;in, The initial fusion features are represented by the first layer MLP, the GELU activation function, and the second layer MLP, respectively, to obtain the case embedding features in Euclidean space. The following relationship exists in the corresponding process: ;in, This represents the case embedding features in Euclidean space. and Both represent weight matrices. and Both represent bias vectors. This indicates that the data has been processed using the GELU activation function. The case embedding features in Euclidean space are linearly projected onto the target hyperbolic cone, and then subjected to L2 normalization and scaling to obtain the normalized case embedding vector in Euclidean space. The following relationship exists in this process: ;in, This represents a normalized case embedding vector in Euclidean space. Indicates a fixed scaling factor. Represents the projection matrix. This indicates taking the L2 norm; the normalized case embedding vector in Euclidean space is mapped to a curve with an exponential mapping at the origin. Poincaré The hyperbolic embedding of the current case is obtained above, and the following relationship exists in the corresponding process: ;in, Represents the hyperbolic embedding of the current case. Represents the hyperbolic tangent function. Represents the curvature parameter. Let the Poincaré sphere be represented; construct a learnable hyperbolic prototype: initialize a set of learnable prototype embeddings on the Poincaré sphere for each disease-state pair; denoted as For the number of diseases, The number of states, including four types: blank, positive, negative, and uncertain; maintain the prototype tensor. ;in, Indicates disease In state prototype embedding, The target hyperbolic taper is represented; the prototype is implemented as a manifold parameter and optimized using Riemann gradient descent, which converts the Euclidean gradient to the Riemann gradient and projects the updated embedding back onto the manifold, ensuring that all prototypes remain within the effective region of the Poincaré sphere throughout the training process; given case embeddings and prototype embedding Using the Poincaré distance Classify; embedded in the Poincaré sphere With Embedded The distance between them is defined as: ;in, Indicates the embedded in the Poincaré ball With Embedded The distance between them This represents the Möbius strip. Represents the inverse hyperbolic tangent function; where the Möbius method is defined as: ;in, Represents the standard Euclidean inner product. Represents the standard Euclidean norm; for each disease and each state Calculate case embeddings and prototype embedding The square of the hyperbolic distance between them is transformed by scaling the negative distance with temperature parameters to obtain the disease. In state The classification scores correspond to the following relationship: ;in, Indicates disease In state Classification score, Represents the temperature parameter, and ； Indicates case embedding and prototype embedding Hyperbolic distance between; for disease In state The probability is calculated using the Softmax function on the classification scores of all states, and the following relationship exists in the corresponding process: ;in, Indicates disease In state The probability, Represents an exponential function. Indicates disease In state The classification score; in reasoning, for each disease The state with the highest probability is selected as the prediction result, and the following relationship exists in the corresponding process: ;in, Indicates the prediction result. This indicates taking the maximum value; the prediction result will be... Mapped to specific disease markers The mapping rule is: All disease markers are assembled in a fixed order to obtain diagnostic perception prompts. The following relationship exists in the correspondence process: ;in, This indicates a diagnostic perception prompt. 、 and All of these indicate specific disease markers.

10. The radiology report generation method integrating clinical semantic modulation and hyperbolic prototype classification according to claim 9, characterized in that, In step 5, the modulated visual representation and diagnostic perception cues are input into a BERT-based decoder for processing to achieve autoregressive generation of radiology reports. This includes the following sub-steps: The modulated visual representation is used as the encoder's hidden state and attended to through a cross-attention layer, enabling the decoder to dynamically focus on the visual region most relevant to the currently generated content at each step of report generation; the labeled diagnostic perception cues are pre-placed in the input sequence to guide the autoregressive generation of the report, allowing the decoder to perceive the diagnostic status of each disease during report generation; the report generation is modeled as a conditional distribution, following a standard autoregressive formula. ;in, Indicates in a given image and clinical information Generate target report marker sequence under the condition of The joint conditional probability distribution, This represents a series of multiplication operations. Indicates the length of the target report tag sequence. Indicates the target report tag sequence, Indicates time step Tags generated from a fixed vocabulary This indicates a previously generated tag.

Images