Multimodal X-ray image diagnosis report generation method based on reinforcement learning optimization

Abstract

The invention discloses a multi-modal X-ray image diagnosis report generation method based on reinforcement learning optimization. The method comprises the following steps: acquiring a multi-source public data set and a private data set in a specific range; performing format standardization processing on the X-ray image, performing normalization processing on text information, and constructing a structured public training data pair and a private enhanced data pair; training an end-to-end multi-modal neural network by using the public training data pair in a supervised fine tuning mode; carrying out reinforcement learning on the first-stage end-to-end multi-mode neural network by adopting a strategy gradient algorithm optimization mode of GRPO and using candidate reports and weighted rewards to obtain a second-stage end-to-end multi-mode neural network; and generating a diagnosis report by using the two-stage end-to-end multi-modal neural network. The method is suitable for personalized application scenes of X-ray image diagnosis, can quickly adapt to specific clinical requirements on the basis of limited professional data, and has good practical value and popularization prospect.

Description

Technical Field

[0001] This invention relates to the field of electronic digital processing, and more particularly to a method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal deep learning models have shown great potential in the field of X-ray image analysis. Traditional X-ray image diagnosis mainly relies on doctors' professional experience and manual analysis, which suffers from problems such as strong subjectivity, low efficiency, and uneven resource allocation. Deep learning-based automatic X-ray image diagnosis technology can effectively improve diagnostic efficiency and accuracy, and reduce the workload of doctors. Multimodal visual language models have made significant progress in natural image understanding tasks, such as visual question answering and multimodal dialogue applications, and their potential in the field of X-ray image reasoning has also attracted widespread attention. These advances are mainly driven by large-scale visual language pre-training and supervised fine-tuning techniques. Supervised fine-tuning enables models to effectively associate visual content with textual information.

[0003] When multimodal visual language models are applied to X-ray image analysis, due to the fundamental differences between X-ray images and natural images, X-ray diagnostic tasks not only require high-precision visual understanding of X-ray images, but also require precise alignment of X-ray visual features with medical language. The generated medical reports need to conform to the reasoning path and result interpretation requirements of clinical logic, which places higher demands on the accuracy, consistency and interpretability of the model.

[0004] Currently, multimodal X-ray image analysis technology mainly employs supervised learning methods, training models with large amounts of labeled data to learn the mapping relationship between image features and diagnostic results. However, existing technologies have the following problems: X-ray image diagnosis is diverse and complex, and supervised learning alone is insufficient to cope with various complex clinical scenarios, especially when dealing with special population data such as pediatric X-ray images, and when dealing with specialized data with relatively small amounts of data, the adaptability and robustness of existing methods are limited. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization.

[0006] The objective of this invention is achieved through the following technical solution: A first aspect of the present invention provides a method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization, comprising the following steps: Obtain public datasets from multiple sources and private datasets within a specific scope. Both public and private datasets include X-ray images and corresponding text information. Standardize the format of the X-ray images and normalize the text information to construct structured public training data pairs and private augmentation data pairs. A supervised fine-tuning approach is used to train an end-to-end multimodal neural network using the publicly available training data. The end-to-end multimodal neural network includes a visual encoder, a feature aligner, and a text generator connected in sequence, resulting in a one-stage end-to-end multimodal neural network. The private reinforcement data pair is input into the first-stage end-to-end multimodal neural network to generate N candidate reports; based on the set reinforcement learning reward mechanism, the weighted reward of each of the N candidate reports is calculated. The GRPO policy gradient algorithm is used for optimization. Candidate reports and weighted rewards are used to perform reinforcement learning on the first-stage end-to-end multimodal neural network to obtain the second-stage end-to-end multimodal neural network. The X-ray image to be used to generate the diagnostic report, along with the corresponding text information, is input into a two-stage end-to-end multimodal neural network to generate the diagnostic report.

[0007] Furthermore, the multi-source public dataset includes X-ray images of different age groups and the body parts of interest, as well as the corresponding text information of the X-ray images; The text information includes a complete diagnostic report or text information that includes incomplete report types; the fragmented incomplete report type text information includes disease category labels and lesion locations; The standardization process for text information in incomplete report types includes: extracting the coordinates of the center point of the lesion location annotation box, determining the region identifier based on the relative position of the center point in the X-ray image to obtain the region location information, and combining the disease category label with the region location information to generate a structured diagnostic description containing location description and disease name.

[0008] Furthermore, the visual encoder employs a multi-level feature extraction architecture to output fixed-dimensional global visual features, including: Preliminary spatial features of X-ray images after format normalization are extracted using a multi-layer convolutional network. The preliminary spatial features are input into a multi-layer self-attention module to obtain correlated spatial features; The correlation space features are input into the MLP block and nonlinearly transformed to obtain the transformed space features; After the transformation space features are passed through the attention pooling module, global visual features of fixed dimensions are obtained. The feature aligner maps global visual features to a dimension consistent with the text embedding space through a multi-layer fully connected network to obtain aligned visual features. The text generator is a large language model based on the Transformer decoder architecture, and it generates diagnostic reports token by token using an autoregressive approach. The text token is obtained by segmenting and embedding the text information. The text token corresponds one-to-one with the text information and expresses the semantic content of the diagnostic report. The text generation process includes: taking aligned visual features and text tokens generated from text information as input to a text generator to obtain the output features of the text generator; the large language model employs a causal mask self-attention mechanism, utilizing only the contextual information of the previous t-1 tokens when predicting the t-th token; the output features are processed by the output projection matrix and softmax operation to obtain the conditional probability distribution of the token; and candidate reports are generated progressively based on the conditional probability distribution. Further, the supervised fine-tuning method uses the publicly available training data to train an end-to-end multimodal neural network, including: During the supervised fine-tuning phase, the publicly available training data pairs are used as training samples, and the optimization objective is to minimize the cross-entropy loss of all tokens, thereby minimizing the difference between the generated candidate reports and the real reports. During training, the parameters of the visual encoder, feature aligner, and text generator are updated.

[0009] Furthermore, the reinforcement learning reward mechanism includes: Templated reward function This is used to evaluate whether the report output by the end-to-end multimodal neural network conforms to the predefined format specifications; the reasoning process is placed between the first tags, and the final diagnostic conclusion is placed between the second tags; Reward function for thought process It is used to evaluate the quality of the thinking and reasoning processes generated by end-to-end multimodal neural networks within labels; Semantic similarity reward function The degree of semantic matching between the report output by the end-to-end multimodal neural network and the reference report; The step of calculating the weighted reward for each of the N candidate reports includes: calculating the reward scores for each candidate report across three categories, and finally obtaining the overall reward value of the sample through a weighted average. in For weight parameters, Normalized, it is used as a weighted reward.

[0010] Furthermore, the method of optimizing the policy gradient algorithm using GRPO employs candidate reports and weighted rewards to perform reinforcement learning on the one-stage end-to-end multimodal neural network, resulting in a two-stage end-to-end multimodal neural network, including: Each input private reinforcement data pair is fed into a one-stage end-to-end multimodal neural network, sampled from the old policy of the one-stage end-to-end multimodal neural network. For each candidate report, a weighted reward is calculated, and the average of the weighted rewards of the G candidate reports is used as the intra-group advantage value. A novel strategy for calculating the GRPO objective function is employed, utilizing within-group dominance and the training process. The optimized loss function is then derived from the calculation results of the GRPO objective function. The optimized loss function is used to perform reinforcement learning on a one-stage end-to-end multimodal neural network, thereby obtaining a two-stage end-to-end multimodal neural network.

[0011] Furthermore, in the GRPO objective function, for each input private reinforcement data pair and a set of candidate reports sampled from the old policy, the probability ratio is calculated using the new policy, and combined with the within-group advantage value, a pruning operation is used to limit the extreme value of the probability ratio, thereby maximizing the advantage-weighted probability ratio of each output.

[0012] Furthermore, a KL divergence regularization term is added to the GRPO objective function to prevent the new strategy from deviating too much from the old strategy.

[0013] The beneficial effects of this invention are: In an exemplary embodiment of the present invention, a multi-stage training framework is constructed. First, a pre-trained multimodal model is supervised and fine-tuned to establish the basic association ability between X-ray images and text. Then, a reinforcement learning mechanism is innovatively introduced to optimize the model. The two-stage training strategy effectively solves the model generalization problem under the condition of scarcity of private X-ray data (private datasets of a specific range).

[0014] This method is particularly suitable for personalized application scenarios in X-ray imaging diagnosis. It can quickly adapt to specific clinical needs based on limited professional data, and has good practical value and prospects for promotion. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a multimodal X-ray image diagnostic report generation method based on reinforcement learning optimization, provided as an exemplary embodiment of the present invention; Figure 2 This is a schematic comparison diagram of large-scale public data and specific private data provided for an exemplary embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0018] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0019] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0020] See Figure 1 , Figure 1 A flowchart illustrating an exemplary embodiment of the present invention is provided for a method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization, comprising the following steps: S1: Obtain public datasets from multiple sources and private datasets within a specific scope. Both public and private datasets include X-ray images and corresponding text information. Standardize the format of the X-ray images and normalize the text information to construct structured public training data pairs and private augmentation data pairs, such as... Figure 2 As shown, although the large-scale public data in step S2 and the specific private data in step S4 are both X-ray images and descriptions of the same region, their features are not the same, specifically including: Publicly available data primarily consists of adult data, while private datasets are collected from children, and the image features differ depending on age. 1. Adults have a mature thoracic cavity structure and flat ribs; children have a narrower thoracic cavity and more rounded ribs. 2. Differences in instruments and markings: Data instruments are more prominently obscured in children's films, while in adult data, intubation, ECG electrodes, and catheterization are less frequently seen; 3. Different imaging positions: The data acknowledges the inclusion of more lateral views. Lateral views provide depth information for adult models, but are easier to obtain when children are positioned; therefore, lateral views are almost nonexistent in the children's set. 4. Exposure and contrast differ: Adults: The grayscale range is large, and some ICU / bedside images are overexposed or underexposed due to rapid imaging; Children are often photographed with fixed parameters, resulting in more uniform overall exposure, but the contrast between bone and soft tissue is higher. 5. Different lesion distribution: Adults mainly have systemic or degenerative lesions such as cardiogenic changes, chronic obstructive lesions, and pleural effusion; but children mainly have infectious lesions and congenital anatomical variations, with frequent complications related to medical devices. Furthermore, due to differences in terminology and usage, the writing styles of public and private data differ. Public data contains a wealth of anatomical and imaging terminology, with complex sentence structures; private data rarely uses complex anatomical terms, and diagnostic opinions usually directly state "pneumonia" or "bronchitis," etc.

[0021] S2: Using supervised fine-tuning, the publicly available training data is used to train an end-to-end multimodal neural network, which includes a visual encoder, a feature aligner, and a text generator connected in sequence, to obtain a one-stage end-to-end multimodal neural network. S3: Input the private reinforcement data pair into the first-stage end-to-end multimodal neural network to generate N candidate reports; calculate the weighted reward of each of the N candidate reports based on the set reinforcement learning reward mechanism; S4: Using the GRPO policy gradient algorithm for optimization, candidate reports and weighted rewards are used to perform reinforcement learning on the first-stage end-to-end multimodal neural network to obtain the second-stage end-to-end multimodal neural network. S5: Input the X-ray image to be used to generate the diagnostic report and the corresponding text information into the two-stage end-to-end multimodal neural network to generate the diagnostic report.

[0022] Specifically, in this exemplary embodiment, the method constructs a multi-stage training framework. First, it performs supervised fine-tuning on the pre-trained multimodal model to establish the basic association ability between X-ray images and text. Then, it innovatively introduces a reinforcement learning mechanism to optimize the model: first, in the supervised fine-tuning stage of step S2, it uses large-scale public data (multi-source public datasets) to establish a basic X-ray knowledge representation; and in the reinforcement learning stage of step S4, it performs fine-tuning on specific private data.

[0023] This invention effectively addresses the model generalization problem under conditions of scarcity of private X-ray data (private datasets within a specific range) through a two-stage training strategy. In the supervised fine-tuning stage, the invention uses public datasets and designs a unified data processing pipeline, using private datasets only during the reinforcement learning training stage, thus requiring only a small amount of private dataset.

[0024] Meanwhile, this exemplary embodiment employs the GRPO algorithm in the reinforcement learning phase. The introduction of reinforcement learning effectively alleviates the overfitting problem in training with small sample data. Through a reward-driven exploration mechanism, the model can learn more robust and generalized diagnostic strategies. Based on advanced reinforcement learning methods such as GRPO, a reasonably designed reward function enables the model to learn thought chain deduction, thereby iteratively optimizing the quality of the model's report generation. This significantly improves the professionalism, accuracy, and clinical applicability of the generated diagnostic reports. The introduction of reinforcement learning effectively solves the quality bottleneck problem of traditional supervised learning methods in medical text generation tasks, providing a new technical path for intelligent X-ray image diagnosis.

[0025] It is particularly noteworthy that this method is especially suitable for personalized application scenarios in X-ray imaging diagnosis, and can quickly adapt to specific clinical needs based on limited professional data, demonstrating good practical value and promising prospects for promotion.

[0026] The following content will elaborate on the specific implementation of each step, using "supervised fine-tuning using publicly available large chest X-ray datasets such as MIMIC-CXR and VinDr-PCXR, followed by reinforcement learning optimization using private pediatric lung X-ray images, ultimately achieving automatic generation of pediatric lung X-ray image diagnostic reports" as a specific example: In step S1, multiple public datasets and a specific range of private datasets are acquired. Both the public and private datasets include X-ray images and corresponding text information. The X-ray images are standardized in format, and the text information is normalized to construct structured public training data pairs and private augmentation data pairs. Specifically, this involves the preprocessing of X-ray image data and text information data.

[0027] More preferably, in an exemplary embodiment, the public dataset from multiple sources includes X-ray images of different age groups and the body parts of interest, as well as text information corresponding to the X-ray images. The text information includes a complete diagnostic report or text information that includes incomplete report types; the fragmented incomplete report type text information includes disease category labels and lesion locations; The standardization process for text information in incomplete report types includes: extracting the coordinates of the center point of the lesion location annotation box, determining the region identifier based on the relative position of the center point in the X-ray image to obtain the region location information, and combining the disease category label with the region location information to generate a structured diagnostic description containing location description and disease name.

[0028] Specifically, in this exemplary embodiment, public datasets from multiple sources and private datasets within a specific scope are processed in the same way. Taking public datasets from multiple sources as an example: First, acquire publicly available X-ray image datasets from multiple sources, including X-ray images and corresponding text information for each (required) age group and (relevant) body part. The content of the text information varies depending on the dataset; in addition to complete diagnostic reports, it may also contain only fragmented, incomplete report-type text information such as disease labels and lesion locations. Standardize the image data, converting the X-ray images into a uniform format. Clean and normalize the text information to construct structured image-text data pairs. For data lacking complete reports, generate standardized diagnostic descriptions based on information such as disease labels and lesion locations.

[0029] Specifically, for datasets containing complete diagnostic reports, image files are extracted and their format converted, while the corresponding diagnostic report text is extracted and language standardized. For datasets containing only disease classification labels and lesion location coordinates, the center point coordinates of the lesion location bounding boxes are extracted. Region identifiers are determined based on the relative position of the center point in the X-ray image. The disease category labels are combined with the region location information to generate a structured diagnostic description containing location descriptions and disease names. This structured diagnostic description can be considered as a subsequently generated diagnostic report and used for supervised fine-tuning training in S2.

[0030] In one specific exemplary embodiment, a multi-source publicly available X-ray image dataset may include two parts: (1) The first part involves obtaining the publicly available adult chest X-ray dataset MIMIC-CXR, which contains a large number of DICOM format X-ray image files and corresponding radiological reports. The data preprocessing process includes image format conversion, text cleaning, and standardization. The DICOM format image data is extracted and converted to PNG format, and the corresponding diagnostic report text is also extracted. The diagnostic reports are then processed for language conversion and standardization, including translating English reports into Chinese, removing irrelevant information, standardizing medical terminology, and standardizing the format, ultimately constructing the publicly available training data pairs for the first part. After preprocessing, more than 200,000 pairs of adult X-ray images and diagnostic reports are obtained, providing sufficient basic data for the subsequent supervised fine-tuning stage.

[0031] (2) The second part involves obtaining the publicly available pediatric chest X-ray dataset VinDr-PCXR, which mainly contains disease classification labels and lesion location annotations. Since the original dataset lacks complete diagnostic reports, a structured diagnostic description needs to be constructed based on the disease category labels and location information. Specifically, the coordinates of the center point of the lesion location annotation box are extracted first. Then, the lesion area is determined based on the ratio of the coordinate position to the image size. The image is divided into four regions: upper left, lower left, upper right, and lower right, and the lesion is assigned to the corresponding region based on the coordinates of its center point. Combining disease category information, a structured report containing a location description and disease name is generated, such as "pneumonia manifestations are visible in the lower left lung field." After processing, more than 10,000 pairs of pediatric X-ray images and corresponding structured reports (text information) were obtained as the second part of the publicly available training data.

[0032] The preprocessed public training data pairs from the first part (adults) and the second part (children) were merged to form a combined training dataset. The dataset was split in a 9:1 ratio, with 10% allocated from the training set as the validation set. To ensure reproducibility, a random seed of 42 was set.

[0033] In step S2, supervised fine-tuning (SFT) is implemented. This step employs an end-to-end multimodal neural network structure, with core components including a vision encoder, an aligner, and a language model, to achieve depth alignment and mapping between X-ray images and diagnostic reports. Specifically: More preferably, in an exemplary embodiment, the visual encoder employs a multi-level feature extraction architecture to output fixed-dimensional global visual features, including: Preliminary spatial features of X-ray images after format normalization are extracted using a multi-layer convolutional network. The preliminary spatial features are input into a multi-layer self-attention module to obtain correlated spatial features; The correlation space features are input into the MLP block and nonlinearly transformed to obtain the transformed space features; After the transformation space features are passed through the attention pooling module, global visual features of fixed dimensions are obtained. The feature aligner maps global visual features to a dimension consistent with the text embedding space through a multi-layer fully connected network to obtain aligned visual features. The text generator is a large language model based on the Transformer decoder architecture, and it generates diagnostic reports token by token using an autoregressive approach. The text token is obtained by segmenting and embedding the text information. The text token corresponds one-to-one with the text information and expresses the semantic content of the diagnostic report. The text generation process includes: taking the aligned visual features and the text token generated from the text information as input to the text generator to obtain the output features of the text generator; the large language model adopts the self-attention mechanism of causal masking, and only uses the context information of the previous t-1 tokens when predicting the t-th token. After the output features are processed by the output projection matrix and softmax operation, the conditional probability distribution of the token is obtained; based on the conditional probability distribution, candidate reports are generated step by step.

[0034] Specifically, in this exemplary embodiment, the visual encoder employs a multi-level feature extraction architecture based on the VisionTransformer structure. First, it extracts the X-ray image after format normalization in step S1 through a multi-layer convolutional network. The initial spatial features z0 are captured to obtain local texture and edge information at different scales: Subsequently, a multi-layer self-attention mechanism is used to model long-range dependencies, enabling the model to focus on the correlations between different regions in the X-ray image. The computation process for each layer is as follows: normalization is performed using Layer Normalization, and then the result is input into the self-attention module. Calculate attention weights to obtain the spatial features of the correlation. : The self-attention mechanism is computed using query vectors, key vectors, and value vectors: in , , , here , , These are the linear transformation weight matrices corresponding to the query vector, key vector, and value vector, respectively. Let be the dimension of the key vector. express The transpose of .

[0035] Finally, the transformation space characteristics are obtained by performing a nonlinear transformation using MLP blocks. .

[0036] This residual connection design ensures stable gradient propagation and effective feature fusion.

[0037] The final layer output undergoes an attention pooling mechanism to obtain a fixed-dimensional global visual feature v: in It is a learnable query vector (not the same as the query vector used in the calculation of the aforementioned self-attention mechanism). and For learnable parameters, It is the hyperbolic tangent function. express The transpose of .

[0038] In a specific exemplary embodiment, the visual encoder employs a VisionTransformer structure based on the SigLIP architecture, comprising 24 Transformer blocks (blocks.0 to blocks.23). Each layer contains a standard self-attention mechanism and a multilayer perceptron (MLP) structure. Specifically, each Transformer block contains two LayerNormalization layers (norm1 and norm2) and a multi-head self-attention module (attn), where the qkv weight matrix has a shape of [3072, 1024] representing the joint projection of the query, key, and value, and the projection output weight proj has a shape of [1024, 1024]. The MLP module contains two fully connected network layers: the first layer fc1 expands the 1024-dimensional features to 4096 dimensions, and the second layer fc2 compresses the 4096-dimensional features back to 1024 dimensions, forming a standard Feed-Forward Network structure.

[0039] In the final stage of visual feature extraction by the visual encoder, the model employs an attention pooling mechanism (attn_pool) to aggregate the serialized visual tokens into a fixed-dimensional global feature representation. The attention pooling layer contains a learnable query vector latent (shape [1, 1, 1024]), a query projection q (shape [1024, 1024]), a key-value projection kv (shape [2048, 1024]), an output projection proj (shape [1024, 1024]), as well as Layer Normalization and MLP modules. This design enables the model to adaptively extract the most relevant information from all visual tokens to form a global visual representation.

[0040] The feature aligner maps global visual features v to a dimension consistent with the text embedding space through a multi-layer fully connected network, which includes a non-linear activation function. This enhances expressive power. Aligned visual features can effectively interact with text tokens in the same semantic space, laying the foundation for subsequent multimodal fusion.

[0041] Feature aligner will align visual features By mapping to the same dimension as the text embedding space using structures such as fully connected networks, aligned visual features are obtained. The specific calculation process is as follows: in, , , , These are learnable parameters. The aligned visual features As a visual cue, it is entered into the text generator along with the text token.

[0042] In a specific exemplary embodiment, the feature aligner employs a two-layer fully connected network structure. The first layer (layers.0) maps 1024-dimensional visual features to 4096 dimensions (weight shape [4096, 1024]), while the second layer (layers.2) retains the 4096-dimensional dimension (weight shape [4096, 4096]), ultimately outputting a 4096-dimensional feature representation consistent with the text generator's embedding space. The layers are connected via non-linear activation functions, enhancing the expressive power of the feature aligner.

[0043] The text generator is a large language model based on the Transformer decoder architecture, and it uses an autoregressive approach to generate diagnostic reports token by token. The specific process is as follows: First, the text information is segmented and embedded to obtain the text token sequence token. Each text token is converted into a 4096-dimensional vector through a word embedding layer (embed_tokens). The text tokens are obtained by word segmentation and embedding of text information (such as the original diagnostic report or label descriptions), with a one-to-one correspondence between the two. The text tokens serve as part of the input to the Transformer decoder (i.e., the text generator) and participate in the autoregressive report generation process.

[0044] The aligned visual feature v' is used as input along with the text token embedding. The token is converted into a vector through a word embedding layer. The correlation between the visual feature and the text is modeled by a multi-head attention mechanism and then input into a multi-layer Transformer decoder for processing to obtain the output features of the text generator. Here, $t$ represents the time step or token sequence position in the current generation process, and $y_{<t}$ represents all tokens with indices less than $t$, that is, the first $t - 1$ generated tokens.

[0045] At each time step, the model adopts a causal masked self-attention mechanism, which only utilizes the context information of the generated tokens and visual features to predict the conditional probability of the next token. The next token can be generated by sampling or selecting the token with the highest probability until a complete diagnostic report is generated or a termination symbol is encountered. Specifically: where is the output projection matrix, is the vocabulary size, is the hidden layer dimension. represents all tokens with indices less than . That is to say, through the output projection operation (lm_head) and the Softmax operation, the probability distribution of the next token is predicted. The causal masked self-attention mechanism is used to ensure that only the information of the generated tokens is utilized when predicting the current token, which is the core mechanism of the Transformer decoder. After the output features pass through the output projection matrix and the softmax operation, the conditional probability of each token is obtained, and the two are continuous generation processes.

[0046] The model samples or selects the token with the highest probability according to the conditional probability at each step, and sequentially generates a complete diagnostic report.

[0047] In a specific exemplary embodiment, the text generator adopts a standard Transformer decoder architecture, which includes 30 Transformer blocks (layers.0 to layers.29). Each layer includes a self-attention mechanism and a feed-forward network. From the parameter shapes, it can be seen that the hidden dimension of the model is 4096, and the intermediate dimension of the feed-forward network is 11008. The vocabulary size is 102400, supporting multi-language processing.

[0048] The text generator converts the input token into a 4096-dimensional vector through the word embedding layer (embed_tokens). After being processed by multiple layers of Transformer, it predicts the probability distribution of the next token through the output head (lm_head). In this method, the aligned visual features After processing by the visual encoder, the visual features are converted into a series of visual tokens. These visual tokens are concatenated with the text tokens to form the input token sequence of the Transformer. Therefore, when the visual features actually enter the Transformer, they are already part of the input token sequence and participate in the autoregressive generation and attention mechanism along with the text tokens.

[0049] The model also includes components for image generation tasks, including a generative vision model (gen_vision_model), a generative aligner (gen_aligner), a generative head (gen_head), and a generative embedding (gen_embed). These components are originally used for image generation in multimodal tasks. Specifically, gen_vision_model is responsible for feature encoding of the input image, gen_aligner is used to align feature representations between different modalities (such as image and text), gen_head is responsible for mapping the generated features back to the image space to achieve image reconstruction, and gen_embed is used to embed various types of information into a high-dimensional feature space. In the diagnostic report generation task of this exemplary embodiment, these components are frozen, their parameters remain unchanged, and they are only used to maintain the integrity of the model structure and compatibility with future multimodal expansions, without participating in the inference and training of the current task.

[0050] More preferably, in an exemplary embodiment, the step of training an end-to-end multimodal neural network using the publicly available training data in a supervised fine-tuning manner includes: During the supervised fine-tuning phase, the publicly available training data pairs are used as training samples, and the optimization objective is to minimize the cross-entropy loss of all tokens, thereby minimizing the difference between the generated candidate reports and the real reports. During training, the parameters of the visual encoder, feature aligner, and text generator are updated.

[0051] Specifically, in this exemplary embodiment, the loss function in the supervised fine-tuning stage of step S2 is designed as cross-entropy loss, with the goal of minimizing the difference between the generated report and the real report. The supervised fine-tuning stage uses publicly available training data pairs as training samples, and the optimization objective is to minimize the cross-entropy loss of all tokens. in For the real report token in the training data, For sequence length, This indicates the input visual features and the preceding... Given the real tokens in the training data, the model predicts the [number]th [unit]. Each token corresponds to the token in the actual report. The goal is to maximize this probability, which is equivalent to minimizing the cross-entropy loss above. This step only involves the real report tokens because a "teacher forcing" mechanism is used during training: the model uses real tokens as input at each step, and the optimization objective is to increase the probability of the model predicting the next real token. Through backpropagation, the model parameters are adjusted so that the difference between the generated token sequence and the real report is minimized during actual autoregressive generation (inference). Therefore, although the loss function only involves real tokens, its essence is to teach the model to generate content that is as close as possible to the real report.

[0052] Furthermore, to simultaneously optimize X-ray image understanding and X-ray report generation capabilities, the parameters of the visual encoder (vision_model.vision_tower), the large language model and text generator (language_model), and the feature aligner are not frozen during training; instead, the parameters of all these modules are updated. This strategy can simultaneously improve the model's visual understanding, cross-modal alignment, and language generation capabilities, making the generated diagnostic reports more accurate and reliable.

[0053] In one specific exemplary embodiment, bfloat16 is used as the training parameter precision, and the maximum allowed input length is 5600 tokens.

[0054] In terms of data processing, the training set is randomly shuffled. When the length of the input sequence is less than the maximum model length, a padding token is added to the right of the sequence; when the length of the input exceeds the maximum allowed length of the model, the excess part is directly deleted from the end.

[0055] For optimizer configuration, the AdamW optimizer was selected, with a learning rate of 1e-4, a weight decay coefficient of 0.1, momentum parameters adam_beta1=0.9, adam_beta2=0.95, and epsilon=1e-8. The maximum gradient norm was set to 1.0 to improve training stability. A cosine annealing learning rate scheduler was used, with a warm-up ratio of 5%.

[0056] In terms of training process control, the training batch size for each GPU was set to 16, gradient checkpointing was used for memory optimization, and training lasted for 50 epochs. An evaluation task was performed every 10,000 steps, and a checkpoint was saved every 1,000 steps, with a maximum of 2 checkpoint files retained.

[0057] For distributed training configuration, four GPUs are used for distributed training, employing DeepSpeed's ZeROStage 3 optimization strategy. ZeRO Stage 3 stores model parameters, gradients, and optimizer states on different GPUs, significantly reducing the memory usage of each GPU. The communication bucket size is set to automatic adjustment, automatically collecting the 16-bit slice parameters from the distributed state into a complete weight file when saving the model checkpoint.

[0058] Through the above process, a supervised fine-tuned one-stage end-to-end multimodal neural network is obtained and used for subsequent reinforcement learning optimization training.

[0059] After completing the supervised fine-tuning in step S2, the one-stage end-to-end multimodal neural network is optimized using private pediatric X-ray image data for reinforcement learning. The private dataset was already processed in step S1, and will be further explained here. For unstructured report images, optical character recognition (OCR) processing is performed to extract key information such as image descriptions and diagnostic results. Data quality is ensured through manual review. X-ray image-report data pairs are constructed for reinforcement learning training. Specifically: First, the target data is preprocessed. Since the reports are unstructured image information, OCR text extraction is performed on the report images, mainly extracting image descriptions and diagnostic results. After manual modification and screening, approximately 1,000 private reinforcement data pairs of private pediatric X-ray images and corresponding text information (reports) are finally obtained.

[0060] In step S3, the private reinforcement data pair is input into the first-stage end-to-end multimodal neural network to generate N candidate reports. Based on the set reinforcement learning reward mechanism, the weighted reward of each of the N candidate reports is calculated. A multi-dimensional reward function system is designed to comprehensively evaluate the quality of the diagnostic reports generated by the model. The core objective of the reward mechanism is to guide the model to generate diagnostic reports that conform to medical logic, possess a reasoning process, and are semantically accurate. The reward mechanism in the reinforcement learning stage is based on a multi-dimensional evaluation design. The core of this mechanism is to guide the model to generate templated text with a reasonable thought process and high-quality final report through the reward function.

[0061] More preferably, in an exemplary embodiment, the reinforcement learning reward mechanism includes: Templated reward function This is used to evaluate whether the report output by the end-to-end multimodal neural network conforms to the predefined format specifications; the reasoning process is placed between the first tags, and the final diagnostic conclusion is placed between the second tags; Reward function for thought process It is used to evaluate the quality of the thinking and reasoning processes generated by end-to-end multimodal neural networks within labels; Semantic similarity reward function The degree of semantic matching between the report output by the end-to-end multimodal neural network and the reference report; The step of calculating the weighted reward for each of the N candidate reports includes: calculating the reward scores for each candidate report across three categories, and finally obtaining the overall reward value of the sample through a weighted average. in For weight parameters, Normalized, it is used as a weighted reward.

[0062] Specifically, in this exemplary embodiment, the templated reward function The evaluation function checks whether the model output conforms to a predefined format specification. Specifically, the model is required to structure the output into two parts: the reasoning process is placed between the first tags, and the final diagnostic conclusion is placed between the second tags. The reward function checks the output format using regular expression matching. Here, `match` is a pseudocode or abstract way of saying "match text using regular expressions," and its implementation varies across different programming languages. If the model's output satisfies the templates `<think>` and `<answer>`, then it can be matched using the regular expression described above, corresponding to the templated reward function. The value is 1, otherwise it is 0.

[0063] In a specific exemplary embodiment, the templated reward function is implemented using a regular expression matching method. The standard template format is defined as follows: the generated text must contain a pair of `` tags and a pair of `<h>` tags, and neither part can be empty. The specific evaluation algorithm uses regular expressions... <think> (.*?)< / think> .*? <answer> (.*?)< / answer> Extract the content from the thinking and answer sections, and check if the extraction was successful and the content is not empty. Output that meets the format requirements will receive a bonus of 1.0 points; otherwise, 0.0 points will be awarded.

[0064] Reward function for thought process This specifically evaluates the quality of inference analysis generated by the model within the labeled text. The reward mechanism can calculate the similarity between the generated thought process and the reference inference text using a pre-trained semantic matching model such as SimCSE. Extract_thinking is a pseudocode or abstract syntax that means "extracting the text from the inference part of the model output". It can be achieved by extracting the text between <\think> and <\think> using regular expressions. Reference_reasoning is the inference text for reference.

[0065] SimCSE is a sentence representation learning model for semantic matching. Through contrastive learning training, it captures deep semantic information of text to calculate the semantic similarity between two text segments. Its core structure is based on a pre-trained Transformer encoder (such as BERT), optimizing sentence vectors through contrastive learning. In its implementation, SimCSE processes each input sentence... via encoder By applying different dropout masks twice (i.e., randomly masking some neurons during each encoding, resulting in slightly different representations of the same sentence in the two encodings), two vectors are obtained. and This is considered a positive sample pair. The model uses cosine similarity... For metric purposes, the InfoNCE loss is used: in For temperature coefficient, This is the batch size. By maximizing the similarity of representations of the same sentence under different enhancements and minimizing the similarity of representations of different sentences, SimCSE can obtain high-quality sentence vectors suitable for semantic matching after training.

[0066] The specific application process of the SimCSE model is as follows: First, the generated thinking section and the reference reasoning text are input into the SimCSE encoder to obtain fixed-dimensional sentence vector representations. Then, the cosine similarity between the two vectors is calculated as the semantic matching score.

[0067] In a specific exemplary embodiment, the calculation of the thought process reward function involves evaluating the quality of medical reasoning. Specific implementation schemes include two methods: The first method uses a pre-trained semantic matching model, such as SimCSE or a sentence embedding model specifically trained in the medical field, as mentioned above, to calculate the semantic similarity between the generated thought process and the reference reasoning text. The second method employs a rule-based evaluation system to assess the completeness of reasoning by checking whether the thought process contains key medical terms (such as "observed," "analysis shows," "in conjunction with clinical practice," etc.) and logical reasoning vocabulary (such as "therefore," "synthetic," "consider," etc.).

[0068] Semantic similarity reward function The generated report and the reference report are measured to determine their semantic similarity. In addition to traditional text similarity metrics such as BLEU, NIST, METEOR, ROUGE-L, and CIDEr, specialized semantic matching models such as SimCSE can be introduced. The specific calculation process is as follows: Extract_answer is a pseudocode or abstract syntax that means "extract the text of the final answer part of the model output". It can be achieved by extracting the text between <\answer> and <\answer> using regular expressions. Reference_answer is the report text for reference. The weights of each indicator, This includes values ​​from various traditional text similarity metrics such as BLEU, NIST, METEOR, ROUGE-L, and CIDEr, as well as scores given by the semantic matching model SimCSE.

[0069] In a specific exemplary embodiment, the semantic similarity reward function employs a multi-metric fusion approach for evaluation. Traditional metrics include BLEU, METEOR, ROUGE-L, and CIDEr, which measure the degree of matching between the generated text and the reference text from different perspectives. BLEU evaluates n-gram overlap, METEOR considers synonym matching, ROUGE-L focuses on the longest common subsequence, and CIDEr is specifically designed for image description tasks. To obtain more accurate semantic evaluation, neural network-based semantic matching models, such as SimCSE or specially trained medical text matching models, are introduced.

[0070] Specifically, for each input X-ray image, the model generates eight different candidate diagnostic reports. This diversified generation strategy helps explore different expressions and diagnostic perspectives. In the GRPO implementation, this is achieved by setting generation parameters (e.g., num_generations=8) and calling the generator multiple times on the same input (using different random seeds and sampling strategies) to generate multiple reports with diverse content in batches.

[0071] For each candidate report, the scores for the three reward categories mentioned above are calculated separately. Finally, a weighted average is used to obtain the overall reward value for the sample. in For weight parameters, Normalized, it is used as a weighted reward.

[0072] Specifically, in this exemplary embodiment, a reward mechanism based on semantic similarity is innovatively introduced, which, compared with traditional supervised learning methods, can better evaluate and optimize the quality of diagnostic reports, making the generated reports more in line with clinical requirements in terms of medical accuracy and language expression.

[0073] In step S4, the supervised fine-tuned one-stage end-to-end multimodal neural network is used as the initial policy model, and the GRPO reinforcement learning algorithm is used to further optimize and train the model. The model parameters are updated using the policy gradient method to maximize the expected reward. Regularization constraints are set to prevent the model from deviating excessively from the initial policy. Through a reasonably designed reward function, the model is guided to learn thought chain reasoning, that is, to perform step-by-step reasoning and analysis before generating diagnostic conclusions, thereby improving the accuracy of the diagnosis. Iterative optimization continues until the model performance converges, resulting in the final two-stage end-to-end multimodal neural network, which is the diagnostic report generation model used subsequently.

[0074] The core of the GRPO algorithm lies in calculating the advantage value through intra-group relative comparisons, effectively alleviating the problems of sparse reward signals and excessive variance in traditional reinforcement learning. Specifically: More preferably, in an exemplary embodiment, the method of optimizing the policy gradient algorithm using GRPO, which uses candidate reports and weighted rewards to perform reinforcement learning on a one-stage end-to-end multimodal neural network to obtain a two-stage end-to-end multimodal neural network, includes: Each input private reinforcement data pair is fed into a one-stage end-to-end multimodal neural network, sampled from the old policy of the one-stage end-to-end multimodal neural network. For each candidate report, a weighted reward is calculated, and the average of the weighted rewards of the G candidate reports is used as the intra-group advantage value. A novel strategy for calculating the GRPO objective function is employed, utilizing within-group dominance and the training process. The optimized loss function is then derived from the calculation results of the GRPO objective function. The optimized loss function is used to perform reinforcement learning on a one-stage end-to-end multimodal neural network, thereby obtaining a two-stage end-to-end multimodal neural network.

[0075] Specifically, in this exemplary embodiment, the process is as follows: for each input (In this exemplary embodiment, the private augmentation data pair is used as multimodal input for X-ray images and prompt text in a specific range of private datasets) from the old strategy sampling candidate outputs (That is, 8 candidate reports, which are obtained by inputting private reinforcement data into the one-stage end-to-end multimodal neural network trained in step S2, consistent with the aforementioned candidate reports), for each output Calculate rewards (This corresponds to the total reward mentioned in step S3) (Based on standardized advantage value calculation of intra-group rewards) : in, The average reward within the group. This is a numerical stability constant. Specifically, since each sample corresponds to 8 candidate reports, multiple overall rewards will be obtained. (In the above formula, the reward) In this step, these scores are averaged to obtain the strength value for the sample. It is used to guide the training process of reinforcement learning.

[0076] More preferably, in an exemplary embodiment, in the GRPO objective function, for each input private reinforcement data pair and a set of candidate reports sampled from the old policy, the probability ratio is calculated using the new policy, and combined with the within-group advantage value, a pruning operation is used to limit the extreme values ​​of the probability ratio, thereby maximizing the advantage-weighted probability ratio of each output. More preferably, in an exemplary embodiment, a KL divergence regularization term is added to the GRPO objective function to prevent the new policy from deviating too much from the old policy.

[0077] Specifically, in this exemplary embodiment, the SFT initial policy refers to the model parameters of the one-stage end-to-end multimodal neural network after supervised fine-tuning (SFT) in step S2, serving as the starting point for reinforcement learning; the old policy This refers to the model parameters currently used to generate the report, which are initially equal to the SFT initial policy and are continuously updated to the new policy from the previous round during training; the new policy This refers to the strategy that is currently being optimized.

[0078] The model is optimized using the following GRPO objective function: in It can be set as the initial strategy for SFT. and For hyperparameters, KL divergence: Consistent with the steps described above, the input q, candidate output, and reward here correspond one-to-one with the definitions above.

[0079] The formula means: by considering each input q and its set of candidate outputs... (These outputs are from the old strategy) The sampled data is equivalent to the aforementioned candidate reports, and a new strategy is used. Calculate the probability ratio and combine it with the standardized within-group odds. The clipping operation (clip function) is used to limit the extreme values ​​of the probability ratio, maximizing the advantage-weighted probability ratio of each output, thereby increasing the probability of high-reward outputs. Simultaneously, the KL divergence regularization term prevents the new policy from deviating too much from the reference policy (such as the SFT initial policy), improving training stability. The clip(x, a, b) function is defined as restricting x to the interval [a, b], i.e., clip(x, a, b) = min(max(x, a), b).

[0080] The strategy probability ratio is calculated using logarithmic probability: in, This represents the token of the i-th output sequence at time t (i.e., the t-th token of the i-th candidate output). This represents the token sequence consisting of all tokens generated before time t in the output sequence. In other words, yes The ordered set of the first t-1 tokens.

[0081] The final optimized loss is .

[0082] It's important to note that the loss function in step S2 is cross-entropy loss, aiming to minimize the difference between the model-generated reports and the real reports, i.e., maximizing the probability of the model generating a real token at each token position. This is a standard maximum likelihood training approach, considering only the reference answer. However, the reinforcement learning stage in step S4 is different. The objective function no longer directly aligns with the real reports. Instead, it generates multiple candidate reports by sampling each input, calculates their rewards, and weights them with a probability ratio (i.e., the ratio of the generation probability of the new policy to that of the old policy on the same candidate report) and an advantage value, thus encouraging the model to increase the probability of outputs with higher rewards. The probability ratio is calculated as shown in the formula above; it is the exponent of the sum of the log probabilities of all tokens, i.e., the ratio of the generation probability of the entire output sequence.

[0083] In a specific exemplary embodiment, in step S4, GRPO reinforcement learning optimization is performed based on the supervised fine-tuned one-stage end-to-end multimodal neural network, with the following settings: Regarding model configuration, due to the relatively small amount of private data, a LoRA (Low-Rank Adaptation) training strategy was adopted to prevent overfitting. The visual encoder and feature aligner were frozen, and only the text generator, i.e., the large language model part, was fine-tuned. The LoRA configuration parameters included: the low-rank matrix dimension was set to 8, the scaling factor was set to 32, the dropout probability was set to 0.05, all linear layers were fine-tuned, and training was performed using bfloat16 precision.

[0084] Regarding the dataset and training strategy, the private augmentation data pairs were divided into training, validation, and test sets in a 6:2:2 ratio. A training data shuffling mechanism was enabled, with a training duration of 50 epochs, an initial learning rate of 1e-4, and a batch size of 16 per GPU. Distributed training was performed using four GPUs, employing a cosine decaying learning rate scheduler, and a warm-start ratio of 5%.

[0085] In terms of the GRPO reinforcement learning algorithm configuration, eight diagnostic texts are generated for each sample for reward calculation, and the reward scores are scaled and normalized. The strength of the KL divergence regularization term in GRPO is set to 0.04 to constrain the policy update magnitude. The PPO gradient clipping range is set to 0.2, the coefficient of the value loss is set to 0.1, and the number of policy updates per round is 4.

[0086] The specific implementation process of the GRPO algorithm is as follows: For each training batch, there are multiple X-ray image inputs. For each input, from the current policy model... Eight candidate reports were generated through sampling. A reward score was calculated for each candidate report, and then a standardized odds value was calculated based on within-group statistics.

[0087] The policy gradient is calculated using importance sampling weights, and the ratio is... The ratio is calculated exponentially from the logarithmic probability difference. To prevent the ratio from becoming too large and causing training instability, a truncation strategy is used to limit the ratio to a certain value. Within the range, of which The final policy loss is the minimum of the losses before and after truncation, ensuring the stability of policy updates.

[0088] The KL divergence regularization term is calculated based on the distribution difference between the current policy and the reference policy (set as the SFT initial policy). This is achieved by setting the KL coefficients. To balance performance improvement and strategy stability.

[0089] Regarding the candidate sample generation strategy, the maximum number of tokens generated each time is 64, and the temperature coefficient is set to 0.9 to control the smoothness of the probability distribution during sampling. `top_k` is set to 50, meaning that only the top 50 tokens with the highest probabilities are sampled in each step. `top_p` is set to 0.9, meaning a set of tokens is dynamically selected until their cumulative probability reaches 0.9 before sampling from them. The duplicate penalty mechanism is disabled, and a sampling method is used instead of beam search for decoding to increase the diversity of the generated results.

[0090] Key monitoring metrics during training include: average reward score, policy KL divergence, gradient norm, and generated text quality. An early stopping mechanism is implemented, halting training when the average reward on the validation set stops increasing for three consecutive epochs to prevent overfitting. Simultaneously, model checkpoints are periodically saved to ensure the recoverability of the training process.

[0091] Additionally, it should be noted that gradient pruning, efficient parameter fine-tuning, and distributed parallelism can be used during training to improve report generation quality and ensure model convergence.

[0092] Through a well-designed reward function, particularly the introduction of thought process rewards and templated rewards, the model learns to conduct systematic observation and analysis before generating a final diagnostic conclusion. The generated reports typically include detailed descriptions of abnormal areas on X-ray images, professional analysis of pathological features, and logical deductions of diagnostic criteria, significantly improving the accuracy and interpretability of the diagnosis. This ability to deduce conclusions through thought processes makes the model's decision-making process more transparent, facilitating understanding and verification by clinicians, and enhancing the system's credibility and practicality.

[0093] For step S5, the X-ray image to be used to generate a diagnostic report and the corresponding text information of the X-ray image are input into a two-stage end-to-end multimodal neural network to generate a diagnostic report.

[0094] It should be noted that the X-ray images and corresponding text information for generating the diagnostic report are also processed to form data pairs, which are then input into a two-stage end-to-end multimodal neural network to generate the diagnostic report. It's important to distinguish that during the model training (reinforcement learning) phase, multiple (e.g., 8) candidate diagnostic reports are generated for each input, and these candidate reports are scored using a reward mechanism (such as semantic similarity scoring) to optimize and improve the model's generation capabilities. However, during the inference phase (practical application), only one diagnostic report is generated and output, directly serving as a clinical or practical reference, without further candidate sampling and screening.

[0095] Through the above implementation methods, this invention achieves image diagnostic report generation based on reinforcement learning optimization for a private pediatric chest X-ray dataset. Compared with traditional supervised learning methods, this method significantly improves the accuracy, professionalism, and clinical applicability of diagnostic reports. The introduction of reinforcement learning mechanisms such as GRPO effectively alleviates the overfitting problem in training with small sample data. A reasonably designed reward function enables the model to learn thought chain deduction, improving the model's generalization ability, diagnostic accuracy, and interpretability in specific X-ray diagnostic scenarios.

[0096] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal X-ray image diagnostic report generation method based on reinforcement learning optimization, characterized in that: Includes the following steps: Obtain public datasets from multiple sources and private datasets within a specific scope. Both public and private datasets include X-ray images and corresponding text information. Standardize the format of the X-ray images and normalize the text information to construct structured public training data pairs and private augmentation data pairs. A supervised fine-tuning approach is used to train an end-to-end multimodal neural network using the publicly available training data. The end-to-end multimodal neural network includes a visual encoder, a feature aligner, and a text generator connected in sequence, resulting in a one-stage end-to-end multimodal neural network. The private reinforcement data pair is input into the first-stage end-to-end multimodal neural network to generate N candidate reports; based on the set reinforcement learning reward mechanism, the weighted reward of each of the N candidate reports is calculated. The GRPO policy gradient algorithm is used for optimization. Candidate reports and weighted rewards are used to perform reinforcement learning on the first-stage end-to-end multimodal neural network to obtain the second-stage end-to-end multimodal neural network. The X-ray image to be used to generate the diagnostic report, along with the corresponding text information, is input into a two-stage end-to-end multimodal neural network to generate the diagnostic report.

2. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 1, characterized in that: The multi-source public dataset includes X-ray images of different age groups and the body parts of interest, as well as the corresponding text information of the X-ray images. The text information includes a complete diagnostic report or text information that includes incomplete report types; the fragmented incomplete report type text information includes disease category labels and lesion locations; The standardization process for text information in incomplete report types includes: extracting the coordinates of the center point of the lesion location annotation box, determining the region identifier based on the relative position of the center point in the X-ray image to obtain the region location information, and combining the disease category label with the region location information to generate a structured diagnostic description containing location description and disease name.

3. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 1, characterized in that: The visual encoder employs a multi-level feature extraction architecture, outputting fixed-dimensional global visual features, including: Preliminary spatial features of X-ray images after format normalization are extracted using a multi-layer convolutional network. The preliminary spatial features are input into a multi-layer self-attention module to obtain correlated spatial features; The correlation space features are input into the MLP block and nonlinearly transformed to obtain the transformed space features; After the transformation space features are passed through the attention pooling module, global visual features of fixed dimensions are obtained. The feature aligner maps global visual features to a dimension consistent with the text embedding space through a multi-layer fully connected network to obtain aligned visual features. The text generator is a large language model based on the Transformer decoder architecture, and it generates diagnostic reports token by token using an autoregressive approach. The text token is obtained by segmenting and embedding the text information. The text token corresponds one-to-one with the text information and expresses the semantic content of the diagnostic report. The text generation process includes: taking the aligned visual features and the text token generated from the text information as input to the text generator to obtain the output features of the text generator; the large language model adopts the self-attention mechanism of causal masking, and only uses the context information of the previous t-1 tokens when predicting the t-th token. After the output features are processed by the output projection matrix and softmax operation, the conditional probability distribution of the token is obtained; based on the conditional probability distribution, candidate reports are generated step by step.

4. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 3, characterized in that: The method of supervised fine-tuning, using the publicly available training data to train an end-to-end multimodal neural network, includes: During the supervised fine-tuning phase, the publicly available training data pairs are used as training samples, and the optimization objective is to minimize the cross-entropy loss of all tokens, thereby minimizing the difference between the generated candidate reports and the real reports. During training, the parameters of the visual encoder, feature aligner, and text generator are updated.

5. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 1, characterized in that: The reinforcement learning reward mechanism described includes: Templated reward function This is used to evaluate whether the report output by the end-to-end multimodal neural network conforms to the predefined format specifications; the reasoning process is placed between the first tags, and the final diagnostic conclusion is placed between the second tags; Reward function for thought process It is used to evaluate the quality of the thinking and reasoning processes generated by end-to-end multimodal neural networks within labels; Semantic similarity reward function The degree of semantic matching between the report output by the end-to-end multimodal neural network and the reference report; The step of calculating the weighted reward for each of the N candidate reports includes: calculating the reward scores for each candidate report across three categories, and finally obtaining the overall reward value of the sample through a weighted average. in For weight parameters, Normalized, it is used as a weighted reward.

6. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 1, characterized in that: The method employing the GRPO policy gradient algorithm for optimization uses candidate reports and weighted rewards to perform reinforcement learning on a one-stage end-to-end multimodal neural network, resulting in a two-stage end-to-end multimodal neural network, including: Each input private reinforcement data pair is fed into a one-stage end-to-end multimodal neural network, sampled from the old policy of the one-stage end-to-end multimodal neural network. For each candidate report, a weighted reward is calculated, and the average of the weighted rewards of the G candidate reports is used as the intra-group advantage value. A novel strategy for calculating the GRPO objective function is employed, utilizing within-group dominance and the training process. The optimized loss function is then derived from the calculation results of the GRPO objective function. The optimized loss function is used to perform reinforcement learning on a one-stage end-to-end multimodal neural network, thereby obtaining a two-stage end-to-end multimodal neural network.

7. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 6, characterized in that: In the GRPO objective function, for each input private reinforcement data pair and a set of candidate reports sampled from the old policy, the probability ratio is calculated using the new policy, and combined with the within-group advantage value, a pruning operation is used to limit the extreme value of the probability ratio, thereby maximizing the advantage-weighted probability ratio of each output.

8. The method for generating multimodal X-ray image diagnostic reports based on reinforcement learning optimization according to claim 7, characterized in that: In the GRPO objective function, a KL divergence regularization term is added to prevent the new strategy from deviating too much from the old strategy.

Images