A chest radiograph report generation method and system based on cross-modal alignment and salient semantic regions

Abstract

The application provides a chest radiograph report generation method and system based on cross-modal alignment and salient semantic regions, and is applied to the technical field of data processing. The chest X-ray chest radiograph image information of a target patient and corresponding initial radiology report text information are processed based on a network model for identifying salient regions of medical semantic information to generate salient regions of medical semantic information and a saliency map; the chest X-ray chest radiograph image of the target patient and the salient region image of medical semantic information are processed based on a mask image modeling module guided by the salient region to generate image block features after mask image reconstruction; the saliency map and the image block features after mask image reconstruction are processed based on a language generation model guided by the saliency map to generate target radiology report information; and the target radiology report information is processed to generate an evaluation result of chest radiograph report generation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for generating chest X-ray reports based on cross-modal alignment and salient semantic regions. Background Technology

[0002] In the medical field, chest X-rays are a widely used diagnostic tool, and chest X-ray reports are crucial for doctors to accurately assess a patient's condition. With the development of medical imaging technology, the traditional method of relying on doctors to manually write chest X-ray reports has gradually revealed many problems. On the one hand, doctors have a heavy daily workload, and manually writing reports consumes a lot of time and energy, easily leading to fatigue, which in turn affects the accuracy and consistency of the reports. Different doctors have individual differences in interpreting chest X-ray images, which can result in significant differences in the descriptions of similar conditions, affecting the accuracy and standardization of diagnosis. For example, different doctors may use different criteria and methods to judge whether lung markings are thickened, and to describe the size and shape of nodules.

[0003] On the other hand, with the gradual application of artificial intelligence (AI) technology in the medical field, computer-aided generation of chest X-ray reports has become a research hotspot. However, some existing AI-based chest X-ray report generation methods often suffer from insufficient understanding of subtle features and complex semantics in the images. For example, some methods struggle to accurately identify subtle abnormalities in early lung lesions, or have difficulty converting image features into accurate and comprehensive text reports, resulting in reports that fail to provide doctors with sufficiently detailed and accurate diagnostic information. Furthermore, in practical clinical applications, chest X-ray reports not only need to accurately describe what is seen in the images but also need to be combined with other clinical information such as the patient's medical history and symptoms. However, most current chest X-ray report generation technologies fail to fully consider the fusion of multimodal information and cannot effectively utilize comprehensive patient information to generate more accurate reports.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore includes information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for generating chest X-ray reports based on cross-modal alignment and salient semantic regions, at least to some extent overcoming the problems existing in the prior art. It utilizes a recognition network to process image and text information, generating salient regions and saliency maps of medical semantic information through steps such as image block segmentation and feature mapping. A masked image modeling module performs mask probability adjustment and other processing on the chest X-ray image and salient region image to generate reconstructed image block features for capturing subtle anomalies. A language generation model, based on the saliency map and image block features, generates the target radiology report through training, matrix multiplication, and other operations. Report quality assessment index values ​​are calculated using natural language generation indicators and medical accuracy indicators to evaluate the report generation effect. The recognition network is trained and optimized through random masking of image block features, completion, and preset loss function processing. Through the collaborative work of multiple modules, key information is extracted from the image and text to generate a radiology report that accurately describes the medical observations.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to one aspect of this application, a method for generating chest X-ray reports based on cross-modal alignment and salient semantic regions is provided, comprising: acquiring chest X-ray image information of a target patient and corresponding initial radiology report text information, and a radiology report generation model, wherein the radiology report generation model comprises a network model for identifying salient regions of medical semantic information, a mask image modeling module guided by salient regions, and a language generation model guided by salient maps; processing the chest X-ray image information of the target patient and the corresponding initial radiology report text information based on the network model for identifying salient regions of medical semantic information to generate salient regions and salient maps of medical semantic information; processing the chest X-ray image of the target patient and the salient region image of medical semantic information based on the mask image modeling module guided by salient regions to generate image block features reconstructed from the mask image; processing the salient map and the image block features reconstructed from the mask image based on the language generation model guided by the salient maps to generate target radiology report information; and processing the target radiology report information to generate an evaluation result of chest X-ray report generation.

[0008] Another aspect of this application discloses a chest X-ray report generation device based on cross-modal alignment and salient semantic regions, characterized in that it comprises: an acquisition module for acquiring chest X-ray image information of a target patient and corresponding initial radiology report text information and a radiology report generation model, wherein the radiology report generation model comprises a network model for identifying salient regions of medical semantic information, a mask image modeling module guided by salient regions, and a language generation model guided by salient maps; a processing module for processing the chest X-ray image information of the target patient and the corresponding initial radiology report text information based on the network model for identifying salient regions of medical semantic information to generate salient regions and salient maps of medical semantic information; processing the chest X-ray image of the target patient and the salient region image of medical semantic information based on the mask image modeling module guided by salient regions to generate image block features reconstructed from the mask image; processing the salient map and the image block features reconstructed from the mask image based on the language generation model guided by the salient maps to generate target radiology report information; and processing the target radiology report information to generate an evaluation result of chest X-ray report generation.

[0009] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a second processor, implements the above-described method for generating chest X-ray reports based on cross-modal alignment and salient semantic regions.

[0010] This application provides a method and system for generating chest X-ray reports based on cross-modal alignment and saliency semantic regions. The server acquires the chest X-ray image of the target patient, the initial radiology report text, and the radiology report generation model (including a recognition network, a mask image modeling module, and a language generation model). The recognition network processes the image and text information and generates saliency regions and saliency maps of medical semantic information through steps such as image block division and feature mapping.

[0011] The masked image modeling module performs mask probability adjustment and other processing on chest X-ray images and salient region images to generate reconstructed image patch features for capturing subtle anomalies. The language generation model, based on the salient map and image patch features, generates the target radiology report through training and matrix multiplication. Report quality assessment metrics are calculated using natural language generation indicators and medical accuracy indicators to evaluate the report generation effect. The recognition network is trained and optimized through random masking of image patch features, completion, and a preset loss function. This method, leveraging the collaborative work of multiple modules, extracts key information from images and text, demonstrating high application value in the field of medical image report generation.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] Figure 1 This document illustrates a flowchart of a chest X-ray report generation method based on cross-modal alignment and saliency semantic regions, provided in an embodiment of this application.

[0014] Figure 2 This illustration shows a schematic diagram of a chest X-ray report generation device based on cross-modal alignment and saliency semantic regions, provided in an embodiment of this application. Detailed Implementation

[0015] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0016] The following is combined Figure 1 This application describes a method for generating chest X-ray reports based on cross-modal alignment and salient semantic regions, according to exemplary embodiments thereof. In one embodiment, this application also proposes a method and system for generating chest X-ray reports based on cross-modal alignment and salient semantic regions. Figure 1 The diagram schematically illustrates a flowchart of a chest X-ray report generation method based on cross-modal alignment and saliency semantic regions according to an embodiment of this application. Figure 1 As shown, this method is applied to a server and includes:

[0017] S101, acquire chest X-ray images of the target patient, as well as the corresponding initial radiology report text information and radiology report generation model.

[0018] In one implementation, it is assumed that there is a patient whose chest X-ray image clearly shows the outline and some details of organs such as the heart and lungs, and the initial radiological report text is "no obvious abnormalities were found in the heart and lungs, and the chest is symmetrical". This information will provide key data support for subsequent model training. Starting from the chest X-ray image information of the target patient, the image block features are randomly masked according to a preset ratio (such as 50%). When extracting image features, the VisionTransformer (ViT)

[181] structure is used as the image encoder to convert the input chest X-ray image I∈R H×W×C Divided into N v Non-overlapping image blocks These image patch features are randomly selected and masked to obtain randomly masked image patch features. The purpose of this step is to simulate information that may be missing in real-world scenarios, allowing the model to learn how to extract key features from partial information. The randomly masked image patch features are then completed based on the masked marker features. During the completion process, the completed image patches become important data for subsequent training. This operation allows the model to attempt to recover the original features of the image patches when faced with incomplete information, helping to improve the model's understanding and learning ability of image features.

[0019] This network model is a cross-modal learning model designed to identify salient regions related to medical semantics by learning the association between images and text. For image feature processing, a ViT structure is used as an image encoder to flatten image patches and map them to visual embeddings via linear transformation. These visual embeddings, along with visual position embeddings, are then input into a Transformer model to obtain image patch features. For text feature processing, BioClinicalBERT is used to tokenize radiology reports into sub-words. After linear transformation and position embedding, these sub-words are input into a Transformer model to obtain text lexical features. Finally, fine-grained cross-modal semantic alignment (FCSA) and pre-mapping aggregation (MbA) strategies are employed to process the image patch features and text lexical features, constructing a common semantic space.

[0020] The completed image patch and the real image patch are processed based on a preset loss function to generate a network model for identifying salient regions of medical semantic information. The preset loss function is: Where, x i and N represents the completed image patch and the real image patch, respectively. v This represents the number of image patches. During training, the completed image patch x... i and real image patches The input is fed into this loss function for calculation. Each calculated loss value reflects the gap between the model's current output and the true situation. The model training process is a continuous adjustment of its parameters to gradually reduce this loss value. Through the backpropagation algorithm, the gradient of the loss function is calculated, which in turn updates the parameters in the model, such as the ViT structure, linear transformation layers, and the Transformer model, optimizing the model towards more accurately reproducing image patch features. For example, in each iteration, based on the gradient information, the linear transformation function f in the image encoder is... v The weights and the Transformer model v The parameters in the model are fine-tuned to make the image patch features output by the model closer to the features of real image patches.

[0021] Preset loss function Lv The loss function directly impacts the model's learning effectiveness on image patch features. A sustained decrease in the loss value indicates continuous model improvement, enabling better capture of detailed information in the image and thus greater accuracy in identifying salient regions of medical semantic information. Conversely, if the loss value does not decrease significantly during training or exhibits increased fluctuations, it may indicate a problem with the model's training, such as an inappropriate learning rate setting or an unreasonable model structure, requiring adjustments to the training process. The calculation result of this loss function not only optimizes the network model itself used to identify salient regions of medical semantic information but also indirectly affects the performance of the entire radiology report generation model. Accurate identification of salient regions is fundamental for the effective operation of the subsequent mask image modeling module guided by salient regions and the language generation model guided by salient maps. If the network model can accurately identify salient regions through loss function optimization, the mask image modeling module can better capture subtle anomalies, and the language generation model can generate more accurate and realistic radiology reports.

[0022] The salient region-guided masking image modeling model belongs to the Transformer-based image modeling module. It is used to process chest X-ray images to capture subtle anomalies and generate more representative image patch features. The VisionTransformer (ViT) is used as the visual encoder to process the input chest X-ray image. The image is divided into multiple non-overlapping image patches, which are then linearly transformed and embedded before being input into the Transformer model to obtain image patch embedding features. These features are used for masking image reconstruction and also provide visual information for subsequent radiological report generation. Key parameters in the processing include the masking probability adjustment parameter for the image patches. For the i-th image patch The formula for calculating its mask probability is: Where I(.) is an indicator function; if the image patch belongs to the salient region S, the mask probability increases. (Set to 0.35). This increases the probability of masking salient regions, helping the model to better learn subtle anomaly features.

[0023] The saliency map-guided language generation model is a Transformer-based language generation model designed to generate radiology reports with sparse clinical semantics using saliency maps and image patch features. It employs a standard Transformer architecture, with 3 layers, 8 attention heads, and a hidden state dimension of 512 in experiments. The model receives a saliency map. Image patch features reconstructed from the masked image As input. First, for the saliency map M... Sand image patch features E I After performing matrix multiplication and normalization, the discriminant representation ω′=Norm(M) is obtained. S E I This representation captures the pathological clues of each chest X-ray image patch. Then, ω' is added as additional input to the language model, gradually incorporating salient semantics into the text report representation through a self-attention mechanism. After training, the model is able to generate target radiological report information based on this input information.

[0024] S102, Based on the network model used to identify salient regions of medical semantic information, the chest X-ray image information of the target patient and the corresponding initial radiological report text information are processed to generate salient regions and salient maps of medical semantic information.

[0025] In one implementation, the chest X-ray image of the target patient is segmented into several non-overlapping image blocks. A linear transformation is then applied to all image blocks to generate visual embedding features. The chest X-ray image of the target patient is segmented into several non-overlapping image blocks. This segmentation forms the basis for subsequent processing; it decomposes the overall information of the image into multiple local information units, facilitating feature extraction and analysis by the model. After segmentation, a linear transformation is applied to all image blocks for mapping. Given an input chest X-ray image I∈R... H×W×C For example, after linear transformation f v Each image patch is mapped to a series of visual embeddings, expressed by the formula: in This represents the [CLS] marker in the Transformer structure. This step converts the raw data of the image patch into a feature vector form that the model can understand and process, preparing for subsequent feature extraction and model training.

[0026] Visual embedding features and visual location embedding features are processed using a network model for identifying salient regions of medical semantic information to generate image patch features. The obtained visual embedding features are then combined with learnable visual location embeddings. Input together into the Transformer model trans v In this study, through multi-layer attention mechanisms and nonlinear transformations of the Transformer model, these features are deeply processed and fused to generate image patch features E containing rich image information. v The calculation formula is: Among them, E v For image patch features, For several non-overlapping image patches, For visual embedding features, Features are embedded for visual location. These image patch features not only contain visual content information of the image patches, but also incorporate location information, enabling the model to perceive the positional relationship of the image patches within the entire image, which helps to represent the features of the image more accurately.

[0027] The initial radiology report text is labeled to generate several sub-word units. A linear transformation is applied to all sub-word units to generate text embedding features. These text embedding features and text position embedding features are then processed using a network model for identifying salient regions of medical semantic information to generate text unit features. Finally, the initial radiology report text is labeled using BioClinicalBERT to represent the radiology report r as a series of sub-word units. Where N t V represents the number of subwords in the report, and V is the size of the vocabulary. Then, all subword lexical units are mapped based on a linear transformation f. t Mapping sub-words to text embeddings yields... in This represents the [CLS] marker in the Transformer structure. This step converts the text information into a vector form that the model can process, laying the foundation for subsequent extraction of semantic features from the text.

[0028] Combine the obtained text embedding features with the learnable text position embeddings Input together into the Transformer model trans t In the middle, after processing by the Transformer model, text lexical features E containing text semantics and positional information are generated. t The formula is Among them, E t For image patch features, It consists of several sub-word elements. For text embedding features, Text location embedding features. These text lexical features can accurately represent the semantic information in radiological reports, providing textual support for cross-modal learning and salient region identification.

[0029] Image patch features and text word features are flattened and linearly projected onto local visual embedding features and local text embedding features, respectively. Max pooling is then used to aggregate these features, generating global image patch features and global text word features. Image patch features E are then processed... v Text lexical features E t Flattening and linear projection onto local visual embedding features and local text embedding features In this step, linear projection is used to further adjust the dimensions and representation of the features, making them more suitable for subsequent aggregation operations.

[0030] Then, the local features of the two modalities are aggregated into their respective global features using a max-pooling operation. The formula is as follows: Where f v ′ and f t '' represents the image patch label feature mapping function and the text lexical label feature mapping function, respectively. Max pooling retains the most salient feature information from each modality, thus generating global image patch features. and text lexical global features This highlights key information and provides a more representative feature representation for the subsequent construction of a public semantic space.

[0031] Global features of image patches and global features of text terms are processed to generate the target semantic space. Next, a bidirectional cross-modal contrastive loss Li is used. bi To learn the common semantic space, the calculation formula is: L bi =λ v L v→t +λ t L t→v , where λ v and λ t These are the weight parameters for the cross-modal contrast loss in both directions. The cross-modal contrast loss L in the image-text direction. v→t The formula is: Cross-modal contrast loss L in the text-image orientation t→v It can be represented as: Among them, L v→t For image-to-text cross-membrane contrast loss, L t→v For text-image cross-membrane contrast loss, N is the minimum batch size for training. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of text terms and global features of image patches. The cosine similarity represents the global features of text lexical units and the global features of image patches, with τ being a temperature parameter. By calculating the bidirectional cross-modal contrastive loss, the model can learn the similarity relationship between images and text in a common semantic space, making image patch features and text lexical features comparable in this space, thereby achieving cross-modal semantic alignment.

[0032] Based on the target semantic space, global and local features of image patches are processed to generate salient regions and salient maps of medical semantic information. After determining the salient map M... S Subsequently, since the aggregation operation is performed in the learned common semantic space, the visual image patches most similar to the global features (top k%) can be considered as containing parts with discriminative semantic information. These parts typically represent what distinguishes a particular chest X-ray image from other chest X-ray images, such as lesions, abnormal organs, surgical scars, and implanted devices. SISRNet uses cosine similarity to identify local features with discriminative semantics and then backtracks to locate their corresponding local image patches. Specifically, by comparing the cosine similarity between the local features and global features of each image patch, image patches with high similarity are found, and the regions corresponding to these image patches are semantically rich salient regions.

[0033] The calculated significance map M S This is a numerical vector reflecting the importance of each image patch. The image patches corresponding to positions with higher values ​​in this vector (i.e., high similarity to global features) are marked; these marked patches constitute salient regions. These salient regions are significant in clinical diagnosis, containing crucial medical information and essential for generating accurate radiology reports. Ultimately, this generates salient regions and saliency maps of medical semantic information. These results provide crucial input for subsequent masked image modeling and language generation models, enabling the models to focus more on important regions in the image when generating reports, thereby improving the accuracy and reliability of the reports.

[0034] S103, based on the mask image modeling module guided by salient regions, processes the chest X-ray image of the target patient and the salient region image of medical semantic information to generate image block features after mask image reconstruction.

[0035] In one implementation, a mask probability adjustment process is performed on the chest X-ray image of the target patient based on a salient region-guided mask image modeling module to generate mask probabilities for image patches. The salient region image of medical semantic information is processed by the same module to generate adjusted mask probabilities for image patches. By increasing the probability of masking salient regions, the model learns subtle anomaly features. In this process, a pre-trained salient region recognition network is used as the input to the chest X-ray image to obtain salient regions S with rich medical-related semantic content. For the i-th image patch... Its mask probability adjustment formula is as follows Where I(.) is an indicator function, if the image patch If the region belongs to the saliency region S, then its mask probability increases. (Set to 0.35); p i The initial mask probabilities follow a uniform distribution U(0,1). This method adjusts the mask probabilities of chest X-ray images, generating mask probabilities for image patches. Simultaneously, images of salient regions with significant medical semantic information are processed to further determine the adjusted mask probabilities of image patches. This process, also based on the aforementioned formula, strengthens the masking operation for salient regions, allowing the model to focus more on feature learning in anomalous regions.

[0036] Image patch features are processed based on the masking probabilities and adjusted masking probabilities of the image patches to generate masked image patches. Specifically, based on the determined masking probabilities, selective masking is performed on the input image patch features, processing only the portions of the image patches that meet the masking conditions to obtain the masked image patches. This process enables the model to simulate scenarios where some information is missing in real-world situations, prompting the model to learn how to extract key features from the remaining information, especially focusing on regions that may contain subtle anomalies.

[0037] The masked image patch and the real image patch are processed to generate pixels for reconstructing the masked image patch. The generated masked image patch and the real image patch are then processed, and an image decoder is used to reconstruct the pixels of the masked image patch using the mean squared error (MSE) loss function. The calculation formula is as follows: Among them, L I To reconstruct the pixels of the masked image patch, and z i Let M represent the masked image patch and the real image patch, respectively, and M be the number of masked image patches. By minimizing this loss function, the model continuously adjusts its parameters to make the pixels of the reconstructed image patch as close as possible to the pixels of the real image patch, thereby improving the accuracy of image feature restoration and laying the foundation for generating more accurate image patch features in the future.

[0038] S104, based on the language generation model guided by the saliency map, processes the saliency map and the image patch features reconstructed from the masked image to generate target radiological report information.

[0039] In one implementation, a language generation model guided by a saliency map is trained based on a preset loss function to generate a trained language generation model. Generating radiology reports presents certain challenges, as sentence semantics are scattered, with a predominance of normal descriptions, making it difficult for the decoder to capture anomalous information. To address this issue, SISRNet employs a Transformer-based language generation model and trains it using a preset loss function. The preset loss function is... Among them, L RLet represent the loss function used to train the language model that generates radiology reports, and l and V represent the length of the radiology report and the size of the vocabulary, respectively. This represents the probability of selecting the j-th word from the vocabulary at the i-th position in the generated report. During training, the model continuously adjusts its parameters to minimize this loss function, thus improving the prediction of the word. As close as possible to the real word y ij This allows the system to learn the ability to accurately generate radiology reports, ultimately leading to the training of a language generation model.

[0040] Matrix multiplication and normalization are performed on the saliency map and the image patch features reconstructed from the masked image to generate discriminative representation information. This discriminative representation information is used to characterize the pathological clues of each chest X-ray image patch, integrating information from the saliency map and image patch features. This provides crucial semantic guidance for subsequent generation of radiological reports in the language model. The saliency map obtained after processing the masked image modeling module guided by the saliency region is shown below. Image patch features reconstructed from the masked image Further processing is performed. First, matrix multiplication is performed on the two, followed by normalization layer processing to obtain the discriminant representation information ω′, calculated as ω′=Norm(M s E1). Discriminant representation information integrates information from saliency maps and image patch features, enabling the capture of pathological clues in each chest X-ray image patch. For example, in actual chest X-ray image analysis, this computational method can combine the importance information of different regions in the image (represented by the saliency map) with the features of the image patch itself, providing crucial semantic guidance for the subsequent generation of radiological reports in the language model, allowing the model to focus more on information from important regions when generating reports.

[0041] S105 processes the target radiology report information to generate an evaluation result for the chest X-ray report.

[0042] In one implementation, based on the target radiology report information and the reference real report information, a report quality assessment index value is generated through calculation processing using natural language generation (NLG) and medical accuracy (CE) indicators. To comprehensively evaluate the quality of the radiology report generated by the SISRNet model, it is necessary to perform calculation processing based on the target radiology report information and the reference real report information using NLG and CE indicators. Commonly used NLG indicators include BLEU, METEOR, and ROUGE, which are mainly used to assess the linguistic similarity between the generated radiology report and the reference real report. Among them, BLEU is a precision-based evaluation metric that analyzes n-grams with a maximum length of 4 and calculates penalties or rewards based on the matching of the generated text with the reference text in terms of length, word selection, and order. METEOR is a precision and recall-based evaluation metric that extends the BLEU-1 metric. It measures the matching of unpaired tuples in the generated text with the reference text based on exact form, stemming form, and meaning, and calculates the harmonic mean of precision and recall for unpaired tuples, biased towards recall. It also uses a multiplication factor to reward unpaired tuples that are in the same order as the reference text. ROUGE-L is a recall-oriented evaluation metric that measures the length of the longest common subsequence between two texts. It calculates the weighted harmonic mean of precision and recall, biased towards recall. It does not require consecutive matching between texts, but rather requires the sequential matching of n-grams that reflects sentence-level word order. However, these NLG metrics mainly focus on lexical similarity and are difficult to reflect the medical accuracy of the report.

[0043] The Medical Accuracy (CE) metric focuses on the medical relevance and accuracy of the generated content. It measures the accuracy of the generated radiology report in terms of medical facts, specifically whether the generated report correctly describes the key medical information contained in the input chest X-ray image. Using the rule-based annotation tool CheXpert, 14 tags related to chest pathology and supporting equipment are extracted for both the generated and reference reports. A high CE score is obtained from CheXpert if the two reports are semantically equivalent. The target radiology report information and the reference real report information are then substituted into the two types of metrics mentioned above for calculation, resulting in the report quality assessment index value. These index values ​​comprehensively reflect the degree of difference between the generated report and the real report in terms of language and medical content.

[0044] The report quality assessment index values ​​are analyzed and interpreted to generate an evaluation result for the generated chest X-ray report. This evaluation result characterizes the linguistic accuracy, medical accuracy, and overall quality level of the report generated by the radiology report generation model. Analysis of the NLG index values ​​reveals the generated report's performance in terms of fluency, grammatical correctness, and vocabulary matching with the real report. For example, a high BLEU score indicates that the generated report is similar to the real report in terms of vocabulary selection and order. The METEOR index provides a more detailed reflection of various forms of tuple matching; its score reflects the accuracy of the generated report's vocabulary comprehension and application.

[0045] The analysis of the medical accuracy index (CE) directly determines the accuracy of the generated report's description of medical information. A high CE score indicates that the generated report accurately describes key medical information in chest X-ray images, demonstrating high reliability in medical diagnosis; conversely, a low CE score indicates insufficient medical accuracy. The combined analysis of the NLG and CE indices comprehensively characterizes the linguistic accuracy, medical accuracy, and overall quality level of the radiology report generation model. If all NLG indices perform well in terms of linguistic accuracy, and the CE index also reaches a high level, then the report generated by the model can be considered to be of high quality. If any one of the indices performs poorly, such as NLG indices showing problems with language expression, or CE indices indicating inaccurate description of medical information, it means that the model has room for improvement in that area. The final evaluation results will provide important evidence for the optimization and improvement of the model.

[0046] The system obtains chest X-ray images of the target patient, initial radiology report text, and radiology report generation model (including recognition network, mask image modeling module, and language generation model) from the server. The recognition network processes the image and text information and generates salient regions and salient maps of medical semantic information through steps such as image block segmentation and feature mapping.

[0047] The masked image modeling module performs mask probability adjustment and other processing on chest X-ray images and salient region images to generate reconstructed image patch features for capturing subtle anomalies. The language generation model, based on the salient map and image patch features, generates the target radiology report through training and matrix multiplication. Report quality assessment metrics are calculated using natural language generation indicators and medical accuracy indicators to evaluate the report generation effect. The recognition network is trained and optimized through random masking of image patch features, completion, and a preset loss function. This method, leveraging the collaborative work of multiple modules, extracts key information from images and text, demonstrating high application value in the field of medical image report generation.

[0048] In one implementation, such as Figure 2 As shown, this application also provides a chest X-ray report generation device based on cross-modal alignment and saliency semantic regions, comprising:

[0049] The acquisition module 201 is used to acquire chest X-ray image information of the target patient and the corresponding initial radiology report text information and radiology report generation model. The radiology report generation model consists of a network model for identifying salient regions of medical semantic information, a mask image modeling module guided by salient regions, and a language generation model guided by salient graphs.

[0050] Processing module 202 is used to process the chest X-ray image information of the target patient and the corresponding initial radiology report text information based on a network model for identifying salient regions of medical semantic information, generating salient regions and salient maps of medical semantic information; processing the chest X-ray image of the target patient and the salient region image of medical semantic information based on a masked image modeling module guided by salient regions, generating image block features after masked image reconstruction; processing the salient map and the image block features after masked image reconstruction based on a language generation model guided by salient maps, generating target radiology report information; and processing the target radiology report information to generate an evaluation result of chest X-ray report generation.

[0051] The various embodiments in this application are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for evaluating the chest X-ray report generation method, electronic device, electronic device, and readable storage medium based on cross-modal alignment and salient semantic regions are relatively simple to describe because they are fundamentally similar to the chest X-ray report generation method embodiments based on cross-modal alignment and salient semantic regions described above. Relevant parts can be referred to the descriptions of the chest X-ray report generation method embodiments based on cross-modal alignment and salient semantic regions described above.

Claims

1. A method for generating chest X-ray reports based on cross-modal alignment and saliency semantic regions, characterized in that, include: The system acquires chest X-ray images of the target patient, along with the corresponding initial radiology report text and a radiology report generation model. The radiology report generation model consists of a network model for identifying salient regions of medical semantic information, a mask image modeling module guided by salient regions, and a language generation model guided by salient maps. This paper uses a network model to identify salient regions of medical semantic information. The chest X-ray image of the target patient and the corresponding initial radiology report text are processed to generate salient regions and saliency maps of medical semantic information. This includes segmenting the chest X-ray image to generate several non-overlapping image patches; mapping all image patches using a linear transformation to generate visual embedding features; processing the visual embedding features and visual location embedding features based on the network model to generate image patch features; and marking the initial radiology report text. The process involves generating several sub-word units; mapping all sub-word units based on linear transformations to generate text embedding features; processing the text embedding features and text location embedding features based on a network model used to identify salient regions of medical semantic information to generate text unit features; flattening and linearly projecting the image patch features and text unit features onto local visual embedding features and local text embedding features respectively, and then aggregating them through max pooling to generate global image patch features and global text unit features; processing the global image patch features and global text unit features to generate the target semantic space, the calculation formula for generating the target semantic space is as follows: ； ； ;in, and These are the weighting parameters for the cross-membrane state contrast loss in two directions. For image-to-text cross-membrane contrast loss, For text-image cross-membrane contrast loss, N is the minimum batch size for training. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of text terms and global features of image patches. Cosine similarity represents the relationship between global features of text terms and global features of image patches. The temperature parameter is used; based on the target semantic space, the global and local features of the image patch are processed to generate the salient regions and salient maps of medical semantic information; The salient region-guided mask image modeling module processes the chest X-ray image of the target patient and the salient region image of medical semantic information to generate image block features after mask image reconstruction. Based on a language generation model guided by saliency maps, the features of saliency maps and image patches reconstructed from masked images are processed to generate target radiological report information; The target radiology report information is processed to generate an evaluation result for the chest X-ray report.

2. The method as described in claim 1, characterized in that, Based on a network model for identifying salient regions of medical semantic information, the system processes chest X-ray images of target patients and corresponding initial radiological report text information to generate salient regions and salient maps of medical semantic information. This also includes: The formula for calculating the generated image patch features is: ； in, For image patch features, For several non-overlapping image patches, For visual embedding features, Embed features for visual location; The formula for calculating the lexical features of generated text is: ； in, For image patch features, It consists of several sub-word elements. For text embedding features, Embed features for text location; The formula for generating the target semantic space is: ； ； ； in, and These are the weighting parameters for the cross-membrane state contrast loss in two directions. For image-to-text cross-membrane contrast loss, For text-image cross-membrane contrast loss, N is the minimum batch size for training. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of an image patch and global features of text terms. Cosine similarity represents the relationship between global features of text terms and global features of image patches. Cosine similarity represents the relationship between global features of text terms and global features of image patches. This refers to the temperature parameter.

3. The method as described in claim 1, characterized in that, A masked image modeling module guided by salient regions processes the chest X-ray image of the target patient and the salient region image of medical semantic information to generate image patch features after masked image reconstruction, including: The mask probability adjustment process is performed on the chest X-ray image of the target patient based on the mask image modeling module guided by salient regions to generate the mask probability of the image block. The salient region image modeling module guided by salient regions is used to process the salient region image of medical semantic information to generate the mask probability after image patch adjustment. Image patch features are processed based on the mask probability of the image patch and the adjusted mask probability of the image patch to generate masked image patches; The mask image patch and the real image patch are processed to generate pixels for reconstructing the mask image patch; Pixel generation of image patch features after reconstructing the masked image based on the reconstructed masked image patch; The formula for calculating the pixels of the generated reconstructed mask image patch is: ; in, To reconstruct the pixels of the masked image patch, and These represent the masked image block and the real image block, respectively, with M being the number of masked image blocks.

4. The method as described in claim 3, characterized in that, Based on a language generation model guided by saliency maps, the saliency maps and image patch features reconstructed from masked images are processed to generate target radiological report information, including: The language generation model guided by the saliency map is trained based on a preset loss function to generate a trained language generation model. Matrix multiplication and normalization are performed on the saliency map and the image patch features reconstructed from the masked image to generate discriminative representation information. The discriminative representation information is used to characterize the pathological clues of each chest X-ray image patch. It integrates the information of the saliency map and the image patch features, and provides key semantic guidance for the subsequent generation of radiological reports in the language model. The discriminative representation information is processed based on the trained language generation model to generate target radiological report information; The formula for generating discriminative representation information is: ; The default loss function is: ; This represents the loss function used to train the language model that generates radiology reports. and These represent the length of the radiology report and the size of the vocabulary, respectively. The representative in the generated report The first position selection vocabulary list The probability of each word.

5. The method as described in claim 1, characterized in that, The target radiology report information is processed to generate an evaluation result of the chest X-ray report, including: Based on the target radiology report information and the reference real report information, the report quality assessment index value is generated by calculating and processing through natural language generation indicators and medical accuracy indicators. The report quality assessment index values ​​are analyzed and interpreted to generate an assessment result for the chest X-ray report. The assessment result for the chest X-ray report is used to characterize the linguistic accuracy, medical accuracy, and overall quality level of the report generated by the radiology report generation model.

6. The method as described in claim 1, characterized in that, Obtain a network model for identifying salient regions of medical semantic information, including: Obtain features of randomly occluded image patches with a preset ratio; Based on the mask marker features, the features of randomly occluded image blocks are completed to generate completed image blocks; The completed image patch and the real image patch are processed based on a preset loss function to generate a network model for recognizing salient regions of medical semantic information. The default loss function is: ; in, and These represent the completed image block and the original image block, respectively. This represents the number of image blocks.

7. A chest X-ray report generation device based on cross-modal alignment and saliency semantic regions, used to implement the method of claim 1, characterized in that, The device includes: The acquisition module is used to acquire chest X-ray images of the target patient and the corresponding initial radiology report text information and radiology report generation model. The radiology report generation model consists of a network model for identifying salient regions of medical semantic information, a mask image modeling module guided by salient regions, and a language generation model guided by salient graphs. The processing module processes the chest X-ray image information of the target patient and the corresponding initial radiology report text information based on a network model used to identify salient regions of medical semantic information, generating salient regions and saliency maps of medical semantic information; it processes the chest X-ray image of the target patient and the salient region image of medical semantic information based on a masked image modeling module guided by salient regions, generating image patch features reconstructed from the masked image; it processes the saliency map and the image patch features reconstructed from the masked image based on a language generation model guided by the saliency map, generating the target radiology report information; and it processes the target radiology report information to generate an evaluation result of the chest X-ray report generation.

8. An electronic device, characterized in that, include: First processor; The processor also includes a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the chest radiograph report generation method based on cross-modal alignment and salient semantic regions as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the second processor, it implements the chest X-ray report generation method based on cross-modal alignment and salient semantic regions as described in any one of claims 1 to 6.

Images