A method for generating an imaging report based on chest X-ray and related equipment

Abstract

The present invention provides an image report generation method based on an X-ray chest film and related equipment, which are applied to the field of data processing technology. The present application obtains image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient, and the initial image report generation model includes an encoder and a decoder; obtains a training sample set for training the initial image report generation model; processes the training sample set based on preset processing rules to generate a training set and a validation set; trains the initial image report generation model based on the training set and the validation set to generate a target image report generation model; processes the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processes the fused visual features and text encoding features based on the target image report generation model to generate an image diagnosis report.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2024110587576 filed with the State Intellectual Property Office on August 2, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the technical field of data processing, and in particular to a method for generating an imaging report based on an X-ray chest film and related equipment. Background Art

[0004] Medical imaging reports, as the primary medium for radiologists to convey patients' medical scan results and diagnostic results, are of great clinical reference value. Therefore, radiologists need to write these reports accurately and in detail. In recent years, deep learning-based image-text matching methods have made great breakthroughs, which can be roughly divided into one-to-one global matching methods and many-to-many fine-grained matching methods. The one-to-one global matching method usually extracts global feature representations of images and sentences, then embeds the global features of images and sentences into a joint space, and measures their similarity by the feature distance of the joint space. This method represents the entire image or text with a global vector, ignoring that the key information in the image may be a few key areas or the key information in the text may be a key word. It lacks fine-grained exploration of image and text features, resulting in mediocre performance.

[0005] Furthermore, research on medical imaging report generation technology is still in its infancy. Existing medical imaging report generation solutions typically use medical knowledge graphs for subtasks such as classification, but fail to integrate them into the model, resulting in low accuracy and reliability in medical imaging report generation. Existing text generation efforts mostly use maximum likelihood estimation (MLE) to train language models. However, this modeling approach often leads to text degradation, meaning the generated radiology reports often contain unnecessary text repetitions, which can occur at the word, phrase, or sentence level. Previous work on automatic report generation has yet to specifically address this issue.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0007] The purpose of this application is to provide an imaging report generation method based on chest X-rays and related equipment, which at least to a certain extent overcomes the problems existing in the prior art, uses a deep learning network to encode the features of a priori knowledge graph, chest X-rays, and clinical information, decodes the received encoded information through a decoder, and uses an algorithm to limit text generation during the decoding process to reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through a loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0008] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0009] According to one aspect of the present application, a method for generating an image report based on an X-ray chest film is provided, comprising: obtaining image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; obtaining a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; processing the training sample set based on preset processing rules to generate a training set and a validation set; training the initial image report generation model based on the training set and the validation set to generate a target image report generation model; processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0010] In one embodiment of the present application, the acquisition of a training sample set for training an initial imaging report generation model includes: acquiring a preset chest X-ray report data set; processing the chest X-ray report data set to obtain radioactive features and correlation co-occurrence frequencies; processing the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information; and generating a priori knowledge graph for characterizing chest images based on the correlation matrix information.

[0011] In one embodiment of the present application, the processing of the radioactive features and the correlation co-occurrence frequency to generate correlation matrix information includes: the method includes a calculation formula for calculating the correlation co-occurrence frequency, and the calculation formula is: Among them, C ij C represents the number of times concept i and concept j co-occur at the report level. j represents the total number of times concept j appears. Pij represents the frequency of concept i appearing when concept j appears. A correlation matrix A can be constructed based on the co-occurrence frequency P: the method includes a calculation formula for calculating the correlation matrix, which is: Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation.

[0012] In one embodiment of the present application, the training sample set is processed based on preset processing rules to generate a training set and a validation set, including: extracting features from the training sample set to generate an original feature library; dividing the training sample set according to the original feature library to generate a training set and a validation set; using a classifier to predict each validation set divided from the original feature library to determine the prediction result; using a preset algorithm to train each training set divided from the original feature library to obtain a validation set class prediction result.

[0013] In one embodiment of the present application, the training of the initial image report generation model based on the training set and the validation set to generate a target image report generation model includes: processing the training set based on a window-constrained contrastive learning loss method to generate a word unit vector value; training the initial image report generation model based on the word unit vector value to generate a target image report generation model; wherein the window-constrained contrastive learning loss method includes a calculation formula for obtaining a word unit vector value, and the calculation formula is: Among them, w is the window constraint parameter, L represents the length of the generated text sequence, i represents the word position index, j represents the comparison word position index, h i represents the feature representation of the i-th generated word, h j represents the feature representation of the jth generated word, ρ is a predefined interval parameter, s(h i ,h i ) represents the similarity between the word and itself, s(h i ,h j ) represents the similarity between the word and the word outside the window.

[0014] In one embodiment of the present application, the target chest X-ray image is processed based on the target image report generation model to generate fused visual features and text coding features, including: the target image report generation model includes a calculation formula for obtaining the fused visual features, and the calculation formula is: Among them, y i Indicates the i-th label value of the current sample, p irepresents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, and i is the main node index; the target imaging report generation model includes a calculation formula for obtaining text encoding features, which is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0015] In one embodiment of the present application, the processing of the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report includes: obtaining a report generation loss function; processing the fused visual features and the text encoding features based on the report generation loss function to generate an image diagnosis report; the target image includes a generation model including a calculation formula for obtaining the report generation loss function, and the calculation formula is: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ij Represents the probability that the i-th word corresponds to the j-th word in the dictionary.

[0016] Another aspect of the present application is an image report generation device based on an X-ray chest film, characterized in that it includes: an acquisition module for acquiring image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; acquiring a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; a processing module for processing the training sample set based on preset processing rules to generate a training set and a verification set; training the initial image report generation model based on the training set and the verification set to generate a target image report generation model; processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0017] According to another aspect of the present application, an electronic device is provided, characterized in that it includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to implement the above-mentioned method for generating an imaging report based on an X-ray chest film by executing the executable instructions.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a second processor, the computer program implements the above-mentioned method for generating an imaging report based on an X-ray chest film.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned method for generating an imaging report based on an X-ray chest film when executed by a third processor.

[0020] The present application provides an imaging report generation method based on an X-ray chest film and related equipment, in which a server obtains image information to be detected and an initial imaging report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient, and the initial imaging report generation model includes an encoder and a decoder; a training sample set for training the initial imaging report generation model is obtained, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients; the training sample set is processed based on preset processing rules to generate a training set and a validation set; the initial imaging report generation model is trained based on the training set and the validation set to generate a target imaging report generation model; the target chest X-ray image is processed based on the target imaging report generation model to generate fused visual features and text encoding features; the fused visual features and text encoding features are processed based on the target imaging report generation model to generate an imaging diagnosis report. A deep learning network is used to encode the features of prior knowledge graphs, chest X-rays, and clinical information. The received encoded information is decoded by a decoder. During the decoding process, an algorithm is used to limit text generation and reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through the loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a method for generating an imaging report based on an X-ray chest film provided in one embodiment of the present application is shown;

[0023] Figure 2 A schematic structural diagram of an X-ray chest film-based imaging report generating device provided in one embodiment of the present application is shown;

[0024] Figure 3 A schematic structural diagram of an electronic device provided in one embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0027] The following combination Figure 1 The following describes a method for generating an imaging report based on a chest X-ray according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are merely provided to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. Rather, the embodiments of the present application can be applied to any applicable scenario.

[0028] In one embodiment, the present application also proposes a method for generating an imaging report based on an X-ray chest film and related equipment. Figure 1 The following schematically shows a flow chart of a method for generating an image report based on an X-ray chest film according to an embodiment of the present application. Figure 1 As shown, the method is applied to the server and includes:

[0029] S101, obtaining image information to be detected and an initial image report generation model that matches the target chest X-ray image.

[0030] In one embodiment, the image information to be detected includes a chest X-ray image of the target patient and the clinical information of the target patient, wherein the clinical information of the target patient includes the information contained in the application form issued by the clinical department when the target patient comes for treatment (for example, symptoms such as chest pain and cough), and the chest X-ray image of the target patient is the image information obtained from the target patient's latest physical examination, which is processed by a subsequent target image report generation model to generate a diverse image report.

[0031] S102: Obtain a training sample set for training an initial image report generation model.

[0032] In one embodiment, a training sample set includes a priori knowledge graph for representing chest images and chest images to be trained, wherein the chest images to be trained include clinical information from different patients. The training sample set includes information on several standard substances. In addition, the data in the training sample set includes patient data from different hospitals. The data is divided into a training group (281 cases) and an internal test group (122 cases) in a 7:3 ratio using a random number table method. The identification information is used to represent the attributes of each type of material information. The initial imaging report generation model is trained based on the training sample set with the identification information to generate a target imaging report generation model. Specifically, the data classification results corresponding to the training sample set are obtained. Since the classification results of each training sample set can be determined in advance, the data can be directly obtained externally. The prediction results are compared with the data classification results to determine a first comparison result; the prediction results of the validation set are compared with the data classification results to determine a second comparison result; and the second comparison result is determined to determine whether the first comparison result meets the preset requirements. If the preset requirements are met, it indicates that the detection results of the current imaging report generation model are relatively accurate. In this case, the current imaging report generation model can be used as the target imaging report generation model.

[0033] In another embodiment, the dataset used in the experiment is the Chinese Chest X-ray Report Generation Dataset (CCXRD). The evaluation indicators adopt indicators in the field of natural language generation (NLG), including BLEU, METEOR, ROUGE-L and CIDEr. In order to comprehensively evaluate the clinical effectiveness (CE), the F1 score, precision and recall rate of the generated report containing the disease were calculated. At the same time, the applicant also reported the evaluation results at the macro and micro levels. In order to evaluate the text quality of the generated report, the applicant used the rep-n indicator to evaluate the repetition of the report, and used the diversity indicator to measure the diversity of the text.

[0034] In the comparative experiments, only three state-of-the-art algorithms, R2Gen, Fluent, and Prior, were compared on the Chinese dataset, as some models are not publicly available or cannot be reproduced. As shown in Table 1, the proposed model outperforms the comparative algorithms on all NLG metrics.

[0035]

[0036] Table 1 compares the NLG indicators of this study and recent studies on Chinese datasets, as shown above.

[0037] Table 2 shows the macro and micro F1 score results, with our method achieving the best performance on both metrics. R2Gen achieved the best micro accuracy, demonstrating that its memory-driven module better preserves information for the primary categories, "no obvious abnormality" and "increased lung texture," leading to strong performance on these categories and improved micro accuracy. However, the lower macro accuracy reflects that our method only performs well on the primary categories, with poor performance on the secondary categories, resulting in the lowest macro accuracy. This suggests that due to the high proportion of normal and "increased lung texture" samples in the dataset, the high micro accuracy value does not fully reflect the model's superiority. Compared to the baseline model, Fluent, our method achieved improvements in both precision and recall, thanks to the enhanced image representation capabilities achieved by using medical prior knowledge, resulting in high-quality text reports. Meanwhile, the macro F1 score for the Chinese dataset was 0.2814, lower than the 0.4199 for the English dataset. This is primarily due to the uneven distribution of Chinese data, with a relatively small proportion of multiple disease labels. The micro F1 value is 0.7194, which is higher than the 0.5904 of the English dataset. This is mainly due to the large proportion of the labels "no obvious abnormality" and "increased lung texture" in the dataset. The generation accuracy of these labels significantly affects the micro F1 value, which is also a notable feature of the Chinese dataset constructed in this paper.

[0038]

[0039] Table 2 shows the comparison of clinical effect indicators between this study and recent studies on Chinese datasets, as shown above.

[0040] The data in Table 3 demonstrate that our proposed method achieves the best performance across all text degradation metrics. Compared to the Prior method, although both utilize prior knowledge, our proposed window-constrained contrastive learning algorithm achieves significant improvement in addressing text degradation, generating text with greater diversity. While the baseline model, Fluent, performs the worst on text diversity metrics, our proposed method achieves the highest text diversity thanks to its window-constrained contrastive learning algorithm, further demonstrating the effectiveness of our algorithm.

[0041]

[0042] Table 3 compares the text degradation indicators of this study and recent studies on Chinese datasets, as shown above.

[0043] In another embodiment, a preset chest X-ray report dataset is obtained; the chest X-ray report dataset is processed to obtain radiological features and correlation co-occurrence frequencies; the radiological features and correlation co-occurrence frequencies are processed to generate correlation matrix information; and a prior knowledge graph for representing chest images is generated based on the correlation matrix information. Radiological concepts and their correlations are extracted from the chest X-ray report dataset using data mining, and relationships between medical concepts are established to form a medical prior knowledge graph. Domain experts then verify the validity of the correlations between concepts one by one, completing the calibration of the correlation matrix and thus completing the construction of the medical prior knowledge graph.

[0044] In another embodiment, the present invention further includes a calculation formula for calculating the correlation co-occurrence frequency, wherein the specific calculation formula is:

[0045] Among them, C ij C represents the number of times concept i and concept j co-occur at the report level. j P represents the total number of times concept j appears. ij Represents the frequency of concept i when concept j appears. Based on the co-occurrence frequency P, the correlation matrix A can be constructed:

[0046] In addition, the present invention also includes a calculation formula for calculating the correlation matrix, and the specific calculation formula is:

[0047]

[0048] Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation. Domain experts check one by one whether the correlation between concept i and concept j is established, complete the calibration of the correlation matrix, and thus complete the construction of the medical prior knowledge graph.

[0049] In another embodiment, when constructing the medical prior knowledge graph, the number of high-frequency radiology concepts, m, is set to 40. When constructing the correlation matrix A, the co-occurrence frequency threshold, τ, is set to 0.5. A pretrained DenseNet121 is used as the image encoder, with an input image size of 256×256 pixels and image augmentation using only rotation operations. Dropout is set to 0.2. The graph convolutional network uses a two-layer GCN network, with 40 nodes in the prior knowledge graph and a feature dimension, e, of 256. The text encoder uses an 8-head, single-layer Transformer layer with a lexicon of 1000, 256 feedforward neurons, and a vector embedding dimension of 256. The classifier, B, uses an attention network with a 256-dimensional embedding vector. The number of output label categories, t, is 114, with the first 14 representing the disease label categories in the dataset and the last 100 representing statistically high-frequency medical terms. In the generator, the number of Transformer layers, N, is 12, the lexicon size, v, is 1000, and the feature dimension, d, is 256. The window parameter w in window-constrained contrastive learning is 4, and the window size 2w+1 is 9. The model is trained using the Adam optimizer, with a learning rate of 3e-4, a weight decay rate of 1e-2, a training cycle of 50 epochs, and a data batch size of 8. Chinese word segmentation uses the AutoTokenizer of the bert-base-chinese model with a vocabulary size of 21,128.

[0050] S103: Process the training sample set based on preset processing rules to generate a training set and a validation set.

[0051] In one implementation, we assume we are using the IU X-Ray dataset (preprocessed to remove unreported samples, convert letters to lowercase, and remove special symbols). This dataset contains 7,470 frontal and lateral chest X-ray images from 3,955 radiological examinations, along with the corresponding radiological reports (with the findings and impressions concatenated as the target text). This serves as our training sample set, where each sample consists of a chest X-ray image and the corresponding report text. We use the pretrained DenseNet121 as the image encoder, with an input image size of 256×256 pixels, to perform feature extraction on each chest X-ray image.

[0052] For an input chest X-ray image, after DenseNet121 encoding, an image feature vector (e.g., dimension 1024) is obtained. The feature vectors of all 7,470 images are combined to form a 7470x1024 matrix, which is the original feature library for the image portion. Clinical information (INDICATION & HISTORY) is input as text into the text encoder. The text encoder uses an 8-head single-layer Transformer layer with a dictionary size of 1000, 256 feedforward neurons, and a vector embedding dimension of 256. For each sample's clinical text information, the text encoder processes it to obtain a text feature vector (e.g., dimension 256). The text feature vectors of all samples are combined to form a 7470x256 matrix, which is the original feature library for the text portion. The original feature libraries of image and text are concatenated along the sample dimension to obtain a combined original feature library, whose dimensions may be 7470x(1024+256).

[0053] The dataset was divided into training, validation, and test sets in a ratio of 7:1:2 (focusing on the training and validation splits here). This resulted in approximately 5,229 samples in the training set and 747 samples in the validation set. From the integrated raw feature library, 5,229 rows (corresponding to the feature vectors of the 5,229 samples) were randomly selected as the training set, and the remaining 747 rows were used as the validation set. The distribution of disease-related features in the training and validation sets was ensured to be roughly similar. For example, the proportion of samples containing various disease conditions (such as lung disease and heart disease) in the training set should be similar to that in the entire dataset; the same was true for the validation set to ensure the effectiveness of subsequent evaluations.

[0054] In this model, classifier B is used for this task. Its network structure is an attention module. The output attention weights are used to calculate the multi-label classification loss during training. The number of output categories is 114 (the first 14 represent the disease labels in the dataset, and the last 100 represent statistically high-frequency medical terms). Classifier B is initialized, and appropriate parameters (such as those in the attention module) are set. The feature vectors of the 747 samples in the validation set are input to classifier B one by one. Classifier B uses the previously learned patterns (the parameters and feature representation relationships learned from the training set samples during training) to make predictions for each sample and output the probability of each sample belonging to each of the 114 categories. For example, for the first sample in the validation set, classifier B might predict a probability of 0.3 for "lung infection" (one of the disease labels), a probability of 0.1 for "cardiac hypertrophy," and so on. For each sample, this set of 114 class probabilities is obtained, which is the prediction result.

[0055] The training algorithm uses the Adam optimizer with a learning rate of 3e-4 and a weight decay rate of 1e-2 for model training. The model calculates the feature vectors and their corresponding labels (the actual disease category and relevant clinical information labels) for the 5229 samples in the training set according to its architecture. Image features are processed through a multi-view image encoder and a graph convolutional neural network (GCN) to obtain visual features that incorporate medical prior knowledge. Text features are fused with visual features and then passed through classifier B to obtain feature representations. The predicted report text is then generated using a text report generator. Simultaneously, corresponding losses (prior knowledge loss, clinical information loss, window constraint contrast loss, and report generation loss) are calculated for each module. Model parameters are then updated using the Adam optimizer based on the overall loss function (which combines these losses). During each training cycle (50 epochs), the model performs a single pass through the entire training set, continuously adjusting parameters to optimize model performance.

[0056] After each epoch in the training process, or after training is completed, the 747 samples in the validation set are input into the trained model. The model is calculated in the same way as the prediction process to obtain the prediction results for each sample in the validation set (such as the category probability output by classifier B and the report text generated by the text report generator). These prediction results can be compared with the true labels of the validation set samples (known disease categories and clinical information) to evaluate the performance of the model on the validation set. For example, natural language generation indicators (BLEU, METEOR, ROUGE-L, CIDEr, etc.) are calculated to measure the quality of the generated report text, and clinical effect indicators (F1 score, precision, recall, etc.) are calculated to evaluate the model's diagnostic accuracy and recall ability for the disease, thereby determining the performance of the model on the validation set, that is, the validation set class prediction results.

[0057] S104: Training the initial image report generation model based on the training set and the validation set to generate a target image report generation model.

[0058] In one embodiment, the training set is processed based on the window-constrained contrastive learning loss method to generate word-unit vector values; the initial image report generation model is trained based on the word-unit vector values ​​to generate a target image report generation model.

[0059] The target image report generation model includes an encoder and a decoder, which represents the feature representation V obtained by the encoder. emb As input, decode to generate radiology report: P word , H word =Generator(V emb); Generator is a radiology report generator, which consists of N Transformer layers and attention modules. The attention weight of the generator output is P word ∈R l×v , represents the probability of the model generating a text sequence, l represents the sequence length, and v represents the dictionary size. word ∈R l×d Is the feature representation corresponding to the generator output text sequence, that is, the word vector value corresponding to each word in the sequence, and d represents the dimension of the feature. The radiology report generator outputs the text sequence probability P word At the same time, it also outputs the feature representation H of each word in the text sequence word ∈R l×d , expanded to H word ={h1,h2,...,h l}, where h l Represents the feature representation corresponding to the l-th generated word.

[0060] Among them, the window-constrained contrastive learning loss method includes a calculation formula for obtaining the word element vector value. The specific calculation formula is: Among them, w is the window constraint parameter, and the window size is set to 2w+1. The optimization goal is to make h i It is similar only to itself, and the features outside the window represent h j Dissimilarities are achieved, thereby achieving calibration of the model in the word unit representation space, prompting the image report generation model to learn discriminative and isotropic word vector representations. w determines the size of the window, which is used to define which words are considered to be within the window (|ij|≤w) and which words are considered to be outside the window (|ij|>w). When calculating the contrast loss, only words outside the window will be used as negative samples to participate in the calculation, thereby constraining the contrast relationship between the word unit in the center of the window and the text outside the window; L represents the length of the generated text sequence, i represents the word unit position index, j represents the comparison word unit position index, and h i represents the feature representation of the i-th generated word, h j represents the feature representation of the jth generated word, and ρ is a predefined interval parameter used to control the interval size in the contrast loss calculation to ensure that the word representation learned by the model has sufficient discriminability. Specifically, it makes h i Similarity with itself (s(h i ,h i )) and h i and the word outside the window h j The similarity (s(h i ,h j )) to keep a certain interval between them, so that the model can learn different word representations and avoid generating repeated or similar texts; s(h i ,hi ) represents the similarity between the word and itself, s(h i ,h j ) represents the similarity between the word and the word outside the window.

[0061] Suppose we have a generated text sequence about a chest X-ray report, "the patient has a normal chest X-ray with clear lungs and no abnormal findings." The length of the generated text sequence (i.e., 8 words in the text are considered 8 tokens) is 2. If we set the window constraint parameter w = 2, then the window size is 2w + 1 = 5.

[0062] Assume that after processing by the model, we obtain the feature representation of each word (represented by a two-dimensional vector): h1 (corresponding to "the") = [0.2, 0.3]; h2 (corresponding to "patient") = [0.4, 0.1]; h3 (corresponding to "has") = [0.1, 0.2]; h4 (corresponding to "a") = [0.3, 0.4]; h5 (corresponding to "normal") = [0.5, 0.2]; h6 (corresponding to "chest") = [0.4, 0.3]; h7 (corresponding to "X-ray") = [0.35, 0.3]; h8 (corresponding to "with") = [0.25, 0.2].

[0063] To calculate the similarity between word units and their relationship with ρ, first calculate the similarity between a word unit and itself. For each word unit, its similarity with itself is 1. For example, s(h1,h1)=1, s(h2,h2)=1, etc.

[0064] Then calculate the similarity s(h i ,h j Taking h3 (“has”) as an example, the words outside its window are h1 (“the”), h7 (“X-ray”), and h8 (“with”).

[0065]

[0066] The impact of different ρ values ​​on loss calculation: Assuming ρ (positive value)

[0067] For i=3 (“has”), compute a portion of the window-constrained contrastive learning loss:

[0068] For i=1 (“the”), since the window constraint is satisfied, we calculate:

[0069] max{0,ρ-s(h3,h3)+s(h3,h1)}=max{0,0.3-1+0.63}=0.03;

[0070] For j=7 ("X-ray"), since |3-7|>2 satisfies the window constraint, we calculate:

[0071] max{0,ρ-s(h3,h3)+s(h3,h7)}=max{0,0.3-1+0.58}=0.01;

[0072] For j=8 (“with”), since |3-8|>2 satisfies the window constraint, we calculate:

[0073] max{0,ρ-s(h3,h3)+s(h3,h8)}=max{0,0.3-1+0.53}=0; (because

[0074] 0.3-1+0.53<0)

[0075] In this case, ρ = 0.3 encourages the model to further increase the similarity gap with the out-of-window tokens, making the feature representation of h3 more discriminative. During training, the model adjusts the feature representation of h3 based on this loss, for example, to [0.12, 0.18] (a new hypothetical value, actually calculated by the model using optimization algorithms such as gradient descent), to reduce the similarity with the out-of-window tokens and thus reduce the loss.

[0076] Assume ρ = -0.2 (negative value)

[0077] For i=3 (“has”), compute a portion of the window-constrained contrastive learning loss:

[0078] For j=1 (“the”), calculate:

[0079] max{0,ρ-s(h3,h3)+s(h3,h1)}=max{0,0.2-1+0.63}=0; (because

[0080] -0.2-1+0.63<0)

[0081] For j=7 ("X-ray"), calculate:

[0082] max{0,ρ-s(h3,h3)+s(h3,h7)}=max{0,-0.2-1+0.58}=0; (because

[0083] -0.2-1+0.58<0)

[0084] For j=8 ("with"), calculate:

[0085] max{0,ρ-s(h3,h3)+s(h3,h8)}=max{0,-0.2-1+0.53}=0; (because

[0086] -0.2-1+0.53<0)

[0087] When ρ = -0.2, the model allows h3 to maintain a certain degree of similarity with the tokens outside the window, which may help capture more subtle semantic associations between tokens. For example, in this example, "has" and "with" are semantically related ("has something with..."), and in this case the model may not over-adjust the feature representation to preserve this potential semantic relationship.

[0088] Impact on model training and word representation: During model training, similar calculations are performed for each word (from 1 to 8) in the entire text sequence, and all results are added together to obtain the total window-constrained contrastive learning loss. Based on this loss value, the model adjusts the feature representation of each token using optimization algorithms such as gradient descent. As training progresses, different values ​​of ρ guide the model to learn different token representations. When ρ takes a positive value, the model tends to separate tokens further in the feature space, making each token more distinct. This helps avoid text degradation issues such as repetition at the word, phrase, or sentence level in generated reports. For example, when describing chest X-ray images, it can accurately distinguish between key semantically distinct tokens such as "normal" and "abnormal," generating more accurate and clear reports.

[0089] When ρ takes a negative value, the model retains a certain degree of similarity between word units, which helps capture the relationships between semantically similar or related word units. For example, when describing symptoms, while "cough" and "respiratory distress" are not exactly the same, they may be related in some cases. An appropriate negative value of ρ allows the model to learn this relationship, allowing the generated report to more comprehensively and accurately describe the patient's symptoms and possible disease conditions, providing more valuable information for clinical diagnosis.

[0090] In another embodiment, multiple data sets are extracted from a training set, each data set comprising a preset number of data samples, at least one of which includes identification information, and the data corresponding to the identification information may be X-ray chest images of different lesion types. An initial imaging report generation model is trained based on the data samples in the multiple data sets to generate a trained imaging report generation model.

[0091] In addition, since the data used to train the imaging report generation model may be too small and the proportion of each stage in the training data is uneven, the StableDiffusion model is used to generate chest images. First, based on the real image used as the target distribution, that is, the stable distribution, according to the principle of the stable diffusion model, a stable diffusion model suitable for chest image generation is constructed. The model consists of multiple deconvolution layers and other neural network layers, and generates chest images in reverse through the reverse diffusion process. During the training process, starting from the stable distribution (that is, the real image distribution), the stable diffusion model is used to perform the reverse diffusion process. The reverse diffusion process will gradually blur the image until a virtual chest image corresponding to the stable distribution is obtained. The virtual chest image generated in this way has corresponding feature information, which can be used for subsequent lesion judgment and material component identification.

[0092] During the back-diffusion process, noise is introduced to control the diversity and randomness of the generated chest images. By adjusting the noise intensity, the clarity and style of the generated images can be influenced, resulting in more diverse virtual chest images. Furthermore, an initial noise setting is performed. Specifically, this initial noise can be random Gaussian noise or uniformly distributed noise. This introduces some randomness in the early stages of image generation, encouraging the model to generate diverse initial images. Next, the noise intensity is adjusted and gradually reduced during the generation process. This can be achieved by controlling the noise parameters in the generative network or by gradually reducing the standard deviation of the added noise during training. This gradual reduction in noise intensity helps the model improve image clarity and quality over the generation process. By introducing initial noise and gradually reducing its intensity, the model can generate more random and diverse images, facilitating the generation of a range of chest images with different characteristics or cases. It also helps stabilize the distribution of learning data for the diffusion model, making the trained model more resilient to noisy or incomplete inputs, thereby improving its robustness and making it more applicable to the various uncertainties encountered in the real world. The trained imaging report generation model is processed based on the validation set to generate a test result. If the data sample containing identification information in the test result indicates that the radiation dose of the current area is in an abnormal state, the trained imaging report generation model is used as the target imaging report generation model.

[0093] S105: Process the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features.

[0094] The multi-view image encoder in the target image report generation model encodes the chest X-ray image of the target patient and inputs the medical prior knowledge into the graph convolutional network. The prior knowledge loss function (T prior) enables the graph convolutional network module to better integrate prior knowledge and image information and enhance the image encoder capability. After the graph convolutional network module obtains the visual features containing prior knowledge, it selects nodes to obtain V prior ∈R k ×e (where k is the number of main nodes and e is the feature dimension), and input it into the classifier A (the network structure is a fully connected network) to obtain the probability P of each category prior =f A (V prior ), P prior ∈R k is the classification probability of k main nodes. The calculation formula for the target image report generation model to obtain the fused visual features is: Among them, y i Indicates the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, i is the main node index, and here y i ∈{0, 1} represents the i-th label value of the current sample, which is 1 if the label exists, otherwise 0; p i Represents the probability value of the i-th element.

[0095] For example, suppose that when processing a target chest X-ray image, the number of main nodes k = 5 and the feature dimension e = 10. After a series of calculations, the label values ​​of the current sample are y1 = 1, y2 = 0, y3 = 1, y4 = 0, y5 = 1 (indicating that the sample has relevant features in the 1st, 3rd, and 5th categories), and the corresponding probability values ​​p1 = 0.7, p2 = 0.3, p3 = 0.8, p4 = 0.2, and p5 = 0.9. The relevant probability values ​​of the fused visual features are calculated according to the formula (only some nodes are calculated in this example):

[0096]

[0097] In the text encoding part, the clinical information of the target patient (such as symptom description, etc.) is input into the text encoder to obtain text encoding features. The clinical information loss function (L txt ) enables the text encoder to fully acquire clinical text information and better integrate it with visual features. The classification probability can be obtained from the previously obtained fusion features through classifier B. Based on this, the clinical information loss function is defined. The calculation formula for obtaining text encoding features of the target imaging report generation model is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ijrepresents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0098] Assume we have a simple case where the total number of disease keywords for classification supervision is T = 4 and the dictionary size is v = 5.

[0099] Suppose there is a clinical information description of "the patient has a cough, accompanied by a low-grade fever, and no difficulty breathing". After being processed by the text encoder, the keywords "cough", "low-grade fever", "difficulty breathing" and other disease-related words have corresponding probability distributions (here it is assumed that the probability values ​​have been obtained after a certain processing, which is actually the result of complex calculations of the model).

[0100] For example, for the keyword "cough" (assuming it corresponds to the first keyword), the probability distribution in the dictionary is as follows: p1 = 0.6 (corresponding to the first word "cough" itself in the dictionary), p2 = 0.1 (corresponding to other words in the dictionary), p3 = 0.1, p4 = 0.1, p5 = 0.1.

[0101] For "low fever" (assuming it corresponds to the second keyword), the probability distribution is: p1 = 0.1, p2 = 0.5 (corresponding to "low fever"), p3 = 0.1, p4 = 0.1, p5 = 0.2.

[0102] For "difficulty breathing" (assuming it corresponds to the third keyword), the probability distribution is: p1 = 0.2, p2 = 0.2, p3 = 0.4 (corresponding to "difficulty breathing"), p4 = 0.1, p5 = 0.1.

[0103] (Assuming that the fourth keyword does not appear in the clinical information, the relevant probability distribution does not consider its impact for the time being)

[0104] At the same time, ij Indicates the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0.

[0105] For the keyword "cough":

[0106] When j = 1, y 11 =1 (because the first word in the dictionary is "cough"). When j = 2, 3, 4, 5, y 1j =0.

[0107] For the keyword "low heat":

[0108] When j = 2, y 22 = 1. When j = 1, 3, 4, 5, y 2j =0.

[0109] For the keyword "difficulty breathing":

[0110] When j = 3, y 33 = 1. When j = 1, 2, 4, 5, y 3j =0.

[0111] Calculate text encoding features according to the formula:

[0112]

[0113] Through this calculation process, using the calculation formula for text encoding features, combined with the probability distribution and label value of clinical information in the dictionary, the text encoding feature value corresponding to the clinical information is obtained (this is just a simplified example; the actual model calculation will be more complex and accurate). This feature value will be used together with the fused visual features for subsequent operations such as generating imaging diagnostic reports, helping to comprehensively consider clinical information and chest X-ray image information, and improving the accuracy and reliability of diagnostic reports. Through this process, the target imaging report generation model can process the target chest X-ray image, generate fused visual features and text encoding features, and provide a foundation for the subsequent generation of imaging diagnostic reports.

[0114] In one embodiment, the multi-view image encoder encodes the target chest X-ray image and inputs it into the graph convolutional network with the medical prior knowledge obtained above, thereby obtaining fused visual features. Medical prior knowledge loss function T prior , this loss function is designed to enable the graph convolutional network module to better integrate medical prior knowledge and image information to enhance the image encoder capability. The graph convolutional network module obtains the visual features v containing medical prior knowledge prior After that, node selection is performed to obtain And input it into classifier A to get the probability of each category: P prior =f A (v″ prior ), the network structure f of classifier A A is a fully connected network. is the classification probability of k main nodes. The target image report generation model includes a calculation formula for obtaining fused visual features. The specific calculation formula is: Among them, y i Indicates the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the k-th medical prior knowledge graph, i is the main node index, and y i ∈{0,1} represents the i-th label value of the current sample, which is 1 if the label exists and 0 otherwise.

[0115] In another embodiment, in the text encoding part, the clinical information is input into the text encoder to obtain the text encoding features, and the clinical information loss function This loss function is designed to enable the text encoder to fully acquire clinical text information and better integrate it with visual features. From a clinical perspective, this is consistent with the practice of radiologists who usually use clinical information as assistance and guidance when observing new chest radiographs. fusion The classification probability can be obtained through classifier B Based on this, the clinical information loss function L can be defined txt ,The target image report generation model includes a calculation formula for obtaining text encoding features, and the specific calculation formula is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0116] S106: Process the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0117] In one embodiment, the loss function φ is generated by obtaining the report word , the loss function supervises the consistency between the report text generated by the model and the reference report text. It is the basic loss function for all chest radiograph report automatic generation tasks. The above content details how to obtain the generated text probability from the generator.

[0118] Based on the report generation loss function, the fused visual features and text encoding features are processed to generate an imaging diagnosis report. During training, the text probability obtained in the generator is used to calculate the generation loss to supervise the generation of the imaging diagnosis report text. During inference, the imaging diagnosis report is generated based on the probability distribution search. In addition, the target image includes a generation model including a calculation formula for obtaining the report generation loss function, which is: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ijrepresents the probability that the i-th word corresponds to the j-th word in the dictionary. An example of the final imaging diagnostic report is as follows: The bilateral lung fields are clear, no obvious consolidation is observed in the lungs, the hilar shadow is small, the mediastinum is centered, the size and shape of the heart shadow are normal, the bilateral diaphragmatic surfaces are smooth, and the costophrenic angles are sharp.

[0119] In this application, the server obtains the image information to be detected and the initial image report generation model that matches the target chest X-ray image, wherein the image information to be detected includes the chest X-ray image of the target patient and the clinical information of the target patient; obtains a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing the chest image and the chest image to be trained, wherein the chest image to be trained includes clinical information of different patients; processes the training sample set based on preset processing rules to generate a training set and a validation set; trains the initial image report generation model based on the training set and the validation set to generate a target image report generation model; processes the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processes the fused visual features and text encoding features based on the target image report generation model to generate an image diagnosis report. A deep learning network is used to encode the features of prior knowledge graphs, chest X-rays, and clinical information. The received encoded information is decoded by a decoder. During the decoding process, an algorithm is used to limit text generation and reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through the loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0120] Optionally, in another embodiment of the above method of the present application, the processing of the training sample set based on a preset processing rule to generate a training set and a validation set further includes:

[0121] Dividing the training set to generate a number of class samples;

[0122] Obtain any number of samples of several categories in the training set;

[0123] Generate adjacent samples based on the distance between the any number of class samples and other number of class samples of the same category, wherein the adjacent samples include a preset number of the any number of class samples;

[0124] Sampling the adjacent samples to generate a preset number of sampling samples;

[0125] Based on any minority class sample and each sampling sample, multiple data groups are generated, wherein each data group includes a preset number of data samples.

[0126] In one embodiment, the distance between any minority class sample and each other minority class sample of the same category can be a Euclidean distance or other cosine distance, which is not specifically limited here. All other minority class samples can be arranged in order of distance from the other minority class samples, from closest to farthest, and then the data in the first few positions of the arrangement result are selected as adjacent samples. The purpose of sampling is to increase the number of minority class samples to achieve a balance between the number of minority class samples and the number of majority class samples in the initial sample set. Here, oversampling can be achieved by a sampling algorithm, which is not specifically limited here. After oversampling, multiple data groups can be determined based on the results of oversampling and combined with the training set. At this point, the number of descriptive data samples corresponding to each attribute category label in the data group is equivalent, and there are no minority class samples or majority class samples. This embodiment first performs a preliminary partitioning of the training set and then merges the minority class samples with the majority class samples.

[0127] Optionally, in another embodiment of the above method of the present application, sampling the adjacent samples to generate a preset number of sampled samples includes:

[0128] Determining a sampling ratio based on the number of samples of each class in the training set;

[0129] determining a sampling ratio based on the sampling ratio;

[0130] The adjacent samples are sampled based on the sampling ratio to generate a preset number of sampling samples.

[0131] In one embodiment, first obtain which data groups are included in the training set, confirm how many number class samples are in each data group, and determine the sampling ratio based on the number of number class samples in each data group, and then use the sampling ratio to determine the sampling ratio. The sampling ratio can be set as needed, for example, it can be 50% or other ratios, which are not specifically limited here. Finally, use the sampling ratio to sample a preset number of adjacent samples to obtain a specified number of sampled samples. All other smaller number class samples can be arranged in order of distance from any smaller number class sample, and then select the preset number of samples ranked first in the arrangement result as the nearest neighbor samples. The preset number can be set according to actual conditions and is not specifically limited here. Based on any smaller number of number class samples and each sampled sample, a new sample is generated and used as a new data group, which can further improve the balance between the smaller number of data groups and the other larger number of data groups.

[0132] By applying the above technical solution, the server first obtains the image information to be detected (including the target patient's chest X-ray image and clinical information) and the initial image report generation model, and obtains a preset chest X-ray report data set. The data set is processed to obtain radioactive features and correlation co-occurrence frequencies, and then generates correlation matrix information. Based on this, a prior knowledge graph representing the chest image is constructed. The training sample set contains this graph and clinical information of different patients. Next, feature extraction is performed on the training sample set to generate an original feature library, which is then divided into a training set and a validation set. A classifier is used to predict the validation set to determine the results, and the training set is trained using a preset algorithm to obtain the validation set class prediction results. The training set is processed using a window-constrained contrastive learning loss method to generate word element vector values ​​to train the initial model to obtain the target model. This method contains a specific calculation formula. The target model processes the target chest X-ray image to generate fused visual features (with a corresponding calculation formula) and text encoding features (also with a calculation formula). After obtaining the report generation loss function (with its calculation formula), the fused features are processed based on this to generate an imaging diagnostic report. This process integrates multiple technologies and computational methods through multi-step data processing, model building, and training, aiming to improve the accuracy and effectiveness of chest X-ray report generation and provide more reliable support for clinical diagnosis. A deep learning network is used to encode features of the prior knowledge graph, chest X-rays, and clinical information. A decoder then decodes the received encoded information. During the decoding process, an algorithm is used to restrict text generation and reduce repetition. During model training, a loss function is used to continuously improve the distance between the output value and the true value, thereby improving the accuracy and reliability of medical imaging report generation.

[0133] In one embodiment, Figure 2 As shown, the present application also provides an imaging report generating device based on chest X-ray, comprising:

[0134] An acquisition module 201 is configured to acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; and to acquire a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for representing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients;

[0135] The processing module 202 is used to process the training sample set based on preset processing rules to generate a training set and a validation set; train the initial image report generation model based on the training set and the validation set to generate a target image report generation model; process the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; process the fused visual features and the text coding features based on the target image report generation model to generate an image diagnosis report.

[0136] The present application embodiment provides an electronic device, such as Figure 3 As shown, the electronic device 3 includes a first processor 300, a memory 301, a bus 302 and a communication interface 303. The first processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the first processor 300, and when the first processor 300 runs the computer program, it executes the method for generating an image report based on an X-ray chest film provided in any of the aforementioned embodiments of the present application.

[0137] The memory 301 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0138] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. The memory 301 is used to store programs, and the first processor 300 executes the programs upon receiving execution instructions. The method for generating an image report based on a chest X-ray film disclosed in any of the aforementioned embodiments of the present application may be applied to the first processor 300 or implemented by the first processor 300.

[0139] The first processor 300 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 300. The above-mentioned first processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 301 , and the first processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0140] The electronic device provided in the above-mentioned embodiment of the present application and the method for generating an imaging report based on an X-ray chest film provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0141] The present application provides a computer-readable storage medium. Figure 4 As shown, the computer-readable storage medium 401 stores a computer program, and when the computer program is read and executed by the second processor 402, the aforementioned method for generating an imaging report based on an X-ray chest film is implemented.

[0142] The technical solution of the embodiments of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing an electronic device (such as an air conditioner, a refrigeration device, a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, ROM, RAM, a magnetic disk, or an optical disk.

[0143] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the method for generating an imaging report based on an X-ray chest film provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0144] An embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is executed by a third processor to implement the method described above.

[0145] The computer program product provided in the above-mentioned embodiments of the present application and the method for generating an imaging report based on an X-ray chest film provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0146] Each embodiment of this application is described in a related manner. Similar portions between the various embodiments may be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the embodiments of the method for generating an image report based on an X-ray chest film, the electronic device, the electronic device, and the readable storage medium are generally similar to the embodiments of the method for generating an image report based on an X-ray chest film, so the description is relatively simple. For related portions, refer to the partial description of the embodiment of the method for generating an image report based on an X-ray chest film.

[0147] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims.

Claims

1. A method for generating an imaging report based on an X-ray chest film, characterized in that: include: Acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; Obtaining a training sample set for training an initial imaging report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; Processing the training sample set based on preset processing rules to generate a training set and a validation set; The initial image report generation model is trained based on the training set and the validation set to generate a target image report generation model, including processing the training set based on a window-constrained contrastive learning loss method to generate a word unit vector value; the initial image report generation model is trained based on the word unit vector value to generate a target image report generation model; wherein the calculation formula for obtaining the word unit vector value is: Among them, w is the window constraint parameter, L represents the length of the generated text sequence, i represents the word position index, j represents the comparison word position index, h i represents the feature representation of the i-th generated word, h i represents the feature representation of the jth generated word, ρ is a predefined interval parameter, s(h i ,h i ) represents the similarity between the word and itself, s(h j ,h j ) represents the similarity between the word and the word outside the window; Processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; The fused visual features and the text encoding features are processed based on the target image report generation model to generate an image diagnosis report.

2. The method according to claim 1, wherein The step of obtaining a training sample set for training an initial imaging report generation model includes: Obtain the preset chest X-ray report data set; Processing the chest X-ray report dataset to obtain radiological features and correlation co-occurrence frequencies; Processing the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information; A priori knowledge graph for characterizing chest images is generated based on the correlation matrix information.

3. The method according to claim 2, wherein The processing of the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information includes: The method includes a calculation formula for calculating the correlation co-occurrence frequency, which is: Among them, C ij represents the number of times concept i and concept j co-occur at the report level, C j represents the total number of times concept j appears, P ij Indicates the frequency of concept i appearing when concept j appears: The method includes a calculation formula for calculating a correlation matrix, which is: Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation.

4. The method according to claim 1, wherein The processing of the training sample set based on the preset processing rules to generate a training set and a validation set includes: Extracting features from the training sample set to generate an original feature library; Divide the training sample set according to the original feature library to generate a training set and a validation set; Use the classifier to divide the original feature library into various verification sets for prediction and determine the prediction results; Use the preset algorithm to divide the original feature library into various training sets for training and obtain the prediction results of the validation set class.

5. The method according to claim 1, wherein The target chest X-ray image is processed based on the target image report generation model to generate fused visual features and text coding features, including: The target image report generation model includes a calculation formula for obtaining fused visual features, which is: Among them, y i Indicates the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, and i is the main node index; The target image report generation model includes a calculation formula for obtaining text encoding features, which is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

6. The method according to claim 1, wherein The processing of the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report includes: Get report to generate loss function; Processing the fused visual features and the text encoding features based on the report generation loss function to generate an imaging diagnosis report; The target image includes a generation model including a calculation formula for obtaining a report generation loss function, the calculation formula being: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ij Represents the probability that the i-th word corresponds to the j-th word in the dictionary.

7. An imaging report generating device based on chest X-ray, characterized in that: For implementing the method of claim 1, the apparatus comprises: an acquisition module, configured to acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; and to acquire a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for representing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients; A processing module is used to process the training sample set based on preset processing rules to generate a training set and a validation set; train the initial image report generation model based on the training set and the validation set to generate a target image report generation model; process the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; and process the fused visual features and the text coding features based on the target image report generation model to generate an image diagnosis report.

8. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; The first processor is configured to execute the method for generating an imaging report based on an X-ray chest film according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the method for generating an imaging report based on an X-ray chest film according to any one of claims 1 to 6 is implemented.

Images