Image report generation method based on X-ray chest radiography and related equipment

Abstract

The invention provides an X-ray chest radiography-based image report generation method and related equipment, and is applied to the technical field of data processing. According to the method, to-be-detected image information and an initial image report generation model matched with a target chest radiograph image are obtained, the to-be-detected image information comprises the chest radiograph image of a target patient and clinical information of the target patient, and the initial image report generation model comprises an encoder and a decoder; obtaining a training sample set for training the initial image report generation model; processing the training sample set based on a preset processing rule to generate a training set and a verification set; training the initial image report generation model based on the training set and the verification set, and generating a target image report generation model; processing the target chest radiograph image based on a target image report generation model to generate fused visual features and text coding features; and processing the fused visual features and the text coding features based on a target image report generation model to generate an image diagnosis report.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2024110587576 filed with the State Intellectual Property Office on August 2, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the technical field of data processing, and in particular to a method for generating an image report based on an X-ray chest film and related equipment. Background Art

[0004] Medical imaging reports are the main medium for radiologists to convey patients' medical scan results and diagnostic results, and have important reference value for clinical diagnosis and treatment. Therefore, radiologists need to write these reports accurately and in detail. In recent years, image-text matching methods based on deep learning have made great breakthroughs, which can be roughly divided into one-to-one global matching methods and many-to-many fine-grained matching methods; one-to-one global matching methods usually extract global feature representations of images and sentences, and then embed the global features of images and sentences into a joint space, and measure their similarity through the feature distance of the joint space. This method represents the entire image or text with a global vector, ignoring that the key information in the image may be a few key areas or the key information in the text may be a key word. It lacks fine-grained exploration of image and text features, resulting in mediocre performance.

[0005] In addition, the current research on medical imaging report generation technology is still in its infancy. In existing medical imaging report generation schemes, medical knowledge graphs are usually used for subtasks such as classification, but are not integrated into the model, resulting in low accuracy and reliability in improving medical imaging report generation. Most existing text generation works use maximum likelihood estimation (MLE) to train language models. However, this modeling method usually leads to text degradation problems, that is, the generated radiology reports often contain unnecessary text repetitions, which may occur at the word level, phrase level, or sentence level. In previous work related to automatic report generation, no work has been specifically studied on this issue.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The purpose of this application is to provide an imaging report generation method based on chest X-ray films and related equipment, which at least overcomes the problems existing in the prior art to a certain extent, uses a deep learning network to encode the features of a priori knowledge graph, chest X-ray, and clinical information, decodes the received encoded information through a decoder, and uses an algorithm to limit text generation during the decoding process to reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through a loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0008] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0009] According to one aspect of the present application, a method for generating an image report based on an X-ray chest film is provided, comprising: obtaining image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; obtaining a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; processing the training sample set based on preset processing rules to generate a training set and a verification set; training the initial image report generation model based on the training set and the verification set to generate a target image report generation model; processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0010] In one embodiment of the present application, the acquisition of a training sample set for training an initial imaging report generation model includes: acquiring a preset chest X-ray report data set; processing the chest X-ray report data set to obtain radioactive features and correlation co-occurrence frequencies; processing the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information; and generating a priori knowledge graph for characterizing chest images based on the correlation matrix information.

[0011] In one embodiment of the present application, the processing of the radioactive features and the correlation co-occurrence frequency to generate correlation matrix information includes: the method includes a calculation formula for calculating the correlation co-occurrence frequency, and the calculation formula is: Among them, C ij Indicates the number of times concept i and concept j co-occur at the report level. j represents the total number of times concept j appears. Pij represents the frequency of concept i appearing when concept j appears. A correlation matrix A can be constructed based on the co-occurrence frequency P: the method includes a calculation formula for calculating the correlation matrix, and the calculation formula is: Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation.

[0012] In one embodiment of the present application, the training sample set is processed based on preset processing rules to generate a training set and a verification set, including: extracting features from the training sample set to generate an original feature library; dividing the training sample set according to the original feature library to generate a training set and a verification set; using a classifier to predict each verification set divided from the original feature library to determine a prediction result; using a preset algorithm to train each training set divided from the original feature library to obtain a verification set class prediction result.

[0013] In one embodiment of the present application, the initial image report generation model is trained based on the training set and the validation set to generate a target image report generation model, including: processing the training set based on a window constrained contrast learning loss method to generate a word unit vector value; training the initial image report generation model based on the word unit vector value to generate a target image report generation model; wherein the window constrained contrast learning loss method includes a calculation formula for obtaining a word unit vector value, and the calculation formula is: Among them, w is the window constraint parameter, L represents the length of the generated text sequence, i represents the word position index, j represents the comparison word position index, and h i represents the feature representation of the i-th generated word, h j represents the feature representation of the jth generated word, ρ is a predefined interval parameter, s(h i ,h i ) represents the similarity between the word and itself, s(h i ,h j ) represents the similarity between the word and the word outside the window.

[0014] In one embodiment of the present application, the target chest X-ray image is processed based on the target image report generation model to generate fused visual features and text coding features, including: the target image report generation model includes a calculation formula for obtaining fused visual features, and the calculation formula is: Among them, y i represents the i-th label value of the current sample, p irepresents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, and i is the main node index; the target imaging report generation model includes a calculation formula for obtaining text encoding features, and the calculation formula is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0015] In one embodiment of the present application, the target image report generation model processes the fused visual features and the text encoding features to generate an image diagnosis report, including: obtaining a report generation loss function; processing the fused visual features and the text encoding features based on the report generation loss function to generate an image diagnosis report; the target image includes a generation model including a calculation formula for obtaining a report generation loss function, and the calculation formula is: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index, which is used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ij Represents the probability that the i-th word corresponds to the j-th word in the dictionary.

[0016] Another aspect of the present application is an image report generation device based on an X-ray chest film, characterized in that it includes: an acquisition module, used to acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; acquiring a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; a processing module, used to process the training sample set based on preset processing rules to generate a training set and a verification set; training the initial image report generation model based on the training set and the verification set to generate a target image report generation model; processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0017] According to another aspect of the present application, an electronic device is provided, characterized in that it comprises: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to implement the above-mentioned method for generating an image report based on a chest X-ray by executing the executable instructions.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a second processor, the above-mentioned method for generating an image report based on a chest X-ray is implemented.

[0019] According to another aspect of the present application, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a third processor, the computer program implements the above-mentioned method for generating an image report based on a chest X-ray.

[0020] The present application provides an image report generation method based on an X-ray chest film and related equipment, wherein a server obtains image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient, and the initial image report generation model includes an encoder and a decoder; a training sample set for training the initial image report generation model is obtained, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients; the training sample set is processed based on preset processing rules to generate a training set and a verification set; the initial image report generation model is trained based on the training set and the verification set to generate a target image report generation model; the target chest X-ray image is processed based on the target image report generation model to generate fused visual features and text encoding features; the fused visual features and text encoding features are processed based on the target image report generation model to generate an image diagnosis report. A deep learning network is used to encode the features of prior knowledge graphs, chest X-rays, and clinical information. The received encoded information is decoded by a decoder. During the decoding process, an algorithm is used to limit text generation and reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through the loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a method for generating an image report based on an X-ray chest film provided in an embodiment of the present application is shown;

[0023] Figure 2 A schematic structural diagram of an imaging report generating device based on an X-ray chest film provided in an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0027] Combine the following Figure 1 To describe the method for generating an image report based on an X-ray chest film according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0028] In one embodiment, the present application also proposes a method for generating an imaging report based on a chest X-ray and related equipment. Figure 1 The following schematically shows a flow chart of a method for generating an image report based on an X-ray chest film according to an embodiment of the present application. Figure 1 As shown, the method is applied to a server, comprising:

[0029] S101, obtaining image information to be detected and an initial image report generation model that matches the target chest X-ray image.

[0030] In one embodiment, the image information to be detected includes a chest X-ray image of the target patient and clinical information of the target patient, wherein the clinical information of the target patient includes the information contained in the application form issued by the clinical department when the target patient comes for treatment (for example, symptoms such as chest pain and cough), and the chest X-ray image of the target patient is the image information obtained from the latest physical examination of the target patient, which is processed by a subsequent target image report generation model to generate a diverse image report.

[0031] S102, obtaining a training sample set for training an initial image report generation model.

[0032] In one embodiment, the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients. The training sample set includes information of several standard substances. In addition, the data of the training sample set includes patient data from different hospitals, and is divided into a training group (281 cases) and an internal test group (122 cases) at a ratio of 7:3 using a random number table method, wherein the identification information is used to characterize the attributes of various types of material information, and the initial image report generation model is trained based on the training sample set with identification information to generate a target image report generation model. Specifically, the data classification result corresponding to the training sample set is obtained, because the classification result of each training sample set can be determined in advance, and the data can be directly obtained from the outside. Compare the prediction result with the data classification result to determine the first comparison result; compare the prediction result of the validation set class with the data classification result to determine the second comparison result; judge whether the second comparison result meets the preset requirements with the first comparison result. When the preset requirements are met, it indicates that the detection result of the current image report generation model is relatively accurate. At this time, the current image report generation model can be used as the target image report generation model.

[0033] In another embodiment, the dataset used in the experiment is the Chinese Chest X-ray Report Generation Dataset (CCXRD). The evaluation indicators adopt indicators in the field of natural language generation (NLG), including BLEU, METEOR, ROUGE-L and CIDEr. In order to comprehensively evaluate the clinical effectiveness (CE), the F1 score, precision and recall rate of the generated report containing the disease were calculated. At the same time, the applicant also reported the evaluation results at the macro and micro levels. In order to evaluate the text quality of the generated report, the applicant used the rep-n indicator to evaluate the repetition of the report, and used the diversity indicator to measure the diversity of the text.

[0034] In the comparative experiments, due to the fact that some models are not public or cannot be reproduced, compared with the comparative experiments on English datasets, only three advanced algorithms, R2Gen, Fluent and Prior, are compared on the Chinese dataset. As shown in Table 1, the model proposed in this paper outperforms the comparative algorithms in all NLG indicators.

[0035]

[0036] Table 1 compares the NLG indicators of this study and recent studies on Chinese datasets, as shown above.

[0037] Table 2 shows the macro and micro F1 score results, and our method performs best in both metrics. In terms of micro precision, R2Gen performs best, which indicates that its memory-driven module does a better job of retaining the information of the two main categories of "no obvious abnormality" and "increased lung texture", resulting in the model's excellent performance in generating this category, thereby improving micro precision, but the lower performance of macro precision reflects that the method only works for the main category and performs poorly for the secondary category, resulting in the lowest macro metric result. This shows that the high value of micro precision does not fully represent the superiority of the model due to the high proportion of normal samples and "increased lung texture" samples in the dataset. Compared with the baseline model Fluent, our method has improved both precision and recall, thanks to the use of medical prior knowledge to enhance the image representation ability, resulting in high-quality text reports. At the same time, the macro F1 value of the Chinese dataset is 0.2814, which is lower than 0.4199 of the English dataset, mainly because the distribution of Chinese data is uneven and the proportion of multiple disease labels is small. The micro F1 value is 0.7194, which is higher than 0.5904 of the English dataset. This is mainly due to the large proportion of labels "no obvious abnormality" and "increased lung texture" in the dataset. The generation accuracy of this label significantly affects the micro F1 value, which is also a notable feature of the Chinese dataset constructed in this paper.

[0038]

[0039] Table 2 is a comparison of the clinical effect indicators of this study and recent studies on the Chinese data set, as shown above.

[0040] The data in Table 3 show that the proposed method has achieved the best performance in all text degradation indicators. Compared with the Prior method, although both use prior knowledge, the window-constrained contrastive learning algorithm proposed in this paper has achieved significant improvements in solving the text degradation problem, and the generated text has higher diversity. Although the baseline model Fluent performs worst in the text diversity indicator, the proposed method achieves the highest text diversity due to the use of the window-constrained contrastive learning algorithm, which further confirms the effectiveness of the proposed algorithm.

[0041]

[0042] Table 3 is a comparison of text degradation indicators between this study and recent studies on Chinese datasets, as shown above.

[0043] In another embodiment, a preset chest X-ray report data set is obtained; the chest X-ray report data set is processed to obtain radiological features and correlation co-occurrence frequencies; the radiological features and correlation co-occurrence frequencies are processed to generate correlation matrix information; and a prior knowledge graph for characterizing chest images is generated based on the correlation matrix information. Radiological concepts and their correlations are extracted from the chest X-ray report data set using data mining, and relationships between medical concepts are established to form a medical prior knowledge graph. Then, domain experts check one by one whether the correlations between concepts hold, complete the calibration of the correlation matrix, and thus complete the construction of the medical prior knowledge graph.

[0044] In another embodiment, the present invention further includes a calculation formula for calculating the correlation co-occurrence frequency, wherein the specific calculation formula is:

[0045] Among them, C ij Indicates the number of times concept i and concept j co-occur at the report level. j P represents the total number of times concept j appears. ij It represents the frequency of concept i when concept j appears. Based on the co-occurrence frequency P, the correlation matrix A can be constructed:

[0046] In addition, the present invention also includes a calculation formula for calculating the correlation matrix, and the specific calculation formula is:

[0047]

[0048] Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation. Domain experts check one by one whether the correlation between concept i and concept j is established, complete the calibration of the correlation matrix, and thus complete the construction of the medical prior knowledge graph.

[0049] In another embodiment, in the stage of constructing the medical prior knowledge graph, the number of high-frequency radiology concepts m is set to 40. When constructing the correlation matrix A, the co-occurrence frequency threshold τ is set to 0.5, the pre-trained Densenet121 is used as the image encoder, the input image size is 256×256 pixels, and the image enhancement only uses the rotation operation; Dropout is set to 0.2. The graph convolution network uses a two-layer GCN network, the number of prior knowledge graph nodes is 40, and the feature dimension e is 256. The text encoder uses an 8-head single-layer Transformer layer, the dictionary size is 1000, the number of feedforward neurons is 256, and the vector embedding dimension is also 256. The classifier B uses an attention network with a 256-dimensional embedding vector, and the number of output label categories t is 114, of which the first 14 represent the disease label categories of the data set, and the last 100 represent the high-frequency medical terms obtained by statistics. In the generator part, the number of Transformer layers N is 12, the dictionary size v is 1000, and the feature dimension d is 256. The window parameter w in window-constrained contrastive learning is 4, and the window size 2w+1 is 9. The model training uses the Adam optimizer, the learning rate is 3e-4, the weight decay rate is 1e-2, the training cycle is 50 epochs, the data batch size is 8, and the Chinese word segmentation uses the AutoTokenizer segmenter of the bert-base-chinese model, and the dictionary size is 21,128.

[0050] S103, processing the training sample set based on preset processing rules to generate a training set and a validation set.

[0051] In one implementation, assume that we use the IU X-Ray dataset (after preprocessing, removing samples without reports, converting letters to lowercase, and removing special symbols), which contains 7,470 frontal and lateral chest X-ray images of 3,955 radiological examinations, and the corresponding radiological reports (the findings and impressions are concatenated as the target text). This is our training sample set, where each sample contains a chest X-ray image and the corresponding text report. Use the pre-trained Densenet121 as the image encoder, the input image size is 256×256 pixels, and perform feature extraction on each chest X-ray image.

[0052] For an input chest X-ray image, after Densenet121 encoding, an image feature vector (e.g., dimension 1024) is obtained. The feature vectors of all 7,470 images are combined to form a 7470x1024 matrix, which is the original feature library of the image part. Clinical information (INDICATION & HISTORY) is used as text input to the text encoder. The text encoder uses an 8-head single-layer Transformer layer, with a dictionary size of 1000, a number of feedforward neurons of 256, and a vector embedding dimension of 256. For the clinical text information in each sample, after being processed by the text encoder, a text feature vector (e.g., dimension 256) is obtained. The text feature vectors of all samples are combined to form a 7470x256 matrix, which is the original feature library of the text part. The original feature libraries of the image and text are spliced ​​on the sample dimension to obtain a comprehensive original feature library, whose dimension may be 7470x(1024+256).

[0053] The dataset is divided into training set, validation set and test set in a ratio of 7:1:2 (here we focus on the division of training set and validation set), so the training set contains about 5229 samples and the validation set contains about 747 samples. In the comprehensive original feature library, 5229 rows (corresponding to the feature vectors of 5229 samples) are randomly selected as the training set, and the remaining 747 rows are used as the validation set. Ensure that the distribution of various disease-related features in the training set and the validation set is roughly similar. For example, in the training set, the proportion of samples containing various disease conditions (such as lung disease, heart disease, etc.) should be similar to that in the entire dataset; the same is true for the validation set to ensure the effectiveness of subsequent evaluation.

[0054] In this model, classifier B is used for this task. Its network structure is an attention module. The output attention weight is used for multi-label classification loss calculation during training. The number of output categories is 114 (the first 14 represent the disease label categories of the dataset, and the last 100 represent the high-frequency medical terms obtained by statistics). Initialize classifier B, set appropriate parameters (such as parameters in the attention module, etc.), and input the feature vectors of 747 samples in the validation set into classifier B one by one. Classifier B predicts each sample based on the previously learned pattern (the parameters and feature representation relationship obtained by learning the training set samples during the training process), and outputs the probability that each sample belongs to each category (114 categories). For example, for the first sample in the validation set, classifier B may predict that its probability of belonging to "pulmonary infection" (one of the disease label categories) is 0.3, and its probability of belonging to "cardiac hypertrophy" is 0.1, etc. For each sample, a set of probability values ​​of 114 categories is obtained, which is the prediction result.

[0055] The training algorithm uses the Adam optimizer with a learning rate of 3e-4 and a weight decay rate of 1e-2 for model training. For the feature vectors of 5229 samples in the training set and their corresponding labels (real disease categories and relevant clinical information labels), the model is calculated according to its architecture. The image features are passed through the multi-view image encoder and the graph convolutional neural network (GCN) to obtain visual features that integrate medical prior knowledge. The text features are fused with the visual features and then passed through the classifier B to obtain feature representation. The predicted report text is then generated through the text report generator. At the same time, the corresponding losses (prior knowledge loss, clinical information loss, window constraint contrast loss, report generation loss) are calculated in each module, and the model parameters are updated through the Adam optimizer according to the total loss function (combining the above losses). In each training cycle (50 epochs in total), the model performs a traversal learning of the entire training set and continuously adjusts the parameters to optimize the model performance.

[0056] After each epoch in the training process, or after the training is completed, the 747 samples in the validation set are input into the trained model, and the model is calculated in the same way as the prediction process to obtain the prediction results of each sample in the validation set (such as the category probability output by classifier B, and the report text generated by the text report generator). These prediction results can be compared with the true labels of the validation set samples (known disease categories and clinical information) to evaluate the performance of the model on the validation set, such as calculating natural language generation indicators (BLEU, METEOR, ROUGE-L, CIDEr, etc.) to measure the quality of the generated report text, and by calculating clinical effect indicators (F1 score, precision, recall, etc.) to evaluate the model's diagnostic accuracy and recall ability for the disease, thereby determining the performance of the model on the validation set, that is, the validation set class prediction results.

[0057] S104: Training the initial image report generation model based on the training set and the validation set to generate a target image report generation model.

[0058] In one implementation, a training set is processed based on a window-constrained contrastive learning loss method to generate word-unit vector values; an initial image report generation model is trained based on the word-unit vector values ​​to generate a target image report generation model.

[0059] The target image report generation model includes an encoder and a decoder, which represents the feature representation V obtained by the encoder. emb As input, decode to generate radiology report: P word , H word =Generator(V emb), where Generator is a radiology report generator, consisting of N Transformer layers and an attention module. The attention weight of the generator output is P word ∈R l×v , represents the probability of the model generating a text sequence, l represents the sequence length, and v represents the dictionary size. word ∈R l×d is the feature representation corresponding to the output text sequence of the generator, that is, the word vector value corresponding to each word in the sequence, and d represents the dimension of the feature. word At the same time, it also outputs the feature representation H of each word in the text sequence word ∈R l×d , which is expanded to H word ={h1,h2,...,h l}, where h l Represents the feature representation corresponding to the lth generated word.

[0060] Among them, the window-constrained contrastive learning loss method includes a calculation formula for obtaining the word element vector value, and the specific calculation formula is: Among them, w is the window constraint parameter, and the window size is set to 2w+1. The optimization goal is to make h i It is similar only to itself, and the features outside the window represent h j Dissimilar, thereby achieving calibration of the model in the word unit representation space, prompting the image report generation model to learn discriminative and isotropic word vector representations. w determines the size of the window, which is used to define which words are considered to be within the window (|ij|≤w) and which words are considered to be outside the window (|ij|>w). When calculating the contrast loss, only words outside the window will be used as negative samples to participate in the calculation, thereby constraining the contrast relationship between the word at the center of the window and the text outside the window; L represents the length of the generated text sequence, i represents the word unit position index, j represents the comparison word unit position index, and h i represents the feature representation of the i-th generated word, h j represents the feature representation of the jth generated word, and ρ is a predefined interval parameter used to control the interval size in the contrast loss calculation to ensure that the word representation learned by the model has sufficient discriminability. Specifically, it makes h i Similarity with itself (s(h i ,h i )) and h i and the word outside the window h j The similarity (s(h i ,h j )) to keep a certain interval between them, so that the model can learn different word representations and avoid generating repeated or similar texts; i ,hi ) represents the similarity between the word and itself, s(h i ,h j ) represents the similarity between the word and the word outside the window.

[0061] Suppose we have a generated text sequence about a chest X-ray report, "the patient has a normal chest X-ray with clear lungs and no abnormal findings", and the length of the generated text sequence (i.e., 8 words in the text are considered as 8 tokens). Set the window constraint parameter w = 2, then the window size is 2w + 1 = 5.

[0062] Assume that after processing by the model, we obtain the feature representation of each word (represented by a two-dimensional vector): h1 (corresponding to "the") = [0.2, 0.3]; h2 (corresponding to "patient") = [0.4, 0.1]; h3 (corresponding to "has") = [0.1, 0.2]; h4 (corresponding to "a") = [0.3, 0.4]; h5 (corresponding to "normal") = [0.5, 0.2]; h6 (corresponding to "chest") = [0.4, 0.3]; h7 (corresponding to "X-ray") = [0.35, 0.3]; h8 (corresponding to "with") = [0.25, 0.2].

[0063] To calculate the similarity between word units and their relationship with ρ, first calculate the similarity between a word unit and itself. For each word unit, its similarity with itself is 1. For example, s(h1,h1)=1, s(h2,h2)=1, etc.

[0064] Then calculate the similarity s(h i ,h j ). Taking h3 (“has”) as an example, the words outside its window are h1 (“the”), h7 (“X-ray”), and h8 (“with”).

[0065]

[0066] The impact of different ρ values ​​on loss calculation: Assuming ρ (positive value)

[0067] For i=3 (“has”), compute part of the window constrained contrastive learning loss:

[0068] For i=1 (“the”), since the window constraint is satisfied, we calculate:

[0069] max{0,ρ-s(h3,h3)+s(h3,h1)}=max{0,0.3-1+0.63}=0.03;

[0070] For j=7 ("X-ray"), since |3-7|>2 satisfies the window constraint, we calculate:

[0071] max{0,ρ-s(h3,h3)+s(h3,h7)}=max{0,0.3-1+0.58}=0.01;

[0072] For j=8 (“with”), since |3-8|>2 satisfies the window constraint, we calculate:

[0073] max{0,ρ-s(h3,h3)+s(h3,h8)}=max{0,0.3-1+0.53}=0; (because

[0074] 0.3-1+0.53<0)

[0075] At this time, ρ = 0.3 encourages the model to further increase the similarity gap with the word outside the window, making the feature representation of h3 more discriminative. During the training process, the model will adjust the feature representation of h3 according to this loss, for example, it may be adjusted to [0.12, 0.18] (new hypothetical value, actually calculated by the model based on optimization algorithms such as gradient descent) to reduce the similarity with the word outside the window, thereby reducing the loss.

[0076] Assume ρ = -0.2 (negative value)

[0077] For i=3 (“has”), compute part of the window constrained contrastive learning loss:

[0078] For j=1 (“the”), calculate:

[0079] max{0,ρ-s(h3,h3)+s(h3,h1)}=max{0,0.2-1+0.63}=0; (because

[0080] -0.2-1+0.63<0)

[0081] For j=7 ("X-ray"), calculate:

[0082] max{0,ρ-s(h3,h3)+s(h3,h7)}=max{0,-0.2-1+0.58}=0; (because

[0083] -0.2-1+0.58<0)

[0084] For j=8 ("with"), calculate:

[0085] max{0,ρ-s(h3,h3)+s(h3,h8)}=max{0,-0.2-1+0.53}=0; (because

[0086] -0.2-1+0.53<0)

[0087] When ρ = -0.2, the model allows h3 to maintain a certain similarity with the tokens outside the window, which may help capture more subtle semantic associations between tokens. For example, in this example, "has" and "with" are semantically related ("has something with..."), and the model may not over-adjust the feature representation to preserve this potential semantic relationship.

[0088] Impact on model training and word representation: During model training, similar calculations are performed for each word (from 1 to 8) in the entire text sequence, and all results are added together to obtain the total window-constrained contrastive learning loss. Based on this loss value, the model adjusts the feature representation of each word through optimization algorithms such as gradient descent. As training progresses, different values ​​will guide the model to learn different word representations. When ρ takes a positive value, the model tends to separate the words further apart in the feature space, making each word more distinctive, which helps avoid text degradation problems such as repetition at the word, phrase, or sentence level in the generated report. For example, when describing a chest X-ray image, it can accurately distinguish between words with different key semantics such as "normal" and "abnormal", generating a more accurate and clear report.

[0089] When ρ takes a negative value, the model will retain the similarity between word units to a certain extent, which is helpful for capturing the relationship between some semantically similar or related word units. For example, when describing symptoms, although "cough" and "respiratorydistress" are not exactly the same, they may be related in some cases. The appropriate negative value of ρ can allow the model to learn this relationship, so that when generating a report, it can more comprehensively and accurately describe the patient's symptoms and possible disease conditions, providing more valuable information for clinical diagnosis.

[0090] In another embodiment, multiple data groups are extracted from the training set, wherein each data group includes a preset number of data samples, wherein at least one data sample includes identification information, and the data corresponding to the identification information may be an X-ray chest image with different lesion types. The initial image report generation model is trained based on the data samples in the multiple data groups to generate a trained image report generation model.

[0091] In addition, since the data of the image report generation model for training may be too small and the proportion of each stage in the training data is uneven, the StableDiffusion model is used to generate chest images. First, based on the real image used as the target distribution, that is, the stable distribution, according to the principle of the stable diffusion model, a stable diffusion model suitable for chest image generation is constructed. The model consists of multiple deconvolution layers and other neural network layers, and the chest image is reversely generated through the reverse diffusion process. During the training process, starting from the stable distribution (that is, the real image distribution), the reverse diffusion process is performed using the stable diffusion model. The reverse diffusion process will gradually blur the image until a virtual chest image corresponding to the stable distribution is obtained. The virtual chest image generated in this way has corresponding feature information, which can be used for subsequent lesion judgment and material component identification.

[0092] In the process of back diffusion, noise is introduced to control the diversity and randomness of the generated chest images. By adjusting the intensity of the noise, the clarity and style of the generated images can be affected, thereby obtaining more diverse virtual chest images. In addition, the initial noise is set. Specifically, this initial noise can be random Gaussian noise or uniformly distributed noise. The purpose is to introduce some randomness in the early stage of generating images to encourage the model to generate diverse initial images. Next, the noise intensity is adjusted and gradually reduced during the generation process. This can be achieved by controlling the noise parameters in the generation network or gradually reducing the standard deviation of the added noise during the training process. Gradually reducing the intensity of the noise helps the model gradually improve the clarity and quality of the image during the generation process. By introducing initial noise and gradually reducing the intensity of the noise, the model can generate more random and diverse images, which helps to generate a series of chest images with different features or cases, and helps to stabilize the distribution of more comprehensive learning data for the diffusion model. The trained model is more likely to cope with the input of noise or incomplete information, thereby improving the robustness of the model and making it more suitable for various uncertain factors in the real world. The trained image report generation model is processed based on the validation set to generate a test result; if the data sample containing identification information in the test result indicates that the radiation dose of the current area is in an abnormal state, the trained image report generation model is used as the target image report generation model.

[0093] S105, processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features.

[0094] The multi-view image encoder in the target image report generation model encodes the chest X-ray image of the target patient and inputs the medical prior knowledge into the graph convolutional network. The prior knowledge loss function (T prior) enables the graph convolutional network module to better integrate prior knowledge and image information and enhance the image encoder capability. After the graph convolutional network module obtains the visual features containing prior knowledge, it selects nodes to obtain V prior ∈R k ×e (where k is the number of main nodes and e is the feature dimension), and input it into the classifier A (the network structure is a fully connected network) to obtain the probability P of each category prior =f A (V prior ), P prior ∈R k is the classification probability of k main nodes. The calculation formula of the target image report generation model to obtain the fused visual features is: Among them, y i represents the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, i is the main node index, and here y i ∈{0, 1} represents the i-th label value of the current sample, which is 1 if the label exists, otherwise 0; p i Represents the probability value of the i-th element.

[0095] For example, suppose that when processing a target chest X-ray image, the number of main nodes k = 5 and the feature dimension e = 10. After a series of calculations, the label values ​​of the current sample are y1 = 1, y2 = 0, y3 = 1, y4 = 0, y5 = 1 (indicating that the sample has relevant features in the 1st, 3rd, and 5th categories), and the corresponding probability values ​​p1 = 0.7, p2 = 0.3, p3 = 0.8, p4 = 0.2, p5 = 0.9. Then the relevant probability values ​​of the fused visual features are calculated according to the formula (here only some nodes are calculated as an example):

[0096]

[0097] In the text encoding part, the clinical information of the target patient (such as symptom description, etc.) is input into the text encoder to obtain the text encoding features. The clinical information loss function (L txt ) enables the text encoder to fully obtain clinical text information and better integrate it with visual features. The classification probability can be obtained from the previously obtained fusion features through classifier B. Based on this, the clinical information loss function is defined. The calculation formula for obtaining text encoding features in the target imaging report generation model is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ijrepresents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0098] Assume that we have a simple case where the total number of disease keywords for classification supervision is T=4 and the dictionary size is v=5.

[0099] Suppose there is a clinical information description as "the patient has a cough, accompanied by a low fever, and no dyspnea". After being processed by the text encoder, there will be corresponding probability distributions for key words related to the disease, such as "cough", "low fever", and "dyspnea" (it is assumed here that the probability value has been obtained after a certain processing, which is actually the result of complex calculations of the model).

[0100] For example, for the keyword "cough" (assuming it corresponds to the first keyword), the probability distribution in the dictionary is as follows: p1=0.6 (corresponding to the first word "cough" itself in the dictionary), p2=0.1 (corresponding to other words in the dictionary), p3=0.1, p4=0.1, p5=0.1.

[0101] For “low fever” (assuming it corresponds to the second keyword), the probability distribution is: p1=0.1, p2=0.5 (corresponding to “low fever”), p3=0.1, p4=0.1, p5=0.2.

[0102] For "difficulty breathing" (assuming it corresponds to the third keyword), the probability distribution is: p1 = 0.2, p2 = 0.2, p3 = 0.4 (corresponding to "difficulty breathing"), p4 = 0.1, p5 = 0.1.

[0103] (Assuming that the fourth keyword does not appear in the clinical information, the relevant probability distribution does not consider its impact for the time being)

[0104] At the same time, ij Indicates the label value of the jth word in the dictionary corresponding to the i-th word. If it is the jth word, it is 1, otherwise it is 0.

[0105] For the keyword "cough":

[0106] When j = 1, y 11 =1 (because the first word in the dictionary is "cough"). When j = 2, 3, 4, 5, y 1j =0.

[0107] For the keyword "low heat":

[0108] When j = 2, y 22 =1. When j = 1, 3, 4, 5, y 2j =0.

[0109] For the keyword "difficulty breathing":

[0110] When j = 3, y 33 =1. When j = 1, 2, 4, 5, y 3j =0.

[0111] Calculate text encoding features according to the formula:

[0112]

[0113] Through such a calculation process, using the calculation formula of text encoding features, combined with the probability distribution and label value of clinical information in the dictionary, the text encoding feature value corresponding to the clinical information is obtained (this is just a simplified example, the actual model calculation will be more complex and accurate). This feature value will be used together with the fused visual features for subsequent operations such as generating imaging diagnostic reports, which will help to comprehensively consider clinical information and chest X-ray image information and improve the accuracy and reliability of diagnostic reports. Through such a process, the target image report generation model can process the target chest X-ray image, generate fused visual features and text encoding features, and provide a basis for the subsequent generation of imaging diagnostic reports.

[0114] In one embodiment, the multi-view image encoder encodes the target chest X-ray image and inputs the image into the graph convolutional network with the medical prior knowledge obtained above, thereby obtaining a fused visual feature. Medical prior knowledge loss function T prior , this loss function is designed to enable the graph convolutional network module to better integrate medical prior knowledge and image information to enhance the image encoder capability. The graph convolutional network module obtains the visual features v containing medical prior knowledge prior After that, node selection is performed to obtain And feed it into classifier A to get the probability of each class: P prior =f A (v″ prior ), the network structure f of classifier A A A fully connected network. is the classification probability of k main nodes. The target image report generation model includes a calculation formula for obtaining fused visual features. The specific calculation formula is: Among them, y i represents the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the k-th medical prior knowledge graph, i is the main node index, and y i ∈{0,1} represents the i-th label value of the current sample, which is 1 if the label exists, otherwise it is 0.

[0115] In another embodiment, in the text encoding part, the clinical information is input into the text encoder to obtain the text encoding features, and the clinical information loss function This loss function is designed to enable the text encoder to fully capture clinical text information and better integrate it with visual features. From a clinical perspective, this is consistent with the practice that radiologists usually use clinical information as an aid and guide when observing new chest radiographs. fusion The classification probability can be obtained through classifier B Based on this, the clinical information loss function L can be defined txt ,The target image report generation model includes a calculation formula for obtaining text encoding features, and the specific calculation formula is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the i-th disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

[0116] S106, processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report.

[0117] In one implementation, the loss function φ is generated by obtaining a report word This loss function supervises the consistency between the report text generated by the model and the reference report text. It is the basic loss function for all chest radiograph report automatic generation tasks. The above content describes in detail how to obtain the generated text probability from the generator.

[0118] Based on the report generation loss function, the fused visual features and text encoding features are processed to generate an image diagnosis report; during training, the text probability obtained in the generator is used to calculate the generation loss to supervise the generation of the image diagnosis report text, and during inference, the image diagnosis report is generated based on the probability distribution search. In addition, the target image includes a generation model including a calculation formula for obtaining the report generation loss function, which is: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index, which is used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ijRepresents the probability that the i-th word corresponds to the j-th word in the dictionary. The final imaging diagnosis report is as follows: the bilateral lung fields are clear, no obvious consolidation shadows are found in the lungs, the hilar shadow is not large, the mediastinum is centered, the size and shape of the heart shadow are normal, the bilateral diaphragmatic surfaces are smooth, and the costophrenic angles are sharp.

[0119] In the present application, the server obtains the image information to be detected and the initial image report generation model that matches the target chest X-ray image, wherein the image information to be detected includes the chest X-ray image of the target patient and the clinical information of the target patient; obtains a training sample set for training the initial image report generation model, wherein the training sample set includes a prior knowledge graph for characterizing the chest image and the chest image to be trained, wherein the chest image to be trained includes the clinical information of different patients; processes the training sample set based on preset processing rules to generate a training set and a verification set; trains the initial image report generation model based on the training set and the verification set to generate a target image report generation model; processes the target chest X-ray image based on the target image report generation model to generate fused visual features and text encoding features; processes the fused visual features and text encoding features based on the target image report generation model to generate an image diagnosis report. A deep learning network is used to encode the features of prior knowledge graphs, chest X-rays, and clinical information. The received encoded information is decoded by a decoder. During the decoding process, an algorithm is used to limit text generation and reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through the loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0120] Optionally, in another embodiment of the method of the present application, the processing of the training sample set based on a preset processing rule to generate a training set and a validation set further includes:

[0121] Dividing the training set to generate a number of numerical samples;

[0122] Obtain any number of samples of several categories in the training set;

[0123] Generate adjacent samples based on the distance between the any number of number-class samples and other number-class samples of the same category, wherein the adjacent samples include a preset number of the any number of number-class samples;

[0124] Sampling the adjacent samples to generate a preset number of sampling samples;

[0125] Based on any minority class sample and each sampling sample, multiple data groups are generated, wherein each data group includes a preset number of data samples.

[0126] In one embodiment, the distance between any number of smaller number of class samples and each other number of smaller number of class samples of the same category can be Euclidean distance or other cosine distance, which is not specifically limited here. All other number of smaller number of class samples can be arranged in order from near to far distance from any other number of smaller number of class samples, and then the data in the first few positions of the arrangement result are selected as adjacent samples. The purpose of sampling is to increase the number of minority class samples to achieve a balance between the number of minority class samples and the number of majority class samples in the initial sample set. Here, oversampling can be achieved by a sampling algorithm, which is not specifically limited here. After oversampling, multiple groups of data groups can be determined based on the results obtained by oversampling and combined with the training set. At this time, the number of descriptive data samples corresponding to each attribute category label in the data group is equivalent, and there are no number of smaller number of class samples or number of larger number of class samples. In this embodiment, the training set is first preliminarily divided, and then the number of smaller number of class samples is merged with the number of larger number of class samples.

[0127] Optionally, in another embodiment of the method of the present application, the step of sampling the adjacent samples to generate a preset number of sampled samples includes:

[0128] Determining a sampling ratio based on the number of samples of each category in the training set;

[0129] Based on the sampling ratio, determining a sampling ratio;

[0130] The adjacent samples are sampled based on the sampling ratio to generate a preset number of sampling samples.

[0131] In one implementation, first obtain which data groups are included in the training set, confirm how many number class samples there are in each data group, and determine the sampling ratio according to the number of number class samples in each data group, and then use the sampling ratio to determine the sampling ratio. The sampling ratio can be set as needed, for example, it can be 50% or other ratios, which are not specifically limited here. Finally, use the sampling ratio to sample a preset number of adjacent samples to obtain a specified number of sampled samples, and all other smaller number of number class samples can be arranged in order from near to far in terms of distance from any smaller number of number class samples, and then select the preset number of samples ranked first in the arrangement result as neighboring samples, and the preset number can be set according to actual conditions, which is not specifically limited here. Based on any smaller number of number class samples and each sampled sample, a new sample is generated and used as a new data group, so that the balance effect between the smaller number of data groups and the other larger number of data groups can be further improved.

[0132] By applying the above technical solution, the server first obtains the image information to be detected (including the target patient's chest X-ray image and clinical information) and the initial image report generation model, and obtains the preset chest X-ray report data set. The data set is processed to obtain the radioactive features and the correlation co-occurrence frequency, and then the correlation matrix information is generated, and the prior knowledge graph representing the chest image is constructed based on this. The training sample set contains the graph and the clinical information of different patients. Then the training sample set is feature extracted to generate the original feature library, and then the training set and the validation set are divided. The classifier is used to predict the validation set to determine the result, and the training set is trained with the preset algorithm to obtain the validation set class prediction result. The training set is processed by the window constraint contrast learning loss method to generate the word element vector value to train the initial model to obtain the target model, and the method contains a specific calculation formula. The target model processes the target chest X-ray image to generate fused visual features (with corresponding calculation formulas) and text encoding features (also with calculation formulas). After obtaining the report generation loss function (with its calculation formula), the fused feature processing is based on this to generate an image diagnosis report. This process integrates multiple technologies and computing methods through multi-step data processing, model building and training, aiming to improve the accuracy and effectiveness of chest X-ray report generation and provide more reliable assistance for clinical diagnosis. The deep learning network is used to encode the features of the prior knowledge graph, chest X-ray, and clinical information, and the decoder is used to decode the received encoded information. During the decoding process, the algorithm is used to restrict text generation and reduce the repetition rate of text generation. During model training, the distance between the output value and the true value is continuously improved through the loss function, thereby improving the accuracy and reliability of medical imaging report generation.

[0133] In one embodiment, if Figure 2 As shown, the present application also provides an imaging report generating device based on an X-ray chest film, comprising:

[0134] The acquisition module 201 is used to acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; and to acquire a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients;

[0135] The processing module 202 is used to process the training sample set based on preset processing rules to generate a training set and a verification set; train the initial image report generation model based on the training set and the verification set to generate a target image report generation model; process the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; process the fused visual features and the text coding features based on the target image report generation model to generate an image diagnosis report.

[0136] The present application embodiment provides an electronic device, such as Figure 3 As shown, the electronic device 3 includes a first processor 300, a memory 301, a bus 302 and a communication interface 303. The first processor 300, the communication interface 303 and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can be run on the first processor 300. When the first processor 300 runs the computer program, it executes the image report generation method based on the X-ray chest film provided in any of the aforementioned embodiments of the present application.

[0137] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0138] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store a program, and the first processor 300 executes the program after receiving an execution instruction. The image report generation method based on a chest X-ray film disclosed in any implementation of the embodiment of the present application may be applied to the first processor 300, or implemented by the first processor 300.

[0139] The first processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the first processor 300. The above-mentioned first processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the first processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0140] The electronic device provided in the above-mentioned embodiment of the present application and the method for generating an image report based on an X-ray chest film provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0141] The present application embodiment provides a computer-readable storage medium, such as Figure 4 As shown, the computer-readable storage medium 401 stores a computer program, and when the computer program is read and executed by the second processor 402, the aforementioned method for generating an image report based on a chest X-ray is implemented.

[0142] The technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which may be an air conditioner, a refrigeration device, a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0143] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the method for generating an image report based on an X-ray chest film provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0144] An embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is executed by a third processor to implement the method described above.

[0145] The computer program product provided in the above-mentioned embodiments of the present application and the method for generating an image report based on an X-ray chest film provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0146] Each embodiment in the present application is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the evaluation of the imaging report generation method based on chest X-ray films, electronic devices, electronic devices, and readable storage medium embodiments, since they are basically similar to the above-mentioned imaging report generation method based on chest X-ray films, the description is relatively simple, and the relevant parts can be referred to the partial description of the above-mentioned imaging report generation method based on chest X-ray films.

[0147] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope defined by the claims.

Claims

1. A method for generating an image report based on an X-ray chest film, characterized in that: include: Acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; Acquire a training sample set for training an initial imaging report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, and the chest images to be trained include clinical information of different patients; Processing the training sample set based on preset processing rules to generate a training set and a validation set; Training the initial image report generation model based on the training set and the validation set to generate a target image report generation model; Processing the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; The fused visual features and the text encoding features are processed based on the target image report generation model to generate an image diagnosis report.

2. The method according to claim 1, characterized in that The step of obtaining a training sample set for training an initial imaging report generation model includes: Obtain a preset chest X-ray report data set; Processing the chest X-ray report data set to obtain radiological features and correlation co-occurrence frequencies; Processing the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information; A priori knowledge graph for characterizing chest images is generated based on the correlation matrix information.

3. The method according to claim 2, characterized in that The processing of the radioactive features and the correlation co-occurrence frequencies to generate correlation matrix information includes: The method includes a calculation formula for calculating the correlation co-occurrence frequency, and the calculation formula is: Among them, C ij represents the number of times concept i and concept j co-occur at the report level, C j represents the total number of times concept j appears, P ij Indicates the frequency of concept i appearing when concept j appears: The method includes a calculation formula for calculating a correlation matrix, which is: Among them, τ is the co-occurrence frequency threshold, if P ij If it is greater than or equal to the threshold τ, it is considered that there is a correlation between concept i and concept j, otherwise there is no correlation.

4. The method according to claim 1, characterized in that The processing of the training sample set based on the preset processing rules to generate a training set and a validation set includes: Extracting features from the training sample set to generate an original feature library; Divide the training sample set according to the original feature library to generate a training set and a validation set; Use the classifier to divide the original feature library into various verification sets for prediction and determine the prediction results; Use the preset algorithm to divide the original feature library into various training sets for training and obtain the prediction results of the validation set class.

5. The method according to claim 1, characterized in that The step of training the initial image report generation model based on the training set and the validation set to generate a target image report generation model includes: Processing the training set based on a window-constrained contrastive learning loss method to generate word-unit vector values; Training the initial image report generation model based on the word element vector value to generate a target image report generation model; The window-constrained contrastive learning loss method includes a calculation formula for obtaining a word element vector value, and the calculation formula is: Among them, w is the window constraint parameter, L represents the length of the generated text sequence, i represents the word position index, j represents the comparison word position index, and h i represents the feature representation of the i-th generated word, h i represents the feature representation of the jth generated word, ρ is a predefined interval parameter, s(h i ,h i ) represents the similarity between the word and itself, s(h j ,h j ) represents the similarity between the word and the word outside the window.

6. The method according to claim 1, characterized in that The target chest X-ray image is processed based on the target image report generation model to generate fused visual features and text coding features, including: The target image report generation model includes a calculation formula for obtaining fused visual features, and the calculation formula is: Among them, y i represents the i-th label value of the current sample, p i represents the probability value of the i-th element, k is the number of main nodes in the medical prior knowledge graph, and i is the main node index; The target image report generation model includes a calculation formula for obtaining text encoding features, and the calculation formula is: Among them, T represents the total number of disease keywords used for classification supervision, i represents the disease keyword index, j represents the category index corresponding to the disease keyword, and y ij represents the true value label of the jth category of the ith disease keyword, p ij represents the probability of the i-th disease keyword in the j-th category, and v represents the total number of words in the dictionary used by the model.

7. The method according to claim 1, characterized in that The step of processing the fused visual features and the text encoding features based on the target image report generation model to generate an image diagnosis report includes: Get report to generate loss function; Processing the fused visual features and the text encoding features based on the report generation loss function to generate an imaging diagnosis report; The target image includes a generation model including a calculation formula for obtaining a report generation loss function, the calculation formula is: Where L represents the number of tokens in the radiology report text sequence generated by the model, i represents the token position index, j represents the dictionary index, which is used to calculate the difference between the probability distribution of each token on different words in the dictionary and the true label, v represents the total number of words in the dictionary used by the model, and y ij Represents the label value of the jth word in the dictionary corresponding to the i-th word. If it is the word, it is 1, otherwise it is 0. ij Represents the probability that the i-th word corresponds to the j-th word in the dictionary.

8. An imaging report generating device based on chest X-ray, characterized in that: The device comprises: An acquisition module is used to acquire image information to be detected and an initial image report generation model that matches a target chest X-ray image, wherein the image information to be detected includes a chest X-ray image of a target patient and clinical information of the target patient; and to acquire a training sample set for training the initial image report generation model, wherein the training sample set includes a priori knowledge graph for characterizing chest images and chest images to be trained, wherein the chest images to be trained include clinical information of different patients; A processing module is used to process the training sample set based on preset processing rules to generate a training set and a verification set; train the initial image report generation model based on the training set and the verification set to generate a target image report generation model; process the target chest X-ray image based on the target image report generation model to generate fused visual features and text coding features; process the fused visual features and the text coding features based on the target image report generation model to generate an image diagnosis report.

9. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; Wherein, the first processor is configured to execute the method for generating an image report based on a chest X-ray film as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the method for generating an image report based on an X-ray chest film according to any one of claims 1 to 7 is implemented.

Images