Method and related equipment for automatically generating structured reports using natural language models

Abstract

The present invention discloses a method and related equipment for automatically generating structured reports using a natural language model. The method comprises: obtaining an input sample image, generating a hidden state sequence and a classification identification sequence based on the sample image by an image diagnosis network, and obtaining a prediction result of whether the structure is abnormal; if the prediction result is no structural abnormality, obtaining a sample image without abnormality from the sample image, and generating a medical image report for the image without abnormality based on the sample image without abnormality by a complete report generation network; if the prediction result is the presence of a structural abnormality, obtaining a sample image with abnormality from the sample image, and generating a medical image report for the image with abnormality based on the sample image with abnormality by a task-aware report generation network. The present invention generates medical image reports based on whether the sample image has abnormalities. By generating reports for different structures separately, the accuracy of the generated report diagnosis is guaranteed, thereby significantly reducing the workload of doctors.

Description

Technical Field

[0001] The present invention relates to the field of image processing information technology, and in particular to a method, system, terminal and computer-readable storage medium for automatically generating structured reports using a natural language model. Background Art

[0002] Medical images, which depict the internal structure of anatomical regions, are often used for medical analysis. Medical reports compiled based on these images can further aid in disease diagnosis or medical research. However, due to a shortage of experienced physicians and an increasing number of patients, doctors face a significant workload of image reading and report writing, inevitably leading to a decline in work quality. Therefore, the use of computer technology to automatically analyze images and generate diagnostic reports—automatically generating medical image reports—is of great significance.

[0003] Because medical image reports can only be written by experienced physicians, medical image report data is often more difficult to obtain than natural image captions. For example, the IU X-Ray dataset contains only approximately 4,000 samples, while the larger MIMIC-CXR dataset contains 270,000 samples, but this is still far less than the over 3 million annotations in natural image caption datasets. Furthermore, current medical image report datasets typically include a large number of images without abnormalities and report samples, resulting in a significant sample imbalance in these datasets.

[0004] In medical image report datasets, the sample imbalance problem is not only reflected in the presence of a large number of normal samples. The degree of sample imbalance in different structures in the medical image report dataset is also different. For example, in chest X-ray images, doctors often describe multiple structures and multiple diseases accordingly, and different types of diseases may appear at the same time. If samples are classified according to different diseases, the sample space becomes exponential (each disease has two possibilities: presence or absence). In this case, reports containing specific diseases are even rarer. Therefore, if the medical image report is treated as a whole to deal with the existing sample imbalance problem, it will not only be more complex, but it will also be easy to miss the diagnosis of structures or diseases with more serious sample imbalance problems.

[0005] Due to the serious sample imbalance problem in the dataset, when the natural image caption generation method is directly applied to the dataset for training, the network tends to output mundane reports that are more common in the dataset, which cannot meet the requirements of medical reports for accurate description of symptoms.

[0006] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0007] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for automatically generating structured reports using a natural language model, aiming to solve the problem in the prior art that due to the serious sample imbalance problem in the data set, the generated medical image reports are inaccurate and cannot meet the requirements of medical reports for accurate description of symptoms.

[0008] To achieve the above object, the present invention provides a method for automatically generating a structured report using a natural language model. The method for automatically generating a structured report using a natural language model comprises the following steps:

[0009] Obtaining an input sample image, generating a hidden state sequence and a classification identification sequence based on the sample image based on an image diagnosis network, and obtaining a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence;

[0010] If the prediction result is that there is no structural abnormality, obtaining a sample image without abnormality from the sample images, and generating a medical image report of the sample image without abnormality based on the complete report generation network;

[0011] If the prediction result is that there is a structural abnormality, an abnormal sample image from the sample images is obtained, and a medical image report with abnormal images is generated based on the abnormal sample image based on the task perception report generation network.

[0012] Optionally, the method for automatically generating a structured report using a natural language model, wherein the step of obtaining an input sample image, generating a hidden state sequence and a classification identification sequence based on the sample image based on an image diagnostic network, and obtaining a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence, further comprises:

[0013] The full report generation network and the task-aware report generation network are pre-trained.

[0014] Optionally, in the method for automatically generating structured reports using a natural language model, the pre-training of the complete report generation network and the task-aware report generation network specifically includes:

[0015] First, the complete report generation network is trained using a set of non-abnormal samples;

[0016] After the complete report generation network training is completed, the task-aware report generation network is trained using the abnormal sample set.

[0017] Optionally, the method for automatically generating a structured report using a natural language model further comprises:

[0018] The image diagnosis network calculates the abnormality probability of each structure in the sample image;

[0019] Determine whether the abnormality probability of each structure is lower than the preset threshold;

[0020] When the abnormal probability of all structures is lower than the preset threshold, the sample image is considered to be a sample image without abnormality;

[0021] When the probability of one or more structures being abnormal is not lower than the preset threshold, the sample image is considered to be an abnormal sample image.

[0022] Optionally, the method for automatically generating a structured report using a natural language model further comprises:

[0023] When training the complete report generation network and the task-aware report generation network, an automatic balanced mask loss function is introduced;

[0024] Balance the number of abnormal samples and normal samples participating in training in different structures, and during training, the normal samples in the i-th region are ranked with a certain probability value p i Discard with probability p i for:

[0025]

[0026] in, and Represent the number of abnormal samples and normal samples in the i-th region respectively. α is a set hyperparameter used to adjust the ratio between normal samples and abnormal samples actually participating in training.

[0027] The calculation process of the automatic balancing mask loss function is as follows:

[0028]

[0029]

[0030] in, represents the automatic balanced mask loss function, represents the loss function of the baseline model, δ(r i ) is used to retain all abnormal samples, Rand(p i ) represents the probability value p i Use the Rand function;

[0031] If the sample is an abnormal sample, the corresponding loss is directly included in the total loss;

[0032] If the sample is a normal sample, then the probability value p i Discard.

[0033] Optionally, in the method for automatically generating a structured report using a natural language model, the image diagnosis network includes an image feature extractor and a Transformer encoder.

[0034] Optionally, in the method for automatically generating structured reports using a natural language model, the task-aware report generation network includes a Transformer decoder.

[0035] Optionally, in the method for automatically generating structured reports using a natural language model, a multi-head adaptive attention module is newly set in the Transformer decoder.

[0036] Optionally, in the method for automatically generating structured reports using a natural language model, the multi-head adaptive attention module is used to realize information interaction between the Transformer decoder output and the Transformer decoder input, the input query is the embedded vector sequence corresponding to the text input by the Transformer decoder, and the key and value are the embedded vector sequence from the image output by the Transformer encoder.

[0037] Optionally, in the method for automatically generating a structured report using a natural language model, the multi-head adaptive attention module acquires language and visual signals, and the calculation process is as follows:

[0038] S L =Linear(Q);

[0039]

[0040] Among them, S L and S V They are language signals and visual signals respectively, Linear is a linear mapping function, Q represents the target query sequence matrix, K represents the sequence matrix to be queried, and K T represents the transposed matrix of K, d k Indicates the dimension of the feature vector corresponding to the key;

[0041] Language and visual signals contain signals corresponding to different time nodes. Different time nodes correspond to the generation process of words at different positions, and the calculation process of different time nodes is performed separately.

[0042] The language signal and the visual signal are combined according to the time node, and the corresponding weight vector is obtained by the normalized exponential function:

[0043] V weight =Softmax(Concat(S V , S L ));

[0044] Among them, V weight is the weight vector;

[0045] By taking the last value of the weight vector and arranging it in the order of time nodes, we get the language feature weight vector β;

[0046] The multi-head adaptive attention module uses a gating mechanism to control visual features and language features. The calculation process is as follows:

[0047] V adap =β×Linear(Q)+(1-β)×Attention(Q,K,V);

[0048] Among them, V adap is the output of the multi-head adaptive attention module, the Linear function is used to obtain language context information, and V represents the self-meaning encoding sequence matrix;

[0049] The multi-head adaptive attention module learns feature representation by mapping inputs, keys, and values to different feature subspaces. The calculation process is as follows:

[0050] head i =AdaptiveAttention(Q i , K i , V i );

[0051] MultiHead(Q,K,V)=Concat(head1,...,head h )W O ;

[0052] Among them, head i represents the matrix, W O represents the weight matrix.

[0053] In addition, to achieve the above-mentioned purpose, the present invention further provides a system for automatically generating structured reports using a natural language model, wherein the system for automatically generating structured reports using a natural language model comprises:

[0054] An image processing and result prediction module is used to obtain an input sample image, generate a hidden state sequence and a classification identification sequence based on the sample image based on the image diagnosis network, and obtain a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence;

[0055] a non-abnormal image report generating module, configured to, if the prediction result is that there is no structural abnormality, obtain a non-abnormal sample image from the sample images, and generate a medical image report of the non-abnormal image based on the non-abnormal sample image based on the complete report generating network;

[0056] An abnormal image report generation module is used to obtain abnormal sample images from the sample images if the prediction result is that there is a structural abnormality, and generate a medical image report with abnormal images based on the abnormal sample images based on the task-aware report generation network.

[0057] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a program for automatically generating structured reports using a natural language model, which is stored in the memory and can be run on the processor. When the program for automatically generating structured reports using a natural language model is executed by the processor, the steps of the method for automatically generating structured reports using a natural language model as described above are implemented.

[0058] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program for automatically generating structured reports using a natural language model, and when the program for automatically generating structured reports using a natural language model is executed by a processor, the steps of the method for automatically generating structured reports using a natural language model as described above are implemented.

[0059] In the present invention, a sample image is input, and a hidden state sequence and a classification identification sequence are generated from the sample image based on an image diagnosis network. A prediction result of whether a structure is abnormal is obtained based on the hidden state sequence and the classification identification sequence. If the prediction result is no structural abnormality, a sample image without abnormality is obtained from the sample image, and a medical image report of a non-abnormal image is generated from the sample image without abnormality based on a complete report generation network. If the prediction result is a structural abnormality, a sample image with abnormality is obtained from the sample image, and a medical image report of an abnormal image is generated from the sample image with abnormality based on a task-aware report generation network. The present invention generates medical image reports based on whether the sample image has abnormalities. By generating reports for different structures separately, the accuracy of the generated report diagnosis is guaranteed, thereby greatly reducing the workload of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a preferred embodiment of the method for automatically generating a structured report using a natural language model according to the present invention;

[0061] Figure 2 1 is a schematic diagram of a framework for generating medical image reports based on whether a sample image has abnormalities in a preferred embodiment of the method for automatically generating structured reports using a natural language model of the present invention;

[0062] Figure 3 2 is a schematic diagram of the two stages of structural anomaly prediction and report generation in a preferred embodiment of the method for automatically generating structured reports using a natural language model of the present invention;

[0063] Figure 4 2 is a schematic diagram of the principle of the attention mechanism in a preferred embodiment of the method for automatically generating structured reports using a natural language model of the present invention;

[0064] Figure 5 2 is a schematic diagram of the principle of a multi-head adaptive attention module in a preferred embodiment of the method for automatically generating structured reports using a natural language model of the present invention;

[0065] Figure 6 1 is a schematic diagram of the principle of a preferred embodiment of a system for automatically generating structured reports using a natural language model according to the present invention;

[0066] Figure 7 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] The method for automatically generating a structured report using a natural language model according to a preferred embodiment of the present invention is as follows: Figure 1 As shown, the method for automatically generating a structured report using a natural language model includes the following steps:

[0069] Step S10: obtaining an input sample image, generating a hidden state sequence and a classification identification sequence according to the sample image based on an image diagnosis network, and obtaining a prediction result of whether the structure is abnormal according to the hidden state sequence and the classification identification sequence.

[0070] Specifically, such as Figure 2As shown, the image diagnosis network consists of an image feature extractor and a Transformer encoder including a classification identification module, which is used to output the hidden state sequence and classification identification sequence required for generating branches (that is, the image diagnosis network generates the hidden state sequence and classification identification sequence according to the sample image), and can also obtain a prediction result of whether the structure is abnormal (for example, structural abnormality refers to an image that is different from a normal human body and may be a state of disease) (that is, the image diagnosis network obtains a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence), that is, the prediction results include two types, namely, no structural abnormality and presence of structural abnormality.

[0071] Furthermore, before step S10, the method further includes pre-training the complete report generation network and the task-aware report generation network. Specifically, the complete report generation network is first trained using a set of non-anomaly samples; after the complete report generation network is trained, the task-aware report generation network is then trained using a set of anomaly samples. During training, both non-anomaly samples and anomaly samples are used. Adjustment of the training samples typically mitigates sample imbalance.

[0072] like Figure 2 As shown, in order to realize the training of two generation branches (the two branches refer to the two processing branches for non-abnormal sample images and abnormal sample images), the present invention divides the entire report training set into two sets of non-abnormal samples and abnormal samples, wherein the non-abnormal sample set is used to train the complete report generation network (branch), and the abnormal sample set is used to train the task-aware report generation network (branch).

[0073] During training, the framework is divided into two phases, each of which trains the two generative networks (branches). In the first training phase, the framework uses a set of non-anomaly samples to train the complete report generation network (branch), and in the second training phase, the framework uses a set of anomaly samples to train the task-aware report generation network (branch). It should be noted that since the non-anomaly samples in the training set do not participate in the training of the task-aware report generation network (branch), the imbalance problem at the sample level is improved. In addition, the parameters of the image feature extraction and classification network parts in the framework are updated in both phases, so the order of training the two generative networks (branches) will affect the parameter update of these parts of the network structure. When designing the training process, the present invention was inspired by the fact that the learning process of humans and animals generally progresses from easy to difficult. In this training process, the two training phases correspond to the network's learning of non-anomaly and anomaly samples, respectively, and non-anomaly samples are easier to learn than anomaly samples. Therefore, during training, the present invention adopts the order of training the complete report generation network (branch) first and then training the task-aware report generation network (branch).

[0074] Furthermore, the present invention can also obtain the prediction result of whether the structure is abnormal in the following way: Figure 3 As shown, in the structural abnormality prediction stage, the image diagnosis network calculates the abnormality probability of each structure in the sample image (for example, lung abnormalities: low lung capacity, bilateral pleural effusion, etc.; heart abnormalities: cardiomegaly, aortic calcification, etc.); based on these abnormality probabilities, it can be determined whether the current sample image has an abnormality; determine whether the abnormality probability of each structure is lower than a preset threshold (for example, the preset threshold is 0.5); when the abnormality probability of all structures is lower than the preset threshold, the sample image is considered to be a sample image without abnormality; when the abnormality probability of one or more structures is not lower than the preset threshold, the sample image is considered to be a sample image with abnormality.

[0075] Furthermore, sample imbalance can impact the diagnostic accuracy of generated reports. Directly training models from the field of natural image captioning often provide trivial solutions based on descriptions, indicating that most structures are normal. Existing methods, however, directly generate complete reports, failing to effectively address sample imbalance differences between different structures. If trained directly using structured reports, even with extreme data imbalance, the network often provides trivial solutions based on descriptions, indicating that most structures are normal.

[0076] In order to solve this problem, the present invention introduces an automatic balancing mask loss function to solve the data imbalance problem. This loss function takes into account the different degrees of imbalance of samples in different structures. That is, when training the complete report generation network and the task-aware report generation network, an automatic balancing mask loss function is introduced. In order to balance the number of abnormal samples and normal samples participating in the training in different structures, the normal samples in the i-th region are given a certain probability value p during training. i Discard with probability p i for:

[0077]

[0078] in, and They represent the number of abnormal samples and normal samples in the i-th region respectively, and α is a set hyperparameter used to adjust the ratio between normal samples and abnormal samples actually participating in the training.

[0079] Based on the above balancing method, the calculation process of the automatic balancing mask loss function is as follows:

[0080]

[0081]

[0082] in, represents the automatic balanced mask loss function, represents the loss function of the baseline model, δ(r i ) is used to retain all abnormal samples, Rand(p i ) represents the probability value p i Use the Rand function.

[0083] Intuitively speaking, if the sample is an abnormal sample, the corresponding loss is directly included in the total loss; if the sample is a normal sample, the probability value p is used. i Discard.

[0084] In this way, the ratio between the number of valid normal samples and the number of abnormal samples included in the total loss calculation will be approximately balanced to α.

[0085] Step S20: If the prediction result is that there is no structural abnormality, obtain a sample image without abnormality from the sample images, and generate a medical image report of the sample image without abnormality based on the complete report generation network.

[0086] Specifically, if Figure 2 and Figure 3As shown, after determining whether the sample image has an abnormality, the report generation phase begins. If the prediction result indicates no abnormality, the complete report generation network (branch) is selected, simulating the process of a doctor filling out a templated report after determining that there is no abnormality. Because there is no abnormality, the medical image report generated for the normal image has a fixed format. This process is relatively simple and the report generation speed is also fast. In this way, normal samples can be generated by the complete report generation branch, avoiding the problem of content redundancy.

[0087] Step S30: If the prediction result is that there is a structural abnormality, obtain an abnormal sample image from the sample images, and generate a medical image report of the abnormal image based on the abnormal sample image based on the task-aware report generation network.

[0088] Specifically, if Figure 2 and Figure 3 As shown, after determining whether the sample image has an abnormality, the system enters the report generation phase. If the prediction result indicates an abnormality, the task-aware report generation network (branch) is selected to generate a task-aware report, simulating the process of a doctor writing a detailed report after diagnosing an abnormality. By generating reports for different structures, the accuracy of the generated reports is guaranteed.

[0089] When people observe their surroundings, they typically focus only on the most important information and ignore all other information. The essence of this phenomenon is the attention mechanism. In deep learning, models typically receive a large amount of information. Indiscriminately processing all of it significantly increases the computational load. However, in reality, only a portion of the information is needed for a specific step or moment. Therefore, networks (such as the Task Perception Report Generation Network) utilize the attention mechanism to filter out the critical information currently required from the large amount of input. This allows for better modeling and processing of information, thereby improving performance.

[0090] The essence of attention is to search for key-value pairs of data for a set of output queries and select the required information, such as Figure 4 As shown in the figure, the general attention mechanism consists of three stages. In the first stage, the input query vector is similarity-measured with the key and the corresponding similarity score is obtained. There are many methods for similarity measurement, which can be dot product operation, additive operation or bilinear model. After obtaining the similarity score, it needs to be normalized and the weight of each key-value pair is obtained. The size of the weight represents the importance of each key-value pair to this set of queries. Finally, the similarity score is weighted and summed with the value to obtain the final attention value.

[0091] Medical image reports typically consist of multiple sentences, each describing a different part of the body. Many types of words can be found in medical image reports, such as "The heart is normal in size. The aorta is tortuous and ectatic. The lungs are clear without focal airspace opacity, pleural effusion, or pneumothorax. The osseous structures are intact." The different words in a report serve different purposes. Some are connectives, such as "the," "is," and "and," used to make the sentence more fluent and readable. Others are medically relevant words, such as "heart," "aorta," and "lungs," which describe the body part, or "normal," "tortuous," and "clear," which describe symptoms. Generating different words requires different information. Connectives like "the," "is," and "and" do not require any visual information but can be inferred solely from the linguistic features of the context. For example, "heart" and "aorta" are likely to be followed by "is" because "is" is needed to connect the structure and its description. However, for medical-related words, contextual features are not enough. For example, when generating words such as "heart" and "aorta", it is necessary to capture their specific location based on image features, while descriptive words such as "normal", "tortuous" and "clear" require more detailed visual information to determine the corresponding symptoms.

[0092] Existing medical report generation algorithms typically use layered long short-term memory networks or Transformer models, often employed in natural image captioning methods, as decoders during the report generation phase. These methods fail to consider the fact that different words require distinct information during decoding, instead providing the same visual features for all different word types. This makes it difficult for the network to understand the characteristics of different words during decoding, thus impacting the quality of the generated report.

[0093] To address this problem, this paper proposes a Multi-Head Adaptive Attention module (MHAA), such as Figure 5As shown, a multi-head adaptive attention module is newly set in the Transformer decoder (the task-aware report generation network includes the Transformer decoder), so that the task-aware report generation network can reduce the amount of calculation when generating a medical image report with abnormal images based on abnormal sample images. That is, the multi-head adaptive attention module is located in the decoder part of the Transformer, and is used to realize the information interaction between the Transformer encoder output and the Transformer decoder input. The input query is the embedded vector sequence corresponding to the text input by the Transformer decoder, and the key and value are the embedded vector sequence from the image output by the Transformer encoder.

[0094] The multi-head adaptive attention module acquires language and visual signals, and the calculation process is as follows:

[0095] S L =Linear(Q);

[0096]

[0097] Among them, S L and S V They are language signals and visual signals respectively, Linear is a linear mapping function, Q represents the target query sequence matrix, K represents the sequence matrix to be queried, and K T represents the transposed matrix of K, d k Indicates the dimension of the feature vector corresponding to the key.

[0098] Language and visual signals contain signals corresponding to different time nodes, and different time nodes correspond to the generation process of words at different positions. Therefore, the calculation process of different time nodes is carried out separately. Figure 5 The calculation process at time node t is given. Subsequently, the language signal and the visual signal are combined according to the time node, and the corresponding weight vector is obtained by the normalized exponential function:

[0099] V weight =Softmax(Concat(S V , S L ));

[0100] Among them, V weight is a weight vector whose length is H×W+1 (H and W are the dimensions of the weight vector), and the final value corresponds to the weight of the language feature.

[0101] By taking out the last value of the weight vector and arranging it in order of time nodes, the language feature weight vector β is obtained.

[0102] Finally, the multi-head adaptive attention module uses a gating mechanism to control visual features and language features. The calculation process is as follows:

[0103] V adap =β×Linear(Q)+(1-β)×Attention(Q,K,V);

[0104] Among them, V adap is the output of the multi-head adaptive attention module, the Linear function is used to obtain language context information, and V represents the self-meaning encoding sequence matrix.

[0105] In addition, the multi-head adaptive attention module learns feature representations by mapping inputs, keys, and values to different feature subspaces. The calculation process is as follows:

[0106] head i =AdaptiveAttention(Q i , K i , V i );

[0107] MultiHead(Q,K,V)=Concat(head1,...,head h )W O ;

[0108] Among them, head i represents the matrix, W O Represents the weight matrix, AdaptiveAttention(Q i , K i , V i ) means to put all the head i Matrix stitching.

[0109] The present invention proposes a dual-branch generation framework, in which the two branches are used to generate reports for samples without abnormalities and samples with abnormalities, respectively. Since the report types output by the two branches are different, the samples without abnormalities and samples with abnormalities are used for training respectively during training. Usually, the adjustment of the training samples alleviates the sample imbalance. At the same time, considering the problem of different sample balances between different structures, the present invention also proposes an automatic balancing mask loss to balance the proportion of samples actually participating in training of different structures. In addition, the present invention also improves the attention mechanism in the model and proposes a multi-head adaptive attention module. When generating each word, the module adaptively controls the contextual information and visual information according to the current state, and controls the influence of the two parts of information on the generation of the current word.

[0110] This paper proposes a method for automatically generating structured reports using a natural language model, enabling accurate diagnostic reports. First, a dual-branch generation framework is constructed to mitigate sample imbalance at the sample level by classifying samples into those with and without anomalies. Second, an automatic masking loss function is proposed to further address the imbalance at the structural level. Furthermore, a multi-head adaptive attention module is proposed to optimize the network's utilization of visual and language features.

[0111] Further, if Figure 6 As shown, based on the above method for automatically generating a structured report using a natural language model, the present invention also provides a system for automatically generating a structured report using a natural language model, wherein the system for automatically generating a structured report using a natural language model includes:

[0112] The image processing and result prediction module 51 is used to obtain an input sample image, generate a hidden state sequence and a classification identification sequence based on the sample image based on the image diagnosis network, and obtain a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence;

[0113] The non-abnormal image report generating module 52 is configured to obtain a non-abnormal sample image from the sample images if the prediction result is that there is no structural abnormality, and generate a medical image report of the non-abnormal image based on the non-abnormal sample image based on the complete report generating network;

[0114] The abnormal image report generation module 53 is used to obtain abnormal sample images from the sample images if the prediction result is that there is a structural abnormality, and generate a medical image report of the abnormal image based on the abnormal sample images based on the task-aware report generation network.

[0115] Further, if Figure 7 As shown, based on the above method and system for automatically generating structured reports using a natural language model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0116] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, a program 40 for automatically generating a structured report using a natural language model is stored on the memory 20. The program 40 for automatically generating a structured report using a natural language model can be executed by the processor 10, thereby realizing the method of automatically generating a structured report using a natural language model in the present application.

[0117] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the method for automatically generating a structured report using a natural language model.

[0118] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0119] In one embodiment, when the processor 10 executes the program 40 for automatically generating a structured report by using a natural language model in the memory 20 , the steps of the method for automatically generating a structured report by using a natural language model as described above are implemented.

[0120] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program for automatically generating structured reports using a natural language model, and when the program for automatically generating structured reports using a natural language model is executed by a processor, the steps of the method for automatically generating structured reports using a natural language model as described above are implemented.

[0121] In summary, the present invention provides a method and related equipment for automatically generating structured reports using a natural language model. The method includes: obtaining an input sample image, generating a hidden state sequence and a classification identification sequence based on the sample image based on an image diagnosis network, and obtaining a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence; if the prediction result is no structural abnormality, obtaining a sample image without abnormality from the sample image, and generating a medical image report for the sample image without abnormality based on the sample image without abnormality based on a complete report generation network; if the prediction result is the presence of a structural abnormality, obtaining a sample image with abnormality from the sample image, and generating a medical image report for the sample image with abnormality based on the sample image with abnormality based on a task-aware report generation network. The present invention generates medical image reports based on whether the sample image has abnormalities. By generating reports for different structures separately, the accuracy of the generated report diagnosis is guaranteed, thereby greatly reducing the workload of doctors.

[0122] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0123] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0124] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for automatically generating a structured report using a natural language model, characterized in that: The method for automatically generating a structured report using a natural language model includes: Pre-training the full report generation network and the task-aware report generation network; The pre-trained complete report generation network and task-aware report generation network specifically include: First, use the non-anomaly sample set to train the complete report generation network; After the complete report generation network training is completed, the task-aware report generation network is trained using the abnormal sample set; When training the complete report generation network and the task-aware report generation network, an automatic balanced mask loss function is introduced; Balance the number of abnormal samples and normal samples participating in training in different structures, and use a certain probability value for the normal samples in the i-th region during training. Discard, probability value for: ； in, and Represent the number of abnormal samples and normal samples in the i-th region respectively. α is a set hyperparameter used to adjust the ratio between normal samples and abnormal samples actually participating in training. The calculation process of the automatic balancing mask loss function is as follows: ； ； in, represents the automatic balanced mask loss function, represents the loss function of the baseline model, The role of is to retain all abnormal samples, Represents the probability value Use the Rand function; If the sample is an abnormal sample, the corresponding loss is directly included in the total loss; If the sample is a normal sample, the probability value is to discard; Obtaining an input sample image, generating a hidden state sequence and a classification identification sequence based on the sample image based on an image diagnosis network, and obtaining a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence; If the prediction result is that there is no structural abnormality, obtaining a sample image without abnormality from the sample images, and generating a medical image report of the sample image without abnormality based on the complete report generation network; If the prediction result is that there is a structural abnormality, an abnormal sample image from the sample images is obtained, and a medical image report with abnormal images is generated based on the abnormal sample image based on the task perception report generation network.

2. The method for automatically generating a structured report using a natural language model according to claim 1, characterized in that: The method for automatically generating a structured report using a natural language model further includes: The image diagnosis network calculates the abnormality probability of each structure in the sample image; Determine whether the abnormality probability of each structure is lower than the preset threshold; When the abnormal probability of all structures is lower than the preset threshold, the sample image is considered to be a sample image without abnormality; When the probability of one or more structures being abnormal is not lower than the preset threshold, the sample image is considered to be an abnormal sample image.

3. The method for automatically generating a structured report using a natural language model according to claim 1, wherein: The image diagnosis network includes an image feature extractor and a Transformer encoder.

4. The method for automatically generating a structured report using a natural language model according to claim 3, wherein: The task-aware report generation network includes a Transformer decoder.

5. The method for automatically generating a structured report using a natural language model according to claim 4, wherein: A new multi-head adaptive attention module is added to the Transformer decoder.

6. The method for automatically generating a structured report using a natural language model according to claim 5, characterized in that: The multi-head adaptive attention module is used to realize the information interaction between the output of the Transformer decoder and the input of the Transformer decoder. The input query is the embedding vector sequence corresponding to the text input by the Transformer decoder, and the key and value are the embedding vector sequence from the image output by the Transformer encoder.

7. The method for automatically generating a structured report using a natural language model according to claim 6, wherein: The multi-head adaptive attention module acquires language and visual signals, and the calculation process is as follows: ； ； in, and They are language signals and visual signals respectively, Linear is a linear mapping function, represents the target query sequence matrix, represents the query sequence matrix, express The transposed matrix of Indicates the dimension of the feature vector corresponding to the key; Language and visual signals contain signals corresponding to different time nodes. Different time nodes correspond to the generation process of words at different positions, and the calculation process of different time nodes is performed separately. The language signal and the visual signal are combined according to the time node, and the corresponding weight vector is obtained by the normalized exponential function: ； in, is the weight vector; By taking the last value of the weight vector and arranging it in the order of time nodes, we get the language feature weight vector β; The multi-head adaptive attention module uses a gating mechanism to control visual features and language features. The calculation process is as follows: ； in, is the output of the multi-head adaptive attention module, and the Linear function is used to obtain language context information. Represents the self-meaning coding sequence matrix; The multi-head adaptive attention module learns feature representation by mapping inputs, keys, and values to different feature subspaces. The calculation process is as follows: ； ； in, represents the matrix, represents the weight matrix.

8. A system for automatically generating structured reports using a natural language model, characterized in that: The system for automatically generating a structured report using a natural language model is applied to the method for automatically generating a structured report using a natural language model according to any one of claims 1 to 7, and the system for automatically generating a structured report using a natural language model comprises: An image processing and result prediction module is used to obtain an input sample image, generate a hidden state sequence and a classification identification sequence based on the sample image based on the image diagnosis network, and obtain a prediction result of whether the structure is abnormal based on the hidden state sequence and the classification identification sequence; a non-abnormal image report generating module, configured to, if the prediction result is that there is no structural abnormality, obtain a non-abnormal sample image from the sample images, and generate a medical image report of the non-abnormal image based on the non-abnormal sample image based on the complete report generating network; An abnormal image report generation module is used to obtain abnormal sample images from the sample images if the prediction result is that there is a structural abnormality, and generate a medical image report with abnormal images based on the abnormal sample images based on the task-aware report generation network.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a program for automatically generating a structured report using a natural language model, which is stored in the memory and can be run on the processor. When the program for automatically generating a structured report using a natural language model is executed by the processor, the steps of the method for automatically generating a structured report using a natural language model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for automatically generating a structured report using a natural language model. When the program for automatically generating a structured report using a natural language model is executed by a processor, the steps of the method for automatically generating a structured report using a natural language model as described in any one of claims 1 to 7 are implemented.

Images