Training method of diagnostic report generation model and image diagnostic report generation method

By acquiring multi-tissue sample images and diagnostic reports, constructing the second sample data and training a multimodal deep learning model, the problem of confusion of diagnostic results in different organs in the prior art is solved, and the accuracy of diagnostic reports is improved.

CN118983046APending Publication Date: 2024-11-19BEIJING KNOWLEDGE ATLAS TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411055685.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When processing new medical images, the existing diagnostic report generation model is prone to confuse diagnostic results of different organs, resulting in errors in the report generation and poor accuracy.

Method used

By obtaining multi-tissue sample images and diagnostic reports, content extraction and construction of second sample data is performed, and a pre-constructed multimodal deep learning model is used for training, and a diagnostic report generation model is generated to reduce the confusion of diagnostic results in different tissues.

Benefits of technology

The generation accuracy of the diagnostic report generation model is improved, the probability that the model will mix the diagnostic results of different tissues is reduced, and the accuracy of the report is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118983046A_ABST
    Figure CN118983046A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a training method of a diagnosis report generation model and an image diagnosis report generation method. The training method of the diagnosis report generation model comprises the steps of obtaining first sample data, performing content extraction on a multi-organization sample diagnosis report in the first sample data, and obtaining a partial organization diagnosis report and a partial organization diagnosis cue word; constructing second sample data based on the partial organization diagnosis report, the partial organization diagnosis cue word and the corresponding multi-organization sample image; and training a pre-constructed multi-modal deep learning model by adopting the first sample data and the second sample data at the same time to obtain a diagnosis report generation model. As the diagnosis report text and the diagnosis cue word in the second sample data also aim at part of the organizations, the independent relation of the diagnosis report content between different relatively independent organizations can be mined in the training process of the diagnosis report generation model; the probability that the model mixes diagnosis results of different tissues together to generate an error result is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a method for training a diagnosis report generation model and a method for generating an imaging diagnosis report. Background Art

[0002] Understanding tissue images (that is, medical images) and writing imaging reports based on tissue images requires radiologists to have extremely high medical and imaging knowledge, and requires a lot of time and energy. In order to assist in the rapid generation of imaging diagnostic reports and reduce the problem of uncontrollable quality of imaging diagnostic reports caused by the uneven level of radiologists, related technologies propose a solution of training a diagnostic report generation model, using the diagnostic report generation model to process tissue images and output diagnostic reports. The existing diagnostic report generation model is trained with tissue images containing multiple tissues and corresponding diagnostic reports as training samples. The tissue images include image content of multiple organ tissue regions (for example, chest CT images include thorax, diaphragm, heart, and lungs), and the diagnostic reports contain diagnostic content of multiple organ tissues. In practical applications, it is precisely because both tissue images and diagnostic reports have multiple contents that the trained diagnostic report generation model has an overall understanding of the correlation between tissue images and diagnostic reports. When the trained diagnostic report generation model processes new medical images, the obtained diagnostic report will mix the diagnostic results of different organ tissues, resulting in report generation errors (from the perspective of the model, the model has hallucinations) and poor accuracy. Summary of the invention

[0003] In order to solve the problem that the diagnostic report generation model obtained by the existing model training method will have model hallucination, the present disclosure provides a diagnostic report generation model training method and an image diagnostic report generation method.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for training a diagnostic report generation model, comprising:

[0005] Acquiring first sample data, wherein the first sample data includes multi-tissue sample images and multi-tissue sample diagnosis reports;

[0006] Extracting contents of the multi-tissue sample diagnosis reports in the first sample data to obtain partial tissue diagnosis reports and partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis reports; and constructing second sample data based on the partial tissue diagnosis reports, the partial tissue diagnosis prompt words, and the corresponding multi-tissue sample images;

[0007] The first sample data and the second sample data are used to simultaneously train a pre-built multimodal deep learning model to obtain a diagnosis report generation model.

[0008] Optionally, before extracting content from the multiple tissue sample diagnosis reports in the first sample data to obtain a partial tissue diagnosis report, the method further includes:

[0009] Determining possible lesion feature descriptions, and constructing text extraction prompt words based on the possible lesion feature descriptions;

[0010] The extracting content from the multiple tissue sample diagnosis reports in the first sample data to obtain a partial tissue diagnosis report includes: inputting the text extraction prompt word and the multiple tissue sample diagnosis report into a large language processing model to obtain the partial tissue diagnosis report.

[0011] Optionally, obtaining the partial tissue diagnosis report and the partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis report includes:

[0012] In the case where the multi-tissue sample diagnosis report has a target lesion characteristic description, obtaining a partial tissue diagnosis report corresponding to the target lesion characteristic description;

[0013] The obtaining of the partial tissue diagnosis prompt word corresponding to the partial tissue diagnosis report includes: upon obtaining the partial tissue diagnosis report of the target lesion characteristic description, generating the partial tissue diagnosis prompt word based on the target lesion characteristic description and / or the partial tissue diagnosis report.

[0014] Optionally, before extracting content from the multiple tissue sample diagnostic reports in the first sample data to obtain a partial tissue diagnostic report, the process includes:

[0015] Constructing hierarchical processing prompt words, wherein the hierarchical processing prompt words include text format requirements and text content requirements;

[0016] The extracting contents of the multiple tissue sample diagnostic reports in the first sample data to obtain a partial tissue diagnostic report includes:

[0017] Inputting the hierarchical processing prompt words and the multi-tissue sample diagnosis report into a large language model to obtain a hierarchical processed multi-tissue sample diagnosis report;

[0018] Extracting content from the hierarchically processed multi-tissue sample diagnostic reports to obtain the partial tissue diagnostic reports;

[0019] The obtaining of the partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis report includes: constructing the partial tissue diagnosis prompt words based on the partial tissue diagnosis report.

[0020] Optionally, the method further comprises: performing region recognition processing on the multi-tissue sample image to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image;

[0021] The edge position data and / or relative size data are used as model input data in the first sample data and the second sample data.

[0022] Optionally, the first sample data and the second sample data are used to simultaneously train a pre-built multimodal deep learning model to obtain a diagnosis report generation model, including:

[0023] The first sample data and the second sample data are used to simultaneously fine-tune the pre-trained multimodal deep learning model to obtain the diagnosis report generation model.

[0024] In a second aspect, an embodiment of the present disclosure provides a method for generating an imaging diagnosis report, comprising:

[0025] Acquiring a tissue image to be interpreted and a diagnostic prompt word, wherein the diagnostic prompt word is used to prompt the content of a diagnostic report obtained by interpreting the tissue image to be interpreted;

[0026] Using a diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word to obtain an imaging diagnostic report;

[0027] The diagnosis report generation model is obtained by using the training method of the diagnosis report generation model as claimed in any one of claims 1 to 5, and the tissue image to be interpreted is a tissue image whose shooting area range is consistent with the image range of the multiple tissue samples.

[0028] Optionally, before using the diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word to obtain the image diagnosis report, the method further includes:

[0029] Performing region identification processing on the tissue image to be interpreted to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image;

[0030] The method comprises: using a diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word, and the edge position data and / or the relative size data to obtain the image diagnostic report.

[0031] In a third aspect, an embodiment of the present disclosure provides a computing device, comprising a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the training method of the diagnostic report generation model as described above and / or the imaging diagnostic report generation method as described above.

[0032] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the training method of the diagnostic report generation model as described in any of the preceding items and / or the imaging diagnostic report generation method as described in the preceding items.

[0033] The embodiment scheme of the present disclosure does not use only the first sample data to train the diagnostic report generation model as in the existing model training method, but uses the first sample data and the second sample data specially constructed for a specific partial tissue. Because the second sample data is specially constructed for a partial tissue, the diagnostic report text therein is a specific diagnostic report for a partial tissue in the multi-tissue sample image, and the corresponding diagnostic prompt words are also prompt words specially set for the partial tissue, so in the diagnostic report generation model training process, the model can implicitly mine the independent relationship between the diagnostic report contents of relatively independent tissues, reducing the probability that the model mixes the diagnostic results of different tissues and generates erroneous results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor, including

[0036] Figure 1 is a flow chart of a training method for a diagnostic report generation model provided by an embodiment of the present disclosure;

[0037] Figure 2 is a flow chart of a method for constructing second sample data provided by some embodiments of the present disclosure;

[0038] Figure 3 is a flow chart of a method for constructing second sample data provided by some embodiments of the present disclosure;

[0039] Figure 4 is a flow chart of a method for generating an imaging diagnosis report provided by an embodiment of the present disclosure;

[0040] Figure 5 It is a structural schematic diagram of a training device for a diagnostic report generation model provided in an embodiment of the present disclosure;

[0041] Figure 6 is a structural schematic diagram of an imaging diagnostic report generating device provided by an embodiment of the present disclosure;

[0042] Figure 7 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0044] The term "including" and its variations used in this document are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. In this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0045] In order to solve the problem that when the existing diagnostic report generation model processes new medical images, the diagnostic report obtained will mix the diagnostic results of different organs and tissues, resulting in report generation errors, the present disclosure embodiment provides a new diagnostic report generation model training method, which improves the generation accuracy of the diagnostic report generation model by improving the training strategy. The diagnostic report generation model training method provided by the present disclosure embodiment is executed by a computing device.

[0046] Figure 1 FIG. 1 is a flow chart of a training method for a diagnostic report generation model provided by an embodiment of the present disclosure. Figure 1 As shown, the training method of the diagnosis report generation model provided by the present disclosure includes S110-S130.

[0047] S110: Acquire first sample data, where the first sample data includes multi-tissue sample images and multi-tissue sample diagnosis reports.

[0048] The first sample data in the embodiment of the present disclosure may be understood as data used in the prior art for training a diagnostic report generation model.

[0049] Each first sample data includes at least two sub-data, namely: (1) a multi-tissue sample image as model input data; and (2) a multi-tissue sample diagnosis report as a reference to the model output data to achieve model parameter adjustment.

[0050] In some embodiments, the first sample data may further include (3) full image diagnosis prompt words. Full image diagnosis prompt words are prompt words used to prompt the generation of a diagnostic report for all tissues of a multi-tissue sample image. In some embodiments of the present disclosure, if the first sample data does not contain the aforementioned full image diagnosis prompt words, the multimodal deep learning model subsequently used for training may default its corresponding prompt words to the full image diagnosis prompt words.

[0051] It should be noted that the "multi-tissue sample image" mentioned in the embodiments of the present disclosure is a coined word, which indicates that there are multiple tissues in one image. However, the meaning of "multi-tissue sample image" is not limited to the above understanding, and it can also be understood that the sample image includes images of multiple organs and / or multiple tissues.

[0052] For example, in one embodiment, the first sample data includes sample data from the open source data set MIMIC-CXR-JPG, wherein each first sample data includes {"prompt":"Output diagnostic report of chest image","text":"R","image":"P"}, wherein "Output diagnostic report of chest image" is the aforementioned full image diagnosis prompt, "R" represents a multi-tissue diagnostic report, and "P" represents a multi-tissue sample image, which is specifically the storage path of multiple groups of sample images.

[0053] According to the above example, the first sample data of an embodiment is as follows.

[0054] {“prompt”: “Output chest imaging diagnosis report”,

[0055] "text":"Image findings: Symmetrical chest, centered mediastinum, enhanced texture of both lungs, nodular shadows in the right lower lung field, no obvious abnormality in the size and shape of the heart.\nImage findings: Enhanced texture of both lungs, please combine with clinical data. Nodular shadows in the right lower lung field, reexamination or further examination is recommended.",

[0056] "image":" / path / to / image"}

[0057] S120: extracting the contents of the multi-tissue sample diagnosis report in the first sample data to obtain a partial tissue diagnosis report and a partial tissue diagnosis prompt word corresponding to the partial tissue diagnosis report; constructing the second sample data based on the partial tissue diagnosis report, the partial tissue diagnosis prompt word, and the corresponding multi-tissue sample images.

[0058] In the disclosed embodiment, content extraction is performed on the multi-tissue sample diagnosis report in the first sample data by using a predetermined text extraction method to extract the diagnosis content related to one tissue in the multi-tissue sample diagnosis report to obtain a partial tissue diagnosis report. For convenience, the predetermined text extraction method will be further analyzed in the following text.

[0059] After obtaining the aforementioned partial tissue diagnosis report, the tissue corresponding to the aforementioned partial tissue diagnosis report can be determined based on the aforementioned partial tissue diagnosis report or other content, and then the corresponding partial tissue diagnosis prompt words are generated. It should be noted that the aforementioned partial tissue can be a single tissue or multiple tissues that have important correlations with each other, which needs to be determined based on the correlation of the tissues in medical diagnosis.

[0060] For example, in an application, a predetermined text extraction method is used to extract the content of the previous specific chest imaging diagnosis report to obtain a diagnosis report of the lung area including "Image findings: nodule shadows are seen in the right lower lung field; Image results: nodule shadows are seen in the right lower lung field." The corresponding generated corresponding part of the tissue diagnosis prompt word is "Output chest image description of lung nodules."

[0061] After obtaining the aforementioned partial tissue diagnosis report and partial tissue diagnosis prompt words, the multi-tissue sample images in the aforementioned first sample data can be used as sample images associated with the aforementioned partial tissue diagnosis report and partial tissue diagnosis prompt words, and the second sample data can be constructed according to the data construction rules of the aforementioned first sample data.

[0062] For example, according to the first sample data constructed as an example in the previous text, the second sample data constructed is as follows.

[0063] {"prompt":"Output the description of lung nodules in chest images",

[0064] "text":"Image findings: Nodular shadows in the right lower lung field. Image findings: Nodular shadows in the right lower lung field.",

[0065] "image":" / path / to / image"}

[0066] S130: Use the first sample data and the second sample data to simultaneously train a pre-built multimodal deep learning model to obtain a diagnosis report generation model.

[0067] After obtaining the aforementioned first sample data and second sample data, the first sample data and the second sample data can be used to train a pre-constructed multimodal deep learning model to obtain a diagnosis report generation model.

[0068] In a specific implementation, the multimodal deep learning model can be a deep learning of various types of architectures, which is not limited by the embodiments of the present disclosure. The multimodal deep learning model can be a model in which all parameters are randomly initialized (that is, a model trained from scratch), or it can be a pre-trained model. Among them, in the case where the multimodal large model is a pre-trained model, it can be retrained by fine-tuning all parameters, or it can be retrained by fine-tuning some parameters to obtain a diagnostic report generation model.

[0069] In a specific implementation, the mathematical method for adjusting the parameters of the multimodal neural network model can be a traditional matrix parameter adjustment method or an adjustment method based on matrix low-rank decomposition, which is not limited in the embodiments of the present disclosure.

[0070] In addition, in a specific embodiment, when the pre-constructed multimodal deep learning model is trained using the first sample data and the second sample number, a more reasonable loss function such as a cross entropy loss function may be used.

[0071] The embodiment scheme of the present disclosure does not use only the first sample data to train the diagnostic report generation model as in the existing model training method, but uses the first sample data and the second sample data specially constructed for a specific partial tissue. Because the second sample data is specially constructed for a partial tissue, the diagnostic report text therein is a specific diagnostic report for a partial tissue in the multi-tissue sample image, and the corresponding diagnostic prompt words are also prompt words specially set for the partial tissue, so in the diagnostic report generation model training process, the model can implicitly mine the independent relationship between the diagnostic report contents of relatively independent tissues, reducing the probability that the model mixes the diagnostic results of different tissues and generates erroneous results.

[0072] According to the above analysis, constructing the second sample data is the key to implementing the training method of the diagnosis report generation model of the embodiment of the present disclosure and constructing a generation model with better diagnosis output results. The following is an analysis of how to construct the second sample data.

[0073] Figure 2 is a flow chart of a method for constructing second sample data provided by some embodiments of the present disclosure. Figure 2 As shown, in some embodiments, the method for constructing the second sample data includes S121-S123.

[0074] S121: Determine possible lesion feature descriptions, and construct text extraction prompt words based on the possible lesion feature descriptions.

[0075] The possible lesion feature description is a lesion feature description determined for the tissue lesion features that may appear in the multi-tissue sample images. In specific implementations, the possible lesion feature description can be determined by radiologists based on experience, or can be determined by automatically screening all multi-tissue sample diagnosis reports. For example, the "pulmonary nodule" mentioned above is a "possible lesion feature description".

[0076] After determining the possible lesion feature description, the possible lesion feature description can be used as the object, and verbs such as "output" and "create" can be used as predicates to construct text extraction prompt words such as "output lung nodule description of chest images (that is, multi-tissue sample images)".

[0077] In some embodiments, in order to ensure the correctness of the subsequent large language processing model in extracting partial tissue diagnosis reports, other restrictive information can also be added to the text extraction prompt words. For example. In some embodiments, in addition to the aforementioned "output chest image (that is, multi-tissue sample image) lung nodule description", the constructed text extraction prompt words also include "if there is no description related to lung nodules, output "none"; the extracted text must ensure correct grammar and smooth sentences, and do not add any information that does not exist in the original report."

[0078] S122: Inputting the text extraction prompt words and the multi-tissue sample diagnosis report into the large language processing model to obtain a partial tissue diagnosis report and partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis report.

[0079] The large language processing model is a pre-trained model used to process or extract complex language content. In specific implementations, the computing device can use various existing large language models that can implement text content interception.

[0080] Specifically, the multi-tissue sample diagnosis report may have the aforementioned possible lesion feature description, or may not have the aforementioned possible lesion feature description.

[0081] In some embodiments, when the computing device determines that a multi-tissue sample diagnosis report contains a target lesion characteristic description, it obtains a partial tissue report corresponding to the target lesion characteristic description, and generates a partial tissue diagnosis prompt word based on the target lesion characteristic description and / or the partial tissue diagnosis report.

[0082] In some embodiments, when the computing device determines that there is no target lesion feature description in the multi-tissue sample diagnosis report, it can have two options: (1) not generating a partial tissue diagnosis report and not generating partial diagnosis prompt words; (2) generating a partial tissue diagnosis report with an empty set of content and generating partial tissue diagnosis prompt words based on the target sideburn feature description.

[0083] S123: Constructing second sample data based on the partial tissue diagnosis report, the partial tissue diagnosis prompt words, and the corresponding multi-tissue sample images.

[0084] Figure 3 is a flow chart of a method for constructing second sample data provided by some embodiments of the present disclosure. Figure 3 As shown, in some embodiments, the method for constructing the second sample data includes S124-S128.

[0085] S124: constructing hierarchical processing prompt words, wherein the hierarchical processing prompt words include text format requirements and text content requirements.

[0086] In the specific implementation, the construction of text hierarchical prompt words needs to be based on the content of the multi-tissue sample image to determine the hierarchical processing prompt words. For example, when the multi-tissue sample image is a chest medical image, it is necessary to construct the hierarchical processing prompt words according to the hierarchical structure of the chest human body system, organ, and tissue. In addition, considering that the hierarchical processing process needs to ensure that the extracted content conforms to the multi-tissue sample diagnosis report, there should be other constraints in the hierarchical processing prompt words.

[0087] For example, in a specific implementation, the hierarchical processing prompt word is {"input":"You are a medical imaging expert. Your task is to extract key local element information from the [imaging report] while strictly complying with the [format requirements] and [content requirements]. [Format requirements]: Please output the key local element information in JSON format. The first-level fields of JSON are "left lung" and "right lung". The second-level fields corresponding to the first-level field "left lung" are "upper lobe" and "lower lobe". The second-level fields corresponding to the first-level field "right lung" are "upper lobe", "middle lobe" and "lower lobe". [Content requirements]: 1. If you determine that the imaging report contains the field information in the format requirements, please extract the description information corresponding to the field from the report and output it to the corresponding field; 2. If you determine that all the field information does not exist in the imaging report, please output "normal" after the non-existent field information.}

[0088] S125: Inputting the hierarchical processing prompt words and the multi-tissue sample diagnosis report into the large language model to obtain the hierarchical processed multi-tissue sample diagnosis report.

[0089] After obtaining the hierarchical processing prompt words, they and the multi-tissue sample diagnosis report are input into the large language model, and the hierarchical processing multi-tissue sample diagnosis report can be obtained. In the specific implementation, the large language processing model hierarchically divides the multi-tissue sample diagnosis report according to the levels determined in the hierarchical processing prompt words to obtain the hierarchical processing multi-tissue sample diagnosis report.

[0090] For example, in some embodiments, a computing device may process a chest imaging report {n[Image Report]: \nImage Performance:; \n1. Lung Texture: Clear and Evenly Distributed. \n2. Lung Field: The bilateral lung fields are clear, and a nodule shadow of about 1 cm in size is seen in the right upper lobe of the lung, with clear boundaries. \n3. No obvious pleural effusion, pneumothorax, or foreign body shadow is seen. \nConclusion: According to the results of the chest X-ray examination, a nodule shadow is seen in the upper lobe of the right lung of the patient. Further examination is recommended to clarify the nature of the nodule, such as CT scan, PET-CT, etc.}, and the hierarchical chest diagnosis report obtained is: {"Left Lung":{"Upper Lobe":"No Abnormality","Lower Lobe":"No Abnormality"},"Right Lung":{"Upper Lobe":"A Nodule Shadow of About 1 cm in Size","Middle Lobe":"No Abnormality","Lower Lobe":"No Abnormality"}}"}.

[0091] S126: extracting content from the hierarchically processed multi-tissue sample diagnosis report to obtain a partial tissue diagnosis report.

[0092] After obtaining the hierarchical multi-tissue sample diagnosis report, the diagnosis content of each part is also hierarchically divided. At this time, the content of the multi-tissue sample diagnosis report can be directly extracted to obtain a hierarchical partial tissue diagnosis report.

[0093] For example, for the hierarchical lung diagnosis report in the previous example, a report of "a nodule of about 1 cm in size is found in the right upper lobe of the lung" can be extracted. If no abnormality is found in other tissues, the corresponding hierarchical partial tissue diagnosis report may not be generated, or a "no abnormality report" may be generated.

[0094] S127: Constructing partial tissue diagnosis prompt words based on the partial tissue diagnosis report.

[0095] After obtaining the aforementioned hierarchical partial tissue diagnosis reports, partial tissue diagnosis prompt words can be constructed in a targeted manner. For example, for the aforementioned "a nodule shadow of about 1 cm in size is found in the right upper lobe of the lung", a prompt word "determine whether there is a nodule in the lung lobe" can be generated.

[0096] S128: Constructing second sample data based on the partial tissue diagnosis report, the partial tissue diagnosis prompt words, and the corresponding multi-tissue sample images.

[0097] The second sample data can be automatically generated by using the aforementioned method of S121-S123 or S124-S128. In other embodiments, the multi-tissue sample diagnosis report in the first sample data can also be manually processed to obtain a partial tissue diagnosis report and a partial tissue diagnosis prompt, and the second sample data can be constructed based on the aforementioned information and the multi-tissue sample images.

[0098] In some embodiments, in addition to executing the aforementioned S110 - S130 , before executing the aforementioned S130 , the computing device may also execute the following S140 - S150 .

[0099] S140: performing region recognition processing on the multi-tissue sample image to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image.

[0100] In a specific implementation, the multi-tissue sample images can be processed for regional recognition based on a predetermined boundary recognition method and a priori tissue boundary shapes to obtain edge position data of each tissue in the multi-tissue sample images. In a specific implementation, the tissue boundaries can be determined by using algorithms such as the Canny edge detection algorithm and the Laplace algorithm. Of course, convolutional neural network methods can also be used to process multi-tissue sample images for regional recognition to obtain edge position data of various tissues in the multi-tissue sample images.

[0101] After obtaining edge position data of various tissues in the multi-tissue sample image, the pixel size of various tissues can be determined. Correspondingly, the relative size data of various tissues can be determined based on the pixel size.

[0102] S150: Using the edge position data and / or the relative size data as model input data in the first sample data and the second sample data.

[0103] After obtaining the edge position data and / or relative size data, the edge position data and / or relative size data can be added to the first sample data and the second sample data as model input data for training the multimodal deep learning model. It should be noted here that the edge position data is the edge data of a certain organization, which is strongly related to a certain organization.

[0104] It can be imagined that by adding edge position data and relative size data to the sample data, the edge position data of the corresponding tissue can be selected for different partial tissue diagnostic prompts during model training to determine the key information in multi-tissue sample images, thereby making the training model more accurate.

[0105] In addition to providing the aforementioned training method for the diagnosis report generation model, the embodiment of the present disclosure also provides a method for generating a needle imaging diagnosis report. Figure 4FIG. 1 is a flow chart of a method for generating an imaging diagnosis report provided by an embodiment of the present disclosure. Figure 4 As shown, the imaging diagnosis report generating method includes S210-S220.

[0106] S210: Obtaining the tissue image to be interpreted and the diagnostic prompt word, where the diagnostic prompt word is used to prompt the content of the diagnostic report obtained by interpreting the tissue image to be interpreted.

[0107] In the disclosed embodiment, the tissue image to be interpreted is a tissue image whose shooting area range is consistent with the multi-tissue sample image range used in the aforementioned model training process. It should be noted that the aforementioned range consistency is not strictly consistent (in actual cases, due to differences in cases, the range cannot be strictly consistent), but only generally consistent. Specifically, the tissues contained in the tissue image to be interpreted and the multi-tissue sample image are generally the same.

[0108] The diagnostic prompt word is input by the user and is intended to instruct the diagnostic report generation model to generate what type of diagnostic report when processing the tissue image to be interpreted, for example, whether to generate a diagnostic report for all tissues in the image or only for some tissues.

[0109] S220: Using a diagnostic report generation model to process the tissue images to be interpreted and the diagnostic prompt words to obtain an imaging diagnostic report.

[0110] After determining the tissue images to be interpreted and the diagnostic prompt words, the two are input into the diagnostic report generation model to obtain the imaging diagnostic report. As previously analyzed, depending on the content of the diagnostic prompt words, the imaging diagnostic report may be a diagnostic report for all tissues, or only a diagnostic report for some tissues.

[0111] In some embodiments, before executing the aforementioned S220, the computing device may further execute the following S230.

[0112] S230: Performing region identification processing on the tissue image to be interpreted to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image.

[0113] The implementation process of S230 is the same as that of S140 in the previous text, which will not be repeated here. For details, please refer to the previous description.

[0114] In the case of executing the aforementioned S230, the aforementioned S220 specifically includes: using a diagnostic report generation model to process the tissue image to be interpreted, the diagnostic prompt words, and the edge position data and / or the relative size data to obtain an image diagnostic report.

[0115] The disclosed embodiment also provides a training device for a diagnosis report generation model. Figure 5Schematic diagram of the structure of the training device for the diagnostic report generation model provided by the embodiment of the present disclosure. Figure 5 As shown, the training device 500 for the diagnosis report generation model includes a sample acquisition unit 501 , a sample construction unit 502 and a model training unit 503 .

[0116] The sample acquisition unit 501 is used to acquire first sample data, where the first sample data includes multi-tissue sample images and multi-tissue sample diagnosis reports.

[0117] The sample construction unit 502 is used to extract the content of the multi-tissue sample diagnosis report in the first sample data to obtain a partial tissue diagnosis report and a partial tissue diagnosis prompt word corresponding to the partial tissue diagnosis report; and construct the second sample data based on the partial tissue diagnosis report, the partial tissue diagnosis prompt word, and the corresponding multi-tissue sample image.

[0118] The model training unit 503 is used to use the first sample data and the second sample data to simultaneously train the pre-built multimodal deep learning model to obtain a diagnosis report generation model.

[0119] In some embodiments, the training device 500 for the diagnostic report generation model further includes a text prompt word construction unit. The text prompt word construction unit is used to determine possible lesion feature descriptions and construct text extraction prompt words based on the possible lesion feature descriptions. The sample construction unit 502 is used to input the text extraction prompt words and the multi-tissue sample diagnostic report into the large language processing model to obtain a partial tissue diagnostic report.

[0120] In some embodiments, when a multi-tissue sample diagnosis report has a target lesion characteristic description, the text prompt word construction unit obtains a partial tissue diagnosis report corresponding to the target lesion characteristic description; and when a partial tissue diagnosis report of the target lesion characteristic description is obtained, a partial tissue diagnosis prompt word is generated based on the target lesion characteristic description and / or the partial tissue diagnosis report.

[0121] In some embodiments, the training device 500 of the diagnosis report generation model also includes a hierarchical processing prompt word construction unit. The hierarchical processing prompt word construction unit is used to construct hierarchical processing prompt words, and the hierarchical processing prompt words include text format requirements and text content requirements. The sample construction unit 502 inputs the hierarchical processing prompt words and the multi-tissue sample diagnosis report into the large language model to obtain the hierarchical processed multi-tissue sample diagnosis report; extracts the content of the hierarchical processed multi-tissue sample diagnosis report to obtain a partial tissue diagnosis report; and constructs the partial tissue diagnosis prompt words based on the partial tissue diagnosis report.

[0122] In some embodiments, the training device 500 for the diagnosis report generation model further includes an image feature extraction unit. The feature extraction unit is used to perform region recognition processing on the multi-tissue sample images, obtain edge position data and / or relative size data of various tissues in the multi-tissue sample images, and use the edge position data and / or relative size data as model input data in the first sample data and the second sample data.

[0123] In some embodiments, the model training unit 503 uses the first sample data and the second sample data to simultaneously fine-tune the pre-trained multimodal deep learning model to obtain a diagnosis report generation model.

[0124] The disclosed embodiment also provides a device for generating an imaging diagnosis report. Figure 6 Schematic diagram of the structure of the imaging diagnosis report generating device provided by the embodiment of the present disclosure. Figure 6 As shown, the image diagnosis report generating device 600 includes an input data acquisition unit 601 and a model processing unit 602. The input data acquisition unit 601 is used to obtain the tissue image to be interpreted and the diagnostic prompt word, and the diagnostic prompt word is used to prompt the content of the diagnostic report obtained by interpreting the tissue image to be interpreted. The model processing unit 602 is used to use the diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word to obtain an image diagnosis report. The aforementioned tissue image to be interpreted is a tissue image whose shooting area range is consistent with the image range of the multi-tissue sample.

[0125] In some embodiments, the image diagnosis report generating device 600 further includes an image feature extraction unit for performing region identification processing on the image of the tissue to be interpreted, and obtaining edge position data and / or relative size data of various tissues in the multi-tissue sample image. Accordingly, the model processing unit 602 uses the diagnosis report generating model to process the image of the tissue to be interpreted, the diagnosis prompt word, and the edge position data and / or relative size data to obtain the image diagnosis report.

[0126] The embodiment of the present disclosure also provides a computing device for implementing the aforementioned method. Figure 7 Schematic diagram of the structure of the computing device provided by the embodiment of the present disclosure. Figure 7 , which shows a schematic diagram of the structure of a computing device 700 suitable for implementing the embodiments of the present disclosure. Figure 7 The computing device shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0127] like Figure 7As shown, the computing device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 702 or a program loaded from a storage device 708 to a random access memory RAM 703. Various programs and data required for the operation of the computing device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output I / O interface 705 is also connected to the bus 704.

[0128] Typically, the following devices may be connected to the I / O interface 705: input devices 705 including, for example, a touch screen, a touch pad, a camera, a microphone, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the computing device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The computing device 700 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0129] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0130] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0131] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0133] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0134] The computer-readable medium may be included in the computing device, or may exist independently without being incorporated into the computing device.

[0135] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester computer, partially on the tester computer, as a separate software package, partially on the tester computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the tester computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0136] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the unit does not, in some cases, constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a diagnostic report generation model, characterized in that: include: Acquiring first sample data, wherein the first sample data includes multi-tissue sample images and multi-tissue sample diagnosis reports; Extracting contents of the multi-tissue sample diagnosis reports in the first sample data to obtain partial tissue diagnosis reports and partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis reports; and constructing second sample data based on the partial tissue diagnosis reports, the partial tissue diagnosis prompt words, and the corresponding multi-tissue sample images; The first sample data and the second sample data are used to simultaneously train a pre-built multimodal deep learning model to obtain a diagnosis report generation model.

2. The training method according to claim 1, characterized in that: Before extracting content from the multiple tissue sample diagnosis reports in the first sample data to obtain a partial tissue diagnosis report, the method further includes: Determining possible lesion feature descriptions, and constructing text extraction prompt words based on the possible lesion feature descriptions; The extracting content from the multiple tissue sample diagnosis reports in the first sample data to obtain a partial tissue diagnosis report includes: inputting the text extraction prompt word and the multiple tissue sample diagnosis report into a large language processing model to obtain the partial tissue diagnosis report.

3. The training method according to claim 2, characterized in that: Obtaining the partial tissue diagnosis report and the partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis report, including: In the case where the multi-tissue sample diagnosis report has a target lesion characteristic description, obtaining a partial tissue diagnosis report corresponding to the target lesion characteristic description; The obtaining of the partial tissue diagnosis prompt word corresponding to the partial tissue diagnosis report includes: upon obtaining the partial tissue diagnosis report of the target lesion characteristic description, generating the partial tissue diagnosis prompt word based on the target lesion characteristic description and / or the partial tissue diagnosis report.

4. The training method according to claim 1, characterized in that: Before extracting content from the multiple tissue sample diagnostic reports in the first sample data to obtain a partial tissue diagnostic report, the method includes: Constructing hierarchical processing prompt words, wherein the hierarchical processing prompt words include text format requirements and text content requirements; The extracting contents of the multiple tissue sample diagnostic reports in the first sample data to obtain a partial tissue diagnostic report includes: Inputting the hierarchical processing prompt words and the multi-tissue sample diagnosis report into a large language model to obtain a hierarchical processed multi-tissue sample diagnosis report; Extracting content from the hierarchically processed multi-tissue sample diagnostic reports to obtain the partial tissue diagnostic reports; The obtaining of the partial tissue diagnosis prompt words corresponding to the partial tissue diagnosis report includes: constructing the partial tissue diagnosis prompt words based on the partial tissue diagnosis report.

5. The training method according to any one of claims 1 to 4, characterized in that: The method further comprises: Performing region recognition processing on the multi-tissue sample image to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image; The edge position data and / or relative size data are used as model input data in the first sample data and the second sample data.

6. The training method according to any one of claims 1 to 4, characterized in that: The pre-built multimodal deep learning model is trained using the first sample data and the second sample data to obtain a diagnosis report generation model, including: The first sample data and the second sample data are used to simultaneously fine-tune the pre-trained multimodal deep learning model to obtain the diagnosis report generation model.

7. A method for generating an imaging diagnosis report, characterized in that: include: Acquiring a tissue image to be interpreted and a diagnostic prompt word, wherein the diagnostic prompt word is used to prompt the content of a diagnostic report obtained by interpreting the tissue image to be interpreted; Using a diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word to obtain an imaging diagnostic report; The diagnosis report generation model is obtained by using the training method of the diagnosis report generation model as claimed in any one of claims 1 to 5, and the tissue image to be interpreted is a tissue image whose shooting area range is consistent with the image range of the multiple tissue samples.

8. The method according to claim 7, characterized in that: Before using the diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word to obtain an image diagnostic report, the method further includes: Performing region identification processing on the tissue image to be interpreted to obtain edge position data and / or relative size data of various tissues in the multi-tissue sample image; The method comprises: using a diagnostic report generation model to process the tissue image to be interpreted and the diagnostic prompt word, and the edge position data and / or the relative size data to obtain the image diagnostic report.

9. A computing device, characterized in that It comprises a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the training method of the diagnosis report generation model as described in any one of claims 1 to 6 and / or the imaging diagnosis report generation method as described in any one of claims 7 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program. When the computer program is executed by the processor, the processor implements the training method of the diagnosis report generation model as described in any one of claims 1 to 6 and / or the imaging diagnosis report generation method as described in any one of claims 7 to 8.

Citation Information

Cited By

  • Image diagnosis report quality evaluation method and device and storage medium

    CN119693962A

  • Radiology report generation method based on hierarchical knowledge integration framework

    CN119785966A