Intelligent generation of medical image diagnostic report

Abstract

The present application relates to the intelligent generation of a medical image diagnostic report. A method according to an embodiment of the present application comprises: acquiring a medical image, and determining a region of interest in the medical image; on the basis of the region of interest determined in the medical image, determining initial diagnostic text by means of a text generation model, and displaying the initial diagnostic text to a user; and in response to input information of the user, adjusting the region of interest and / or the initial diagnostic text, so as to generate a diagnostic report.

Description

Intelligent generation of medical image diagnosis report TECHNICAL FIELD

[0001] The present specification relates to the field of computer technology, and in particular to intelligent generation of medical image diagnosis report. BACKGROUND

[0002] In the process of diagnosing diseases through medical images, doctors need to check each part of the human body on the medical image to determine the abnormal part, and manually record the lesion information of the abnormal part on the diagnosis report according to the lesion condition of the abnormal part.

[0003] SUMMARY

[0004] The present specification provides a medical image diagnosis report generation method, including: acquiring a medical image and determining a region of interest in the medical image; determining an initial diagnosis text according to the determined region of interest in the medical image through a text generation model, and displaying the initial diagnosis text to a user; and generating a diagnosis report in response to input information of the user, adjusting the region of interest and / or the initial diagnosis text.

[0005] In some embodiments, determining the initial diagnosis text according to the determined region of interest in the medical image includes: determining one or more preset human tissues; for each human tissue in the one or more human tissues, determining an image region corresponding to the human tissue in the medical image in which the region of interest is determined; determining a description text of the human tissue according to the image region corresponding to the human tissue through the text generation model; and determining the initial diagnosis text according to the description texts of the one or more human tissues.

[0006] In some embodiments, generating a diagnosis report in response to input information of the user, adjusting the initial diagnosis text includes: acquiring the input information of the user, performing semantic recognition on the input information, determining a human tissue corresponding to the input information as a region to be updated; determining a description text corresponding to the region to be updated in the initial diagnosis text as a text to be updated; updating the text to be updated in the initial diagnosis text through the input information to obtain an updated initial diagnosis text; and generating the diagnosis report according to the updated initial diagnosis text.

[0007] In some embodiments, the method further includes: in response to a selection operation of the user on the initial diagnosis text, determining the description text selected by the user; determining an image region of a human tissue corresponding to the description text selected by the user in the medical image in which the region of interest is determined as a display region; and displaying the display region to the user.

[0008] In some embodiments, in response to the input information of the user, adjusting the region of interest, generating the diagnostic report comprises: determining, according to the input information, an adjustment operation of the user on the region of interest in the medical image; updating the region of interest in the medical image according to the adjustment operation to obtain an updated region of interest; and re-generating, by the text generation model, the initial diagnostic text as a diagnostic report according to the updated region of interest.

[0009] In some embodiments, the input information comprises text information or voice information; and in response to the input information of the user, adjusting the initial diagnostic text comprises: respectively extracting features of the input information and the initial diagnostic text to determine semantic features of the input information and semantic features of the initial diagnostic text; determining whether the semantics of the input information and the semantics of the initial diagnostic text are opposite; if the semantics of the input information and the semantics of the initial diagnostic text are opposite, replacing the initial diagnostic text with the input information; and if the semantics of the input information and the semantics of the initial diagnostic text are not opposite, merging the input information and the initial diagnostic text.

[0010] In some embodiments, determining whether the semantics of the input information and the semantics of the initial diagnostic text are opposite comprises: determining a first similarity between the semantic features of the input information and the semantic features of the initial diagnostic text; in response to determining that the first similarity is less than a first preset value, determining that the semantics of the input information and the semantics of the initial diagnostic text are opposite; and in response to determining that the first similarity is greater than or equal to the first preset value, determining that the semantics of the input information and the semantics of the initial diagnostic text are not opposite.

[0011] In some embodiments, if the semantics of the input information and the semantics of the initial diagnostic text are not opposite, merging the input information and the initial diagnostic text comprises: determining whether a second similarity between the semantic features of the input information and the semantic features of the initial diagnostic text is greater than a second preset value; in response to determining that the second similarity is greater than or equal to the second preset value, fusing the semantics of the input information and the initial diagnostic text; and in response to determining that the second similarity is less than the second preset value, splicing the semantics of the input information and the initial diagnostic text.

[0012] In some embodiments, in response to the input information of the user, the adjustment on the region of interest and / or the initial diagnosis text comprises: normalizing the input information by a language model to obtain normalized information, wherein the normalization operation comprises at least one of converting non-medical terms into medical terms, deleting repeated words, and correcting misspelled words; and adjusting the initial diagnosis text according to the normalized information.

[0013] In some embodiments, the method further comprises: extending the diagnosis report by a pre-trained language model to obtain an extended diagnosis report, wherein the extended diagnosis report comprises the diagnosis report and examination suggestions.

[0014] In some embodiments, the input information comprises text information or voice information; and before the adjustment on the initial diagnosis text in response to the input information of the user, the method further comprises: obtaining the input information of the user, performing intent recognition on the input information, determining a user intent of the input information, wherein the user intent is classified into a relevant intent, an instruction intent, and an irrelevant intent; when the input information indicates that the user intent is the relevant intent, retaining the input information for the adjustment on the initial diagnosis text; when the input information indicates that the user intent is the instruction intent, determining a corresponding instruction of the input information from preset instructions, and performing an operation step of the corresponding instruction of the input information; and when the input information indicates that the user intent is the irrelevant intent, ignoring the input information, and not using the input information for the adjustment on the initial diagnosis text.

[0015] The specification provides a medical image diagnosis report generation device, the device comprising: an acquisition module, acquiring a medical image and determining a region of interest in the medical image; a determination module, determining an initial diagnosis text according to the determined region of interest in the medical image by a text generation model, and displaying the initial diagnosis text to a user; and an adjustment module, adjusting the region of interest and / or the initial diagnosis text in response to input information of the user, and generating a diagnosis report.

[0016] The specification provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the medical image diagnosis report generation method.

[0017] The specification provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the medical image diagnosis report generation method when executing the program.

[0018] The above at least one technical solution adopted by the specification can achieve the following beneficial effects:

[0019] In the method for generating a medical image diagnosis report provided in the present specification, a medical image is acquired, a region of interest in the medical image is determined, an initial diagnosis text is determined from the region of interest in the medical image by a text generation model, the initial diagnosis text is displayed to a user, the region of interest and / or the initial diagnosis text are adjusted in response to input information of the user, and a diagnosis report is generated. Through the method, in the whole process of generating the diagnosis report, only the initial diagnosis text generated by the text generation model needs to be checked and modified by the user, and the diagnosis report can be obtained, thereby improving the generation efficiency of the diagnosis report and the diagnosis efficiency of the user. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the present specification, constitute a part of the present specification, and the illustrative embodiments of the present specification and their description serve to explain the present specification, and do not constitute an improper limitation on the present specification. In the drawings:

[0021] FIG. 1 is a flowchart of a medical image diagnosis report intelligent generation method according to an embodiment of the present specification.

[0022] FIG. 2 is an initial diagnosis text generation interface provided in an embodiment of the present specification.

[0023] FIG. 3 is a diagnosis report generation interface provided in an embodiment of the present specification.

[0024] FIG. 4 is a flowchart of an initial diagnosis text determination method provided in an embodiment of the present specification.

[0025] FIG. 5 is an initial diagnosis text generation interface provided in an embodiment of the present specification.

[0026] FIG. 6 is an initial diagnosis text linkage display interface provided in an embodiment of the present specification.

[0027] FIG. 7 is an initial diagnosis text generation interface provided in an embodiment of the present specification.

[0028] FIG. 8 is an initial diagnosis text linkage display interface corresponding to FIG. 7 provided in an embodiment of the present specification.

[0029] FIG. 9 is an initial diagnosis text linkage display interface provided in an embodiment of the present specification.

[0030] FIG. 10 is an initial diagnosis text linkage display interface provided in an embodiment of the present specification.

[0031] FIG. 11 is an intent recognition process provided in an embodiment of the present specification.

[0032] FIG. 12 is a diagram of an extended diagnostic report generation interface provided in an embodiment of the present specification.

[0033] FIG. 13 is a diagram of a device scenario provided in an embodiment of the present specification.

[0034] FIG. 14 is a diagram of a medical image diagnostic report intelligent generation apparatus provided in the present specification.

[0035] FIG. 15 is a diagram of an electronic device corresponding to FIG. 1 provided in the present specification. DETAILED DESCRIPTION

[0036] For the purposes of the present specification, technical solutions and advantages, the technical solutions of the present specification will be described in detail below with reference to the embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of the present application.

[0037] The technical solutions provided by the embodiments of the present specification will be described in detail below with reference to the drawings.

[0038] FIG. 1 is a flow diagram of a medical image diagnostic report generation method provided in the present specification, including steps S100 to S104.

[0039] S100: Obtain a medical image and determine a region of interest in the medical image.

[0040] All steps in the medical image diagnostic report intelligent generation method provided in the present specification can be implemented by any electronic device with computing function, such as a terminal, a server, and the like. For ease of description, the medical image diagnostic report intelligent generation method provided in the present specification will be described below with the server as the execution subject.

[0041] In the medical image diagnostic report intelligent generation method provided in the present specification, the number, type, or scanning site of the medical image used for generating the diagnostic report is not limited.

[0042] The server can obtain at least one medical image, such as an X-ray image, a magnetic resonance imaging (MRI) image, a computed tomography (CT) image, a positron emission computed tomography (PET-CT) image, and the like, from a certain image acquisition device, or obtain multiple medical images of different types from multiple image acquisition devices.

[0043] After the server obtains the medical image, the server determines a region of interest in the medical image. The region of interest is a region in the medical image that needs to be focused on, including a region in the medical image that is likely to have a lesion, a marked region for anatomical positioning, a related normal tissue structure region for comparison with the lesion region, and the like. Due to the differences in medical images, the server can determine one or more regions of interest.

[0044] After the region of interest is determined, the server can analyze the medical image according to the marked region of interest and output a diagnosis report.

[0045] There can be various methods for determining the region of interest, and the server can determine the region of interest by a deep learning method, such as a target analysis model for marking the region of interest in the medical image that is trained based on a region-based convolutional neural network (R-CNN), a single shot multibox detector (SSD), and the like. There are many mature technologies in this field at present, and therefore, the description is not expanded here. Alternatively, the server can obtain an image result containing the region of interest marked by artificial labeling to determine the region of interest.

[0046] S102: Determine an initial diagnosis text according to the region of interest determined in the medical image by a text generation model, and display the initial diagnosis text to the user.

[0047] The text generation model is trained to align the image features of the medical image with the features of the diagnosis conclusion of the medical image, and the image features of the medical image and the text features of the diagnosis conclusion of the medical image have similarity in the feature space. Therefore, the server can analyze the image features of the medical image according to the position, size, shape, and grayscale value of the region of interest in the medical image by the text generation model according to the region of interest determined in the medical image, and determine the initial diagnosis text. In some other embodiments, the text generation model can be a small model for detecting a certain region of interest, which can detect a region of interest frame, such as the position and size of a lesion (region of interest). In addition to inputting the medical image including the region of interest to the text generation model, at least one of the detection result of the region of interest, the mask segmentation result, and the image analysis result can be input to the text generation model.

[0048] The specific network structure of the text generation model is not limited in the specification, and can be any network structure such as a transformer network, a convolutional neural network, or a recurrent neural network.

[0049] The server determines the medical image of the region of interest, inputs the text generation model, and obtains the initial diagnosis text given by the text generation model. The initial diagnosis text contains a diagnosis conclusion of the abnormal cause of the region of interest. The server displays the initial diagnosis text to the user to check whether the initial diagnosis text has errors or omissions. The user in the specification refers to medical personnel or researchers with certain medical knowledge.

[0050] The generation of the initial diagnosis text is based on the text generation model, and no manual operation is required in the generation process of the initial diagnosis text. The initial diagnosis text is a preliminary diagnosis conclusion of the medical image. The user can choose to modify the initial diagnosis text or correct the deficiencies of the initial diagnosis text according to the specific circumstances of the initial diagnosis text after viewing the initial diagnosis text displayed by the server, thereby obtaining a diagnosis report and improving the generation efficiency of the diagnosis report.

[0051] S104: In response to the input information of the user, the region of interest and / or the initial diagnosis text are adjusted to generate a diagnosis report.

[0052] When the user finds that the content in the initial diagnosis text has errors or omissions, the user can adjust the region of interest and / or the initial diagnosis text. The server determines the diagnosis report according to the adjusted region of interest and / or the initial diagnosis text. The method of the specification supports the user to adjust the region of interest and the user to adjust the initial diagnosis text.

[0053] In one or more embodiments of the specification, the user can adjust the region of interest determined in the medical image in step S100 to indirectly adjust the initial diagnosis text.

[0054] The server determines the user's adjustment operation on the region of interest in the medical image according to the input information of the user. The input information can be a click, sketch, or frame selection operation performed by the user operating a mouse, a handwriting board, or other input devices. The adjustment information determined by the server according to the input information of the user can include the increase of the region of interest caused by the frame selection of a new region of interest, the deletion of at least one region of interest determined in step S100, the modification of the size of the marking frame of the region of interest determined in step S100, and the like.

[0055] The server updates the region of interest in the medical image according to the above adjustment operation. The medical image with the updated region of interest is input into the text generation model in step S102 again to obtain the initial diagnosis text output by the text generation model as the diagnosis report.

[0056] In one or more embodiments of the present specification, the user can directly adjust the initial diagnosis text displayed by the server in step S102. That is, the server can directly adjust the text content of the initial diagnosis text according to the input information of the user.

[0057] The server extracts features from the input information to determine the semantic features of the input information, and extracts features from the initial diagnosis text to determine the semantic features of the initial diagnosis text.

[0058] In the present embodiment, the input information of the user can be text information or voice information. If the input information is text information, the server can extract features from the text information to determine the semantic features of the text information.

[0059] If the input information is voice information, the server can first convert the voice information into text information, and then extract features from the converted text information to determine the semantic features of the voice information. Alternatively, the server can extract features from the audio data of the obtained voice information to determine the semantic features of the voice information.

[0060] Then, the server determines whether the semantics of the input information and the initial diagnosis text are opposite according to the semantic features of the input information and the semantic features of the initial diagnosis text. That is, the semantics are opposite, which means that the meanings of the content described by the input information and the initial diagnosis text conflict with each other, and the content described by the input information and the content described by the initial diagnosis text cannot occur at the same time. For example, the input information represents an abnormality, and the initial diagnosis text represents no abnormality, which means that the semantics are opposite.

[0061] If yes, the initial diagnosis text is replaced by the input information. If no, the input information and the initial diagnosis text are merged. When the input information is voice information, the server replaces the initial diagnosis text with the text information corresponding to the voice information or merges the initial diagnosis text with the text information corresponding to the voice information.

[0062] The user can choose to indirectly update the initial diagnosis text by adjusting the region of interest in the medical image or directly adjust the initial diagnosis text according to the actual situation. In the generation process of a diagnosis report, the server can adjust the initial diagnosis text according to the adjustment operation selected by the user according to the adjustment process of one of the above two embodiments, or can adjust the initial diagnosis text according to the adjustment process of the above two embodiments.

[0063] The method in the specification supports user operations of adjusting the region of interest and the initial diagnosis text, and the user can flexibly select a specific way of adjusting operation according to actual conditions, thereby improving user experience.

[0064] When the region of interest is adjusted, the server only needs to obtain a selection operation of the user on a region in the image, and regenerate the initial diagnosis text according to the adjustment of the region of interest by the user. The user can adjust the region of interest in the process of observing the medical image, and the adjustment of the initial diagnosis text can be realized, without spending extra time to input voice information or text information, and the efficiency is higher.

[0065] When the initial diagnosis text is adjusted, the user directly inputs a correct diagnosis conclusion to update the initial diagnosis text when finding that there is a description error or a missed lesion site in the initial diagnosis text. This adjustment method is more direct, and the user operation is more convenient. In particular, when the user inputs voice information to modify the initial diagnosis text, no manual operation needs to be performed in the whole process of generating the diagnosis report, and the generation efficiency of the diagnosis report is improved.

[0066] FIG. 2 is an initial diagnosis text generation interface provided in an embodiment of the specification. In the interface shown in FIG. 2, the left column is a display interface of the initial diagnosis text, and the right column is a display interface of the medical image. The shaded rectangle represents the region of interest. In the left column, the initial diagnosis text displayed to the user, an input box, and a "determine" operation icon are included. The input box is provided with "text input" and "voice input" operation icons.

[0067] The user can adjust the region of interest in the medical image in the right column, so that the server regenerates the initial diagnosis text according to the adjusted region of interest by the user, or the user can adjust the initial diagnosis text through text information or voice information by operating the "text input" or "voice input" icon in the left column.

[0068] After the user adjusts the initial diagnosis text, and determines that the regenerated initial diagnosis text is accurate, the user can operate the "determine" icon to generate a diagnosis report. An example of a diagnosis report interface can be as shown in FIG. 3, which is a diagnosis report generation interface provided in an embodiment of the specification.

[0069] Based on the medical image diagnosis report intelligent generation method shown in FIG. 1, in the generation process of the diagnosis report, only a small amount of modification of the initial diagnosis text by the user is needed to obtain the diagnosis report of the medical image. Compared with the traditional handwritten diagnosis report method, the intelligent generation method of the diagnosis report provided in the specification greatly improves the generation efficiency of the diagnosis report.

[0070] To make the diagnostic report more orderly and convenient for users to view, in one or more embodiments of the present specification, the initial diagnostic text is generated based on preset human tissues.

[0071] In step S102, the server determines the initial diagnostic text according to the method shown in FIG. 4.

[0072] FIG. 4 is a flowchart of an initial diagnostic text determination method provided in an embodiment of the present specification. As shown in FIG. 4, the method includes steps S200 to S206.

[0073] S200: Determine one or more preset human tissues.

[0074] The present specification does not limit the manner of determining the one or more preset human tissues. Any human structure classification manner can be used for determination, or the scanning range of the medical image can be used for determination, or the patient's medical history and other personal conditions can be used for determination, etc.

[0075] After determining the one or more preset human tissues, when presenting the initial diagnostic text to the user, the diagnostic result of each human tissue can be presented to enhance the orderliness and readability of the initial diagnostic text.

[0076] S202: For each human tissue in the one or more human tissues, determine the image region corresponding to the human tissue in the medical image in which the region of interest is determined.

[0077] In order to display the diagnostic results of the human tissues respectively, the text generation model determines the initial diagnostic text by detecting each human tissue according to the image region corresponding to the human tissue. Therefore, the server determines the image region corresponding to each human tissue in the medical image in which the region of interest is determined.

[0078] S204: Determine the description text of the human tissue according to the image region corresponding to the human tissue by the text generation model.

[0079] S206: Determine the initial diagnostic text according to the description text of the one or more human tissues.

[0080] The server analyzes the image region corresponding to the human tissue by using the text generation model. The text generation model generates the description text corresponding to the image region of the human tissue according to the determination of the region of interest in the image region corresponding to the human tissue. In some embodiments, for example, the human tissue is a lung, and the determined region of interest is a region where a lung lesion is located. For example, the list format of the detected lesion (region of interest) is [lesion type, lung position where the lesion is located, image layer number where the lesion is located, lesion attribute, size], but the application is not limited to this list format. The list of detected lesions is as follows:

[0081] Lesion 1 (region of interest 1): [pulmonary nodule, left upper lobe, IM23, ground glass, 3mm*4mm];

[0082] Lesion 2 (region of interest 2): [pulmonary nodule, left upper lobe, IM28, ground glass, 3*5mm];

[0083] Lesion 3 (region of interest 3): [pulmonary nodule, right upper lobe, IM35, ground glass, 6mm*7mm];

[0084] Lesion 4 (region of interest 4): [cyst, right lower lobe, IM110, lucent shadow, 8mm*9mm].

[0085] By using the text generation model, the description text of the lung can be determined as follows: multiple ground glass nodule shadows are visible in the left upper lobe (IM23, IM28) and the right upper lobe (IM35), and the largest one has a maximum diameter of about 7mm; a cystic lucent shadow is visible in the right lower lobe (IM110), and the size is about 8mm*9mm.

[0086] In other embodiments, for example, the human tissue is an abdomen, and the determined region of interest is a liver organ. The region of interest (liver organ) can be normal tissue for comparison with other lesion regions. The human tissue described in the foregoing can include a human body part or structure, and the image region corresponding to the human tissue can include an image region corresponding to an organ or part of the human body, such as an image region corresponding to the liver or an image region corresponding to the abdomen. The region of interest in the image region corresponding to the human tissue can be a lesion, or it can be a region of interest of a certain normal part or structure.

[0087] If no region of interest is determined in the image region of the human tissue, the text generation model gives a description text representing the normal state of the human tissue. If one or more regions of interest exist in the image region of the human tissue, the text generation model analyzes the features of the one or more regions of interest and obtains a diagnostic conclusion of the abnormal state of the human tissue, thereby giving a description text of the image region of the human tissue.

[0088] The server determines the initial diagnosis text according to the description text of the image area of each human tissue.

[0089] The server can determine the initial diagnosis text according to the above steps S200-S206. In one or more embodiments of the present specification, in order to further facilitate the user to quickly determine the disease detected by the medical image according to the initial diagnosis text, when the initial diagnosis text is displayed to the user in the above step 102, the server can display the description text indicating that the human tissue is abnormal to the user in a more eye-catching manner to remind the user to pay attention.

[0090] Specifically, for the description text corresponding to each human tissue, the server determines whether the description text indicates that the human tissue is abnormal. If yes, the description text is displayed to the user in a preset first display effect. If no, the description text is displayed to the user in a preset second display effect. The first display effect is more eye-catching than the second display effect.

[0091] FIG. 5 is an initial diagnosis text generation interface provided in an embodiment of the present specification, wherein A-E represent five preset human tissues. As shown in FIG. 5, the description text corresponding to the three human tissues A, D, and E is displayed in bold, indicating that the description text of the three human tissues A, D, and E indicates that the three human tissues A, D, and E are abnormal, and the description text corresponding to the two human tissues B and C is not displayed in bold, indicating that the description text of the two human tissues B and C indicates that the two human tissues B and C are normal. In this way, the user can be reminded to focus on the three human tissues A, D, and E.

[0092] In the present embodiment, the server displays the description text indicating that the human tissue is abnormal to the user in a more eye-catching manner, so that the user can more quickly notice the description text of the patient's human tissue with lesions and first check the description text of the human tissue with lesions.

[0093] If the server determines the initial diagnosis text according to the above steps S200-S206. In one or more embodiments of the present specification, the server can display the initial diagnosis text to the user in a list, table, or other formatted structure.

[0094] In the formatted structure, in order to further indicate which human tissue each description text specifically corresponds to, in another embodiment, the server can also determine the part identifier of each human tissue, so that the user can distinguish the human tissue corresponding to the description text through the part identifier. The specific form of the part identifier is not limited in the present embodiment and can be a graphical identifier, a textual identifier, etc.

[0095] For each human tissue, the server displays the part identifier of the human tissue and the description text corresponding to the human tissue to the user.

[0096] On the basis of the embodiment, in order to facilitate the user to check the correctness of the description text of each human tissue, the server can determine the description text corresponding to the part identifier selected by the user in response to the selection operation of the user on the part identifier, determine the image region associated with the description text corresponding to the part identifier in the medical image of the region of interest, and display the image region to the user.

[0097] FIG. 6 is a diagram of an initial diagnosis text linkage display interface provided in an embodiment of the present specification, wherein A-E represent five preset human tissues, and the shaded rectangle in front of A-E represents a part identifier. The part identifiers of the human tissues are different, so as to facilitate the user to distinguish different human tissues. As shown in FIG. 6, when the user clicks the part identifier corresponding to D, the server displays the image region corresponding to the human tissue D in the right column in response to the selection operation of the user.

[0098] As described in the above step S100, the region of interest in the medical image can be determined by a trained target analysis model, which includes but is not limited to a detection model, a segmentation model, a classification model, etc.

[0099] When the server determines the region of interest by the target analysis model, the initial diagnosis text can be determined according to the method shown in FIG. 4.

[0100] Generally, the user has reservations about the result of the region of interest determined by the target analysis model, because the result of the region of interest (such as the category, size, position, range, nature, brightness, etc.) will affect the diagnosis conclusion of the disease, so the image region of the region of interest determined by the target analysis model generally needs to be checked by the user. Generally, the region of interest is the detection result of the region where the lesion may exist, and the user needs to review whether the region of interest is consistent with the actual situation of the patient.

[0101] In order to facilitate the user to check the description text of the human tissue corresponding to the image region where the region of interest exists in the initial diagnosis text, in one or more embodiments of the present specification, the server displays the description text of the human tissue corresponding to the image region where the region of interest exists determined by the target analysis model to the user with a preset third display effect in the initial diagnosis text. The description text of the human tissue corresponding to the image region where the region of interest exists determined by the target analysis model is displayed to the user with a preset fourth display effect.

[0102] The present specification does not limit the specific display effects of the third display effect and the fourth display effect, as long as the third display effect and the fourth display effect are different, so as to facilitate the user to distinguish. For example, the difference between the third display effect and the fourth display effect can be whether there is a preset identifier, color difference, font difference, etc.

[0103] In one example, the difference between the third display effect and the fourth display effect can be whether there is a preset identifier. The present example is described below in conjunction with FIG. 7. FIG. 7 is an initial diagnosis text generation interface provided in an embodiment of the present specification. As shown in FIG. 7, A-E represent different human tissues, and [AI] represents a preset identifier. In the initial diagnosis text, the description text of each human tissue is displayed to the user in sections. The description text of the human tissues A, C, and D is marked with the [AI] identifier in the upper right corner, indicating that the image region corresponding to the human tissues A, C, and D is determined to have a region of interest by the target analysis model.

[0104] Based on the present example, in order to facilitate the user to check the description text corresponding to the image region having the region of interest, the server can determine the description text corresponding to the preset identifier selected by the user, determine the image region associated with the description text corresponding to the preset identifier in the medical image having the region of interest, and display the image region having the region of interest to the user in response to the selection operation of the user on the preset identifier.

[0105] FIG. 8 is an initial diagnosis text linkage display interface corresponding to FIG. 7 provided in an embodiment of the present specification. As shown in FIG. 8, the user performs a selection operation on the [AI] identifier in the upper right corner of the description text of the human tissue D. The server displays the image region corresponding to the human tissue D in the right column in response to the operation of the user.

[0106] In step S102 described above, if the server determines the initial diagnosis text by the method described in S200-S206, the initial diagnosis text is composed of the description text of each human tissue.

[0107] In step S104, when the server adjusts the initial diagnosis text according to the input information of the user, the server can first determine the description text that needs to be adjusted according to the input information, and only adjust the description text that needs to be adjusted, so as to improve the generation efficiency of the diagnosis report.

[0108] In one or more embodiments of the present specification, the input information of the user includes text information or voice information.

[0109] Specifically, the server obtains the input information of the user, performs semantic recognition on the input information, and determines the human tissue corresponding to the input information as the part to be updated.

[0110] Similar to step S104 described above, if the input information is text information, the server can perform semantic recognition on the text information. If the input information is voice information, the server can first convert the voice information into text information, and then perform semantic recognition on the text information. Alternatively, the server can also perform semantic recognition on the audio data of the voice information.

[0111] Then, the server determines, in the initial diagnosis text, description text of human tissue corresponding to the to-be-updated part as to-be-updated text. The to-be-updated text in the initial diagnosis text is updated according to the input information, and a diagnosis report is determined according to the updated initial diagnosis text. The updating of the to-be-updated text in the initial diagnosis text can include modification, deletion, and addition of the to-be-updated text.

[0112] In one or more embodiments of the present specification, the server adjusts the region of interest according to the input information of the user.

[0113] Specifically, the server obtains input information of the user, determines an adjustment operation of the user on the region of interest in the medical image, and determines an image region in the medical image corresponding to the adjustment operation. Then, the server determines, in the initial diagnosis text, description text of human tissue corresponding to the image region as to-be-updated text. According to the adjusted image region, the description text corresponding to the image region is regenerated in the initial diagnosis text to determine an updated initial diagnosis text as a diagnosis report.

[0114] In step S102 described above, if the server determines the initial diagnosis text by the method described in S200-S206, that is, the initial diagnosis text is composed of description texts of each human tissue.

[0115] In one or more embodiments of the present specification, in order to facilitate the user to check the correctness of the initial diagnosis text through the medical image, the server can determine, in response to a selection operation of the user on the description text of each human tissue in the initial diagnosis text, an image region corresponding to the selected description text of human tissue in the medical image, and display the image region to the user.

[0116] FIG. 9 is an initial diagnosis text linkage display interface provided in an embodiment of the present specification, as shown in FIG. 9, when the user clicks the description text corresponding to the human tissue D, the server responds to the selection operation of the user to display the image region corresponding to the human tissue D in the right column.

[0117] Further, in order to facilitate the user to view the image region more finely, in one or more embodiments of the present specification, the server can also display the scale parameters of the image region to the user while displaying the image region, the scale parameters include corresponding parameters for adjusting image brightness, contrast, gray value range, etc., and the scale parameters corresponding to different types of medical images are different, such as window width and window level in CT images. The server can also display interface elements for adjusting the scale parameters of the image region to the user while displaying the image region, such as zoom-in and zoom-out instruction icons, image rotation icons, etc.

[0118] On the basis of the embodiment, the server can also automatically match the scale parameter and the gray parameter that are more conducive to viewing the image region according to the size of the image region, the properties of the corresponding human tissue, and the like before showing the image region to the user, and show the image region to the user with the matched scale parameter and gray parameter that are more conducive to viewing the image region.

[0119] FIG. 10 is a diagram of an initial diagnosis text linkage display interface provided in an embodiment of the present specification. In FIG. 10, the medical image is taken as a CT image. When the user clicks the description text corresponding to the human tissue D, the server displays the image region corresponding to the human tissue D in the right column in response to the selection operation of the user, and displays the zoom-in, zoom-out, and rotation icons in sequence to the right of the image region, and displays the setting icons of the window width and window level below the image region.

[0120] In the step S104, when the server adjusts the initial diagnosis text according to the input information of the user, it is necessary to determine whether the semantics of the input information is opposite to the semantics of the initial diagnosis text.

[0121] In one or more embodiments of the present specification, the server can determine whether the semantics of the input information is opposite to the semantics of the initial diagnosis text according to the similarity between the semantic features of the input information and the semantic features of the initial diagnosis text.

[0122] Specifically, the server determines whether the similarity is less than a first preset value. If yes, it is determined that the semantics of the input information is opposite to the semantics of the initial diagnosis text, and the replacement operation of replacing the initial diagnosis text with the input information is performed. The specific value of the first preset value can be set according to the requirement. For example, the initial diagnosis text is “trachea and bronchus are unobstructed”, and the input information is “left lower lobe bronchial dilatation with inflammation”. The former is a description indicating normal lung, and the latter is a description indicating abnormal lung. After the server determines that the semantics of the two is opposite, the replacement operation of replacing the initial diagnosis text with the input information is performed.

[0123] If no, it is determined that the semantics of the input information is not opposite to the semantics of the initial diagnosis text, and the merging operation of the input information and the initial diagnosis text is performed.

[0124] In the step S104, when the server determines that the semantics of the input information is not opposite to the semantics of the initial diagnosis text, the input information and the initial diagnosis text are merged.

[0125] In one or more embodiments of the present specification, in order to increase the accuracy of the merging of the input information and the initial diagnosis text and make the expression in the initial diagnosis text more concise and enhance the reading experience of the user, the server can further divide the merging operation into two types of splicing and fusion.

[0126] Specifically, when the server determines that the semantics of the input information is not opposite to the semantics of the initial diagnosis text, it is further determined whether the similarity between the semantic features of the input information and the semantic features of the initial diagnosis text is greater than a second preset value. The specific value of the second preset value can also be set according to the needs.

[0127] If yes, it indicates that the input information and the initial diagnosis text are descriptions of the same type of disease, and the semantics of the input information and the initial diagnosis text are fused. For example, the initial diagnosis text is "left 2 posterior rib fracture", and the input information is "left 5 posterior rib fracture". The semantics of the input information and the initial diagnosis text are similar, and the server combines the two to determine the updated initial diagnosis text as "left 2, 5 posterior rib fracture".

[0128] If no, it indicates that the input information and the initial diagnosis text are not descriptions of the same type of disease, and the semantics of the input information and the initial diagnosis text are spliced. For example, the initial diagnosis text is "sternal body shows patchy bone destruction", and the input information is "left 5 posterior rib fracture". The server determines that the semantics of the input information and the initial diagnosis text are neither opposite nor similar, and splices the two to determine the updated initial diagnosis text as "sternal body shows patchy bone destruction, left 5 posterior rib fracture".

[0129] In step S102 described above, if the input information of the user is text information or voice information, the server can further preprocess the input information through a language model after obtaining the input information to standardize the expression of the input information and make more accurate adjustments to the initial diagnosis text. When the input information is voice information, the server converts the voice information into text information and then preprocesses the text information. The language model mentioned here and in the following can be obtained by training any neural network model suitable for natural language processing tasks.

[0130] In one or more embodiments of the present specification, the preprocessing can be content normalization.

[0131] Specifically, the server inputs the input information into a pre-trained first language model to perform normalization operation on the input information. The normalization operation can include converting non-medical terms into medical terms, deleting repeated words, correcting misspelled words, etc. to determine the standardized information corresponding to the input information, and adjusting the initial diagnosis text according to the standardized information.

[0132] For example, the input information determined by the server through the text information or the voice information is "trachea normal", which is a colloquial description of the normal trachea part. The standardized description should be "trachea, bronchus unobstructed". The server can determine that the standardized information corresponding to the input information is "trachea, bronchus unobstructed" through the first language model, and use the standardized information to adjust the initial diagnosis text.

[0133] Through the embodiment, the server adjusts the initial diagnosis text according to the standardized information, so that the final obtained diagnosis report content is more standardized and has stronger readability.

[0134] In one or more embodiments of the present specification, the preprocessing can also be intent recognition.

[0135] Specifically, the server determines the user intent of the input information through the pre-trained second language model. In one or more embodiments of the present specification, the server determines the user intent corresponding to the input information through the second language model in the preset user intent classification.

[0136] The preset user intent classification at least includes a relevant intent and an irrelevant intent. The relevant intent is an intent related to the diagnosis conclusion of the medical image, and the irrelevant intent is an intent irrelevant to the diagnosis conclusion of the medical image. The input information of the irrelevant intent can be a user's mistaken input of text information, or voice information of casual chatting, etc. If the input information corresponding to the irrelevant intent appears in the diagnosis report, it will affect the quality of the diagnosis report.

[0137] If the server determines that the user intent corresponding to the input information is the relevant intent, the initial diagnosis text is adjusted according to the input information. If the server determines that the input information is the irrelevant intent, the input information is ignored, and the initial diagnosis text is not adjusted according to the input information.

[0138] Through the embodiment, the server filters out the information irrelevant to the generation of the diagnosis report input by the user through intent recognition before adjusting the initial diagnosis text, prevents irrelevant information from being introduced into the diagnosis report, and improves the generation quality of the diagnosis report.

[0139] In one or more embodiments of the present specification, the server can also activate the instruction corresponding to the input information according to the input information in the preset instructions, and respond to the user according to the instruction corresponding to the input information.

[0140] Therefore, the preset user intent can also include an instruction intent. When the server determines that the user intent of the input information is the instruction intent, the instruction corresponding to the instruction intent is determined in the preset instructions, and the operation steps of the instruction corresponding to the instruction intent are executed.

[0141] The preset instructions in this embodiment can be various instructions that a user can achieve by operating a graphical user interface. For example, "empty the initial diagnosis text result", "regenerate the initial diagnosis text", "determine to generate a diagnosis report", and the like.

[0142] Generally, after the user checks the initial diagnosis text displayed by the server and adjusts the initial diagnosis text, the user needs to operate the graphical user interface to execute the instruction "determine to generate a diagnosis report" so that the server stores the diagnosis report determined by the user or performs other operations such as exporting. Through this embodiment, the user can input the text or voice "determine to generate a diagnosis report" to control the server to execute the instruction "determine to generate a diagnosis report".

[0143] FIG. 11 is a process diagram of an intent recognition provided in an embodiment of the present specification. As shown in FIG. 11, the server recognizes the input information of the user, and after intent recognition, if the input information is irrelevant intent, the input information is ignored, if it is determined that the input information is instruction intent, and it is determined that the instruction corresponding to the instruction intent is "determine to generate a diagnosis report", the user is responded to the instruction.

[0144] In FIG. 11, input information 1 is "description of A part", and input information 2 is "description of D part". After the server determines that the user intent of the input information is relevant intent, the input information 1 is used to adjust the description text of the A organization, and the input information 2 is used to adjust the description text of the D organization.

[0145] If it is determined that the input information is relevant intent, it is used to adjust the initial diagnosis text,

[0146] Through this embodiment, the user can obtain the same response effect as operating the image user interface by inputting information, without the user going to the interface to find the operation elements of each instruction, thereby increasing the convenience of user operation.

[0147] When the input information of the user is voice information, the user usually needs to operate the interface element representing voice input in the graphical user interface, so that the server responds to the operation of the user to start the listening mode and start receiving the voice information input by the user. In this embodiment, the wake-up word corresponding to the instruction for starting the listening mode can be pre-set, and the user can input the wake-up word through voice information to start the listening mode, so that the server can use the voice information input by the user to adjust the initial diagnosis text. The wake-up word here can be set to any word according to the demand.

[0148] Therefore, when the input information is voice information, the user can not have to perform any manual operation, and the operation portability and the diagnosis report generation efficiency are greatly improved.

[0149] Further, when the user intention indicated by the input information is an instruction intention, in order to improve the fault tolerance of the method to the non-standard input of the user, the server can determine the instruction corresponding to the input information according to the semantics of the input information. For example, when the input text information or voice information is "generate a report", "determine to generate", or other information consistent with the semantics of the preset instruction "determine to generate a diagnostic report", the server can also respond to the user with the corresponding instruction "determine to generate a diagnostic report".

[0150] In one embodiment, the server can perform intention recognition on the input text information or voice information in the case of determining the initial diagnostic text according to the method described in S200-S206.

[0151] The adjustment of the initial diagnostic text according to the input information can be implemented in the following steps: the server performs intention recognition on the input information, determines the user intention indicated by the input information is a related intention, determines the human tissue corresponding to the input information, and determines the description text corresponding to the human tissue in the initial diagnostic text.

[0152] Then, the server extracts features from the input information to determine the semantic features of the input information. The server also extracts features from the description text corresponding to the human tissue to determine the semantic features of the description text corresponding to the human tissue. The server determines whether the semantics of the input information and the semantics of the description text corresponding to the human tissue are opposite according to the semantic features of the input information and the semantic features of the description text corresponding to the human tissue.

[0153] If the semantics of the input information and the semantics of the description text corresponding to the human tissue are opposite, the description text corresponding to the human tissue is replaced by the input information to obtain an updated initial diagnostic text. If the semantics of the input information and the semantics of the description text corresponding to the human tissue are not opposite, the input information is combined with the initial diagnostic text to obtain an updated initial diagnostic text.

[0154] Through this embodiment, when adjusting the initial diagnostic text, the server quickly determines the description text that needs to be adjusted according to the preset human tissues, which not only ensures the accuracy of the adjustment of the diagnostic text, but also improves the generation efficiency of the diagnostic report.

[0155] After the above step S104, the server can further expand the generated diagnostic report through a pre-trained third language model.

[0156] In many cases, the diagnosis conclusion given by the user according to the diagnosis report of the medical image is not the final conclusion of the patient's condition. After obtaining the diagnosis conclusion of the medical image, the patient needs to combine the diagnosis conclusions of other medical examination items to give the final conclusion of the patient's condition. Therefore, the diagnosis report of the medical image usually contains not only the diagnosis conclusion of the condition, but also the examination suggestions given by the doctor according to the diagnosis conclusion, such as perfecting some medical examination items, etc.

[0157] In one or more embodiments of the present specification, in order to further reduce the time spent by the user in entering the diagnosis report, the server can generate examination suggestions for the patient according to the relevant description of the region of interest in the diagnosis report through the third language model, and combine the diagnosis report and the examination suggestions as an extended diagnosis report.

[0158] In the knowledge base, various historical conditions and examination suggestions corresponding to each historical condition are pre-stored. The server inputs the diagnosis report into the third language model, which determines the condition contained in the diagnosis report according to the coding features of the diagnosis report in the knowledge base, and outputs the condition contained in the diagnosis report and the examination suggestions corresponding to each condition. Then, the server displays the extended diagnosis report to the user.

[0159] In one or more embodiments of the present specification, after obtaining the diagnosis report containing the examination suggestions, the server displays the extended diagnosis report and the examination report obtained in step S104 which does not contain the examination suggestions to the user, so that the user can view the correspondence between the extended diagnosis report and the diagnosis report obtained in step S104 to determine the correctness of each generated examination suggestion.

[0160] Specifically, the server determines, as a target sentence, the sentence corresponding to the selected sentence in the extended diagnosis report in the diagnosis report obtained in step S104 in response to the selection operation of the user on the sentence in the extended diagnosis report. Then, the target sentence is displayed to the user with a preset highlighting display effect.

[0161] The present embodiment does not limit the specific form of the preset highlighting display effect, which can be set as a highlighting, flashing, or other display effect according to specific needs.

[0162] Because the extended diagnosis report is the output result of the third language model, although the extended diagnosis report contains the diagnosis report and the examination suggestions, the content of the extended diagnosis report output by the third language model is not completely the same as the content of the diagnosis report obtained in step S104.

[0163] The extended diagnostic report content output by the third language model can be a result of further optimization such as summarizing and normalizing the language of the diagnostic report obtained in step S104.

[0164] Through the embodiment, the user can quickly notice the corresponding sentence in the diagnostic report obtained in step S104 based on the selected sentence in the extended diagnostic report, thereby improving the checking efficiency of the user.

[0165] Meanwhile, since the target sentence is based on the sentence selected by the user in the extended diagnostic report, the user can check the correctness of the inspection suggestion through the target sentence, thereby enhancing the user's trust in the decision-making process of the third language model and thus enhancing the user experience.

[0166] FIG. 12 is a diagnostic report generation interface provided in an embodiment of the present specification, as shown in FIG. 12, the diagnostic report and the extended diagnostic report are displayed together, and when the user clicks on a sentence in the extended diagnostic report, the corresponding sentence in the diagnostic report, i.e., the target sentence, is also highlighted.

[0167] After the above step S104, the server can further optimize and extend the text content of the generated diagnostic report through a pre-trained fourth language model.

[0168] In the above step S104, the diagnostic report determined by the server is obtained by modifying the initial diagnostic text generated by the text generation model. In order to improve the accuracy of the diagnostic report, in one or more embodiments of the present specification, the server inputs the diagnostic report in S104 into a language model to further optimize and extend the content of the diagnostic report, so that the language of the diagnostic report is more concise, the text errors are eliminated, and the repeated words are removed.

[0169] The method described in the embodiment can be set as a default execution process or a selectable execution process in the application process.

[0170] If set as a default execution process, the server can automatically execute the embodiment to determine the final diagnostic report after executing the above step S104.

[0171] If set as a selectable execution process, the server can set an interface element for starting the process on a graphical user interface operated by the user, and execute the process to optimize and extend the diagnostic report in S104 in response to the selection operation of the user on the interface element.

[0172] Since the setting is the default process, the generation time of the diagnostic report is increased while improving the quality of the diagnostic report. By setting the execution process to be selectable, the user can flexibly select whether to optimize and expand the diagnostic report according to the actual situation, thereby further enhancing the user experience.

[0173] The specification provides a medical image diagnostic report generation device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The intelligent generation device also includes a display device and an input device. The processor executes the medical image diagnostic report intelligent generation method provided by the specification, and the input device is used to obtain the input information of the user.

[0174] Due to the particularity of medical images, a special display is usually used for display. Compared with ordinary displays, the special display generally has higher resolution and more accurate display of color and contrast, which facilitates the user to observe the medical image accurately and in detail. If the medical image and the initial diagnosis text or the diagnostic report are displayed on an ordinary display, it is not conducive to the user's observation of the medical image.

[0175] Therefore, in one or more embodiments of the specification, the display device of the intelligent generation device provided by the specification includes two displays, namely a first display and a second display. The first display is a special display for medical images and is used to display medical images. The second display is an ordinary display and is used to display the initial diagnosis text or the diagnostic report to the user.

[0176] When the input information of the user is text information or voice information, the text information or voice information is displayed synchronously on the first display and the second display. The specification does not limit the specific implementation of synchronous display, which can be displayed in the same way as the input box in the display page of the second display by displaying a floating window, scrolling subtitles, etc. on the first display. In order to facilitate the user to check the correctness of the input information obtained by the input device while viewing the medical image through the first display. Instead of frequently switching the display device to increase the length of the diagnosis process when the user views the medical image, thereby further improving the generation efficiency of the diagnostic report and improving the user's diagnosis efficiency.

[0177] FIG. 13 is a device scene diagram provided in an embodiment of the specification, as shown in FIG. 13, the left display is the first display, and the right display is the second display. In the interface of the first display, the input information display box above the medical image is consistent with the content of the input box in the interface of the second display.

[0178] The specification also provides a corresponding medical image diagnostic report intelligent generation device, as shown in FIG. 14.

[0179] FIG. 14 is a schematic diagram of a medical image diagnosis report generation apparatus provided in the present specification, comprising: an acquisition module 300 configured to acquire a medical image and determine a region of interest in the medical image; a determination module 302 configured to determine, by a text generation model, an initial diagnosis text according to the region of interest determined in the medical image, and display the initial diagnosis text to a user; and an adjustment module 304 configured to adjust the region of interest and / or the initial diagnosis text in response to input information of the user, and generate a diagnosis report.

[0180] Optionally, the determination module 302 is configured to determine one or more preset human tissues, for each human tissue of the one or more human tissues, determine an image region corresponding to the human tissue in the medical image in which the region of interest is determined, determine a description text of the human tissue by a text generation model according to the image region corresponding to the human tissue, and determine the initial diagnosis text according to the description texts of the one or more human tissues.

[0181] Optionally, the adjustment module 304 is configured to determine an adjustment operation of the user on the region of interest in the medical image according to the input information, update the region of interest in the medical image to obtain an updated region of interest according to the adjustment operation, and regenerate the initial diagnosis text as a diagnosis report by the text generation model according to the updated region of interest.

[0182] Optionally, the input information comprises text information or voice information, and the adjustment module 304 is configured to perform feature extraction on the input information and the initial diagnosis text respectively, determine semantic features of the input information and semantic features of the initial diagnosis text, judge whether the semantics of the input information and the semantics of the initial diagnosis text are opposite, replace the initial diagnosis text by the input information if the semantics of the input information and the semantics of the initial diagnosis text are opposite, and merge the input information and the initial diagnosis text if the semantics of the input information and the semantics of the initial diagnosis text are not opposite.

[0183] Optionally, the adjustment module 304 is configured to acquire the input information of the user, perform semantic recognition on the input information, determine a human tissue corresponding to the input information as a region to be updated, determine a description text corresponding to the region to be updated in the initial diagnosis text as a text to be updated, update the text to be updated in the initial diagnosis text by the input information to obtain an updated initial diagnosis text, and generate a diagnosis report according to the updated initial diagnosis text.

[0184] Optionally, the determining module 302 is configured to, in response to the selection operation performed by the user on the initial diagnosis text, determine the description text selected by the user, determine the image area of the human tissue corresponding to the description text selected by the user as a display area in the medical image of the region of interest, and display the display area to the user.

[0185] Optionally, the input information includes text information or voice information, the adjusting module 304 is configured to acquire the input information of the user, perform intent recognition on the input information, determine the user intent of the input information, the user intent includes a relevant intent, an instruction intent and an irrelevant intent, when the input information indicates that the user intent is the relevant intent, the input information is reserved for adjusting the initial diagnosis text, when the input information indicates that the user intent is the instruction intent, in each instruction of the preset instructions, the instruction corresponding to the input information is determined, and the operation step of the instruction corresponding to the input information is performed, when the input information indicates that the user intent is the irrelevant intent, the input information is ignored, and the input information is not used for adjusting the initial diagnosis text.

[0186] The specification also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the medical image diagnosis report intelligent generation method provided in FIG. 1.

[0187] The specification also provides a schematic structural diagram of the electronic device shown in FIG. 15. As shown in FIG. 15, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the medical image diagnosis report intelligent generation method described in FIG. 1. Of course, in addition to the software implementation, the specification does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0188] For a technical improvement, it can be obvious whether the improvement is in hardware (e.g., improvement of circuit structures of diodes, transistors, switches, etc.) or in software (e.g., improvement of method processes). However, with the development of technology, many improvements of method processes nowadays can be considered as direct improvements of hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method processes into hardware circuits. Therefore, it cannot be said that an improvement of a method process cannot be implemented by a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "onto" a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used when developing programs, and the original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there are many kinds of HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should be clear to those skilled in the art that only a little logical programming of the method processes in the above-mentioned hardware description languages and programming into integrated circuits can easily obtain hardware circuits that implement the logical method processes.

[0189] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0190] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0191] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in one or more software and / or hardware in implementing the present specification.

[0192] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0193] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0194] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0196] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0197] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0198] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0199] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or other elements inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0200] The specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0201] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

Claims

1. A medical image diagnosis report generation method, comprising: obtaining a medical image and determining a region of interest in the medical image; determining, by a text generation model, an initial diagnosis text according to the determined region of interest in the medical image, and displaying the initial diagnosis text to a user; adjusting the region of interest and / or the initial diagnosis text in response to input information of the user, and generating a diagnosis report.

2. The method of claim 1, wherein, The determining of the initial diagnosis text according to the determined region of interest in the medical image comprises: determining one or more preset human tissues; for each of the one or more human tissues, determining an image region corresponding to the human tissue in the medical image in which the region of interest is determined; determining, by the text generation model, a description text of the human tissue according to the image region corresponding to the human tissue; determining the initial diagnosis text according to the description texts of the one or more human tissues.

3. The method of claim 2, wherein, The adjusting of the initial diagnosis text in response to the input information of the user and the generating of the diagnosis report comprise: obtaining the input information of the user, performing semantic recognition on the input information, determining a human tissue corresponding to the input information as a region to be updated; determining, in the initial diagnosis text, a description text corresponding to the region to be updated as a text to be updated; updating the text to be updated in the initial diagnosis text by the input information to obtain an updated initial diagnosis text; generating the diagnosis report according to the updated initial diagnosis text.

4. The method of claim 2 or 3, further comprising: determining a description text selected by the user in response to a selection operation of the user on the initial diagnosis text; determining, in the medical image in which the region of interest is determined, an image region of a human tissue corresponding to the description text selected by the user as a display region; displaying the display region to the user.

5. The method of claim 1, wherein, The adjusting of the region of interest in response to the input information of the user and the generating of the diagnosis report comprise: determining an adjustment operation of the user on the region of interest in the medical image according to the input information; updating the region of interest in the medical image according to the adjustment operation to obtain an updated region of interest; re-generating, by the text generation model, the initial diagnosis text according to the updated region of interest as the diagnosis report.

6. The method of claim 1, wherein, The input information comprises text information or voice information; The adjusting of the initial diagnosis text in response to the input information of the user comprises: performing feature extraction on the input information and the initial diagnosis text respectively to determine semantic features of the input information and semantic features of the initial diagnosis text; determining whether the semantics of the input information and the semantics of the initial diagnosis text are opposite; if the semantics of the input information and the semantics of the initial diagnosis text are opposite, replacing the initial diagnosis text by the input information. If the semantics of the input information is not opposite to the semantics of the initial diagnosis text, the input information is merged with the initial diagnosis text.

7. The method of claim 6, wherein, It is judged whether the semantics of the input information is opposite to the semantics of the initial diagnosis text, comprising: determining a first similarity between the semantic features of the input information and the semantic features of the initial diagnosis text, in response to determining that the first similarity is less than a first preset value, it is determined that the semantics of the input information is opposite to the semantics of the initial diagnosis text; in response to determining that the first similarity is greater than or equal to the first preset value, it is determined that the semantics of the input information is not opposite to the semantics of the initial diagnosis text.

8. The method of claim 6, wherein, If the semantics of the input information is not opposite to the semantics of the initial diagnosis text, the input information is merged with the initial diagnosis text, comprising: determining whether a second similarity between the semantic features of the input information and the semantic features of the initial diagnosis text is greater than a second preset value, in response to determining that the second similarity is greater than or equal to the second preset value, the semantics of the input information and the initial diagnosis text are fused; in response to determining that the second similarity is less than the second preset value, the semantics of the input information and the initial diagnosis text are spliced.

9. The method of claim 1, wherein, In response to the input information of the user, the method for adjusting the region of interest and / or the initial diagnosis text, comprising: normalizing the input information through a language model to obtain standardized information, wherein the normalization operation includes at least one of converting non-medical terms into medical terms, deleting repeated words, and correcting misspelled words; adjusting the initial diagnosis text according to the standardized information.

10. The method of claim 1, further comprising: The diagnostic report is expanded through a pre-trained language model to obtain an expanded diagnostic report, and the expanded diagnostic report includes the diagnostic report and examination suggestions.

11. The method of claim 1, wherein, The input information includes text information or voice information; Before adjusting the initial diagnosis text in response to the input information of the user, the method further comprises: obtaining the input information of the user, performing intent recognition on the input information, and determining the user intent of the input information, the user intent including relevant intent, instruction intent and irrelevant intent; when the input information indicates that the user intent is relevant intent, the input information is retained for adjusting the initial diagnosis text; when the input information indicates that the user intent is instruction intent, among the preset instructions, the instruction corresponding to the input information is determined, and the operation steps of the instruction corresponding to the input information are executed; when the input information indicates that the user intent is irrelevant intent, the input information is ignored and is not used for adjusting the initial diagnosis text.

12. A medical image diagnosis report generation device, comprising: an acquisition module for determining a region of interest in a medical image; a determination module for determining an initial diagnosis text based on the determined region of interest in the medical image through a text generation model, and displaying the initial diagnosis text to a user; An adjusting module, in response to the input information of the user, adjusts the region of interest and / or the initial diagnosis text to generate a diagnosis report. 13.A computer readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the method of any one of claims 1-11.

14. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the method of any one of claims 1-11.

Images