Image processing device, image display method, image display program, recording medium, and image diagnosis system

Abstract

An image processing device 1 comprises: an identification result reception unit 102 that receives a lesion identification result a from an image; a text generation unit 102a that generates, as text, basis information of the identification; a classification unit 103 that classifies constituent elements of the text into an object to be emphasized and an object not to be emphasized; an emphasis unit 104 that emphasizes the object to be emphasized; and an output unit 105 that outputs at least the object to be emphasized in the text to a monitor.

Description

Image processing device, image display method, image display program, recording medium, and image diagnosis system

[0001] In recent years, advances in IT technology have led to the practical application of image diagnosis using AI (Artificial Intelligence) in various fields. In image diagnosis, a technique for presenting evidence information for inferences made by AI is known (see, for example, Patent Literature 1).

[0002] Patent No. 7161979

[0003] In the medical field, there is a possibility that AI for computer-aided diagnosis (CAD), which is used in endoscopic diagnosis, will be required to provide all supporting information to the user, the doctor. However, the time available for diagnosis is limited.

[0004] Therefore, the present invention aims to provide an image processing device, an image display method, an image display program, a storage medium, and an image diagnostic system that can generate a display image that allows medical professionals and others to quickly identify the underlying information when referring to the image diagnostic results of medical images obtained by CAD or the like.

[0005] An image processing device according to one aspect of the present invention includes a classification result receiving unit that receives a classification result of a lesion from an image, and a sentence generating unit that generates sentences based on information on the basis of the classification. The image processing device also includes a classification unit that classifies components of the sentences into emphasis targets and non-emphasis targets. The image processing device also includes an emphasis unit that emphasizes the emphasis targets. The image processing device also includes an output unit that outputs at least the emphasis targets of the sentences to a monitor.

[0006] An image processing device according to one aspect of the present invention includes one or more processors that receive a classification result for an image, generate a sentence based on the classification basis information, classify components of the sentence into emphasis targets and non-emphasis targets, highlight the emphasis targets, and output at least the emphasis targets of the sentence and the classification result to a monitor.

[0007] An image display method according to one aspect of the present invention receives the results of image classification, generates the basis information for the classification as text, classifies the components of the text into highlight targets and non-highlight targets, highlights the highlight targets, and outputs at least the highlight targets of the text to a monitor.

[0008] An image display program according to one aspect of the present invention receives the results of image classification, generates the basis information for the classification as text, classifies the components of the text into emphasis targets and non-emphasis targets, emphasizes the emphasis targets, generates a diagram linked to the emphasis targets, and outputs at least the emphasis targets of the text to a monitor.

[0009] A storage medium according to one aspect of the present invention stores an image display program that receives the results of image classification, generates the basis information for the classification as text, classifies the components of the text into emphasis targets and non-emphasis targets, emphasizes the emphasis targets, generates a diagram associated with the emphasis targets, and outputs at least the emphasis targets of the text to a monitor.

[0010] An image diagnostic system according to one aspect of the present invention includes a classification unit that receives evidence information for the differentiation of a lesion and classifies components of a sentence that constitutes the evidence information into emphasis targets and non-emphasis targets. The image diagnostic system also includes an emphasis unit that emphasizes the emphasis targets. The image diagnostic system further includes an output unit that outputs at least the emphasis targets of the sentence to a monitor.

[0011] According to the present invention, when a medical professional or the like refers to the image diagnosis results of a medical image obtained by CAD or the like, it is possible to generate a display image that allows the medical professional or the like to quickly identify the basis information.

[0012] 1 is a block diagram illustrating an example of the configuration of a medical system including an image processing device according to a first embodiment of the present invention. A schematic block diagram illustrating a detailed configuration of an endoscopic device. A block diagram illustrating a functional configuration of an image processing device according to the first embodiment. A flowchart illustrating an example of a discrimination algorithm. A diagram illustrating an example of a display image according to the first embodiment. A diagram illustrating an example of a display image according to the first embodiment. A diagram illustrating an example of a display image according to the first embodiment. A diagram illustrating an example of a display image according to the first embodiment. A diagram illustrating an example of a display image according to the second embodiment. A diagram illustrating an example of a display image according to the second embodiment. A diagram illustrating an example of a display image according to the second embodiment. A diagram illustrating an example of a display image according to the third embodiment. A diagram illustrating an example of a display image according to the fourth embodiment. A diagram illustrating an example of a display image according to the fifth embodiment. A diagram illustrating an example of a display image according to the fifth embodiment. A diagram illustrating an example of a display image according to the sixth embodiment. A diagram illustrating an example of a display image according to the seventh embodiment. A diagram illustrating an example of a display image according to the seventh embodiment. A diagram illustrating an example of a display image according to the seventh embodiment. A diagram illustrating an example of a display image according to the eighth embodiment.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment Hereinafter, a first embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0014] Fig. 1 is a block diagram illustrating an example of the configuration of a medical system including an image processing device according to a first embodiment of the present invention. The medical system according to this embodiment is mainly composed of an image processing device 1, an endoscope device 3, a display device 41, and a network 5. As shown in Fig. 1, the system may also include a server 2. In the medical system shown in Fig. 1, medical images acquired by the endoscope device 3 or the like are analyzed, and information is output to assist a doctor in making a diagnosis by differentiating lesions and the like.

[0015] The image processing device 1 differentiates lesions from medical images and outputs the results of the differentiation. The image processing device 1 includes a processor 11 and a storage device 12.

[0016] The processor 11 is connected to the network 5 and includes a central processing unit (hereinafter referred to as CPU) 111 and hardware circuits 112 such as ROM and RAM. The processor 11 may include an integrated circuit such as an FPGA instead of the CPU or separate from the CPU.

[0017] The storage device 12 stores various software programs. Each software program is read and executed by the CPU 111 of the processor 11. Note that all or part of the various programs may be stored in the ROM of the processor 11.

[0018] The storage device 12 stores a program (not shown) used to control the operation of the image processing device 1, and an image display program 121. The image display program 121 is a program that performs processing related to the generation of a display image based on the results of lesion differentiation for medical images. The storage device 12 also stores various setting values ​​and parameters required for executing the image display program 121. The storage device 12 is also configured to be able to store medical images, etc. acquired from the server 2 and the endoscope device 3.

[0019] The server 2 is connected to the network 5 and includes a processor 21 and a storage device 22. The processor 21 includes a CPU and the like. The storage device 22 is a large-capacity storage device such as a hard disk drive. The server 2 accumulates medical images output from an endoscope device 3 and the like connected to the network 5. The server 2 also has a function of transmitting medical images in response to a request from an image processing device 1 and the like connected to the network 5.

[0020] The endoscope device 3 is an instrument for observing the inside of a subject's body. Fig. 2 is a schematic block diagram illustrating the detailed configuration of the endoscope device 3. The endoscope device 3 includes an endoscope 31 and a video processor 32.

[0021] The endoscope 31 has an elongated insertion section 311 , an operation section 312 provided at the base end of the insertion section 311 , and a universal cable 313 extending from the operation section 312 .

[0022] An image sensor 314 is provided at the tip of the insertion section 311. Reflected light from the subject illuminated by illumination light emitted from an illumination window (not shown) is incident on the imaging surface of the image sensor 314 via an observation window (not shown). The image sensor 314 outputs an imaging signal of the subject image.

[0023] The operation unit 312 has various operation members 312a including a bending knob for up-down directions, a bending knob for left-right directions, various operation buttons, etc. The user of the endoscope 31 can operate these operation members 312a to bend a bending portion (not shown) provided in the insertion portion 311, or give instructions to record endoscopic images.

[0024] The endoscope 31 is connected to the video processor 32 by a connector 313 a provided at the base end of a universal cable 313 .

[0025] An imaging signal from the imaging element 314 is supplied to the video processor 32 via a signal line 315 that passes through the insertion portion 311 , the operation portion 312 and the universal cable 313 .

[0026] The video processor 32 includes a control unit 321, an image processing unit 322, a communication interface (hereinafter abbreviated as communication I / F) 323, and an operation panel 324. The video processor 32 is connected to the network 5.

[0027] The control unit 321 controls the overall operation and execution of various functions of the endoscope device 3. The control unit 321 includes a CPU, ROM, RAM, etc., and programs for various operations and controls are recorded in the ROM.

[0028] The CPU reads out from the ROM and executes a program corresponding to an operation signal from the various operation members 312a of the operation unit 312 or the operation panel 324, thereby causing the entire operation of the endoscope device 3 and the execution of various functions.

[0029] Under the control of the control unit 321, the image processing unit 322 drives the imaging element 314 and receives an imaging signal from the imaging element 314. Under the control of the control unit 321, the image processing unit 322 generates an endoscopic image based on the imaging signal and outputs the image to the control unit 321.

[0030] The communication I / F 323 is a circuit that connects the control unit 321 to the network 5. The control unit 321 outputs the endoscopic image to the server 2 via the communication I / F 323.

[0031] The operation panel 324 has various buttons and the like for the user to specify the execution of various functions. The operation panel 324 is, for example, a display device with a touch panel. The user can operate the operation panel 324 to instruct the endoscope device 3 to execute a desired function.

[0032] The control unit 321 generates an image signal of an endoscopic image to be displayed on the display device 41, and outputs the image signal to the display device 41. The endoscopic image based on the image signal is displayed on the screen of the display device 41.

[0033] The display device 41 is included in the notification device 4. The notification device 4 may include an audio device 42 in addition to the display device 41. The display device 41 may include a monitor or the like. The video processor 32 may transmit image data to the display device 41 via the network 5, or may transmit image data to the display device 41 via the network 5 and the image processing device 1. When the evidence described below is presented as text, depending on the evidence, it may be sensitive information for the subject undergoing the endoscopic examination, such as text suggesting the possibility of cancer. For this reason, the relative angle or height between the display device 41 and the examination table on which the subject lies may be adjusted so that the evidence information is not visible to the subject. Furthermore, goggles with a monitor may be used as the display device 41 instead of a stationary monitor.

[0034] Next, a function of generating a display image from a medical image in the image processing device 1 will be described. Fig. 3 is a block diagram for explaining the functional configuration of the image processing device according to the first embodiment. The image processing device 1 is configured to have a basis information generation unit 102, a sentence generation unit 102a, a classification unit 103, an emphasis unit 104, and an output unit 105. The image processing device 1 may further include a detection unit 100 or a discrimination unit 101, or at least one of these may be separate from the detection unit 100.

[0035] The discrimination unit 101 is equipped with artificial intelligence (AI) that has undergone deep learning for CADx (Computer-Aided Diagnosis). The discrimination results by the discrimination unit 101, such as whether it is a lesion, blood vessel, or nerve, or the type of lesion, infection history with Helicobacter pylori or the like, type of blood vessel, type of organ, or location within the organ, are output to the display device 41 via the output unit 105.

[0036] The result of the discrimination of the medical image by the discrimination unit 101 is output to the basis information generation unit 102, which serves as a discrimination result receiving unit. The basis information generation unit 102 estimates the basis of the discrimination, that is, why the discrimination unit 101 output such a discrimination result. Although not shown, a configuration that allows the display of the basis information to be turned on and off may be provided, so that the result of the discrimination is output to the basis information generation unit 102 only when the display of the basis information is turned on.

[0037] The sentence generation unit 102a converts the basis for classification into sentences that can be understood by humans. The classification unit 103 classifies multiple types of sentences into emphasis targets and non-emphasis targets. Although a specific example will be described later, when there are multiple types of sentences generated by the sentence generation unit 102a, each sentence may be classified into emphasis targets and non-emphasis targets, or emphasis targets and non-emphasis targets may be classified within a single sentence. When classifying emphasis targets and non-emphasis targets within a single sentence, the classification may be performed on a phrase-by-phrase basis, a word-by-word basis, or a combination of these. It is also possible to switch emphasis targets and non-emphasis targets over time.

[0038] The highlighting unit 104 generates information for highlighting the highlight target according to the classification result, and outputs the information to the display device 41 via the output unit 105 as a highlighting process. Examples of highlighting methods used by the highlighting unit 104 include the following: "underlining the highlight target and not underlining the non-highlight target," "making the font of the highlight target bolder than that of the non-highlight target," "making the font size of the highlight target larger than that of the non-highlight target," "making the font type of the highlight target different from that of the non-highlight target," "making the color of the highlight target different from that of the non-highlight target," "making the tilt angle of the font of the highlight target different from that of the non-highlight target," or "making the blinking speed of the highlight target faster than that of the non-highlight target."

[0039] As an example of a method of using color to emphasize text, the color of the text can be selected to match the background color, as shown in Table 1. [Table 1]

[0040] The highlighting unit 104 may include a diagram generating unit 104a. The diagram generating unit 104a generates a highlighted diagram by adding marker images to the medical image after differentiation at locations corresponding to the components to be highlighted. The marker image may be an icon, such as an arrow, that indicates the portion to be highlighted, or a bounding box that surrounds the portion to be highlighted, or may be linked to the evidence information. For example, if the evidence information includes information regarding the outline of the lesion, a frame tracing the outline of the lesion may be displayed. The marker image is preferably an image that is visually consistent with the highlighting process of the component to be highlighted. For example, the display color of the marker image is preferably similar or the same color as the display color of the component to be highlighted. Similar colors refer to colors that fall within a predetermined range on the color wheel.

[0041] The detection results of the lesion position and contour by the detection unit 100 may be input to the image generation unit 104a, and the detection information of the lesion position and contour may be used to generate a marker image. The detection unit 100 may be equipped with artificial intelligence (AI) that has undergone deep learning for Computer-Aided Detection (CADe).

[0042] Specifically, it is assumed that the sentence generation unit 102a generates multiple types of sentences, namely, a "sentence verbalizing the first basis," a "sentence verbalizing the second basis," and a "sentence verbalizing the third basis," from the basis information generated by the basis information generation unit 102. When the classification unit 103 determines that the "sentence verbalizing the first basis" is to be emphasized and the "sentence verbalizing the second basis" and the "sentence verbalizing the third basis" are not to be emphasized, the highlighting unit 104 uses the above-mentioned highlighting method or the like to highlight the "sentence verbalizing the first basis" so that, when viewed by a doctor's eyes, it stands out more than the "sentence verbalizing the second basis" and the "sentence verbalizing the third basis."

[0043] Of course, it is not essential to emphasize each sentence as described above. For example, when the sentence generation unit 102a generates a "sentence verbalizing the reason" including a "first word," a "second word," and a "third word," the classification unit 103 may determine that the "first word" is to be emphasized and that the "second word" and the "third word" are not to be emphasized.

[0044] The classification unit 103 may add timing information to the emphasis target information and non-emphasis target information and output the information to the emphasis unit 104. Even if there are multiple locations to which the doctor should pay attention, the doctor can be made to pay attention by emphasizing the locations in order at different timings, thereby reducing the burden on the doctor while allowing the doctor to input information.

[0045] For example, it is possible to output information such as, "At a first timing, the 'sentence verbalizing the first basis' is to be emphasized, and the 'sentence verbalizing the second basis' and the 'sentence verbalizing the third basis' are to be de-emphasized," "At a second timing later than the first timing, the 'sentence verbalizing the second basis' is to be emphasized, and the 'sentence verbalizing the first basis' and the 'sentence verbalizing the third basis' are to be de-emphasized," and "At a third timing later than the second timing, the 'sentence verbalizing the third basis' is to be emphasized, and the 'sentence verbalizing the first basis' and the 'sentence verbalizing the second basis' are to be de-emphasized."

[0046] The order of highlighting can be determined, for example, by utilizing a diagnostic sequence. The diagnostic sequence can be customized and registered by the user, or a diagnostic algorithm can be utilized. Figure 4 shows the diagnostic procedure using MESDA-G as an example of a diagnostic algorithm. MESDA-G is a diagnostic algorithm used to differentiate gastric cancer. First, it determines color changes (whitish or reddish) or morphological changes (prominence, flatness, or depression) on the gastric mucosal surface from images. Next, it identifies the demarcation line (DL) between the lesion and the background mucosa (S1). If a DL is not present, it diagnoses the lesion as non-cancerous (benign). If a DL is present, it then observes the presence of an irregular microvascular pattern (IMVP) and an irregular microsurface pattern (IMSP) (S2). If irregular microvessels and / or microsurface patterns are present within the borderline, the diagnosis is cancer. If irregular microvessels and / or microsurface patterns are not present within the borderline, the diagnosis is non-cancerous.

[0047] As a specific example, let us consider a case where the differentiation unit 101 differentiates gastric cancer from a medical image and obtains a result of cancer with a diagnostic certainty of 90%. From the evidence information generated by the evidence information generation unit 102, the sentence generation unit 102a generates multiple types of sentences, such as the sentence "There is a clear boundary" as the "first evidence," the sentence "Has an irregular texture" as the "second evidence," and the sentence "The color is reddish" as the "third evidence." Furthermore, let us assume that these sentences are merged to generate the evidence information sentence "There is a clear boundary, has an irregular texture, and is reddish in color." In this case, in accordance with the diagnostic algorithm, the classification unit first highlights sentences, phrases, or words related to the color of the gastric mucosal surface. Then, the classification unit next highlights sentences, phrases, or words related to the boundary between the lesion and the background mucosa. Then, the classification unit next highlights sentences, phrases, or words related to the microvascular pattern. Therefore, the elements contained in the sentence "It has clear boundaries, irregular texture, and is reddish in color" are emphasized in the order of "redness," "clear boundaries," and "irregular texture."

[0048] When there are similar changes, such as "changes in the color of the gastric mucosal surface" and "morphological changes," the timing of highlighting may be set to be simultaneous, or the order may be determined by user settings.

[0049] It should be noted that the diagnostic algorithm may be switched to perform differentiation depending on the type of organ. When the target organ is the large intestine, for example, the JNET classification, NICE classification, Sano classification, Jikei classification, Showa classification, Hiroshima classification, Kudo-Tsuruta classification, Akita-Nisseki classification, etc. are used. When the target organ is the esophagus, for example, the Inoue classification, Japan Esophageal Society classification, Arima classification, BING, etc. are used. When the target organ is the stomach, for example, the MESDA-G, Koyama classification, Jikei classification, Yao classification, Yagi classification, etc. are used. In addition to the type of organ, the diagnostic algorithm may be switched to perform differentiation depending on the target lesion, the type of endoscope used to capture the medical image, etc.

[0050] When determining the order of emphasis in accordance with the diagnostic sequence, for example, a keyword can be set for each component of the diagnostic sequence, and the order of emphasis for each component of the evidence information can be determined based on the degree of match between each component of the evidence information and the keyword.

[0051] For example, when using the above-mentioned MESDA-G, the keywords "red, bleeding, protuberance, flat, depression" related to the first keyword in the diagnostic order, "boundary" related to the second keyword in the diagnostic order, and "capillary, capillary, texture, texture" related to the third keyword in the diagnostic order are stored in the classification unit 103, or this information stored in a separate configuration can be accessed by the classification unit 103. Then, if the sentence generation unit generates sentences such as "There is a clear boundary" and "The color is reddish," the sentence "The color is reddish" which has a higher degree of match with the keyword related to the first keyword in the diagnostic order is emphasized first in the chronological order.

[0052] The sentence generator 102a may generate sentences using medical terms contained in a medical dictionary set by the user or the like. For example, prior to image diagnosis, an arbitrary dictionary may be selected and set from multiple medical dictionaries, and the dictionary may be used to generate sentences for evidence information. The generated sentences are output to the classifier 103.

[0053] The output unit 105 generates and outputs a display image to be displayed on the display device 41. The display image includes the differentiation result (disease name, diagnostic certainty), the text of the evidence information that has been emphasized, and an emphasized diagram. FIGS. 5A, 5B, 5C, and 5D are diagrams illustrating an example of a display image according to the first embodiment. FIG. 5A shows an example of a display image 411 displayed on the display device 41. The display image 411 includes two display areas D1 and D2. The emphasized diagram G1 generated by the diagram generation unit 104a is arranged in the display area D1, and the differentiation result (disease name, diagnostic certainty) and the text of the evidence information generated by the emphasis unit 104 are arranged in the display area D2. FIG. 5A shows a display image in which the "first evidence" is emphasized. In this case, the marker image L1 in the emphasized diagram G1 is added to the portion corresponding to the "first evidence." In the display image 411, the bolded portion of the text of the evidence information and the marker image L1 indicated by a bold line are displayed in green, for example. The other parts of the evidence information text are displayed in black, for example. By making the display color of the component to be highlighted different from the display colors of the other components in the evidence information text, the user can quickly identify the evidence information to be highlighted. Furthermore, by making the display color of the component to be highlighted the same as the display color of the marker image L1, the user can quickly identify the evidence information to be highlighted and the location of the corresponding lesion.

[0054] It is desirable to use different colors for the display colors of the components to be emphasized and the marker images for each component to be emphasized. Fig. 5B is an example of a display image in the above-mentioned example of gastric cancer differentiation, where the "first basis" "there is a clear boundary" is to be emphasized. Fig. 5C is an example of a display image in the above-mentioned example of gastric cancer differentiation, where the "second basis" "has an irregular texture" is to be emphasized. Fig. 5D is an example of a display image in the above-mentioned example of gastric cancer differentiation, where the "third basis" "the color is reddish" is to be emphasized.

[0055] In the display image 411 shown in Fig. 5B, the emphasis (bold part) of the sentence of the basis information and the marker image L1 indicated by a thick line are displayed in, for example, green. In the display image 411 shown in Fig. 5C, the emphasis (left italic part) of the sentence of the basis information and the marker image L2 indicated by a double line are displayed in, for example, blue. In the display image 411 shown in Fig. 5D, the emphasis (right italic part) of the sentence of the basis information and the marker image L3 indicated by a thick dotted line are displayed in, for example, red. In this way, when the components classified as the emphasis part are different, they may be displayed using different colors.

[0056] In this way, when generating a display image based on the results of discrimination of a medical image, the image processing device of the first embodiment differentiates the display color of the component to be emphasized from the display color of the other components in the text of the evidence information to be displayed in the display image. This allows the user to quickly identify the evidence information to be emphasized. Furthermore, the image processing device of the first embodiment adds marker images to the medical image on which discrimination has been performed at locations corresponding to the component to be emphasized, thereby generating an emphasized image. At this time, the display color of the marker image is made the same as the display color of the component to be emphasized. This allows the user to quickly identify the evidence information to be emphasized and the location of the corresponding lesion.

[0057] In generating a sentence for evidence information in the sentence generator 102a, in addition to the setting of the medical dictionary, the setting of the description method related to color may be switchable. For example, two modes, a normal mode and a detailed mode, may be set, and when the detailed mode is set, a sentence that describes in detail the color of a component that includes color may be generated (Second Embodiment).

[0058] In the first embodiment described above, the display color of the components classified as those to be emphasized in the display image is set to a color different from the display color of the components not to be emphasized, as a process for visually emphasizing the components classified as those to be emphasized. In contrast, in the present embodiment, the texture of the components to be emphasized in the display image is made different from the texture of the components not to be emphasized. That is, the process performed by the highlighting unit 104 in this embodiment differs from that in the first embodiment.

[0059] The image processing apparatuses in this embodiment and the following embodiments have the same configuration as the image processing apparatus 1 in the first embodiment. The same components are denoted by the same reference numerals and the description thereof will be omitted.

[0060] The highlighting unit 104 in this embodiment visually highlights the components to be highlighted by underlining the characters of the components to be highlighted or by making the characters bolder than the characters of the components not to be highlighted. Figures 6A, 6B, and 6C are diagrams illustrating an example of a display image according to the second embodiment. Figure 6A is an example of a display image in the above-described example of gastric cancer differentiation, where the "first basis" "clear boundary" is to be highlighted. Figure 6B is an example of a display image in the above-described example of gastric cancer differentiation, where the "second basis" "has an irregular texture" is to be highlighted. Figure 6C is an example of a display image in the above-described example of gastric cancer differentiation, where the "third basis" "has a reddish color" is to be highlighted.

[0061] 6A to 6C, the marker image L4 in the highlighting diagram G1 is added to a location corresponding to the component to be highlighted. In the display image 411, the component to be highlighted in the text of the evidence information is shown in bold and underlined. The other parts of the text of the evidence information are shown in thin, ununderlined text. In this way, by making the texture of the characters of the component to be highlighted in the text of the evidence information different from the texture of the other components, the user can quickly identify the evidence information to be highlighted.

[0062] In the above description, the texture of the component to be emphasized is made different from that of the other components depending on the thickness of the characters and whether or not they are underlined. However, the size, font, or inclination of the characters may also be made different. These may also be combined. (Third embodiment)

[0063] In the first embodiment described above, a display image is generated in which an emphasis image is arranged when a specific component is to be emphasized. Therefore, when another component is to be emphasized, it is necessary to generate a display image in which an emphasis image corresponding to that component is arranged. In contrast, in the present embodiment, emphasis images are arranged in which each component is to be emphasized, which is different from the first embodiment.

[0064] The display image 411 generated by the output unit 105 in this embodiment includes text of the evidence information on which emphasis processing has been performed, a medical image on which differentiation has been performed, and an emphasized image. Fig. 7 is a diagram illustrating an example of a display image according to the third embodiment. The display image 411 has three display areas D1, D2, and D3. The emphasized images G1, G2, and G3 are arranged in the display area D1, and the text of the evidence information is arranged in the display area D2. Furthermore, the medical image G0 on which differentiation has been performed is arranged in the display area D3.

[0065] The sentences of the evidence information arranged in the display area D2 are visually highlighted in different ways for each component. For example, the "first evidence" "has a clear boundary" is displayed in green, the "second evidence" "has an irregular texture" is displayed in blue, and the "third evidence" "is reddish in color" is displayed in red. A marker image L1 in the highlighted image G1 arranged in the display area D1 is added to a location corresponding to the "first evidence." The color of the marker image L1 is the same as the display color of the "first evidence." A marker image L2 in the highlighted image G2 is added to a location corresponding to the "second evidence." The color of the marker image L2 is the same as the display color of the "second evidence." A marker image L3 in the highlighted image G3 is added to a location corresponding to the "third evidence." The color of the marker image L3 is the same as the display color of the "third evidence."

[0066] In this way, by arranging the highlighted images corresponding to the respective constituent elements of the sentence of the evidence information in one display image, it is possible to quickly compare the constituent elements (fourth embodiment).

[0067] In the first embodiment described above, a display image is generated in which an emphasis image is arranged when a specific component is to be emphasized. In contrast, in the present embodiment, a display image is generated in which a reference image including a lesion similar to the component is also arranged. The reference image is, for example, an atlas image published in a medical atlas or the like, which includes a lesion similar to the component.

[0068] The display image 411 generated by the output unit 105 in this embodiment includes text of the evidence information that has been subjected to the emphasis process, a medical image that has been subjected to the differentiation process, and a reference image. FIG. 8 is a diagram illustrating an example of a display image according to the fourth embodiment. The display image 411 has three display areas D1, D2, and D4. An emphasis image G1 is arranged in the display area D1, and text of the evidence information is arranged in the display area D2. Furthermore, a reference image G11 is arranged in the display area D4. A marker image L11 is added to the reference image G11. The marker image L11 is added to a portion of the reference image G11 that corresponds to the component to be emphasized. The marker image L11 is an image that is visually consistent with the marker image L1. For example, the display color of the marker image L11 is the same as the display color of the marker image L1 and the display color of the component to be emphasized.

[0069] In this way, by arranging the text of the evidence information, the highlighted image, and the reference image in one display image, comparison with other cases can be performed quickly (Fifth embodiment).

[0070] This embodiment differs from the above-described embodiments in that the display images are sequentially switched based on the order of diagnoses in the differentiation. For example, when differentiation is performed using the diagnostic procedure according to MESDA-G shown in FIG. 4, a display image (FIG. 9A) is first displayed in which the components based on the diagnosis shown in S1 are highlighted. Next, a display image (FIG. 9B) is displayed in which the components based on the diagnosis shown in S2 are highlighted. FIGS. 9A and 9B are diagrams illustrating an example of a display image according to the fifth embodiment.

[0071] The differential diagnosis results shown in the display area D1 preferably include the disease name, diagnostic certainty, and the judgment result for each diagnosis (e.g., "DL+" in FIG. 9A or "IMVP / IMSP+" in FIG. 9B). In this case, it is preferable to display the judgment result in the same color as the display color of the component to be highlighted.

[0072] The display color of the component to be highlighted and the display color of the marker image may be switched depending on the judgment status, such as red if the judgment result is "+" and green if the judgment result is "-" (sixth embodiment).

[0073] This embodiment differs from the above-described embodiment in that, as a result of the differentiation, the basis information is displayed on the display image only if the lesion is determined to be a disease, and the basis information is not displayed if the lesion is determined not to be a disease.

[0074] Fig. 10 is a diagram illustrating an example of a display image according to the sixth embodiment. The display image when the lesion is determined to be a disease is the same as that of the above-described embodiments (e.g., Fig. 5B). When the lesion is determined not to be a disease, a display screen such as that shown in Fig. 10 is generated. That is, an emphasized image G1a is arranged in the display area D1, and the rejected disease name and diagnostic certainty are arranged in the display area D2. The emphasized image G1a is generated by adding a marker image L5 to a location on the medical image corresponding to the lesion.

[0075] In this way, if the result of the differentiation is that the object is not a disease, the amount of information in the displayed image is reduced, allowing the user to quickly perform image diagnosis (seventh embodiment).

[0076] This embodiment differs from the above-described embodiments in that the display images are sequentially switched and displayed based on the importance of the basis for diagnosis. Figures 11A, 11B, and 11C are diagrams illustrating examples of display images according to the seventh embodiment. Figures 11A, 11B, and 11C are diagrams illustrating examples of display images according to the seventh embodiment, in which the importance of the basis for diagnosis is added to the display images shown in Figures 5B, 5C, and 5D, respectively.

[0077] For example, if the DL identification result has the highest importance for disease diagnosis (e.g., importance 100), the IMVP / IMSP observation result has the next highest importance (e.g., importance 70), and the color change of the gastric mucosal surface has the lowest importance (e.g., importance 30), the display image shown in Fig. 11A is displayed first. Next, the display image shown in Fig. 11B is displayed. Finally, the display image shown in Fig. 11C is displayed.

[0078] In this way, by displaying an image in which the components are highlighted in descending order of importance of the basis for judgment, the user can quickly perform image diagnosis (Eighth Embodiment).

[0079] This embodiment differs from the above-described embodiment in that, as a result of the differentiation, the lesion is determined to be a disease and the diagnostic certainty is high (higher than a set value), and the basis information is displayed on the display image only when the lesion is determined to be not a disease, or when the lesion is determined to be a disease but the diagnostic certainty is low.

[0080] Fig. 12 is a diagram illustrating an example of a display image according to the eighth embodiment. The display image when the lesion is determined to be a disease is the same as that of the above-described embodiments (e.g., Fig. 5B). The display image when the lesion is determined not to be a disease is, for example, the image shown in Fig. 10. When the lesion is determined to be a disease but the diagnostic certainty is low, for example, a display screen shown in Fig. 12 is generated. That is, an emphasized image G1a is arranged in the display area D1, and the determined disease name and diagnostic certainty are arranged in the display area D2. The emphasized image G1a is an image generated by adding a marker image L5 to a location corresponding to the lesion in the medical image.

[0081] In this way, even if the lesion is determined to be a disease as a result of differentiation but the diagnostic certainty is low, the amount of information in the displayed image is reduced in the same way as when the lesion is determined not to be a disease, allowing the user to quickly perform image diagnosis (ninth embodiment).

[0082] In the above embodiment, the case where the basis is presented as text on the display device 41 has been introduced, but the present invention is not limited to this, and the basis information may be input to the doctor by voice. Voice may also be combined with Examples 1 to 8.

[0083] In this case, the emphasis unit 104 synthesizes voice data that has been subjected to emphasis processing, such as increasing the volume of the emphasis target compared to the non-emphasis target, or reading out only the emphasis target first, followed by reading out all of the evidence, and transmits the synthesized voice data to the audio device 42 via the output unit 105. The audio device 42 may be a general speaker, a directional speaker, earphones, headphones, or a bone conduction device. When a directional speaker, earphones, headphones, or a bone conduction device is used, the evidence information can be transmitted only to a specific party (the doctor) through these devices, thereby reducing the possibility that the subject will see the evidence information.

[0084] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0085] In addition, even if the operational flows in the claims, specifications, and drawings are described using "first," "next," etc. for convenience, this does not mean that they must be performed in that order. Furthermore, it goes without saying that the steps that make up these operational flows can be omitted as appropriate if they do not affect the essence of the invention.

[0086] Of the technologies described here, the controls mainly described in the flowcharts can often be set by a program, and may be stored on a recording medium or a recording unit. The method of recording on this recording medium or recording unit may be recording at the time of product shipment, using a distributed recording medium, or downloading via the Internet.

[0087] In the embodiments, the parts described as "parts" (sections or units) may be configured by combining dedicated circuits or multiple general-purpose circuits, or, if necessary, by combining a processor such as a microcomputer or CPU that operates according to pre-programmed software, or a sequencer such as an FPGA. It is also possible to design the device so that an external device takes over part or all of the control, in which case a wired or wireless communication circuit is involved. Communication may be via Bluetooth (registered trademark), Wi-Fi, a telephone line, or USB. The dedicated circuit, the general-purpose circuit, and the control unit may be integrated into an ASIC.

Claims

1. An image processing device including: a differentiation result receiving unit that receives the result of differentiation of a lesion from an image; a sentence generating unit that generates the basis information of the differentiation as sentence; a classification unit that classifies components of the sentence into emphasis targets and non-emphasis targets; an emphasis unit that emphasizes the emphasis targets; and an output unit that outputs at least the emphasis targets of the sentence to a monitor.

2. The image processing device of claim 1, wherein the output unit also outputs the results of the discrimination to a monitor, and includes an image generation unit that generates an emphasized image linked to the object to be emphasized, and the output unit links the emphasized image to the results of the discrimination and outputs it to the monitor so that the output state of the object to be emphasized and the display of the emphasized image are linked.

3. The image processing device of claim 1, wherein the output unit outputs both the emphasized and non-emphasized portions of the text, and the emphasis unit performs at least one of the following: underlining the emphasized portions and not underlining the non-emphasized portions; making the font of the emphasized portions bolder than that of the non-emphasized portions; making the font size of the emphasized portions larger than that of the non-emphasized portions; using a different font type for the emphasized portions than that of the non-emphasized portions; using a different color for the emphasized portions than that of the non-emphasized portions; making the tilt angle of the emphasized portions different from that of the non-emphasized portions; and making the blinking rate of the emphasized portions faster than that of the non-emphasized portions.

4. The image processing device according to claim 1, wherein the image generating unit sets the color of the outline used in the highlighted image to a similar color to the highlighted object.

5. The image processing device according to claim 1, wherein the image generating unit generates, as the emphasized image, an image that surrounds or points out a portion of the image that corresponds to the explanatory text to be emphasized.

6. The image processing device described in claim 1, wherein the classification unit classifies the emphasis target into a first emphasis target and a second emphasis target, the emphasis unit emphasizes the second emphasis target in a different appearance or timing from the first emphasis target, the image generation unit generates a first emphasis image linked to the first emphasis target and a second emphasis image linked to the second emphasis target, and the output unit links the emphasis state of the first emphasis target with the display of the first emphasis image, and links the emphasis state of the second emphasis target with the display of the second emphasis image.

7. The image processing device according to claim 6, wherein the highlighting unit displays the second emphasis target in a color different from that of the first emphasis target as a method of highlighting the second emphasis target in a different appearance from that of the first emphasis target.

8. The image processing device according to claim 6, wherein the highlighting unit highlights the first highlight target and the second highlight target at different times, and displays the first highlight target and the second highlight target that are not at the time of highlighting in the same manner as the non-highlighted target.

9. An image processing device as described in claim 6, wherein the emphasis unit emphasizes the first emphasis target and the second emphasis target at different times, and the output unit does not output to the monitor any of the first emphasis target and the second emphasis target that is not at the emphasis timing.

10. The image processing device according to claim 6, wherein the emphasis unit emphasizes the first emphasis target and the second emphasis target at different times, and the order of emphasis is based on a predetermined diagnostic order.

11. The image processing device according to claim 1, wherein the highlighting unit highlights the object to be highlighted when the discrimination result matches a predetermined condition.

12. An image processing device includes one or more processors, which receive the image classification results, generate the basis information for the classification as text, classify the components of the text into items to be emphasized and items not to be emphasized, emphasize the items to be emphasized, and output at least the items to be emphasized from the text to a monitor.

13. An image display method comprising: receiving an image classification result; generating a sentence based on information on the basis of the classification; classifying components of the sentence into items to be emphasized and items not to be emphasized; emphasizing the items to be emphasized; and outputting at least the items to be emphasized from the sentence to a monitor.

14. An image display program that receives the results of image classification, generates basis information for the classification as text, classifies components of the text into emphasis targets and non-emphasis targets, emphasizes the emphasis targets, generates a diagram associated with the emphasis targets, and outputs at least the emphasis targets of the text to a monitor.

15. A storage medium storing an image display program that receives the results of image classification, generates information on the basis of the classification as text, classifies the components of the text into emphasis targets and non-emphasis targets, emphasizes the emphasis targets, generates a diagram associated with the emphasis targets, and outputs at least the emphasis targets of the text to a monitor.

16. An imaging diagnostic system including: a classification unit that receives evidence information for the differentiation of lesions and classifies the components of the text that make up the evidence information into emphasis targets and non-emphasis targets; an emphasis unit that emphasizes the emphasis targets; and an output unit that outputs at least the emphasis targets from the text to a monitor.

Images