Medical image processing device, medical image processing method, and program

Abstract

Provided are a medical image processing device, a medical image processing method, and a program that enable a plurality of regions of interest and graphic information items to be clearly displayed in a medical image. A processor of a medical image processing device (112) performs: an image acquisition process for acquiring a medical image; a region information acquisition process for acquiring region information items regarding a plurality of regions of interest that include positions and categorizations of the plurality of regions of interest included in the medical image; a display control process for causing a display unit to display, so as to be superimposed on the medical image, a plurality of graphic information items indicating the results of categorizations of the plurality of regions of interest; and a superimposing position determination process for determining, on the basis of a relative positional relationship among the plurality of regions of interest, superimposing positions of the plurality of graphic information items to be displayed by the display control process.

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program, and particularly to a medical image processing apparatus, a medical image processing method, and a program for superimposing and displaying information regarding a region of interest in a medical image on the medical image.

Background Art

[0002] In recent years, by automatically detecting regions of interest such as organs and lesions shown in medical images using AI (Artificial Intelligence) and displaying information (graphical information) regarding the detected regions of interest on a monitor, assistance is provided for doctors' examinations and diagnoses.

[0003] For example, Patent Document 1 describes a technique for superimposing and displaying findings of doctors or the like on an endoscopic image to assist in examinations and diagnoses using an endoscope.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, there may be cases where a plurality of regions of interest are detected in a medical image. When a plurality of regions of interest are detected, depending on the positions of the detected regions of interest, the graphical information to be displayed may overlap and be displayed. Also, depending on the positions of the detected regions of interest, the graphical information may be displayed overlapping the region of interest, and the region of interest to be observed may be hidden. Thus, when the graphical information is displayed, the region of interest and the graphical information become difficult to see.

[0006] The present invention has been made in view of these circumstances, and its purpose is to provide a medical image processing apparatus, a medical image processing method, and a program that can display multiple areas of interest and graphic information in a medical image in an easily viewable manner. [Means for solving the problem]

[0007] A medical image processing apparatus, which is one aspect of the present invention for achieving the above objective, is a medical image processing apparatus comprising a processor, the processor performing: an image acquisition process for acquiring a medical image; a region information acquisition process for acquiring region information relating to a plurality of regions of interest, including the location and category classification of a plurality of regions of interest included in the medical image; a display control process for displaying a plurality of graphic information indicating the category classification of a plurality of regions of interest superimposed on a medical image on a display unit; and a superimposition position determination process for determining the superimposition position of the plurality of graphic information to be displayed by the display control process based on the relative positional relationship of the plurality of regions of interest.

[0008] According to this embodiment, the superposition position of each graphic information to be displayed is determined by the display control process based on the relative positional relationship of multiple areas of interest, so that the areas of interest and graphic information can be displayed in an easy-to-view manner.

[0009] Preferably, the processor detects multiple areas of interest contained in the medical image, estimates the categorization of the detected areas of interest, and performs region information generation processing to generate region information.

[0010] Preferably, at least one of the multiple regions of interest is an anatomical region.

[0011] Preferably, at least one of the multiple regions of interest is an annotation drawn by the user on the medical image.

[0012] Preferably, the superposition position determination process determines the superposition position of the graphic information of at least one of the multiple areas of interest based on the positions of the other areas of interest.

[0013] Preferably, the superposition position determination process determines the superposition position of the graphic information of at least one of the multiple regions of interest based on the superposition position of the graphic information of the other regions of interest.

[0014] Preferably, the image acquisition process acquires multiple medical images that are sequentially arranged in time.

[0015] Preferably, the superposition position determination process determines the superposition position of graphic information in the current medical image based on the superposition position of graphic information in past medical images among multiple medical images.

[0016] Preferably, the superposition position determination process determines the superposition position of the graphic information in the current medical image within a region where the distance from the superposition position of the graphic information in past medical images is within a first threshold.

[0017] Preferably, the display control process displays a leader line up to the overlapping position of the graphic information of at least one of the multiple areas of interest.

[0018] Preferably, the processor performs an inclusion relationship acquisition process to acquire inclusion relationship information for multiple regions of interest based on region information, and the overlap position determination process determines the overlap position of multiple graphic information based on the inclusion relationship information.

[0019] Preferably, the superimposition position determination process determines whether to display the leader line of at least one of the multiple areas of interest based on the inclusion relationship information, and switches the display on or off.

[0020] Preferably, the display control process displays the graphic information of the areas of interest that are in an inclusion relationship, in a nested format that indicates the inclusion relationship, based on the inclusion relationship information.

[0021] Preferably, the display control process displays information indicating the range of the area of ​​interest based on the area information.

[0022] Preferably, the information indicating the range of the target area is a bounding box, and the display control process displays the graphic information corresponding to the bounding box.

[0023] Preferably, the graphic information is composed of character information indicating category classification.

[0024] Another aspect of the present invention, a medical image processing method, is a medical image processing method using a medical image processing apparatus including a processor, including an image acquisition step of acquiring a medical image performed by the processor, an area information acquisition step of acquiring area information regarding a plurality of target areas including the positions and category classifications of the plurality of target areas included in the medical image, a display control step of superimposing and displaying on a display unit a plurality of graphic information indicating the category classifications of the plurality of target areas on the medical image, and a superimposition position determination step of determining a superimposition position of the plurality of graphic information to be displayed by the display control step based on the relative positional relationship of the plurality of target areas.

[0025] Another aspect of the present invention, a program, is a program for causing a medical image processing apparatus including a processor to execute a medical image processing method, causing the processor to execute an image acquisition step of acquiring a medical image, an area information acquisition step of acquiring area information regarding a plurality of target areas including the positions and category classifications of the plurality of target areas included in the medical image, a display control step of superimposing and displaying on a display unit a plurality of graphic information indicating the category classifications of the plurality of target areas on the medical image, and a superimposition position determination step of determining a superimposition position of the plurality of graphic information to be displayed by the display control step based on the relative positional relationship of the plurality of target areas.

Advantages of the Invention

[0026] According to this aspect, since the superimposition position of each graphic information to be displayed by the display control process is determined based on the relative positional relationship of the plurality of target areas, the target areas and the graphic information can be displayed in an easy-to-see manner.

Brief Description of the Drawings

[0027] [Figure 1]Figure 1 is a schematic diagram showing the overall configuration of an ultrasound endoscope system equipped with a medical image processing device. [Figure 2] Figure 2 is a block diagram showing an embodiment of an ultrasonic processor device. [Figure 3] Figure 3 shows an example of displaying the area of ​​interest and its bounding box. [Figure 4] Figure 4 illustrates an example of a conventional display of graphic information. [Figure 5] Figure 5 illustrates other examples of conventional graphic information displays. [Figure 6] Figure 6 shows an example of how graphic information is displayed. [Figure 7] Figure 7 is a flowchart illustrating a medical image processing method. [Figure 8] Figure 8 is a block diagram showing an embodiment of an ultrasonic processor device. [Figure 9] Figure 9 shows an example of a display format. [Figure 10] Figure 10 shows other examples of display formats. [Figure 11] Figure 11 illustrates the display of regional information between ultrasound image P1 and ultrasound image P2. [Figure 12] Figure 12 illustrates the display of regional information between ultrasound image P1 and ultrasound image P2. [Figure 13] Figure 13 illustrates an example of a display format. [Modes for carrying out the invention]

[0028] Preferred embodiments of the medical image processing apparatus, medical image processing method, and program according to the present invention will be described below with reference to the attached drawings.

[0029] Figure 1 is a schematic diagram showing the overall configuration of an ultrasound endoscope system equipped with the medical image processing device of the present invention.

[0030] As shown in Figure 1, the ultrasound endoscopy system 102 includes an ultrasound scope 110, an ultrasound processor device 112 for generating ultrasound images, an endoscope processor device 114 for generating endoscope images, a light source device 116 for supplying illumination light to the ultrasound scope 110 to illuminate the inside of the body cavity, and a monitor (display unit) 118 for displaying ultrasound images and endoscope images. The following description will focus on the processing of ultrasound images, which are an example of medical images.

[0031] The ultrasound scope 110 comprises an insertion section 120 inserted into the body cavity of the subject, a handheld control section 122 connected to the proximal end of the insertion section 120 for operation by the operator, and a universal cord 124 with one end connected to the handheld control section 122. The other end of the universal cord 124 is provided with an ultrasound connector 126 connected to an ultrasound processor device 112, an endoscope connector 128 connected to an endoscope processor device 114, and a light source connector 130 connected to a light source device 116.

[0032] The ultrasound scope 110 is detachably connected to the ultrasound processor unit 112, the endoscope processor unit 114, and the light source unit 116 via these connectors 126, 128, and 130. In addition, the light source connector 130 is connected to the air and water supply tube 132 and the suction tube 134.

[0033] The monitor 118 receives the respective video signals generated by the ultrasound processor unit 112 and the endoscope processor unit 114 and displays the ultrasound image and the endoscope image. The display of the ultrasound image and the endoscope image can be switched between as needed and displayed on the monitor 118, or both images can be displayed simultaneously.

[0034] The handheld control unit 122 is equipped with an air supply / water supply button 136 and a suction button 138 side by side, as well as a pair of angle knobs 142 and a treatment instrument insertion port 144.

[0035] The insertion section 120 has a tip, a base, and a longitudinal axis 120a. Starting from the tip, it is composed of a tip body 150 made of a rigid material, a curved section 152 connected to the base end of the tip body 150, and a slender, long, flexible flexible section 154 connecting the base end of the curved section 152 to the tip end of the hand-operated section 122. Specifically, the tip body 150 is located on the tip side of the insertion section 120 in the direction of the longitudinal axis 120a. The curved section 152 is remotely bent by rotating a pair of angle knobs 142 provided on the hand-operated section 122. This allows the tip body 150 to be directed in a desired direction.

[0036] The tip body 150 is equipped with an ultrasonic transducer 162 and a bag-shaped balloon 164 that encloses the ultrasonic transducer 162. The balloon 164 can be inflated or deflated by water being supplied from the water tank 170 or by the suction pump 172 drawing water out of the balloon 164. The balloon 164 is inflated until it contacts the inner wall of the body cavity in order to prevent attenuation of ultrasound and ultrasonic echoes (echo signals) during ultrasound observation.

[0037] Furthermore, the tip body 150 is fitted with an endoscope observation unit (not shown) which has an observation unit equipped with an objective lens and an image sensor, and an illumination unit. The endoscope observation unit is located behind the ultrasonic probe 162 (on the handheld control unit 122 side).

[0038] Figure 2 is a block diagram showing an embodiment of the ultrasonic processor device 112.

[0039] The ultrasound processor device 112 shown in Figure 2 recognizes the location and category classification of the region of interest within the ultrasound image based on sequentially acquired time-series ultrasound images, and informs the user (physician, etc.) of the recognition result. The ultrasound processor device 112 also functions as an image processing device that processes ultrasound images.

[0040] The ultrasonic processor device 112 shown in Figure 2 consists of a transmitting / receiving unit 202, an image generation unit 204, a region information generation unit 206, a superposition position determination unit 208, a display control unit 210, a CPU (Central Processing Unit) 210, and a memory 214. The processing of each unit is realized by one or more processors (not shown).

[0041] The CPU 212 operates based on various programs, including an ultrasonic image processing program, stored in the memory 214, and comprehensively controls the transmitting / receiving unit 202, the image generation unit 204, the region information generation unit 206, the superimposed position determination unit 208, the display control unit 210, and the memory 214, and also functions as a part of each of these units.

[0042] The ultrasound image acquisition unit (image acquisition unit) performs image acquisition processing. The transmitting / receiving unit 202 and the image generation unit 204, which function as the ultrasound image acquisition unit, sequentially acquire time-series ultrasound images.

[0043] The transmitting unit of the transmitting / receiving unit 202 generates multiple drive signals to be applied to multiple ultrasonic transducers of the ultrasonic probe 162 of the ultrasonic scope 110, and applies the multiple drive signals to the multiple ultrasonic transducers by assigning a delay time to each of the multiple drive signals based on a transmission delay pattern selected by a scanning control unit (not shown).

[0044] The receiver of the transmitting / receiving unit 202 amplifies multiple detection signals output from multiple ultrasonic transducers of the ultrasonic probe 162 and converts the analog detection signals into digital detection signals (also called RF (Radio Frequency) data). This RF data is input to the image generation unit 204.

[0045] The image generation unit 204 performs reception focus processing by assigning delay times to multiple detection signals represented by RF data based on the reception delay pattern selected by the scanning control unit, and then adding these detection signals together. This reception focus processing forms sound line data in which the focus of the ultrasonic echo is narrowed.

[0046] The image generation unit 204 further applies STC (Sensitivity Timegain Control) to the sound line data to correct for attenuation due to distance according to the depth of the ultrasonic reflection position, and then generates envelope data by performing envelope detection processing using a low-pass filter or the like, and stores the envelope data for one frame, more preferably multiple frames, in a cine memory (not shown). The image generation unit 204 then applies preprocessing such as log (logarithmic) compression and gain adjustment to the envelope data stored in the cine memory to generate a B-mode image.

[0047] In this way, the transmitting / receiving unit 202 and the image generation unit 204, which function as an ultrasound image acquisition unit, sequentially acquire time-series B-mode images (hereinafter referred to as "ultrasound images").

[0048] The region information generation unit 206 performs region information generation processing, which includes detecting a region of interest within the ultrasound image based on the ultrasound image, and estimating and classifying the region of interest into one of several categories (types) based on the ultrasound image. By performing these processes, the region information generation unit 206 generates region information including the location and category classification of the region of interest. The region information generation unit 206 can generate region information using various methods. For example, the region information generation unit 206 may generate region information using AI (Artificial Intelligence). Here, the region of interest is defined as having at least one anatomical region. Furthermore, at least one of the regions of interest may include annotations drawn by the user on the medical image. In this example, the case where region information is generated by the region information generation unit 206 is described, but the ultrasound processor device 112 may acquire region information generated externally (region information acquisition processing).

[0049] The region information generation unit 206 performs category classification, for example, classifying the type of organ detected as a region of interest within the ultrasound image (B-mode tomographic image). The region information generation unit 206 classifies the detected region of interest into, for example, the pancreas (indicated as "Panc" in the figure), the main pancreatic duct (indicated as "MPD" in the figure), the superior mesenteric vein (indicated as "SMV" in the figure), or the gallbladder (indicated as "GB" in the figure). The category classification is then displayed as graphic information superimposed on the ultrasound image. Here, graphic information consists of text information indicating the category classification. Specific examples of graphic information include "Panc," "MPD," "SMV," and "GB."

[0050] The superposition position determination unit 208 performs a superposition position determination process to determine the superposition position of the graphic information. Based on the relative positional relationship of multiple areas of interest, the superposition position determination unit 208 determines the superposition position (display position) of each graphic information to be displayed by the display control unit 210. Specifically, the superposition position determination unit 208 determines the superposition position of the graphic information of at least one of the multiple areas of interest based on the positions of the other areas of interest. That is, the superposition position determination unit 208 determines the superposition position of the graphic information of one area of ​​interest other than the positions of the other areas of interest so that the graphic information of one area of ​​interest does not overlap with that of the other areas of interest. As a result, the superposition position determination unit 208 can superimpose the graphic information onto the ultrasound image without the graphic information overlapping with each other. In other words, the overlap position determination unit 208 determines the overlap position of the graphic information of one area of ​​interest so that the graphic information of one area of ​​interest does not overlap with the graphic information of another area of ​​interest, by determining the position of the graphic information of one area of ​​interest other than the position of the graphic information of the other areas of interest. As a result, the overlap position determination unit 208 can determine the overlap position of the graphic information so that the graphic information does not overlap with the area of ​​interest.

[0051] The display control unit 210 performs display control processing and displays the ultrasound image on the monitor 118, which is the display unit. The display control unit 210 also overlays graphic information onto the ultrasound image and displays it on the monitor 118. The display control unit 210 displays the graphic information on the monitor 118 based on the position determined by the overlay position determination unit 208.

[0052] Next, the display format of the ultrasonic image displayed on the monitor 118 by the display control unit 210 and the graphic information superimposed on the ultrasonic image will be described in detail.

[0053] First, we will explain the display of the area of ​​interest detected in the region information section and the bounding that indicates the extent of that area of ​​interest.

[0054] Figure 3 shows an example of displaying the area of ​​interest detected by the area information generation unit 206 and the bounding box corresponding to that detected area of ​​interest.

[0055] The display control unit 210 displays the ultrasound image P on the monitor 118. The region information generation unit 206 detects the regions of interest C1, C2, C3, and C4 in the ultrasound image P and generates information (region information) regarding the positions of the regions of interest C1, C2, C3, and C4. Based on the region information, the display control unit 210 overlays information (bounding boxes) indicating the ranges of the regions of interest C1, C2, C3, and C4 onto the ultrasound image P and displays them on the monitor 118. Specifically, the display control unit 210 displays bounding box B1 corresponding to region of interest C1, bounding box B2 corresponding to region of interest C2, bounding box B3 corresponding to region of interest C3, and bounding box B4 corresponding to region of interest C4 on the monitor 118. In this way, by enclosing the area of ​​interest C1-C4 with bounding boxes B1-B4 and highlighting it, the user can easily recognize the location of the area of ​​interest C1-C4.

[0056] Next, we will explain how to display the geometric information for the areas of interest C1 to C4.

[0057] Figure 4 illustrates an example of a conventional display of graphic information.

[0058] Conventionally, the display of graphic information for focus regions C1 to C4 was often done by displaying it at a predetermined position. For example, the graphic information was displayed at the center of the bounding box that indicated the position of the focus region. Specifically, the graphic information for focus region C1 ("Panc") F1 was displayed at the center of bounding box B1, the graphic information for focus region C2 ("MPD") F2 was displayed at the center of bounding box B2, the graphic information for focus region C3 ("SMV") F3 was displayed at the center of bounding box B3, and the graphic information for focus region C4 ("GB") F4 was displayed at the center of bounding box B4. Thus, conventionally, regardless of the position of the focus region, the graphic information was often displayed at a predetermined position (for example, the center of the bounding box). However, when the graphic information is displayed at the center of the bounding box, depending on the position of the detected focus region, the graphic information may overlap and become difficult to see. For example, in the case shown in Figure 4, graphic information F1 and graphic information F2 are displayed close together, and graphic information F1 overlaps with the bounding box B2, making it difficult to see. Also, when graphic information is displayed in the center of the bounding box in this way, small areas of interest may become difficult to see due to the graphic information. In the case shown in Figure 4, graphic information F2 overlaps with the area of ​​interest C2, making the area of ​​interest C2 difficult to see.

[0059] Figure 5 illustrates other examples of conventional graphic information displays.

[0060] In the case shown in Figure 5, graphic information is displayed at a predetermined position outside the bounding box. Even when graphic information is displayed outside the bounding box, it is displayed in a position where the user can clearly understand its relationship to the corresponding area of ​​interest. For example, graphic information is displayed along the bounding box, near the corresponding bounding box. In the case shown in Figure 5, graphic information is displayed along the bounding box, in the upper right corner of the bounding box as viewed from the figure. Displaying graphic information outside the bounding box in this way prevents the graphic information F2 from being superimposed on the small area of ​​interest C2, as explained in Figure 4, and thus obscuring the area of ​​interest C2 that is the object of observation. However, since graphic information F1 to F4 are displayed at predetermined positions (upper right) of the bounding boxes B1 to B4, depending on the position of the detected area of ​​interest, the graphic information may overlap with each other, or the graphic information may be superimposed on the area of ​​interest. For example, graphic information F2 and graphic information F3 are displayed overlapping the area of ​​interest C1 and the bounding box B1, respectively.

[0061] As explained above, when graphic information is displayed at predetermined positions, depending on the position of the detected area of ​​interest, the graphic information may overlap with each other, or the graphic information may overlap with the area of ​​interest.

[0062] Therefore, the present invention suppresses the overlapping of graphic information and the display of graphic information overlapping the area of ​​interest, thereby enabling a clear display of both the graphic information and the area of ​​interest.

[0063] <First Embodiment> Next, a first embodiment of the present invention will be described.

[0064] Figure 6 shows an example of the display of graphic information in this embodiment.

[0065] In this embodiment, the superposition position determination unit 208 determines the superposition position of the graphic information according to the position of the area of ​​interest. This makes it possible to suppress the overlapping of graphic information with each other and the overlapping of graphic information with the area of ​​interest.

[0066] The superposition position determination unit 208 determines the superposition position of graphic information F1 and graphic information F4 at a predetermined position (outside the bounding box and in the upper right position when facing the figure) because even if graphic information F1 and graphic information F4 are displayed at a predetermined position, they will not overlap with other areas of interest or other graphic information. On the other hand, if graphic information F3 is displayed in the upper right of the predetermined bounding box, the superposition position determination unit 208 displays graphic information F3 in the lower left of the bounding box because it would overlap with area of ​​interest C1 and bounding box B1, hindering observation of area of ​​interest C1. Also, if graphic information F2 is displayed in the upper right of the predetermined bounding box, the superposition position determination unit 208 displays it in the lower left of the bounding box because it would overlap with graphic information F1.

[0067] In this way, the superposition position determination unit 208 determines the superposition position of graphic information F3 according to the positions of the areas of interest C1, C2, C4 and graphic information F1, F2, and F4. The superposition position determination unit 208 also determines the superposition position of graphic information F2 according to the positions of the areas of interest C1, C3, C4 and graphic information F1, F3, and F4. As a result, graphic information F1 to F4 are displayed on the monitor 118 in an easy-to-view manner, without overlapping with each other or overlapping with the areas of interest.

[0068] Next, we will explain a medical image processing method using a medical image processing device. In this medical image processing method, each step is executed by the processor executing a program.

[0069] Figure 7 is a flowchart illustrating a medical image processing method.

[0070] First, the transmitting / receiving unit 202 and the image generation unit 204, which function as an ultrasound image acquisition unit, acquire an ultrasound image (Step S10: Image acquisition step). Then, the region information generation unit 206 generates region information including the location and category classification of the region of interest included in the ultrasound image, and the region information is acquired (Step S11: Region information acquisition step). Next, the superimposition position determination unit 208 determines whether or not there are multiple regions of interest detected in the region information (Step S12). If there is only one region of interest in the region information, the superimposition position determination unit 208 superimposes the graphic information onto the ultrasound image at a predetermined position and displays it. On the other hand, if there is multiple regions of interest in the region information, the superimposition position determination unit 208 determines the superimposition position of the graphic information based on the relative positional relationship of the multiple regions of interest (Step S13: Superimposition position determination step). Then, the display control unit 210 superimposes the graphic information onto the ultrasound image based on the determination of the superimposition position and displays it on the monitor 118 (Step S14: Display control step).

[0071] As described above, according to this embodiment, when the region information contains information about multiple regions of interest, the display control unit 210 determines and displays the superposition position of each graphic information to be displayed based on the relative positional relationship of the multiple regions of interest. As a result, this embodiment can display the regions of interest and graphic information in an easy-to-view manner, even when multiple regions of interest are detected.

[0072] <Second Embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, the inclusion relationship of the detected area of ​​interest is obtained, and graphic information is displayed based on that inclusion relationship.

[0073] Figure 8 is a block diagram showing an embodiment of the ultrasonic processor device 112 of this embodiment. Note that parts that have already been described in Figure 2 are denoted by the same reference numerals and their descriptions are omitted.

[0074] The ultrasonic processor device 112 shown in Figure 8 consists of a transmitting / receiving unit 202, an image generation unit 204, a region information generation unit 206, an inclusion relationship acquisition unit 216, an overlap position determination unit 208, a display control unit 210, a CPU (Central Processing Unit) 210, and a memory 214. The processing of each unit is realized by one or more processors (not shown).

[0075] The inclusion relationship acquisition unit 216 performs inclusion relationship acquisition processing and acquires inclusion relationship information for multiple areas of interest based on the area information. Specifically, the inclusion relationship acquisition unit 216 acquires inclusion relationships between areas, such as when one area is included in another, based on the area information. For example, if the area information generation unit 206 detects the area of ​​the pancreas and the main pancreatic duct, the inclusion relationship acquisition unit 216 acquires the inclusion relationship that the main pancreatic duct is included in the pancreas, based on the positional relationship of the detected areas of interest or the classified category. The inclusion relationship acquisition unit 216 can acquire the inclusion relationships of areas of interest by various methods. For example, the inclusion relationship acquisition unit 216 stores table data indicating inclusion relationships in advance and acquires the inclusion relationship based on that table data and category classification. Then, the superposition position determination unit 208 determines the superposition position of the graphic information based on the inclusion relationship. In addition, the display control unit 210 can change the display form of the graphic information based on the inclusion relationship.

[0076] Figure 9 shows an example of a display configuration to which the present invention is applied. Note that parts already described in Figure 6 are denoted by the same reference numerals, and their descriptions are omitted.

[0077] The inclusion relationship acquisition unit 216 acquires region information from the ultrasound image P1 and acquires the inclusion relationship that region of interest C1 is contained within region of interest C2. Based on the category classification of the region information, the inclusion relationship acquisition unit 216 determines that region of interest C1 is the pancreas and region of interest C2 is the main pancreatic duct, and based on the stored table data, acquires the inclusion relationship information that region of interest C2 is contained within region of interest C1.

[0078] In the example shown in Figure 9, the display control unit 210 displays the graphic information F2 and the area of ​​interest C2 in association with a leader line M. That is, according to the inclusion relationship information, the area of ​​interest C2 is included in the area of ​​interest C1, so if the graphic information F2 is displayed at a predetermined position (upper right of the bounding box B2), the graphic information F2 will be displayed overlapping the area of ​​interest C1. Therefore, the overlap position determination unit 208 determines the overlap position of the graphic information F2 to be outside the range of the area of ​​interest C1. The overlap position determination unit 208 also determines whether or not to display the leader line based on the inclusion relationship, and the display control unit 210 displays the leader line M to indicate the correspondence between the graphic information F2 and the area of ​​interest C2.

[0079] Furthermore, in the example shown in Figure 9, the position of the area of ​​interest C2 is indicated by the leader line M, so the bounding box B2 is hidden. This prevents the bounding box B2 from overlapping the area of ​​interest C1, thereby reducing the overall clutter of the image display.

[0080] As explained above, the superposition position determination unit 208 can also determine the superposition position of graphic information based on the inclusion relationship. By determining the superposition position of graphic information based on the inclusion relationship in this way, it is possible to prevent the graphic information from being displayed overlapping the area of ​​interest. Furthermore, if the graphic information is displayed away from the corresponding area of ​​interest, the display control unit 210 can indicate the correspondence between the graphic information and the area of ​​interest by displaying a leader line.

[0081] Figure 10 shows another example of the display configuration of this embodiment. As explained in Figure 9, the superimposed position determination unit 208 has an inclusion relationship between the area of ​​interest C1 and the area of ​​interest C2.

[0082] The display control unit 210 displays graphic information F1 and graphic information F2 using nested display N, based on the inclusion relationship between the area of ​​interest C1 and the area of ​​interest C2. By displaying graphic information F1 and graphic information F2 using nested display N in this way, it is possible to avoid displaying graphic information F2 overlapping the area of ​​interest C1, and to indicate the inclusion relationship between graphic information F1 and graphic information F2.

[0083] As explained above, the display control unit 210 can also display graphic information in a nested format based on the inclusion relationship. By using this nested format to display graphic information with an inclusion relationship, it is possible to prevent the graphic information from overlapping the area of ​​interest. Furthermore, the nested format allows the user to see the inclusion relationship of the graphic information.

[0084] <Third Embodiment> Next, a third embodiment of the present invention will be described. In this embodiment, adjustments are made to the movement of the superposition position of graphic information between image frames that constitute a video.

[0085] Ultrasound image acquisition processing performed by the transmitting / receiving unit 202 and the image generation unit 204 sequentially acquires a series of ultrasound images in chronological order.

[0086] The region information generation unit 206 generates region information for each ultrasound image in a sequence of ultrasound images. The superposition position determination unit 208 determines the superposition position of graphic information for each ultrasound image based on the region information generated by the region information generation unit 206 for each ultrasound image.

[0087] Figures 11 and 12 illustrate the display of region information between ultrasound image P1 and ultrasound image P2.

[0088] Ultrasound images P1 and P2 are sequential in time, and the superposition positions of graphic information F1 to F4 are determined by the superposition position determination unit 208 in ultrasound images P1 and P2. The superposition position determination unit 208 determines the superposition positions of graphic information F1 to F4 in each of ultrasound images P1 and P2, according to the positions of the areas of interest C1 to C4 and the positions of graphic information F1 to F4.

[0089] Between ultrasound image P1 and ultrasound image P2, graphic information F1, F3, and F4 have moved. Specifically, graphic information F1 was located in the upper left of bounding box B1 in ultrasound image P1, but is located to the right of bounding box B1 in ultrasound image P2. Also, graphic information F3 was located in the lower left of bounding box B3 in ultrasound image P1, but is located in the upper left of bounding box B3 in ultrasound image P2. Furthermore, graphic information F4 was located in the upper right of bounding box B4 in ultrasound image P1, but is located in the lower right of bounding box B4 in ultrasound image P2. If the superposition position of graphic information changes significantly between ultrasound images P1 and P2, which are sequentially consecutive, visibility will decrease. Therefore, in this embodiment, the superposition position is determined so that the graphic information does not change significantly between ultrasound image P1 and ultrasound image P2. Specifically, the superposition position determination unit 208 determines the superposition position of graphic information in ultrasound image P2 (current ultrasound image) based on the superposition position of graphic information in ultrasound image P1 (past ultrasound image). For example, as described below, the superposition position determination unit 208 determines the superposition position of graphic information in a region where the distance from the superposition position of graphic information in ultrasound image P1 is within a first threshold.

[0090] Figure 12 shows an example of the display configuration in the monitor 118 of this embodiment.

[0091] In the case shown in Figure 12, the superposition position determination unit 208 determines the superposition position of the graphic information within a region where the distance is within a first threshold. Specifically, each of the graphic information F1, F3, and F4 is superimposed at a position moved within a region within the first threshold from its position in the ultrasound image P1. In this way, by determining the superposition position of the graphic information within a region within a first threshold from its superposition position in the ultrasound image P1, the superposition position of the graphic information does not change significantly between frames, allowing for a clear and easy-to-view display.

[0092] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described. In this embodiment, if annotations are drawn by the user while the ultrasound image P is displayed on the monitor 118, the region information is displayed while avoiding the area of ​​the annotations.

[0093] Figure 13 illustrates an example of the display configuration of this embodiment. Note that parts already described in Figure 6 are denoted by the same reference numerals, and their descriptions are omitted.

[0094] The user may add annotations directly to the ultrasound image P when it is displayed on the monitor 118. The user can add annotations to the ultrasound image using an operating unit (not shown) connected to the ultrasound processor.

[0095] The region information generation unit 206 detects an annotated region as the region of interest C5. In this case, the category classification of the region of interest C5 is the annotation. The region information generation unit 206 then generates region information that includes the location and category classification (annotation) of the region of interest C5. Note that if the annotation is detected as the region of interest C5, the graphic information is not displayed.

[0096] The superposition position determination unit 208 then superimposes the graphic information based on the region information so that the graphic information does not overlap with the regions of interest C1 to C5. Specifically, if graphic information F1 is superimposed at a predetermined position (upper right of bounding box B1), it will overlap with region of interest C5, so graphic information F1 is superimposed at the upper left of bounding box B1. In this way, even annotations added while the ultrasound image is being displayed can be displayed clearly by ensuring that the graphic information does not overlap.

[0097] <Other> The above description focused on ultrasound images as an example of medical images, but the present invention is not limited to ultrasound images. For example, the present invention is also applicable to endoscopic images, which are another example of medical images.

[0098] In the above embodiment, the hardware structure of the processing unit that performs various processes is a variety of processors as shown below. These various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as a processing unit; a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which has a circuit configuration specifically designed to perform a particular process.

[0099] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0100] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0101] Each of the above-described configurations and functions can be appropriately implemented using any hardware, software, or a combination thereof. For example, the present invention can also be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-temporary recording medium) that records such a program, or a computer on which such a program can be installed.

[0102] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0103] 102: Ultrasound Endoscopy System 110: Ultrasonic scope 112: Ultrasonic processor device 114: Endoscope processor device 116:Light source device 118: Monitor 120: Insertion part 120a: Long axis 122: Handheld control unit 124: Universal Code 126: Ultrasonic connector 128: Endoscope connector 130: Connector for light source 132: Tube 134: Tube 136: Air / Water Supply Button 138: Suction button 142: Angle knob 144: Insertion port for treatment instruments 150: Tip body 152: Curved section 154: Soft part 162:Ultrasonic probe 164: Balloon 170: Water supply tank 172: Suction pump 202: Transceiver Unit 204: Image generation unit 206: Area information generation unit 208: Superposition position determination unit 210: Display Control Unit 212:CPU 214: Memory 216: Inclusion relationship acquisition unit

Claims

1. A medical image processing device equipped with a processor, The aforementioned processor, Image acquisition processing to obtain medical images, A region information acquisition process that acquires region information relating to the multiple regions of interest included in the medical image, including the location and categorization of the multiple regions of interest, A display control process that overlays multiple graphic information indicating the categorization of the multiple areas of interest onto the medical image and displays it on the display unit; A superposition position determination process that determines the superposition position of the plurality of graphic information to be displayed by the display control process based on the relative positional relationship of the plurality of areas of interest, A medical image processing device that performs the following: The image acquisition process acquires a plurality of medical images that are sequentially arranged in time, The superposition position determination process is a medical image processing device that determines the superposition position of graphic information in the current medical image within a region where the distance from the superposition position of graphic information in past medical images is within a first threshold.

2. The medical image processing apparatus according to claim 1, wherein the processor detects the plurality of areas of interest contained in the medical image, estimates the category classification of the detected plurality of areas of interest, and performs area information generation processing to generate area information.

3. The medical image processing apparatus according to claim 1 or 2, wherein at least one of the multiple regions of interest is an anatomical region.

4. The medical image processing apparatus according to any one of claims 1 to 3, wherein at least one of the multiple areas of interest is an annotation drawn on the medical image by the user.

5. The medical image processing apparatus according to any one of claims 1 to 4, wherein the superposition position determination process determines the superposition position of graphic information of at least one of the plurality of areas of interest based on the positions of the other areas of interest.

6. The medical image processing apparatus according to any one of claims 1 to 5, wherein the superposition position determination process determines the superposition position of graphic information of at least one of the plurality of areas of interest based on the superposition position of graphic information of other areas of interest among the plurality of areas of interest.

7. The medical image processing apparatus according to any one of claims 1 to 6, wherein the display control process displays a leader line up to the superposition position of graphic information of at least one of the plurality of areas of interest.

8. The processor performs an inclusion relationship acquisition process to acquire inclusion relationship information of the multiple areas of interest based on the area information. The medical image processing apparatus according to any one of claims 1 to 7, wherein the superposition position determination process determines the superposition position of the plurality of graphic information based on the inclusion relationship information.

9. The medical image processing apparatus according to claim 8, wherein the superimposition position determination process determines whether to display the leader line of at least one of the plurality of areas of interest based on the inclusion relationship information, and switches whether or not to display it.

10. The medical image processing apparatus according to claim 9, wherein the display control process displays graphic information of the areas of interest that are in an inclusion relationship in a nested format that indicates the inclusion relationship, based on the inclusion relationship information.

11. The medical image processing apparatus according to any one of claims 1 to 10, wherein the display control process displays information indicating the range of the area of ​​interest based on the area information.

12. The information indicating the extent of the area of ​​interest is a bounding box. The medical image processing apparatus according to claim 11, wherein the display control process displays the graphic information in correspondence with the bounding box.

13. The medical image processing apparatus according to any one of claims 1 to 12, wherein the graphic information is composed of textual information indicating the category classification.

14. A medical image processing method using a medical image processing device equipped with a processor, The following is performed on the aforementioned processor: The image acquisition process for obtaining medical images, A region information acquisition step, which acquires region information relating to the multiple regions of interest, including the location and categorization of the multiple regions of interest included in the medical image, A display control step which involves displaying multiple graphic information indicating the categorization of the multiple areas of interest on the display unit, superimposed on the medical image; A superposition position determination step that determines the superposition position of the plurality of graphic information to be displayed by the display control step based on the relative positional relationship of the plurality of areas of interest, A medical image processing method including, In the image acquisition step, a plurality of medical images that are sequentially consecutive are acquired. A medical image processing method in which, in the superposition position determination step, the superposition position of the graphic information of the current medical image is determined to be within a region where the distance from the superposition position of the graphic information in past medical images is within a first threshold.

15. A program that causes a medical image processing device equipped with a processor to execute a medical image processing method, The aforementioned processor, The image acquisition process for obtaining medical images, A region information acquisition step, which acquires region information relating to the multiple regions of interest, including the location and categorization of the multiple regions of interest included in the medical image, A display control step which involves displaying multiple graphic information indicating the categorization of the multiple areas of interest on the display unit, superimposed on the medical image; A superposition position determination step that determines the superposition position of the plurality of graphic information to be displayed by the display control step based on the relative positional relationship of the plurality of areas of interest, A program that executes, In the image acquisition step, a plurality of medical images that are sequentially consecutive are acquired. The superposition position determination step is a program that determines the superposition position of the graphic information of the current medical image in a region where the distance from the superposition position of the graphic information in past medical images is within a first threshold.

16. A non-temporary and computer-readable recording medium on which the program described in claim 15 is recorded.

Images