Endoscope system, method of operating the endoscope system, program, and recording medium.

Abstract

One aspect of the present invention provides an endoscope system capable of appropriately displaying the recognition results of a region of interest. [Solution] An endoscope system according to one aspect of the present invention is an endoscope system comprising a processor and an ultrasound endoscope capable of capturing ultrasound images, wherein the processor is capable of displaying on a display device the position information and type information of the region of interest in the B-mode image of the ultrasound image superimposed on the ultrasound image, and the position in which the position information is displayed changes in accordance with the change in the position of the region of interest over time, while the position in which the type information is displayed is maintained regardless of the change in the position of the region of interest over time.

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus, an endoscope system, a medical image processing method, and a medical image processing program, and particularly relates to a technique for displaying a recognition result of a region of interest.

Background Art

[0002] It is known to notify a recognition result of a region of interest by a medical image processing apparatus as support for a user such as a doctor when observing and diagnosing a medical image. For example, Patent Document 1 describes displaying a bounding box or the like at the position of the recognized region of interest. Patent Document 1 also describes displaying characters indicating the region of interest outside the image area.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When performing observations using medical devices such as endoscopes and ultrasound machines, physicians operate the equipment to display the desired area on the screen while viewing organs, blood vessels, etc. (areas of interest, anatomical areas), but this is an extremely difficult operation. Therefore, to simplify the operation of the equipment, it is conceivable to use image recognition technology such as AI (Artificial Intelligence) to detect organs, blood vessels, etc. and present them to the user (for example, displaying the recognition results on the screen). It is preferable that such a display can be freely turned on or off at the user's discretion. However, depending on the type of medical device, the user may need to use both hands to operate it, so it is not always possible for the user to turn the display on or off. Therefore, it is preferable that the display of recognition results assists the user's observation without hindering it. However, with conventional technologies such as those described in Patent Document 1 above, it has been difficult to appropriately display the recognition results of the area of ​​interest.

[0005] This invention has been made in view of these circumstances, and aims to provide a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program that can appropriately display the recognition results of a region of interest. [Means for solving the problem]

[0006] To achieve the above-mentioned objectives, a medical image processing apparatus according to a first aspect of the present invention is a medical image processing apparatus comprising a processor, the processor performing an image acquisition process for sequentially acquiring time-series medical images, a focus region recognition process for recognizing the location and type of focus region from the medical image, and a display control process for displaying on a display device the location information indicating the location of the focus region and type information indicating the type of focus region superimposed on the medical image, wherein in the display control process, the processor changes the position where the location information is displayed according to the change in the location of the focus region over time, and maintains the position where the type information is displayed regardless of the change in the location of the focus region over time.

[0007] One possible method for displaying the recognition results of a region of interest in a medical image is to display location information indicating the location of the region of interest, and type information such as the name of the region of interest. However, since the location of the region of interest in a medical image changes as observation progresses (over time), if the display position of the type information is changed according to the change in the location of the region of interest, the display position of the type information will also change significantly when the change in the location of the region of interest is large, reducing visibility. On the other hand, if the display position of the location information is maintained regardless of the change in the location of the region of interest, the location of the region of interest cannot be displayed correctly.

[0008] From this perspective, the medical image processing device related to the first response can support the user's observation by displaying location information and type information, while reducing the risk of hindering the user's observation by displaying location information and type information.

[0009] Thus, according to the first embodiment, the recognition result of the area of ​​interest can be appropriately displayed. In addition, in the first embodiment, for some areas of interest, the display position of type information may be linked to the change in the position of the area of ​​interest, while for other areas of interest, the display position of type information may be maintained regardless of the change in the position of the area of ​​interest. Furthermore, whether or not to link the display position of type information to the change in the position of the area of ​​interest may be set according to the type of area of ​​interest (characteristics of the anatomical area).

[0010] In the first embodiment, the type of region of interest may be, for example, the name of an organ or blood vessel, or the classification result of a lesion, but is not limited to these examples. In the first embodiment, the region of interest may be recognized using a detector configured by machine learning.

[0011] In the first embodiment and in each of the following embodiments, "acquisition of time-series medical images" includes sequentially acquiring multiple medical images captured at a predetermined frame rate. The acquisition may be in real time or not. For example, medical images that have been captured and recorded in advance may be acquired.

[0012] The medical image processing device according to the first embodiment can be implemented, for example, as the processor part of a medical image processing system, but is not limited to this embodiment. "Medical image" refers to an image obtained as a result of photographing or measuring a living body such as the human body for the purpose of diagnosis, treatment, measurement, etc. Examples include endoscopic images, ultrasound images, CT images (CT: Computed Tomography), and MRI images (MRI: Magnetic Resonance Imaging). Medical images are also called medical images. Furthermore, in the first embodiment and each of the following embodiments, the "Region of Interest (ROI)" may be a lesion area or candidate lesion area in a medical image, an organ or blood vessel, a post-treatment area, a treatment instrument, etc. The "Region of Interest" is sometimes called the "area of ​​concern."

[0013] In the second embodiment, the medical image processing apparatus, in the first embodiment, has a processor that, in display control processing, displays a figure or character as positional information at the location of the area of ​​interest in the medical image. The second embodiment defines a specific form of positional information.

[0014] In the third embodiment, the medical image processing apparatus, in the first or second embodiment, has a processor that, in the display control processing, displays characters indicating the type of area of ​​interest as type information. The third embodiment defines a specific form of the type information.

[0015] The medical image processing device according to the fourth embodiment, in any one of the first to third embodiments, has a processor that displays a figure or character set according to the type of area of ​​interest as location information. The fourth embodiment defines a specific form of location information, and by displaying location information according to the type of area of ​​interest, the user can easily grasp the recognition result of the area of ​​interest.

[0016] In the fifth embodiment, the medical image processing device, in any one of the first to fourth embodiments, has a processor that displays location information in association with type information during display control processing. According to the fifth embodiment, the user can easily understand the relationship between location information and type information.

[0017] In the sixth embodiment, the medical image processing apparatus, in any one of the first to fifth embodiments, displays a line segment as positional information in the display control process, where one endpoint is the location of the area of ​​interest and the other endpoint is the location of the type information. According to the sixth embodiment, the user can easily grasp the relationship between the positional information and the type information.

[0018] In the medical image processing apparatus according to the seventh embodiment, in any one of the first to sixth embodiments, the processor displays the type of the area of ​​interest recognized in the area of ​​interest recognition process as type information in the display control process. In the seventh embodiment, "type of area of ​​interest recognized in the area of ​​interest recognition process" means the type of area of ​​interest that was actually recognized.

[0019] In the medical image processing apparatus according to the eighth embodiment, in any one of the first to sixth embodiments, the processor displays the types of focus regions that can be recognized by the focus region recognition process as type information during the display control process. In the eighth embodiment, "types of focus regions that can be recognized by the focus region recognition process" means types of focus regions that may be recognized even if they are not actually recognized. These "types of recognizable focus regions" may differ depending on the type of medical device and the configuration of the recognition device.

[0020] In the ninth embodiment, the medical image processing apparatus, in the eighth embodiment, causes the processor to display type information corresponding to areas of interest that are not actually recognized among the areas of interest that can be recognized in the area of ​​interest recognition process, with a second notification power lower than the first notification power for type information indicating areas of interest that are actually recognized in the area of ​​interest recognition process. If all types of recognizable areas of interest are displayed with the same notification power, it may be difficult for the user to grasp the types of areas of interest that have actually been recognized. Therefore, by changing the notification power as in the ninth embodiment, the user can easily grasp the types of areas of interest that have actually been recognized.

[0021] In the medical image processing apparatus according to the tenth embodiment, in any one of the first to ninth embodiments, the processor, in display control processing, displays position information inside the image signal display area of ​​the medical image and type information outside the image signal display area of ​​the medical image.

[0022] To achieve the above-mentioned objectives, the endoscopic system according to the 11th aspect of the present invention comprises a medical image processing device described in any one of the first to tenth aspects, an endoscope scope inserted into a subject and equipped with an imaging unit for sequentially capturing medical images, and a display device. Since the endoscopic system according to the 11th aspect comprises a medical image processing device described in any one of the first to tenth aspects, the recognition results of the area of ​​interest can be appropriately displayed.

[0023] In the 11th embodiment, the endoscopic system is an ultrasound endoscopic scope that acquires ultrasound images of a subject as medical images.

[0024] In order to achieve the above object, a medical image processing method according to a 13th aspect of the present invention is a medical image processing method executed by a medical image processing apparatus including a processor. The processor executes an image acquisition step of sequentially acquiring time-series medical images, a target region recognition step of recognizing the position and type of a target region from the medical images, and a display control step of superimposing position information indicating the position of the target region and type information indicating the type of the target region on the medical image and causing the display device to display the same. In the display control step, the processor changes the position where the position information is displayed according to the change in the position of the target region over time, and maintains the position where the type information is displayed without depending on the change in the position of the target region over time. According to the 13th aspect, similar to the 1st aspect, appropriate notification can be performed according to the drawing information. Note that the medical image processing method according to the 13th aspect may further execute the same processing as that of the 2nd to 10th aspects.

[0025] In order to achieve the above object, a medical image processing program according to a 14th aspect of the present invention is a medical image processing program that causes a medical image processing apparatus including a processor to execute a medical image processing method. The medical image processing method includes an image acquisition step of sequentially acquiring time-series medical images, a target region recognition step of recognizing the position and type of a target region from the medical images, and a display control step of superimposing position information indicating the position of the target region and type information indicating the type of the target region on the medical image and causing the display device to display the same. In the display control step, the position where the position information is displayed is changed according to the change in the position of the target region over time, and the position where the type information is displayed is maintained without depending on the change in the position of the target region over time. According to the 14th aspect, similar to the 1st aspect, appropriate notification can be performed according to the drawing information. Note that the medical image processing program according to the 14th aspect may be a program that further causes the same processing as that of the 2nd to 10th aspects to be executed. Note that a non-temporary recording medium recording the computer-readable code of the program of these aspects can also be cited as an aspect of the present invention.

Advantages of the Invention

[0026] As described above, according to the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program according to the present invention, the recognition result of the target region can be appropriately notified.

Brief Description of the Drawings

[0027] [Figure 1] FIG. 1 is an external view of an endoscope system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the main configuration of an ultrasonic processor device. [Figure 3] FIG. 3 is a flowchart showing the procedure of a medical image processing method according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a setting screen for the display mode. [Figure 5] FIG. 5 is a diagram showing an example of the superimposed display of position information and type information. [Figure 6] FIG. 6 is a diagram showing a comparative example of the superimposed display of position information and type information. [Figure 7] FIG. 7 is a diagram showing another example of the superimposed display of position information and type information. [Figure 8] FIG. 8 is a diagram showing yet another example of the superimposed display of position information and type information. [Figure 9] FIG. 9 is a diagram showing yet another example of the superimposed display of position information and type information. [Figure 10] FIG. 10 is a diagram showing yet another example of the superimposed display of position information and type information. [Figure 11] FIG. 11 is a diagram showing yet another example of the superimposed display of position information and type information. [Figure 12] FIG. 12 is a diagram showing an example of setting the display mode according to the characteristics of the anatomical region.

Embodiments for Carrying Out the Invention

[0028] Hereinafter, embodiments of the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program according to the present invention will be described in detail with reference to the attached drawings.

[0029] [First Embodiment] [Overall configuration of an endoscopy system including medical image processing equipment] Figure 1 is an external view of the endoscope system according to the first embodiment. As shown in Figure 1, the endoscope system 2 (endoscope system, medical image acquisition device) comprises an ultrasound scope 10 (endoscope scope, ultrasound endoscope scope), an ultrasound processor device 12 (medical image processing device) that generates ultrasound images (medical images), an endoscope processor device 14 (medical image processing device) that generates endoscopic images (medical images), a light source device 16 that supplies illumination light (observation light) to the ultrasound scope 10 to illuminate the inside of the body cavity, and a monitor 18 (display device) that displays ultrasound images and endoscopic images.

[0030] The ultrasound scope 10 comprises an insertion section 20 inserted into the body cavity of the subject, a handheld control section 22 connected to the proximal end of the insertion section 20 for operation by the operator, and a universal cord 24 with one end connected to the handheld control section 22. The other end of the universal cord 24 is provided with an ultrasound connector 26 connected to an ultrasound processor device 12, an endoscope connector 28 connected to an endoscope processor device 14, and a light source connector 30 connected to a light source device 16.

[0031] The ultrasound scope 10 is detachably connected to the ultrasound processor unit 12, the endoscope processor unit 14, and the light source unit 16 via these connectors. In addition, the light source connector 30 is connected to a tube 32 for air and water supply and a tube 34 for suction.

[0032] The light source device 16 consists of illumination light sources (for example, a red light source, a green light source, a blue light source, and a violet light source that emit narrowband light of red, green, blue, and violet, respectively), an aperture, a focusing lens, and a light source control unit, and these light sources can be used to produce observation light such as normal light (white light), special light (narrowband light, etc.), and combinations thereof.

[0033] The monitor 18 receives the respective video signals generated by the ultrasound processor unit 12 and the endoscope processor unit 14 and displays the ultrasound image and the endoscope image. The display of the ultrasound image and the endoscope image can be switched between as needed and displayed on the monitor 18, or both images can be displayed simultaneously.

[0034] The handheld control unit 22 is equipped with an air supply / water supply button 36 and a suction button 38 side by side, as well as a pair of angle knobs 42 and a treatment instrument insertion port 44.

[0035] The insertion section 20 has a tip, a base, and a longitudinal axis 20a. Starting from the tip, it consists of a tip body 50 made of a rigid material, a curved section 52 connected to the base of the tip body 50, and a slender, long, flexible flexible section 54 connecting the base of the curved section 52 to the tip of the hand-operated section 22. Specifically, the tip body 50 is located on the tip side of the insertion section 20 in the direction of the longitudinal axis 20a. The curved section 52 is bent by rotating a pair of angle knobs 42 provided on the hand-operated section 22. This allows the user to orient the tip body 50 in the desired direction.

[0036] The tip body 50 is equipped with an ultrasonic transducer 62 (imaging unit) and a bag-shaped balloon 64 that encloses the ultrasonic transducer 62. The balloon 64 can be inflated or deflated by water being supplied from the water tank 70 or by the water being sucked out of the balloon 64 by the suction pump 72. The balloon 64 is inflated until it contacts the inner wall of the body cavity in order to prevent attenuation of ultrasound and ultrasonic echoes (echo signals) during ultrasound observation.

[0037] Furthermore, the tip body 50 is fitted with an endoscope observation unit (not shown) which has an observation unit and an illumination unit equipped with an objective lens and an image sensor. The endoscope observation unit is located behind the ultrasonic probe 62 (on the handheld control unit 22 side).

[0038] The endoscope system 2 can sequentially acquire (sequentially image) endoscopic images (optical images) and ultrasound images with the configuration described above. The endoscope system 2 may also acquire endoscopic images and ultrasound images from the recording unit 120 or from a server or database (not shown).

[0039] [Medical image processing device] Figure 2 is a block diagram showing the main components of an ultrasonic processor device.

[0040] The ultrasound processor device 12 (medical image processing device, processor) shown in Figure 2 is a device that recognizes a region of interest (object) within a medical image based on sequentially acquired time-series medical images and displays the recognition result on a display device. It consists of a transmitting / receiving unit 100 (processor, image acquisition unit), an image generation unit 102 (processor, image acquisition unit), a CPU 104 (processor, CPU: Central Processing Unit), a region of interest recognition unit 106 (processor, region of interest recognition unit), a communication control unit 110 (processor), a display control unit 112 (processor, display unit), a memory 118, and a recording unit 120 (recording device). The processing of each of these units is realized by one or more processors, as will be described later.

[0041] The CPU 104 operates based on various programs, including the medical image processing program according to the present invention, stored in the memory 118, and comprehensively controls the area of ​​interest recognition unit 106, the communication control unit 110, and the display control unit 112, and also functions as a part of each of these units. The memory 118 includes a non-temporary recording medium such as ROM (Read Only Memory) on which the medical image processing program and the like are recorded, and a temporary recording medium such as RAM (Random Access Memory) used as a temporary storage area.

[0042] The transmitting / receiving unit 100 and the image generation unit 102, which function as image acquisition units, sequentially acquire time-series medical images (image acquisition processing, image acquisition process).

[0043] The transmitting unit of the transmitting / receiving unit 100 generates multiple drive signals to be applied to multiple ultrasonic transducers of the ultrasonic probe 62 of the ultrasonic scope 10, and applies the multiple drive signals to the multiple ultrasonic transducers by assigning a delay time to each of the multiple drive signals based on a transmission delay pattern selected by a scanning control unit (not shown).

[0044] The receiver of the transmitting / receiving unit 100 amplifies multiple detection signals output from multiple ultrasonic transducers of the ultrasonic probe 62 and converts the analog detection signals into digital detection signals (also called RF (Radio Frequency) data). This RF data is input to the image generation unit 102.

[0045] The image generation unit 102 performs reception focus processing by assigning delay times to multiple detection signals represented by RF data based on the reception delay pattern selected by the scanning control unit, and then adding these detection signals together. This reception focus processing forms sound line data in which the focus of the ultrasonic echo is narrowed.

[0046] The image generation unit 102 applies STC (Sensitivity Time Control) to the sound line data to correct for attenuation due to distance according to the depth of the ultrasonic reflection position, and then generates envelope data by performing envelope detection processing using a low-pass filter or the like. The image generation unit 102 stores envelope data for one frame, more preferably multiple frames, in a cine memory (not shown). The image generation unit 102 then applies preprocessing such as log (logarithmic) compression and gain adjustment to the envelope data stored in the cine memory to generate a B-mode image.

[0047] In this way, the transmitting / receiving unit 100 and the image generation unit 102 sequentially acquire time-series B-mode images (hereinafter referred to as "medical images").

[0048] The area of ​​interest recognition unit 106 performs the following processes: recognizing information about the location of the area of ​​interest within a medical image based on the medical image (detection process, area of ​​interest recognition process, area of ​​interest recognition step); and classifying the area of ​​interest into one of several classes based on the medical image (classification process, classification step). For example, it can be configured using a trained model (a model trained using an image set composed of images of living organisms) constructed by machine learning, such as a CNN (Convolutional Neural Network) or an SVM (Support Vector Machine). In this embodiment, the area of ​​interest is, for example, an organ or blood vessel in a medical image (a tomographic image of a B-mode image), such as the pancreas, main pancreatic duct, spleen, splenic vein, splenic artery, gallbladder, etc.

[0049] An example of a layer configuration when the region recognition unit 106 is constructed using a CNN is described below. The CNN includes an input layer, an intermediate layer, and an output layer. The input layer takes a medical image generated by the image generation unit 102 as input and outputs features. The intermediate layer includes a convolutional layer and a pooling layer, and calculates other features by taking the features output by the input layer as input. These layers have a structure in which multiple "nodes" are connected by "edges" and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The output layer recognizes the region of interest in the input medical image based on the features output from the intermediate layer and outputs the result.

[0050] In this example, the area of ​​interest recognition unit 106, upon sequential input of time-series medical images, recognizes (detects) the location of the area of ​​interest for each input medical image, outputs information related to that location (location information), and also recognizes (classifies) which of several classes the area of ​​interest belongs to, and outputs information indicating the recognized class (class information, type information).

[0051] The display control unit 112 displays time-series medical images (endoscopic images, ultrasound images) acquired sequentially by the transmitting / receiving unit 100 and the image generation unit 102 on the monitor 18 (display device). In this example, a moving image showing an ultrasound tomography is displayed on the monitor 18. The display control unit 112 also displays the target object on the monitor 18 with a notification power determined by the notification power determination unit 108.

[0052] Medical image processing using the functions described above will be explained in detail later.

[0053] <Realization of functions by various processors> The functions of the ultrasonic processor device 12 described above can be realized using various processors and recording media. These various processors include, for example, a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) to realize various functions. Furthermore, these various processors also include a GPU (Graphics Processing Unit), a processor specialized for image processing, and programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), whose circuit configurations can be changed after manufacturing. When processing images as in the present invention, a configuration using a GPU is effective. Additionally, dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform particular processing, are also included in the above-mentioned "various processors."

[0054] The functions of each part may be realized by a single processor, or by multiple processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Furthermore, multiple functions may be realized by a single processor. Examples of configuring multiple functions with a single processor include, firstly, as exemplified by computers, a single processor composed of one or more CPUs and software, where this processor realizes multiple functions; and secondly, as exemplified by System-on-a-Chip (SoC), a processor that realizes the functions of the entire system on a single IC (Integrated Circuit) chip. Thus, various functions are configured as hardware structures using one or more of the aforementioned processors. More specifically, the hardware structures of these various processors are electrical circuits combining circuit elements such as semiconductor elements. These electrical circuits may be circuits that realize the aforementioned functions using logical operations such as logical OR, logical AND, logical negation, exclusive OR, and combinations thereof.

[0055] When the aforementioned processor or electrical circuit executes software (program), the code of the software to be executed is stored in a non-temporary recording medium such as ROM (Read Only Memory), which is readable by the computer (for example, the various processors and electrical circuits constituting the ultrasound processor device 12, and / or a combination thereof), and the computer refers to the software. The software stored in the non-temporary recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used for setting display modes and notification modes, weight parameters used in the detector 223, etc.). The code may be recorded in a non-temporary recording medium such as various magneto-optical recording devices or semiconductor memory instead of ROM. When processing using software, for example, RAM (Random Access Memory) may be used as a temporary storage area, and data stored in an EEPROM (Electrically Erasable and Programmable Read Only Memory), not shown, may also be referred to. The memory 118 may be used as the "non-temporary recording medium," or the recording unit 120 may be used.

[0056] The recording unit 120 records ultrasound images and endoscopic images (medical images), detection results of the area of ​​interest, and processing conditions (conditions for detection and notification). Other information may also be recorded. The communication control unit 110 controls the acquisition of medical images, etc., from other medical image acquisition devices connected to the endoscope system 2, external servers, or databases.

[0057] [Medical Image Processing Procedures] The medical image processing (execution of the medical image processing method and medical image processing program according to the present invention) in the endoscope system 2 with the above-described configuration will now be explained. Figure 3 is a flowchart showing the procedure of the medical image processing method according to the first embodiment. Note that the procedure described below may be executed in any order as necessary.

[0058] [Initial settings] The display control unit 112 (processor) sets the conditions necessary for executing the medical image processing method / program based on user operations via an operation unit (keyboard, mouse, touch panel, microphone, etc.) not shown, and / or pre-set processing conditions (for example, default processing conditions) (step S100: initial setup step). The display control unit 112 sets, for example, the display manner of location information and type information (type of characters and symbols, color, etc.), emphasis level, etc. The user can set the processing conditions by turning radio buttons on / off or selecting from pull-down menus via an operation unit on a screen like the one in Figure 4 (this does not show all processing condition settings). The display control unit 112 can display such a screen on a display device such as the monitor 18. These settings include, for example, what kind of shapes or characters (type, color, etc.) to use to display location information and type information, and whether or not to display a list of the types of recognizable areas of interest. The display control unit 112 may set processing conditions not only at the start of processing but also during the execution of the following steps.

[0059] [Acquisition of ultrasound images and recognition of areas of interest] The transmitting / receiving unit 100 and the image generation unit 102 sequentially acquire time-series ultrasound images (medical images) (step S110: image acquisition process, image acquisition step), and the display control unit 112 displays the acquired ultrasound images on the monitor 18 (step S120: display control process, display control step). The area of ​​interest recognition unit 106 recognizes the position and type of the area of ​​interest in the ultrasound image (step S130: area of ​​interest recognition process, area of ​​interest recognition step). The area of ​​interest recognition unit 106 can, for example, define the position of the area of ​​interest as the center position of the rectangle surrounding the area of ​​interest, and define the information indicating that position (coordinates in the image, etc.) as "position information". In this embodiment, information indicating the type of organ or blood vessel can be defined as "class information, type information".

[0060] [Overlay display of location and type information] When the area of ​​interest recognition unit 106 detects an area of ​​interest (YES in step S140), the display control unit 112 determines the display mode of the position information and type information based on the conditions set in step S100 (step S150: display control processing, display control step), and displays the position information and type information superimposed on the ultrasound image on the display device (monitor 18, etc., the same applies hereinafter) according to the determined display mode (step S160: display control processing, display control step). For example, the display control unit 112 determines a display mode such as "superimposing a cross (figure; position information) at the center of the rectangle surrounding the area of ​​interest and displaying the name of the area of ​​interest that was actually recognized (detected) as text," and displays the image in that display mode. The processes from steps S110 to S160 are performed sequentially as time-series ultrasound images are acquired. Note that the recognition of the area of ​​interest and the display control processing may be performed for all of the time-series ultrasound images (i.e., for every frame), or for a part of the time-series ultrasound images (for some frames).

[0061] Figure 5 shows an example of the superimposed display of location information and type information. In part (a) of Figure 5, the pancreas 530 (area of ​​interest) and splenic vein 540 (area of ​​interest) are detected (recognized) in the ultrasound image, and figures 532 and 542 (figures, location information) indicating the locations of these areas of interest are superimposed within the image signal display area 510 (image signal display area, ultrasound image) of the display screen 500. On the other hand, the characters 550B ("Pancreas", type information) and 552B ("SV (Splenic Vein)", type information) indicating the names of the areas of interest are superimposed in area 520, which is outside the image signal display area 510. In addition, the same figures as figures 532 and 542 indicating the location information are displayed in area 520 as legends 550A and 552A (type information) alongside the characters 550B and 552B, and the display control unit 112 associates the location information with the type information in this way.

[0062] Part (b) of Figure 5 shows the state in which the position of the area of ​​interest (pancreas 530 and splenic vein 540) in the ultrasound image has changed as time has passed since the state shown in part (a) of the same figure (or, in addition to the passage of time, the observation position and direction have changed due to the operation of the ultrasound scope 10). In this state, the display control unit 112 (processor) changes the position in which the figures 532 and 542 (position information) are displayed according to the change in the position of the area of ​​interest due to the passage of time, but the position in which the characters 550B and 552B (type information) are displayed is maintained regardless of the change in the position of the area of ​​interest due to the passage of time.

[0063] In the display configuration shown in Figure 5, a graphic (location information) indicating the position of the area of ​​interest can assist the user's observation. Furthermore, since a simple graphic (an "X") is displayed as location information, the screen display is less likely to interfere with observation compared to displaying the names of organs, etc., as text. In addition, if the position of the area of ​​interest changes significantly on the screen over time, only the graphic (location information) moves, while the position of the text (type information) remains unchanged, allowing the user to easily understand the type of area of ​​interest.

[0064] Figure 6 shows a comparative example of the superimposed display of location information and type information, with parts (a) and (b) of the figure showing the changes over time. In the comparative example shown in Figure 6, both location information and type information are represented in text, and the display position of this information is linked to the change in the position of the area of ​​interest. Therefore, compared to the display configuration according to the present invention shown in Figure 5, the screen display is more likely to interfere with observation.

[0065] Figure 7 shows another example of the superimposed display of location information and type information. In the example shown in Figure 7, the shape of the symbol indicating the location information is set according to the type of area of ​​interest (in addition to the shape of the symbol, or instead of the shape, the color of the symbol may be set according to the type of area of ​​interest). Specifically, the display control unit 112 displays a figure 532 (cross mark) composed of diagonal line segments as location information for the pancreas 530, and a symbol 542A (cross) composed of horizontal and vertical line segments as location information for the splenic vein 540. Furthermore, the display control unit 112 displays these symbols alongside the characters 550B and 552B as legends 550A and 552C (type information). While users spend most of their time during examinations or observations looking at ultrasound images, it can be difficult for them to shift their gaze outside the image area (outside the image signal display area 510, area 520). However, even in such cases, by setting the shape of the symbols indicating positional information according to the type of area of ​​interest, as shown in Figure 7, users can easily understand the type of area of ​​interest by looking at the symbols displayed as positional information. Users can also adjust the shape and color of the symbols via a screen like the one shown in Figure 4, and the display control unit 112 can determine the display mode based on this user operation.

[0066] Figure 8 shows yet another example of the superimposed display of location information and type information. In the examples described above for Figures 5 and 7, an "X" or a cross is displayed as a symbol to indicate location information, but in the example shown in Figure 8, the display control unit 112 displays a figure 533 (location information) representing the initial letter P of "Pancreas" and a figure 543 (location information) representing the initial letter b of "blood vessel". The display control unit 112 also displays these symbols alongside the characters 551B and 553B (type information) as legends 551A and 553A (type information). With this display, the user can easily understand the type of area of ​​interest by looking at the symbols displayed as location information.

[0067] Figure 9 shows yet another example of the superimposed display of location information and type information. When type information is displayed only for the area of ​​interest that has actually been detected, the displayed characters may change as the type of the detected area of ​​interest changes over time, which may be annoying to the user. Therefore, in the example shown in Figure 9, the display control unit 112 displays the types of areas of interest that can be recognized by the area of ​​interest recognition process as characters 560 (type information) in a list (list display) in area 520. The "types of areas of interest that can be recognized by the area of ​​interest recognition process" can be determined according to the type and configuration of the endoscope system (for example, whether it is an ultrasound endoscope or an optical endoscope, whether it is an upper endoscope or a lower endoscope, whether the observation light is normal light or special light, etc.). The display control unit 112 also displays line segments 570 and 572 (line segments, leader lines) as location information, where one endpoint is the location of the area of ​​interest (pancreas 530, splenic vein 540) and the other endpoint is the location of the character 560 (type information). The display control unit 112 may display these line segments 570 and 572 with a different line type, as shown in Figure 9. Alternatively, the display control unit 112 may change the line color in addition to the line type (the same applies to Figures 10 and 11, which will be explained below).

[0068] With this type of display, users can easily link the ultrasound image screen with the organ name by following the line segment (type information of the type of focus information). Users can perform setting operations for list display and line segment display via a screen such as Figure 4, and the display control unit 112 can determine the display mode based on the user's operation.

[0069] Figure 10 shows yet another example of the superimposed display of location information and type information. In the example shown in Figure 10, the display control unit 112 displays the type information indicating areas of interest that are not actually recognized among the areas of interest that can be recognized in the area of ​​interest recognition process (second notification power) at a lower level than the type information indicating areas of interest that are actually recognized (first notification power). Specifically, the display control unit 112 displays "Pancreas" and "SV" corresponding to the pancreas 530 and splenic vein 540 (areas of interest that are actually recognized) as characters 562A, and displays other areas of interest (areas of interest that are not actually recognized) as characters 562B, which are smaller than characters 562A, thereby making the notification power of character 562B (second notification power) lower than the notification power of character 562A (first notification power).

[0070] Figure 11 shows yet another example of superimposed display of location and type information (another method of making the second notification power lower than the first notification power). In the example shown in Figure 11, the display control unit 112 displays "Pancreas" and "SV" corresponding to the pancreas 530 and splenic vein 540 as characters 562A, and displays other areas of interest (areas of interest that are not actually recognized) as grayed-out characters 562C, thereby making the notification power of character 562C (second notification power) lower than the notification power of character 562A (first notification power).

[0071] In addition to the examples shown in Figures 10 and 11, the display control unit 112 can use methods to reduce the notification intensity, such as making the characters thinner, increasing the transparency of the character color, or decreasing the brightness of the character color. The display control unit 112 may also use a combination of these methods. In the endoscope system 2, such notifications can improve the visibility of the area of ​​interest that has actually been recognized. The user can set how to reduce the notification intensity via a screen such as the one shown in Figure 4, and the display control unit 112 can determine the display mode based on the user's operation.

[0072] [Display changes according to the characteristics of the anatomical area] Depending on the anatomical region (area of ​​focus), it is necessary to always display text on the endoscopic ultrasound screen. For example, the pancreas is the main organ observed in endoscopic ultrasound, and it is an organ that is easily overlooked because its boundary with surrounding tissue is unclear. Therefore, it is preferable to display text on the screen for the pancreas so that the user can always easily understand its location. On the other hand, the splenic vein (SV) is a blood vessel used as a basis for determining what area is being shown in the endoscopic ultrasound image, and therefore it does not need to be displayed to the user at all times. Also, since the splenic vein may appear very small depending on the scene, it is preferable to prevent the area from being obscured by superimposed shapes.

[0073] Therefore, in the first embodiment, the screen display mode may be set according to the characteristics of the anatomical region. Specifically, the display control unit 112 may set the type of information to be displayed and whether to link the display position of the information to the change in the position of the region of interest over time, according to the type of region of interest (characteristics of the anatomical region). Figure 12 is an example of such a screen display, in which the display control unit 112 superimposes the characters 534 (position information, type information) on the pancreas 530 and the graphic 542 (position information) on the splenic vein 540. In addition, the display control unit 112 displays the legend 552 (type information) and the characters 552B ("SV"; type information) side by side in the region 520. In the example in Figure 12, the display control unit 112 links the display positions of the characters 534 and the graphic 542 to the change in the position of the region of interest (pancreas 530, splenic vein 540) over time in the ultrasound image, while the display positions of the legend 552 and the characters 552B are maintained regardless of the change in the position of the region of interest. In other words, in the first embodiment, the display control unit 112 can link the display position of not only the position information but also the type information for some areas of interest (the pancreas 530 in the example of Figure 12) to the change in the display position of the area of ​​interest over time, while maintaining the display position of the type information for other areas of interest (the splenic vein 540 in the example of Figure 12) regardless of the change in the display position of the area of ​​interest over time. This makes it possible to provide appropriate notification according to the type of area of ​​interest (characteristics of the anatomical area).

[0074] Furthermore, the user can configure the screen display mode according to the characteristics of the anatomical region via a screen like the one shown in Figure 4, and the display control unit 112 can determine the display mode based on the user's operation.

[0075] As described above, the endoscope system according to the first embodiment can appropriately notify the recognition result of the area of ​​interest.

[0076] [Application to other medical images] In the first embodiment described above, the case in which recognition is performed using an endoscopic ultrasound image, which is one form of a medical image, was explained. However, the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program according to the present invention can also be applied to cases using medical images other than endoscopic ultrasound images, such as ultrasound images acquired with an ultrasound device other than an endoscope (such as an endoscope for the body surface) or endoscopic images acquired with an optical endoscope that images a subject with white light and / or special light.

[0077] Although embodiments and other examples of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are possible. [Explanation of symbols]

[0078] 2 Endoscopic System 10. Ultrasound scope 12. Ultrasonic processor device 14 Endoscope processor device 16 Light source device 18 monitors 20 Insertion part 20a Long axis 22 Handheld control unit 24 Universal Code 26 Ultrasonic Connectors 28 Endoscope connectors 30 Light source connectors 32 tubes 34 tubes 36 Air / Water Supply Button 38 Suction button 42 Angle knob 44. Insertion port for treatment instruments 50 Tip body 52 Curved section 54 Soft part 62 Ultrasonic probe 64 Balloons 70 Water supply tank 72 Suction pump 100 Transmitter / Receiver Unit 102 Image generation unit 104 CPU 106 Area of ​​Interest Recognition Unit 108 News Power Decision Department 110 Communication Control Unit 112 Display Control Unit 118 memory 120 Records Section 223 Detectors 500 display screen 510 Image signal display area 520 areas 530 Pancreas 532 Shapes 533 Shapes 534 characters 540 Splenic vein 542 Shapes 543 Shapes 550B characters 551B characters 552B characters 553B characters 560 characters 562A character 562B characters 562C characters 570 line segments 572 line segments S100~S170 Steps in the medical image processing method

Claims

1. A medical image processing device equipped with a processor, The aforementioned processor, Image acquisition processing that sequentially acquires medical images in time series, A focus region recognition process that recognizes the location and type of the focus region from the aforementioned medical image, A display control process that superimposes position information indicating the location of the area of ​​interest and type information indicating the type of the area of ​​interest onto the medical image and displays them on a display device; Execute, The processor is a medical image processing apparatus that, in the display control processing, changes the position where the position information is displayed according to the change in the position of the area of ​​interest over time, and maintains the position where the type information is displayed regardless of the change in the position of the area of ​​interest over time.

2. The medical image processing apparatus according to claim 1, wherein the processor, in the display control processing, displays a figure or character as positional information at the position of the area of ​​interest in the medical image.

3. The medical image processing apparatus according to claim 1 or 2, wherein the processor displays characters indicating the type of the area of ​​interest as type information in the display control processing.

4. The medical image processing apparatus according to any one of claims 1 to 3, wherein the processor displays a figure or character set according to the type of area of ​​interest as position information.

5. The medical image processing apparatus according to any one of claims 1 to 4, wherein the processor displays the position information in association with the type information in the display control processing.

6. The medical image processing apparatus according to any one of claims 1 to 5, wherein the processor, in the display control processing, displays a line segment as position information, where one endpoint is at the location of the area of ​​interest and the other endpoint is at the location of the type information.

7. The medical image processing apparatus according to any one of claims 1 to 6, wherein the processor, in the display control processing, displays the type of the area of ​​interest recognized in the area of ​​interest recognition processing as type information.

8. The medical image processing apparatus according to any one of claims 1 to 6, wherein the processor, in the display control processing, displays the type of the area of ​​interest that can be recognized in the area of ​​interest recognition processing as type information.

9. The medical image processing apparatus according to claim 8, wherein the processor displays the type information corresponding to the areas of interest that are not actually recognized among the areas of interest that can be recognized in the area of ​​interest recognition processing with a second notification power lower than the first notification power for the type information indicating areas of interest that have actually been recognized in the area of ​​interest recognition processing.

10. The medical image processing apparatus according to any one of claims 1 to 9, wherein the processor, in the display control processing, displays the position information inside the image signal display area of ​​the medical image and displays the type information outside the image signal display area of ​​the medical image.

11. A medical image processing apparatus according to any one of claims 1 to 10, An endoscope scope inserted into a subject, comprising an imaging unit that sequentially captures the medical images, The display device and, An endoscopic system equipped with [the following features].

12. The endoscopic system according to claim 11, wherein the endoscope is an ultrasound endoscope that acquires an ultrasound image of the subject as the medical image.

13. A medical image processing method performed by a medical image processing device equipped with a processor, The aforementioned processor, The image acquisition process involves sequentially acquiring medical images in a time series, A focus region recognition step that recognizes the location and type of the focus region from the aforementioned medical image, A display control step that superimposes position information indicating the position of the area of ​​interest and type information indicating the type of the area of ​​interest onto the medical image and displays them on a display device; Execute, The processor, in the display control step, changes the position where the position information is displayed according to the change in the position of the area of ​​interest over time, and maintains the position where the type information is displayed regardless of the change in the position of the area of ​​interest over time.

14. A medical image processing program that causes a medical image processing device equipped with a processor to execute a medical image processing method, The aforementioned medical image processing method is The image acquisition process involves sequentially acquiring medical images in a time series, A focus region recognition step that recognizes the location and type of the focus region from the aforementioned medical image, A display control step that superimposes position information indicating the position of the area of ​​interest and type information indicating the type of the area of ​​interest onto the medical image and displays them on a display device; Includes, A medical image processing program in which, in the display control step, the position for displaying the position information is changed according to the change in the position of the area of ​​interest over time, and the position for displaying the type information is maintained regardless of the change in the position of the area of ​​interest over time.

15. A non-temporary and computer-readable recording medium on which the program described in claim 14 is recorded.

Images