Medical image processing device, endoscope system, medical image processing method, and medical image processing program

Abstract

The purpose of the present invention is to provide a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program that are capable of providing an appropriate notification in accordance with rendered information. A medical image processing device according to one embodiment of the present invention is provided with a processor, wherein the processor executes: an image acquisition process for sequentially acquiring medical images in a time series; a rendered information acquisition process for acquiring rendered information rendered in the medical image; a rendered information recognition process for recognizing the acquired rendered information; a target object recognition process for recognizing a target object from a medical image; a notification power determination process for determining the notification power of the target object by using the rendered information; and a display process for displaying the target object on a display device at the determined notification power.

Description

[Technical Field]

[0001] The present invention relates to a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program, and more particularly to a technique for notifying the recognition result of a target object. [Background technology]

[0002] It is known that a medical image processing device notifies a user of a detection result of a region of interest to assist a user such as a doctor in observing or diagnosing a medical image. For example, Patent Document 1 describes controlling the notification power according to the size of a recognized target object. Furthermore, Patent Document 2 describes drawing figures, symbols, characters, etc. in an endoscopic image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2020 / 054541 publication [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-007145 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing observations using medical equipment such as ultrasound equipment, doctors may draw shapes, letters, measurement lines, etc. on medical images to measure organ size or determine the location of a lesion (information drawn in this manner may be referred to as "drawn information" hereinafter). However, when assisting diagnosis by presenting (reporting) recognition results of information (presence / absence, location, type, etc.) about target objects such as organs or lesions using AI (Artificial Intelligence), such as a trained model constructed by machine learning, there is a possibility that the drawn letters, measurement lines, etc. may adversely affect the AI's recognition, resulting in an incorrect result being output. However, with conventional technologies such as those described in Patent Documents 1 and 2 above, it has been difficult to provide appropriate notification when drawn information has been drawn.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a medical image processing device, an endoscopic system, a medical image processing method, and a medical image processing program that can provide appropriate notifications based on drawing information. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, a medical image processing device according to a first aspect of the present invention is a medical image processing device equipped with a processor, which executes an image acquisition process for sequentially acquiring time-series medical images, a drawing information acquisition process for acquiring drawing information drawn on the medical images, a drawing information recognition process for recognizing the acquired drawing information, a target object recognition process for recognizing a target object from the medical image, a reporting power determination process for determining the reporting power of the target object using the recognized drawing information, and a display process for displaying the target object on a display device with the determined reporting power.

[0007] As described above, when drawing information is drawn on a medical image, the reliability of the recognition result of the object may be reduced. Therefore, in the medical image processing device according to the first aspect, when reporting the object recognized from the medical image, the drawing information drawn on the medical image is acquired and recognized, and the recognized drawing information is used to determine the reporting power of the object. This allows appropriate reporting to be performed according to the drawing information.

[0008] In the first aspect, the drawing information "drawn on the medical image" may mean that the drawing information is embedded in the image (a state in which it is part of the image and cannot be separated), or that the drawing information is associated with the image but is separate information and can be separated from the image (for example, a state in which it is in a separate file from the image data). Furthermore, "recognizing the drawing information" may mean, for example, grasping the presence or absence of drawing information, its quantity, position, size, shape, etc. in the medical image. Furthermore, the higher the "informing power" of a target object, the easier it is for the user to recognize the target object. When multiple target objects are recognized in a medical image, the processor may determine the informing power for each target object.

[0009] In the first aspect and each of the following aspects, "acquiring time-series medical images" includes sequentially acquiring a plurality of medical images captured at a predetermined frame rate. The acquisition may be in real time or not. For example, medical images that have been captured and recorded in advance may be acquired.

[0010] The medical image processing device according to the first aspect can be realized, for example, as a processor part of a medical image processing system, but is not limited to this aspect. Note that a "medical image" refers to an image obtained as a result of photographing or measuring a living body such as a human body for the purpose of diagnosis, treatment, measurement, etc., and examples thereof include endoscopic images, ultrasound images, CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Medical images are also called medical images. Furthermore, in the first aspect and each of the following aspects, the "target object" may be a "region of interest (ROI)" in a medical image, such as a lesion area or a suspected lesion area, an organ or blood vessel, a post-treatment area, or a treatment tool. A "region of interest" may also be called a "region of interest."

[0011] In the medical image processing device according to the second aspect, in the first aspect, when drawing information is recognized in the drawing information recognition process, the processor lowers the informing power compared to when no drawing information is recognized in the informing power determination process, because when drawing information is recognized, the reliability of the recognition result may be impaired.

[0012] A medical image processing device according to a third aspect is the first or second aspect, in which a processor acquires drawing information from a medical imaging device connected to the medical image processing device in a drawing information acquisition process. The third aspect defines one aspect of a drawing information acquisition method. The medical imaging device is, for example, an endoscope device that captures endoscopic images using optics or ultrasound, or an ultrasound device that acquires ultrasound images by contacting an ultrasound probe with the body surface of a subject.

[0013] A medical image processing device according to a fourth aspect is any one of the first to third aspects, wherein the processor performs image processing on the medical image in the drawing information acquisition process to acquire the drawing information. 4 The above aspect defines another aspect of the drawing information acquisition method.

[0014] A medical image processing device according to a fifth aspect is the fourth aspect, in which the processor acquires the drawing information using a trained model configured by machine learning in the drawing information acquisition process. The fifth aspect specifically defines the configuration for acquiring the drawing information, and the trained model can be, for example, a convolutional neural network (CNN) or a support vector machine (SVM).

[0015] A medical image processing device according to a sixth aspect is any one of the first to fifth aspects, in which the processor acquires drawing information by comparing a medical image with drawable information that can be drawn on the medical image. Not just any information can be drawn on a medical image; the information that can be drawn is limited to a certain extent depending on the type of medical image, the part of the subject, etc. Therefore, by comparing a medical image with drawable information (information that can be drawn on a medical image) as in the sixth aspect, accurate drawing information can be acquired, thereby making it possible to notify the target object with appropriate notification power.

[0016] A medical image processing device according to a seventh aspect is any one of the first to sixth aspects, in which the processor acquires, as drawing information, information about figures, characters, or symbols drawn on the medical image in the drawing information acquisition process. The seventh aspect defines a specific aspect of the drawing information.

[0017] In a medical image processing device according to an eighth aspect, in the seventh aspect, the processor acquires, as drawing information, at least one of position, size, shape, line type, color, and type relating to at least one of figures, characters, and symbols in a drawing information acquisition process. The eighth aspect further specifies aspects of the drawing information.

[0018] A medical image processing device according to a ninth aspect is any one of the first to eighth aspects, wherein the processor reduces the notification power of the target object in accordance with the level of discrimination of the drawing information compared to when the medical image does not contain drawing information. The "discrimination power" of the drawing information refers to the degree of influence on the recognition result of the target object, and may depend on the type, number, amount, position, size, shape, etc. of the drawing information. If this discrimination power is high, the reliability of the recognition result may be low. Therefore, by reducing the notification power of the target object in accordance with the level of discrimination of the drawing information compared to when the medical image does not contain drawing information, as in the ninth aspect, it is possible to notify the target object with appropriate notification power.

[0019] A medical image processing device according to a tenth aspect is the ninth aspect, wherein the processor reduces the notification power of an object recognized as having a discrimination power of drawing information equal to or greater than a threshold value, compared to a case where the medical image does not have drawing information. 10 The second aspect defines a criterion for lowering the notification power of the target object.

[0020] In the medical image processing device of the 11th aspect, in any one of the first to eighth aspects, when drawing information is acquired in the drawing information acquisition process, the processor lowers the reporting power of the target object compared to when the medical image does not have drawing information, regardless of the level of discrimination power of the drawing information.

[0021] A medical image processing device according to a twelfth aspect is any one of the ninth to eleventh aspects, wherein the processor determines the discrimination ability according to at least one of the position, size, shape, line type, color, and type of figures, characters, or symbols in the medical image. In the twelfth aspect, the processor may determine the discrimination ability taking into account the number of figures, etc.

[0022] A medical image processing device according to a thirteenth aspect is any one of the first to twelfth aspects, wherein the processor determines the informing power of the target object based on the positional relationship between the target object in the medical image and the drawing information in the informing power determination process. For example, if the target object is far from the drawing information, the processor can increase the informing power of the target object, and if the target object is close to the drawing information, the processor can decrease the informing power of the target object.

[0023] In order to achieve the above object, the present invention 14 The endoscope system according to the present invention includes first to second 13 The medical image processing device according to any one of the above aspects, an endoscope to be inserted into a subject, the endoscope having an imaging unit that sequentially captures medical images, and a display device.

[0024] No. 15 The endoscope system according to the present embodiment is 14 In this aspect, the endoscope is an ultrasonic endoscope that acquires ultrasonic images of a subject as medical images.

[0025] In order to achieve the above object, the present invention 16 The medical image processing method according to the aspect is a medical image processing method executed by a medical image processing device having a processor. ,time The medical image processing method according to the fifteenth aspect includes an image acquisition step of sequentially acquiring a series of medical images, a drawing information acquisition step of acquiring drawing information drawn on the medical images, a drawing information recognition step of recognizing the acquired drawing information, a target object recognition step of recognizing a target object from the medical images, a notification power determination step of determining a notification power of the target object using the recognized drawing information, and a display step of displaying the target object on a display device with the determined notification power. According to the fifteenth aspect, as in the first aspect, appropriate notification can be performed according to the drawing information. The medical image processing method according to the fifteenth aspect may further perform processing similar to that of the second to thirteenth aspects.

[0026] In order to achieve the above object, the present invention 17The medical image processing program according to this aspect is a medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method, and the medical image processing method includes an image acquisition step of sequentially acquiring time-series medical images, a drawing information acquisition step of acquiring drawing information drawn on the medical images, a drawing information recognition step of recognizing the acquired drawing information, a target object recognition step of recognizing a target object from the medical image, an informing power determination step of determining the informing power of the target object using the recognized drawing information, and a display step of displaying the target object on a display device with the determined informing power. 17 According to this aspect, it is possible to give an appropriate notification in accordance with the drawing information, similar to the first aspect. 17 The medical image processing program according to this aspect may be a program that further executes the same processes as those of the second to thirteenth aspects. Note that a non-transitory recording medium on which computer-readable code of the program according to these aspects is recorded can also be cited as an aspect of the present invention. [Effects of the Invention]

[0027] As described above, the medical image processing device, endoscope system, medical image processing method, and medical image processing program according to the present invention can provide an appropriate notification in accordance with the drawing information. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is an external view of an endoscope system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the main part of the ultrasonic processor device. [Figure 3] FIG. 3 is a flowchart showing an outline of the medical image processing method according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the drawing information. [Figure 5] FIG. 5 is a flowchart showing the details of the informing power determination process. [Figure 6] FIG. 6 is a diagram showing the relationship between the discriminability of drawing information and the informing power of a target object. [Figure 7] FIG. 7 is a diagram showing how a target object is notified when there is no drawing information. [Figure 8] FIG. 8 is a diagram showing how a target object is notified when drawing information is available. [Figure 9] FIG. 9 is a diagram showing a display example in which notification is performed with a notification power according to the positional relationship between the drawing information and the target object. DETAILED DESCRIPTION OF THE INVENTION

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, with reference to the accompanying drawings, embodiments of a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program according to the present invention will be described in detail.

[0030] [Overall configuration of an endoscope system including a medical image processing device] Fig. 1 is an external view of an endoscope system. As shown in Fig. 1, an endoscope system 2 (endoscopic system, medical image capturing device) includes an ultrasound scope 10 (endoscopic scope, ultrasound endoscope scope), an ultrasound processor device 12 (medical image processing device) that generates ultrasound images, an endoscope processor device 14 (medical image processing device) that generates endoscopic images, a light source device 16 that supplies illumination light (observation light) to the ultrasound scope 10 to illuminate the inside of a body cavity, and a monitor 18 (display device) that displays ultrasound images and endoscopic images.

[0031] The ultrasound scope 10 comprises an insertion section 20 that is inserted into a body cavity of a subject, a handheld operation section 22 that is connected to the proximal end of the insertion section 20 and that is operated by the surgeon, and a universal cord 24 having one end connected to the handheld operation section 22. The other end of the universal cord 24 is provided with an ultrasound connector 26 that is connected to the ultrasound processor device 12, an endoscope connector 28 that is connected to the endoscope processor device 14, and a light source connector 30 that is connected to the light source device 16.

[0032] The ultrasonic scope 10 is detachably connected to the ultrasonic processor 12, the endoscope processor 14, and the light source device 16 via these connectors. In addition, a tube 32 for supplying air and water and a tube 34 for suction are connected to the light source connector 30.

[0033] The light source device 16 is composed of illumination light sources (for example, a red light source, a green light source, a blue light source, and a purple light source that emit narrowband light of red, green, blue, and purple, respectively), an aperture, a condenser lens, a light source control unit, etc., and these light sources can produce normal light (white light), special light (narrowband light, etc.), and combinations thereof as observation light.

[0034] The monitor 18 receives the video signals generated by the ultrasonic processor 12 and the endoscope processor 14 and displays the ultrasonic image and the endoscopic image. The monitor 18 can display only one of the ultrasonic image and the endoscopic image by switching as needed, or can display both images simultaneously.

[0035] The handheld operation section 22 is provided with an air / water supply button 36 and a suction button 38, a pair of angle knobs 42, and a treatment tool insertion port 44.

[0036] The insertion section 20 has a distal end, a proximal end, and a longitudinal axis 20a, and is composed of, in order from the distal end, a distal end main body 50 made of a hard member, a bending section 52 connected to the proximal end of the distal end main body 50, and a thin, long, flexible soft section 54 connecting the proximal end of the bending section 52 to the distal end of the handheld operation section 22. That is, the distal end main body 50 is provided on the distal end side of the longitudinal axis 20a of the insertion section 20. The bending section 52 is remotely bent by rotating a pair of angle knobs 42 provided on the handheld operation section 22. This allows the user to point the distal end main body 50 in a desired direction.

[0037] The tip body 50 is fitted with an ultrasonic probe 62 (imaging unit) and a bag-shaped balloon 64 that encases the ultrasonic probe 62. The balloon 64 can be inflated or deflated by supplying water from a water tank 70 or by suctioning water from the balloon 64 with a suction pump 72. The balloon 64 is inflated until it abuts against the inner wall of the body cavity to prevent attenuation of the ultrasonic waves and ultrasonic echoes (echo signals) during ultrasonic observation.

[0038] The distal end body 50 is provided with an objective lens, an imaging element, etc. Ta An endoscopic observation unit (not shown) having an observation unit and an illumination unit is attached to the ultrasound probe 62. The endoscopic observation unit is provided behind the ultrasound probe 62 (on the handheld operation unit 22 side).

[0039] The endoscope system 2 can sequentially acquire (capture) endoscopic images (optical images) and ultrasound images with the above-described configuration. Note that the endoscope system 2 may acquire the endoscopic images and ultrasound images from the recording unit 120 or a server or database (not shown).

[0040] [Medical image processing device] FIG. 2 is a block diagram showing the configuration of the main part of an ultrasound processor device that functions as a medical image processing device according to the present invention.

[0041] 2 recognizes a region of interest (object) in a medical image based on sequentially acquired time-series medical images, and displays the object on a display device with a notification power corresponding to the drawing information. The ultrasound processor device 12 (medical image processing device, processor) shown in FIG. 2 is a device that recognizes a region of interest (object) in a medical image based on sequentially acquired time-series medical images, and displays the object on a display device with a notification power corresponding to the drawing information. The device is composed of a transceiver unit 100 (processor, image acquisition unit), an image generation unit 102 (processor, image acquisition unit), a CPU 104 (processor, CPU: Central Processing Unit), an object recognition unit 106 (processor, object recognition unit), a notification power determination unit 108 (processor, notification power determination unit), a communication control unit 110 (processor), a display control unit 112 (processor, display unit), a drawing information acquisition unit 114 (processor, drawing information acquisition unit), a drawing information recognition unit 116 (processor, drawing information recognition unit), a memory 118, and a recording unit 120 (recording device). The processing of each of these units is realized by one or more processors, as described below.

[0042] The CPU 104 operates based on various programs including the medical image processing program according to the present invention stored in the memory 118, and controls the target object recognition unit 106, the informing power determination unit 108, the display control unit 112, the drawing information acquisition unit 114, and the drawing information recognition unit 116, and also functions as a part of each of these units. The memory 118 includes a non-temporary recording medium such as a ROM (Read Only Memory) in which the medical image processing program and the like are recorded, and a temporary recording medium such as a RAM (Random Access Memory) used as a temporary storage area.

[0043] The transmitting / receiving unit 100 and the image generating unit 102, which function as an image acquiring unit, sequentially acquire time-series medical images (image acquisition processing, image acquisition step).

[0044] The transmitting section of the transmitting / receiving unit 100 generates a plurality of drive signals to be applied to a plurality of ultrasonic transducers of the ultrasonic probe 62 of the ultrasonic scope 10, and applies the plurality of drive signals to the plurality of ultrasonic transducers by giving each of the drive signals a delay time based on a transmission delay pattern selected by a scanning control unit (not shown).

[0045] The receiving section of the transmitting / receiving unit 100 amplifies the multiple detection signals output from the multiple ultrasonic transducers of the ultrasonic probe 62, and converts the analog detection signals into digital detection signals (also called RF (Radio Frequency) data). This RF data is input to the image generating unit 102.

[0046] The image generation unit 102 performs reception focusing processing by adding together the detection signals represented by the RF data and applying delay times to the detection signals based on the reception delay pattern selected by the scan control unit. This reception focusing processing forms sound ray data in which the focus of the ultrasonic echo is narrowed.

[0047] The image generation unit 102 corrects the sound ray data for attenuation due to distance according to the depth of the ultrasonic wave reflection position using STC (Sensitivity Time Control), then generates envelope data by performing envelope detection processing using a low-pass filter or the like, and stores the envelope data for one frame, or more preferably for multiple frames, in a cine memory (not shown).The image generation unit 102 performs pre-processing such as log (logarithmic) compression and gain adjustment on the envelope data stored in the cine memory to generate a B-mode image.

[0048] In this way, the transmitting and receiving unit 100 and the image generating unit 102 sequentially acquire time-series B-mode images (hereinafter referred to as "medical images").

[0049] The target object recognition unit 106 performs a process (detection process, target object recognition process, target object recognition step) of recognizing information about the position of a target object (region of interest) in a medical image based on the medical image, and a process (classification process, classification step) of classifying the target object into one of multiple classes based on the medical image, and can be configured using a trained model (a model trained using an image set consisting of images of a living body) configured by machine learning, such as a CNN (Convolutional Neural Network) or an SVM (Support Vector Machine).The target object in this embodiment is, for example, a region of interest such as an organ or blood vessel in a medical image (a tomographic image of a B-mode image), such as the pancreas, main pancreatic duct, spleen, splenic vein, splenic artery, gallbladder, etc.

[0050] An example of the layer configuration when the target object recognition unit 106 is configured using CNN will be described below. CNN includes an input layer, an intermediate layer, and an output layer. The input layer inputs the medical image generated by the image generation unit 102 and outputs features. The intermediate layer includes a convolutional layer and a pooling layer, and inputs the features output by the input layer to calculate other features. These layers have a structure in which multiple "nodes" are connected by "edges," and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The output layer recognizes the target object (region of interest) in the input medical image based on the features output from the intermediate layer, and outputs the results.

[0051] In this example, when the target object recognition unit 106 sequentially inputs time-series medical images, it recognizes (detects) the position of the area of ​​interest for each input medical image and outputs information related to that position, and also recognizes (classifies) which of multiple classes the area of ​​interest belongs to and outputs information indicating the recognized class (class information).

[0052] The target object recognition unit 106 can determine, for example, the center position of a rectangle surrounding the region of interest as the position of the region of interest. In addition, in this embodiment, information indicating the type of organ or blood vessel can be determined as "class information."

[0053] The drawing information acquisition unit 114 (processor, drawing information acquisition unit) acquires drawing information drawn on the medical image. The drawing information acquisition unit 114 may acquire the drawing information by image processing, may acquire it from another medical imaging device connected to the endoscope system 2, or may acquire the drawing information by matching the medical image with drawable information. The drawing information may be acquired by a combination of these means. When acquiring the drawing information by image processing, the drawing information acquisition unit 114 may use a trained model (CNN, SVM, etc.) configured by machine learning. The drawing information recognition unit 116 recognizes the drawing information, and the informing power determination unit 108 determines the informing power of the target object using the recognized drawing information.

[0054] The display control unit 112 causes the monitor 18 (display device) to display time-series medical images (endoscopic images, ultrasound images) sequentially acquired by the transmitting / receiving unit 100 and the image generating unit 102. In this example, a moving image showing an ultrasound tomographic image is displayed on the monitor 18. The display control unit 112 also causes the monitor 18 to display the target object at the alert power determined by the alert power determining unit 108.

[0055] Medical image processing using the above-mentioned functions will be described in detail later.

[0056] <Functional implementation using various processors> The functions of the ultrasound processor device 12 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. When processing images as in the present invention, a configuration using a GPU is effective. Furthermore, dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which are processors with a circuit configuration specifically designed to execute specific processing, are also included in the "various processors" described above.

[0057] The functions of each unit may be realized by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by a single processor. Examples of multiple functions configured by a single processor include: a first configuration, as typified by a computer, in which a single processor is configured by combining one or more CPUs and software, and this processor realizes multiple functions; a second configuration, as typified by a system-on-chip (SoC), in which a processor is used to realize the functions of the entire system on a single IC (Integrated Circuit) chip; and various functions are thus configured as hardware structures using one or more of the various processors described above. Furthermore, the hardware structures of these various processors are, more specifically, electrical circuits combining circuit elements such as semiconductor devices. These electrical circuits may be electrical circuits that realize the above-mentioned functions using logical operations such as logical sum, logical product, logical negation, exclusive OR, and combinations of these.

[0058] When the above-mentioned processor or electric circuit executes the software (program), the code readable by the computer (for example, various processors and electric circuits constituting the ultrasonic processor device 12, and / or a combination thereof) of the software to be executed is stored in a non-transitory recording medium such as a ROM (Read Only Memory), and the computer refers to the software. The software stored in the non-transitory recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used to set the display mode and notification mode, Target object recognition unit 106The code may be recorded in a non-transitory recording medium such as various magneto-optical recording devices or semiconductor memory, instead of a ROM. When processing using software, for example, a RAM (Random Access Memory) is used as a temporary storage area, and for example, an EEPROM (not shown) is used as a temporary storage area. Electrically It is also possible to refer to data stored in an erasable and programmable read-only memory. The memory 118 or the recording unit 120 may be used as the "non-transitory recording medium."

[0059] The recording unit 120 records ultrasound images and endoscopic images (medical images), drawing information, detection results of target objects, processing conditions (conditions for detection and notification), etc. Other information may also be recorded. The communication control unit 110 controls the acquisition of medical images, etc. from other medical imaging devices connected to the endoscope system 2, external servers, or databases.

[0060] [Overview of medical image processing] The medical image processing (execution of the medical image processing method and medical image processing program according to the present invention) in the endoscope system 2 configured as described above will now be described. Fig. 3 is a flowchart showing an outline of the medical image processing method according to the first embodiment. Note that the order of the steps described below may be changed as necessary.

[0061] The transmitting / receiving unit 100 and the image generating unit 102 sequentially acquire ultrasound images (medical images) in time series (step S110: image acquisition processing, image acquisition step), and the drawing information acquiring unit 114 acquires drawing information drawn on the ultrasound images (step S120: drawing information acquisition processing, drawing information acquiring step). The drawing information acquiring unit 114 acquires information about figures, characters, or symbols drawn on the ultrasound images (for example, at least one of the position, size, shape, line type, color, or type related to at least one of the figures, characters, or symbols) as drawing information. The drawing information acquiring unit 114 may acquire the drawing information from a medical imaging device connected to the endoscope system 2, may acquire the drawing information using a trained model configured by machine learning, or may acquire the drawing information by comparing the ultrasound images with drawable information (information that can be drawn on the ultrasound images, which are medical images). The drawable information is, for example, character strings that can be drawn (such as names of organs or blood vessels as "Pancreas" or "SV" and their abbreviations, or comments by the radiologist), types of symbols (such as arrows, circles, ellipses, and irregular curves), or characteristics of measurement lines (such as dotted or solid lines). The drawing information acquisition unit 114 may acquire drawing information using a plurality of these means.

[0062] FIG. 4 is a diagram showing an example of the drawing information, in which an ultrasound image 602 is displayed on a screen 600 of the monitor 18 (in FIG. 4, the target object is not shown in order to clearly show the drawing information). In the example shown in part (a) of FIG. 4, an arrow 604 (drawing information), which is one form of symbol, and character A character string 606 (drawing information) in one form is drawn on the ultrasound image 602. Note that "Panc" is an abbreviation for "pancreas" and means the pancreas, and "SV" is an abbreviation for " splenic On the other hand, in the example shown in part (b) of FIG. 4, a measurement line 608 and a measurement figure 610, which are one form of figure, are drawn on the ultrasound image 602.

[0063] The target object recognition unit 106 recognizes the target object (step S130: target object recognition processing, target object recognition step), and the informing power determination unit 108 determines the informing power of the target object using the recognized drawing information (step S140: informing power determination processing, informing power determination step). Details of the processing in step S140 will be described later. The display control unit 112 displays the target object on the monitor 18 (display device) at the determined informing power (step S150: display processing, display step).

[0064] [Details of the notification power determination process / notification power determination step] 5 is a flowchart showing details of the informing power determination process / informing power determination step in step S140 of FIG. 3. The drawing information recognition unit 116 (processor) determines whether drawing information has been recognized in the drawing information (acquired in step S120) (step S200: drawing information recognition process / drawing information recognition step). "Recognizing drawing information" means, for example, grasping the presence or absence of drawing information and its contents, and the "contents of drawing information" includes at least one of the position, size, shape, line type, color, or type of at least one of figures, characters, or symbols such as the above-mentioned arrow 604 or character string 606. It is preferable that the drawing information recognition unit 116 further grasps the quantity of drawing information and the positional relationship between the target object and the drawing information.

[0065] If the drawing information recognition unit 116 does not recognize the drawing information (if the drawing information does not exist), the judgment in step S200 is denied, and the alarm power determination unit 108 (processor) sets the first alarm power as the alarm power of the target object (step S220: alarm power determination process, alarm power determination step).

[0066] If the drawing information recognition unit 116 recognizes some drawing information (if drawing information exists), the determination in step S200 is affirmative, and the informing power determination unit 108 reduces the informing power of the target object below the first informing power (when the medical image does not have drawing information) regardless of the level of discrimination power of the drawing information. Specifically, the informing power determination unit 108 determines the discrimination power of the drawing information (step S210: informing power determination process, informing power determination step), and reduces the informing power of the target object below the first informing power (informing power when the medical image does not have drawing information) depending on the level of discrimination power. More specifically, the alert power determination unit 108 determines whether the discrimination power of the drawing information is greater than or equal to a threshold (step S230: alert power determination process, alert power determination step), and if the judgment is positive, sets a third alert power as the alert power of the target object (step S240: alert power determination process, alert power determination step), and if the judgment is negative, sets a second alert power as the alert power of the target object (step S250: alert power determination process, alert power determination step).

[0067] Figure 6 is a diagram showing the relationship between the discriminability of drawing information and the informing power of a target object. The left side of Figure 6 shows the level of discriminability of drawing information, with the bottom being zero (when no drawing information is present) and the discriminability increasing toward the top of the figure. On the other hand, the right side of Figure 6 shows the level of informing power of a target object, with the informing power increasing toward the bottom of the figure. As shown in Figure 6, the informing power when the discriminability of drawing information is zero (no drawing information) is "first informing power," the informing power when the discriminability is below a threshold is "second informing power," and the informing power when the discriminability is equal to or greater than the threshold is "third informing power."

[0068] The "discriminability" of the drawing information can be considered to be the degree of influence on the recognition result of the target object, and can depend on the type, number, amount, position, size, shape of the drawing information, or the distance to the target object, etc. If this discriminability is high, there is a possibility that the reliability of the recognition result will be low, so by determining the notification power of the target object according to the discriminability as described above, it is possible to provide an appropriate notification according to the drawing information.

[0069] [Example of displaying a target object with the determined notification power] An example of displaying a target object with the informing power determined by the above-described process will be described below. Fig. 7 is a diagram showing an example of display when there is no drawing information (the discriminability of the drawing information is zero; an example of display with the first informing power), in which an area of ​​interest 502 (target object; in this example, the pancreas) is present in an ultrasound image 500. The display control unit 112 then uses a bounding box 504 and a character string 506 to indicate the position and type of the area of ​​interest 502.

[0070] On the other hand, Fig. 8 is a diagram showing an example of a display when drawing information is present, and is an example in which the informing power is lower than that of the example in Fig. 7 (an example of a display with a second informing power or a third informing power). In the example shown in part (a) of Fig. 8, the display control unit 112 displays the lines of the bounding box 504A as dotted lines thinner than those of the bounding box 504 (by changing the line thickness and line type). In the example shown in part (b) of Fig. 8, only the corner portions of the bounding box 504B are displayed. In the example shown in part (c) of Fig. 8, the size of the character string 506A is reduced (the character string may be an initial or an abbreviation). The aspects shown in Fig. 8 may be combined. In addition, the color or brightness of the bounding box or the character string may be changed.

[0071] FIG. 9 is a diagram showing a display example in which notification is performed with an informing power according to the positional relationship (distance) between the target object and the drawing information. Regions of interest 502 and 503 are present in an ultrasound image 500. Multiple measurement lines 508 (drawing information) and a figure 510 (drawing information) are drawn near the region of interest 502, while the region of interest 503 is far from these pieces of drawing information. The discriminative power of these pieces of drawing information may affect the recognition result of the target object, and it can be considered that the influence on the region of interest 502, which is closer to the drawing information, is greater than the influence on the region of interest 503. Therefore, in the example of FIG. 9 , the lines of the bounding box 505 for the region of interest 503 are made thicker to increase the relative informing power, while the lines of the bounding box 504C for the region of interest 502 are made thinner, and only the corners are displayed to decrease the relative informing power. By determining the informing power based on the positional relationship (distance) between the target object and the drawing information in this way, an appropriate notification can be performed according to the discriminative power of the drawing information.

[0072] [Application to other medical images] In the first embodiment described above, recognition is performed using an ultrasonic endoscopic image, which is one type of medical image (medical image). However, the medical image processing device, endoscopic system, medical image processing method, and medical image processing program according to the present invention can also be applied to cases where medical images other than ultrasonic endoscopic images are used, such as ultrasonic images acquired by ultrasonic devices other than endoscopes (such as superficial endoscopic devices) and endoscopic images acquired by optical endoscopic devices that image a subject using white light and / or special light.

[0073] Although the embodiment and other examples of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible. [Explanation of symbols]

[0074] 2 Endoscopy System 10 Ultrasound scope 12 Ultrasonic processor device 14. Endoscope processor device 16 Light source device 18 monitors 20 Insertion section 20a Longitudinal axis 22 Handheld operation unit 24 Universal Code 26 Ultrasonic connector 28 Endoscope connector 30 Light source connector 32 tubes 34 tubes 36 Air and water supply button 38 Suction button 42 Angle knob 44 Treatment tool insertion port 50 Tip body 52 Curved section 54 Soft part 62 Ultrasonic probe 64 Balloon 70 Water Tank 72 Suction pump 100 Transmitter / Receiver 102 Image generation unit 104 CPU 106 Target object recognition unit 108 Information Power Determination Department 110 Communication control unit 112 Display control unit 114 Drawing information acquisition unit 116 Drawing information recognition unit 118 memory 120 records Department 5 00 Ultrasound image 502 Areas of Interest 503 Areas of Interest 504 Bounding Box 504A Bounding Box 504B Bounding Box 504C Bounding Box 505 Bounding Box 506 String 506A string 508 Measurement Line 510 Shapes 600 screens 602 Ultrasound Images 604 Arrow 606 String 608 Measurement Line 610 Measurement figures S110~S250 Steps of medical image processing method

Claims

1. 1. A medical imaging device comprising a processor, The processor: an image acquisition process for sequentially acquiring time-series medical images; a drawing information acquisition process for acquiring drawing information drawn on the medical image; a drawing information recognition process for recognizing the acquired drawing information; a target object recognition process for recognizing a target object from the medical image; a notification power determination process for determining notification power of the target object using the recognized drawing information; a display process of displaying the target object on a display device with the determined informing power; A medical imaging device that performs

2. The medical image processing apparatus according to claim 1 , wherein the processor, when recognizing the drawing information in the drawing information recognition process, lowers the informing power compared to when not recognizing the drawing information in the informing power determination process.

3. The medical image processing apparatus according to claim 1 or 2, wherein the processor, in the drawing information acquisition process, acquires the drawing information from a medical imaging device connected to the medical image processing apparatus.

4. The medical image processing apparatus according to claim 1 , wherein the processor, in the drawing information acquisition process, performs image processing on the medical image to acquire the drawing information.

5. The medical image processing device according to claim 4 , wherein the processor acquires the drawing information using a trained model configured by machine learning in the drawing information acquisition process.

6. The medical image processing apparatus according to claim 1 , wherein the processor acquires the drawing information by comparing the medical image with drawing information that can be drawn on the medical image.

7. The medical image processing apparatus according to claim 1 , wherein the processor acquires, as the drawing information, information relating to a figure, a character, or a symbol drawn on the medical image in the drawing information acquisition process.

8. The medical image processing device according to claim 7 , wherein the processor acquires, in the drawing information acquisition process, at least one of the position, size, shape, line type, color, or type of at least one of the figure, the character, or the symbol as the drawing information.

9. The medical image processing device according to claim 1 , wherein the processor reduces the informing power of the target object in accordance with the level of discriminability of the drawing information compared to when the medical image does not have the drawing information.

10. The medical image processing device according to claim 9 , wherein the processor lowers the notification power for a target object recognized as having the discrimination power of the drawing information equal to or greater than a threshold value, compared to when the medical image does not have the drawing information.

11. A medical image processing device as described in any one of claims 1 to 8, wherein when the processor acquires the drawing information in the drawing information acquisition process, the processor lowers the alert power of the target object compared to when the medical image does not have the drawing information, regardless of the high level of discriminability of the drawing information.

12. The medical image processing device according to any one of claims 9 to 11, wherein the processor determines the discrimination ability according to at least one of the position, size, shape, line type, color, and type of figure, character, or symbol in the medical image.

13. The medical image processing device according to claim 1 , wherein the processor, in the notification power determination process, determines the notification power of the target object based on a positional relationship between the target object in the medical image and the drawing information.

14. A medical image processing device according to any one of claims 1 to 13; an endoscope to be inserted into a subject, the endoscope including an imaging unit that sequentially captures the medical images; the display device; An endoscope system comprising:

15. The endoscope system according to claim 14, wherein the endoscope is an ultrasonic endoscope that acquires ultrasonic images of the subject as the medical images.

16. 1. A medical image processing method performed by a medical image processing device having a processor, comprising: an image acquisition step of sequentially acquiring time-series medical images; a drawing information acquisition step of acquiring drawing information drawn on the medical image; a drawing information recognition step of recognizing the acquired drawing information; a target object recognition step of recognizing a target object from the medical image; a notification power determination step of determining notification power of the target object using the recognized drawing information; a display step of displaying the target object on a display device with the determined informing power; A medical image processing method comprising:

17. A medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method, The medical image processing method includes: an image acquisition step of sequentially acquiring time-series medical images; a drawing information acquisition step of acquiring drawing information drawn on the medical image; a drawing information recognition step of recognizing the acquired drawing information; a target object recognition step of recognizing a target object from the medical image; a notification power determination step of determining notification power of the target object using the recognized drawing information; a display step of displaying the target object on a display device with the determined informing power; Medical imaging programs, including:

18. A non-transitory computer-readable recording medium having the program according to claim 17 recorded thereon.

Images