Medical image processing apparatus, endoscope system, medical image processing method, and recording medium

Abstract

An object of the present application is to provide a medical image processing apparatus, an endoscope system, a medical image processing method, and a recording medium, which can appropriately display a recognition result of a region of interest. A medical image processing apparatus according to an aspect of the present application includes a processor configured to execute: an image acquisition process configured to sequentially acquire medical images in time series; a region of interest recognition process configured to recognize a position and a type of a region of interest from the medical images; and a display control process configured to cause position information indicating the position of the region of interest and type information indicating the type of the region of interest to be displayed on a display device in superposition with the medical images. In the display control process, the processor is configured to cause a position at which the position information is displayed to change in accordance with a change in the position of the region of interest due to the passage of time, and to maintain a position at which the type information is displayed regardless of the change in the position of the region of interest due to the passage of time.

Description

Technical Field

[0001] This invention relates to medical image processing apparatus, endoscope system, medical image processing method and medical image processing program, and in particular to technology for displaying the recognition results of regions of interest. Background Technology

[0002] As an aid for doctors and other users in observing or diagnosing medical images, there are known technologies for notifying the identification results of regions of interest based on medical image processing devices. For example, Patent Document 1 describes a technology that displays a bounding box at the location of the identified region of interest. Additionally, Patent Document 1 also describes a technology that displays characters indicating the type of region of interest outside the image area.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-069300 Summary of the Invention

[0006] The technical problem to be solved by the invention

[0007] When using medical devices such as endoscopes and ultrasound machines for observation, doctors operate the devices, identifying organs or blood vessels (areas of interest, anatomical regions) on the screen while simultaneously projecting the desired area onto the image. This is a very difficult operation. Therefore, to simplify device operation, image recognition technologies such as AI (Artificial Intelligence) can be used to detect organs or blood vessels and provide prompts to the user (e.g., displaying the recognition results). Ideally, such a display should be freely open / closed based on the user's judgment. However, depending on the type of medical device, users sometimes operate with both hands, so the display cannot always be opened / closed by the user's input. Therefore, it is preferable that the display of the recognition results assists the user's observation without causing obstruction. However, in prior art such as Patent Document 1 mentioned above, it is difficult to appropriately display the recognition results of the area of ​​interest.

[0008] The present invention was made in view of the following circumstances, and its object is to provide a medical image processing apparatus, endoscope system, medical image processing method and medical image processing program capable of appropriately displaying the recognition results of the region of interest.

[0009] means for solving technical problems

[0010] To achieve the above objectives, the medical image processing apparatus according to the first aspect of the present invention includes a processor, wherein the processor performs: image acquisition processing to sequentially acquire time-series medical images; region of interest identification processing to identify the location and type of the region of interest from the medical images; and display control processing to display location information indicating the location of the region of interest and type information indicating the type of the region of interest overlapping with the medical images on a display device. In the display control processing, the processor causes the position of the display location information to change according to the change in the location of the region of interest caused by the passage of time, and maintains the position of the display type information regardless of the change in the location of the region of interest caused by the passage of time.

[0011] As a method for displaying the identification results of regions of interest in medical images, one consideration is to display location information indicating the location of the region of interest, as well as type information such as the name of the region of interest. However, since the location of the region of interest in a medical image changes as observation progresses (over time), if the display position of the type information changes according to the change in the location of the region of interest, then when the location of the region of interest changes significantly, the display position of the type information will also change drastically, reducing visual recognition. On the other hand, if the display position of the location information is maintained regardless of the change in the location of the region of interest, the location of the region of interest cannot be displayed correctly.

[0012] From this perspective, the medical image processing device involved in the first approach can assist the user's observation by displaying location information and type information, while reducing the possibility that the display of location information and type information may hinder the user's observation.

[0013] Thus, according to the first method, the identification results of the region of interest can be appropriately displayed. Furthermore, in the first method, for a portion of the region of interest, not only the location information but also the type information can be displayed in conjunction with changes in the location of the region of interest; for other regions of interest, the display position of the type information remains unchanged regardless of changes in the location of the region of interest. Alternatively, the display position of the type information can be set to be linked to changes in the location of the region of interest based on the type of the region of interest (the characteristics of the anatomical region).

[0014] In the first approach, the type of region of interest is, for example, the name of an organ or blood vessel, or the classification result of a lesion, but is not limited to these examples. Alternatively, in the first approach, a detector constructed using machine learning can also be used to identify the region of interest.

[0015] In the first method and the following methods, "acquisition of time-series medical images" includes sequentially acquiring multiple medical images captured at a defined frame rate. Acquisition may be real-time or not. For example, pre-captured and recorded medical images may also be acquired.

[0016] The medical image processing device involved in the first approach can be implemented as a processor part of a medical image processing system, but is not limited to this approach. Furthermore, "medical image" refers to an image obtained as a result of photographing or measuring a living organism such as a human body for purposes such as diagnosis, treatment, or measurement; examples include endoscopic images, ultrasound images, CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Medical images are also called medical images. Additionally, in the first approach and the following approaches, the "Region of Interest (ROI)" can be a lesion area or potential lesion area, an organ or blood vessel, a treated area, or a treatment device in the medical image. The "Region of Interest" is sometimes also referred to as the "Region of Concern."

[0017] The second approach involves a medical image processing device that, based on the first approach, displays graphics or characters as positional information at the location of the region of interest in the medical image during display control processing. The second approach specifies the concrete method for providing this positional information.

[0018] The third approach involves a medical image processing device that, based on the first or second approach, displays characters indicating the type of the region of interest as type information during display control processing. The third approach specifies the concrete method for providing this type information.

[0019] The fourth approach involves a medical image processing device that, based on any of the first to third approaches, displays graphics or characters set according to the type of the region of interest as location information. The fourth approach specifies the concrete method for displaying location information; by displaying location information corresponding to the type of region of interest, the user can easily grasp the recognition result of the region of interest.

[0020] The medical image processing device involved in the fifth method, based on any of the first to fourth methods, displays position information in association with type information during display control processing. According to the fifth method, the user can easily grasp the relationship between position information and type information.

[0021] The medical image processing device involved in the sixth method is based on any of the first to fifth methods. In the display control processing, the processor displays a line segment with one endpoint representing the location of the region of interest and the other endpoint representing the location of type information as position information. According to the sixth method, the user can easily grasp the relationship between position information and type information.

[0022] The medical image processing apparatus involved in the seventh method is based on any of the first to sixth methods. In the display control processing, the processor displays the type of the region of interest identified by the region of interest identification processing as type information. Furthermore, in the seventh method, "the type of the region of interest identified by the region of interest identification processing" refers to the type of the actually identified region of interest.

[0023] The medical image processing apparatus involved in the eighth method is based on any of the first to sixth methods. In the display control processing, the processor displays the type of the region of interest that can be identified by the region of interest identification processing as type information. In the eighth method, "the type of region of interest that can be identified by the region of interest identification processing" refers to the type of region of interest that has the possibility of being identified even if it is not actually identified. This "type of region of interest that can be identified" can vary depending on the type of medical device or the structure of the identifier.

[0024] The medical image processing apparatus of the ninth method, based on the eighth method, displays information about unidentified areas within a region of interest that can be identified through region-of-interest recognition processing using a second notification force that is lower than the first notification force used to represent information about the type of regions of interest that can be identified through region-of-interest recognition processing. If all recognizable regions of interest are displayed with the same notification force, it can be difficult for users to grasp the actual type of the identified region of interest. Therefore, by changing the notification force as described in the ninth method, users can easily grasp the type of the actually identified region of interest.

[0025] The medical image processing apparatus involved in the tenth method is based on any of the first to ninth methods. In the display control processing, the processor displays position information inside the image signal display area of ​​the medical image and displays type information outside the image signal display area of ​​the medical image.

[0026] To achieve the above-mentioned objectives, the endoscopic system according to the eleventh aspect of the present invention includes: a medical image processing apparatus as described in any of the first to tenth aspects; an endoscopic observer inserted into a patient and having an imaging unit for sequentially capturing medical images; and a display device. The endoscopic system according to the eleventh aspect, having the medical image processing apparatus as described in any of the first to tenth aspects, is therefore capable of appropriately displaying the identification results of the region of interest.

[0027] The endoscopic system involved in the twelfth method is based on the eleventh method, and the endoscopic observer is an ultrasonic endoscopic observer that acquires ultrasound images of the subject as medical images.

[0028] To achieve the aforementioned objectives, the medical image processing method according to the thirteenth aspect of the present invention is executed by a medical image processing apparatus equipped with a processor. The processor executes: an image acquisition step, sequentially acquiring time-series medical images; a region of interest identification step, identifying the location and type of a region of interest from the medical images; and a display control step, displaying location information indicating the location of the region of interest and type information indicating the type of the region of interest overlapping the medical images on a display device. In the display control step, the processor causes the position of the displayed location information to change according to changes in the location of the region of interest over time, while maintaining the position of the displayed type information regardless of the changes in the location of the region of interest over time. According to the thirteenth aspect, the identification result of the region of interest can be displayed appropriately in the same manner as in the first aspect. Furthermore, the medical image processing method according to the thirteenth aspect can also perform the same processing as in the second to tenth aspects.

[0029] To achieve the above objectives, the medical image processing program according to the fourteenth aspect of the present invention enables a medical image processing apparatus equipped with a processor to execute a medical image processing method, wherein the medical image processing method includes: an image acquisition step, sequentially acquiring time-series medical images; a region of interest identification step, identifying the position and type of a region of interest from the medical images; and a display control step, displaying position information indicating the position of the region of interest and type information indicating the type of the region of interest on a display device, overlapping the medical images. In the display control step, the position of the display position information changes according to the change in the position of the region of interest due to the passage of time, while maintaining the position of the display type information regardless of the change in the position of the region of interest due to the passage of time. According to the fourteenth aspect, the identification result of the region of interest can be displayed appropriately in the same way as in the first aspect. Furthermore, the medical image processing program according to the fourteenth aspect may also be a program that further performs the same processing as in the second to tenth aspects. In addition, a non-transitory recording medium containing computer-readable code of the programs of these aspects may also be cited as an aspect of the present invention.

[0030] Invention Effects

[0031] As described above, the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program according to the present invention can appropriately notify the recognition results of the region of interest. Attached Figure Description

[0032] Figure 1 This is an external view of the endoscope system according to the first embodiment.

[0033] Figure 2 This is a diagram showing the main structural components of an ultrasonic processor device.

[0034] Figure 3 This is a flowchart showing the sequence of the medical image processing methods involved in the first embodiment.

[0035] Figure 4 This is an example image showing the display settings screen.

[0036] Figure 5 This is a diagram illustrating an example of overlapping display of location and type information.

[0037] Figure 6 This is a diagram showing a comparative example of overlapping display of location and type information.

[0038] Figure 7 This is another example of an overlay display of location and type information.

[0039] Figure 8 This is another example of an overlapping display of location and type information.

[0040] Figure 9 This is another example of an overlapping display of location and type information.

[0041] Figure 10 This is another example of an overlapping display of location and type information.

[0042] Figure 11 This is another example of an overlapping display of location and type information.

[0043] Figure 12 This diagram illustrates an example of how the screen display is set according to the characteristics of the anatomical region. Detailed Implementation

[0044] Hereinafter, with reference to the accompanying drawings, the embodiments of the medical image processing apparatus, endoscope system, medical image processing method and medical image processing program involved in the present invention will be described in detail.

[0045] [First Implementation Method]

[0046] [The overall structure of an endoscope system, including medical image processing equipment]

[0047] Figure 1 This is an external view of the endoscope system according to the first embodiment. (Example) Figure 1As shown, the endoscope system 2 (endoscope system, medical image camera device) includes an ultrasound observer 10 (endoscope observer, ultrasound endoscope observer), an ultrasound processor device 12 (medical image processing device) for generating ultrasound images (medical images), an endoscope processor device 14 (medical image processing device) for generating endoscopic images (medical images), a light source device 16 for supplying illumination light (observation light) to the ultrasound observer 10 for illuminating the body cavity, and a monitor 18 (display device) for displaying ultrasound images and endoscopic images.

[0048] The ultrasound observer 10 includes an insertion part 20 that is inserted into the body cavity of the subject, a hand-operated part 22 connected to the base of the insertion part 20 for operation by the operator, and a universal plug 24 connected at one end to the hand-operated part 22. At the other end of the universal plug 24 are provided an ultrasound connector 26 connected to the ultrasound processor device 12, an endoscope connector 28 connected to the endoscope processor device 14, and a light source connector 30 connected to the light source device 16.

[0049] The ultrasonic observer 10 is detachably connected to the ultrasonic processor 12, the endoscope processor 14, and the light source 16 via these connectors. Additionally, a gas / water supply tube 32 and a suction tube 34 are connected to the light source connector 30.

[0050] The light source device 16 consists of a light source for illumination (e.g., a red light source, a green light source, a blue light source, and a purple light source that respectively illuminate narrow band light of red, green, blue, and purple), an aperture, a condenser lens, and a light source control unit, etc. These light sources can be used to set ordinary light (white light), special light (narrow band light, etc.), and combinations thereof as observation light.

[0051] The monitor 18 receives the image signals generated by the ultrasound processor 12 and the endoscope processor 14 and displays the ultrasound image and the endoscope image. Regarding the display of the ultrasound image and the endoscope image, it is possible to switch to display only one image on the monitor 18, or to display both images simultaneously.

[0052] The hand-held control unit 22 is provided with a gas supply and water supply button 36 and a suction button 38, and is also provided with a pair of angle buttons 42 and a treatment device insertion port 44.

[0053] The insertion part 20 has a front end, a base end, and a long axis 20a. Starting from the front end, it consists of a front end body 50 made of a rigid component, a curved part 52 connected to the base end of the front end body 50, and a flexible part 54. The flexible part 54 connects the base end of the curved part 52 to the front end of the hand-operated part 22 and is elongated. Specifically, the front end body 50 is located at the front end of the insertion part 20 in the direction of its long axis 20a. Furthermore, the curved part 52 can be bent by rotating a pair of bend knobs 42 provided on the hand-operated part 22. This allows the user to orient the front end body 50 in a desired direction.

[0054] An ultrasonic probe 62 (camera unit) and a bag-shaped capsule 64 covering the ultrasonic probe 62 are mounted on the front end body 50. The capsule 64 can expand or contract by supplying water from the water tank 70 or by suctioning water from the capsule 64 by the suction pump 72. During ultrasonic observation, the capsule 64 expands to abut against the inner wall of the body cavity to prevent the attenuation of ultrasonic waves and ultrasonic echoes (echo signals).

[0055] Additionally, an endoscopic observation section (not shown) with an observation section and an illumination section is mounted on the front end body 50. The observation section includes an objective lens and an imaging element. The endoscopic observation section is located behind the ultrasonic probe 62 (on the side of the hand-operated section 22).

[0056] The endoscope system 2 can sequentially acquire (sequentially capture) endoscopic images (optical images) and ultrasound images through the structure described above. Furthermore, the endoscope system 2 can also acquire endoscopic images or ultrasound images from the recording unit 120, a server (not shown), or a database.

[0057] [Medical Image Processing Device]

[0058] Figure 2 This is a block diagram showing the main structural components of an ultrasonic processor device.

[0059] Figure 2 The ultrasound processor device 12 (medical image processing device, processor) shown is a device that identifies regions of interest (objects) within medical images based on sequentially acquired time-series medical images and displays the identification results on a display device. It comprises a transceiver unit 100 (processor, image acquisition unit), an image generation unit 102 (processor, image acquisition unit), a CPU 104 (processor, CPU: Central Processing Unit), a region of interest identification unit 106 (processor, region of interest identification unit), a communication control unit 110 (processor), a display control unit 112 (processor, display unit), a memory 118, and a recording unit 120 (recording device). The processing of these units, as described below, is implemented by one or more processors.

[0060] The CPU 104 operates based on various programs stored in the memory 118, including the medical image processing program according to the present invention, and uniformly controls the region of interest identification unit 106, the communication control unit 110, and the display control unit 112, and also functions as part of these units. The memory 118 includes non-temporary recording media such as ROM (Read-Only Memory) for recording medical image processing programs, and temporary recording media such as RAM (Random Access Memory) used as temporary storage areas.

[0061] The transceiver unit 100 and the image generation unit 102, which function as image acquisition units, sequentially acquire time-series medical images (image acquisition and processing, image acquisition process).

[0062] The transmitting unit of the transceiver unit 100 generates multiple drive signals to multiple ultrasonic transducers of the ultrasonic probe 62 of the ultrasonic observer 10, and assigns a delay time to each of the multiple drive signals based on the transmission delay mode selected by the scanning control unit (not shown), and applies the multiple drive signals to the multiple ultrasonic transducers.

[0063] The receiving unit of the transceiver unit 100 amplifies the multiple detection signals output from the multiple ultrasonic transducers of the ultrasonic probe 62, converting the analog detection signals into digital detection signals (also known as RF (Radio Frequency) data). This RF data is then input to the image generation unit 102.

[0064] The image generation unit 102 assigns a delay time to each of the multiple detection signals represented by RF data based on the reception delay mode selected by the scanning control unit, and performs reception focusing processing by adding these detection signals together. Through this reception focusing processing, acoustic ray data with a reduced focus of the ultrasonic echo is formed.

[0065] The image generation unit 102 corrects for attenuation due to distance in the acoustic data by using STC (Sensitivity Time Control) based on the depth of the ultrasonic wave reflection location. Then, it performs envelope detection processing using a low-pass filter or similar means to generate envelope data. This envelope data, consisting of one frame or more frames, is stored in a movie memory (not shown). The image generation unit 102 performs preprocessing on the envelope data stored in the movie memory, such as Logarithmic compression or gain adjustment, to generate a B-mode image.

[0066] In this way, the transceiver unit 100 and the image generation unit 102 sequentially acquire time-series B-mode images (hereinafter referred to as "medical images").

[0067] The region of interest identification unit 106 performs processing based on medical image recognition and information related to the location of the region of interest within the medical image (detection processing, region of interest identification processing, region of interest identification procedure), and processing to classify the region of interest into any one of multiple categories based on the medical image (classification processing, classification procedure). For example, it can be constructed using a learned model (a model learned using an image set composed of images obtained from photographs of an organism) constructed through machine learning, such as a CNN (Convolutional Neural Network) or SVM (Support Vector Machine). In this embodiment, the region of interest is, for example, an organ or blood vessel within the medical image (a tomographic image of a B-mode image), such as the pancreas, main pancreatic duct, spleen, splenic vein, splenic artery, gallbladder, etc.

[0068] An example of the layer structure of the region of interest identification unit 106 composed of a CNN will be explained. The CNN includes an input layer, intermediate layers, and an output layer. The input layer takes into account the medical image generated by the image generation unit 102 and outputs feature values. The intermediate layers include convolutional layers and pooling layers, which take into account the feature values ​​output by the input layer and calculate other feature values. These layers form a structure of multiple "nodes" connected by "edges," maintaining multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The output layer identifies the region of interest appearing in the input medical image based on the feature values ​​output from the intermediate layers and outputs the result.

[0069] In this example, when a time series of medical images are input sequentially, the region of interest identification unit 106 identifies (detects) the location of the region of interest for each input medical image, outputs information related to that location (location information), and identifies (classifies) which category the region of interest belongs to among multiple categories, outputting information representing the identified category (category information, type information).

[0070] The display control unit 112 displays time-series medical images (endoscopic images, ultrasound images) sequentially acquired by the transceiver unit 100 and the image generation unit 102 on the monitor 18 (display device). In this example, a dynamic image representing an ultrasound tomography image is displayed on the monitor 18. Additionally, the display control unit 112 displays the object on the monitor 18 with a notification force determined by a notification force determination unit (not shown).

[0071] For medical image processing that utilizes the above functions, details will be provided later.

[0072] <Implementation of Functions Based on Various Processors>

[0073] The functions of the ultrasonic processor device 12 described above can be implemented using various processors and recording media. Among these various processors are, for example, general-purpose processors that execute software (programs) to implement various functions, namely CPUs (Central Processing Units). Additionally, among the various processors described above, there are processors specifically designed for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and other processors whose circuit structures can be changed after manufacturing, i.e., programmable logic devices (PLDs). When performing image processing as described in this invention, using a GPU structure is effective. Furthermore, processors with circuit structures specifically designed for performing specific processing, such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits, are also included among the "various processors" described above.

[0074] The functions of each part can be implemented by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, a single processor can implement multiple functions. As examples of a single processor performing multiple functions, firstly, there are forms such as computers, where a single processor is constructed using a combination of one or more CPUs and software, and that processor performs multiple functions. Secondly, there are forms such as System-on-Chip (SoC), where a single IC (Integrated Circuit) chip is used to implement the overall system functions. In this way, one or more of the above-mentioned processors are used as hardware structures to construct various functions. More specifically, the hardware structures of these various processors are circuits composed of circuit elements such as semiconductor components. These circuits can also be circuits that implement the above functions using logical operations such as AND, product, NOT, XOR, and combinations thereof.

[0075] When the aforementioned processor or circuit executes the software (program), the code that can be read by the computer executing the software (e.g., various processors, circuits, and / or combinations thereof constituting the ultrasonic processor device 12) is stored in a non-transitory recording medium such as ROM (Read Only Memory), and the computer refers to the software. The software pre-stored in the non-transitory recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used in setting the display mode and notification mode, weight parameters used in the region of interest identification unit 106, etc.). Alternatively, the code may not be recorded in ROM, but in a non-transitory recording medium such as various optical-magnetic recording devices or semiconductor memories. When processing the software, for example, RAM (Random Access Memory) may be used as a temporary storage area, or, for example, data stored in EEPROM (Flectrically Erasable and Programmable Read Only Memory) not shown may be referenced. As a "non-transitory recording medium", memory 118 or recording unit 120 may be used.

[0076] The recording unit 120 records ultrasound images and endoscopic images (medical images), detection results of the area of ​​interest, and processing conditions (conditions for conducting detection or notification). Other information may also be recorded at the same time. The communication control unit 110 controls the acquisition of medical images from other medical imaging devices connected to the endoscope system 2, external servers, or databases.

[0077] [Order of Medical Image Processing]

[0078] The medical image processing (execution of the medical image processing method and medical image processing program involved in the present invention) in the endoscope system 2 with the above structure will be described. Figure 3 This is a flowchart illustrating the sequence of the medical image processing methods according to the first embodiment. Furthermore, the order described below can be switched as needed.

[0079] [Initial Settings]

[0080] The display control unit 112 (processor) sets the conditions required for the execution of the medical image processing method / program based on user operations via an operation unit (not shown) (keyboard, mouse, touch panel, microphone, etc.) and / or pre-set processing conditions (e.g., default processing conditions) (step S100: initial setting process). The display control unit 112 sets, for example, the display method (type of character or symbol, color, etc.) and emphasis level of position and type information. For example, the user... Figure 4 In such a screen (not representing all processing condition settings), processing conditions can be set by turning radio buttons on / off and selecting from drop-down menus via the operation unit. The display control unit 112 can display such a screen on a display device such as a monitor 18. These settings include, for example, what graphics or characters (type, color, etc.) are used to display position and type information, and whether to display a list (overview) of the types of identifiable areas of interest. Furthermore, the display control unit 112 can set processing conditions not only at the start of processing but also during the execution of subsequent steps.

[0081] [Acquisition of ultrasound images and identification of regions of interest]

[0082] The transceiver unit 100 and the image generation unit 102 sequentially acquire time-series ultrasound images (medical images) (step S110: image acquisition processing, image acquisition process), and the display control unit 112 causes the monitor 18 to display the acquired ultrasound images (step S120: display control processing, display control process). The region of interest identification unit 106 identifies the location and type of the region of interest in the ultrasound image (step S130: region of interest identification processing, region of interest identification process). For example, the region of interest identification unit 106 can set the center position of the rectangle surrounding the region of interest as the location of the region of interest, and set the information representing the location (coordinates within the image, etc.) as "location information". In addition, in this embodiment, the information representing the type of organ or blood vessel can be set as "category information, type information".

[0083] [Overlapping display of location and type information]

[0084] When the region of interest identification unit 106 detects a region of interest (YES in step S140), the display control unit 112 determines the display mode of the location information and type information based on the conditions set in step S100 (step S150: display control processing, display control step), and displays the location information and type information on the display device (monitor 18, etc., hereinafter the same) by overlaying the ultrasound image with the determined display mode (step S160: display control processing, display control step). For example, the display control unit 112 determines a display mode such as "overlaying a cross (graphic; location information) at the center of the rectangle surrounding the region of interest, and displaying the name of the actually identified (detected) region of interest as a character", and displays it in this display mode. The processing of steps S110 to S160 is performed sequentially as the time-series ultrasound images are acquired. Furthermore, the region of interest identification and display control processing can be performed on all ultrasound images of the time series (i.e., on each frame), or on a portion of the ultrasound images of the time series (on a portion of the frames).

[0085] Figure 5 This is a diagram illustrating an example of overlapping display of location and type information. In Figure 5 In part (a), the pancreas 530 (region of interest) and splenic vein 540 (region of interest) are detected (identified) in the ultrasound image. Graphics 532 and 542 (graphics, location information) indicating the location of these regions of interest are overlaid and displayed inside the image signal display area 510 (image signal display area, ultrasound image) of the display screen 500. On the other hand, characters 550B ("Pancreas", type information) and 552B ("SV (Splenic Vein)", type information) indicating the name of the region of interest are overlaid and displayed outside the image signal display area 510, i.e., area 520. In addition, graphics identical to those indicating the location information, 532 and 542, are displayed side by side with characters 550B and 552B in area 520 as examples 550A and 552A (type information). Thus, the display control unit 112 associates the location information and the type information.

[0086] Figure 5 Part (b) indicates from Figure 5 The state shown in part (a) is the state after time has elapsed (or, in addition to the elapsed time, the observation position or orientation has changed due to the operation of the ultrasound observer 10) and the position of the region of interest (pancreas 530 and splenic vein 540) in the ultrasound image has changed. In this state, as far as the display control unit 112 (processor) is concerned, the positions of the display graphics 532 and 542 (position information) change according to the change in the position of the region of interest caused by the elapsed time, but the positions of the display characters 550B and 552B (type information) remain unchanged regardless of the change in the position of the region of interest caused by the elapsed time.

[0087] In such Figure 5 In the display method shown, a graphic (position information) indicating the location of the area of ​​interest assists the user's observation. Furthermore, because a simple graphic (cross) is displayed as position information, the screen display is less likely to obstruct observation compared to displaying the names of organs, etc., as characters. Additionally, even when the location of the area of ​​interest changes significantly within the screen over time, the user can easily grasp the type of the area of ​​interest because the position of the characters (type information) is maintained only by moving the graphic (position information).

[0088] Figure 6 This is a diagram showing a comparative example of overlapping display of location and type information. Figure 6 Parts (a) and (b) indicate the changes over time. Figure 6In the comparative example shown, both location information and type information are represented by characters, and the display position of this information is linked to changes in the position of the area of ​​interest. Therefore, it is similar to... Figure 5 The display method of the present invention shown in the illustration can easily obstruct observation.

[0089] Figure 7 This is another example of an overlay display of location and type information. In Figure 7 In the example shown, the shape of the symbol representing location information is set according to the type of area of ​​interest (or, in addition to the shape of the symbol, or instead of the shape, the color of the symbol is set according to the type of area of ​​interest). Specifically, the display control unit 112 displays a graphic 532 (cross) composed of line segments in a direction inclined relative to the pancreas 530 as location information, and displays a symbol 542A (cross) composed of line segments in a direction horizontal and vertical to the splenic vein 540 as location information. Furthermore, the display control unit 112 displays these symbols as examples 550A and 552C (type information) alongside characters 550B and 552B. Users spend most of their time observing ultrasound images during examinations or observations, and sometimes it is difficult to move their gaze outside the image area (outside the image signal display area 510, area 520), but even in such cases, by using... Figure 7 As shown, the shape of the symbol representing location information is set according to the type of area of ​​interest, allowing users to easily understand the type of area of ​​interest by seeing the symbol displayed as location information. Furthermore, users can access information through methods such as... Figure 4 The user can set the shape or color of the symbols on the screen, and the display control unit 112 can determine the display mode based on the user's operation.

[0090] Figure 8 This is another example of an overlay display of location and type information. About Figure 5 , Figure 7 In the examples above, the symbols used to indicate location information are crosses or crosses, but... Figure 8 In the example shown, the display control unit 112 displays graphic 533 (location information) representing the initial letter P of "Pancreas" and graphic 543 (location information) representing the initial letter b of "blood vessel". Furthermore, the display control unit 112 displays these symbols side-by-side with characters 551B and 553B (type information) as examples 551A and 553A (type information). Through this display, the user can see the symbols displayed as location information and easily grasp the type of the area of ​​interest.

[0091] Figure 9This diagram illustrates another example of overlapping display of location and type information. When only type information is displayed for the actually detected area of ​​interest, the displayed characters may change as the type of the detected area of ​​interest changes over time, which can be cumbersome for the user. Therefore, in... Figure 9 In the example shown, the display control unit 112 sets the type of the region of interest that can be identified through the region of interest identification processing to character 560 (type information) and displays it in a list (overview) in area 520. The "type of the region of interest that can be identified through the region of interest identification processing" can be determined based on the type or structure of the endoscope system (e.g., ultrasound endoscope or optical endoscope, upper endoscope or lower endoscope, observation light is ordinary light or special light, etc.). Additionally, the display control unit 112 displays line segments 570 and 572 (line segments, leader lines) with one endpoint representing the location of the region of interest (pancreas 530, splenic vein 540) and the other endpoint representing the location of character 560 (type information) as location information. The display control unit 112 can also be as follows... Figure 9 The line type is changed to display these line segments 570 and 572. Additionally, the display control unit 112 can also change the line color in addition to the line type (as explained below). Figure 10 , Figure 11 (The same applies).

[0092] Based on this display, users can easily access organ names (type information of the area of ​​interest) by simply following the line segments, thus facilitating the binding of ultrasound images to organ names. For example, users can access this information via... Figure 4 The display screen can be set to display in a list or as a line segment, and the display control unit 112 can determine the display mode based on the user's operation.

[0093] Figure 10 This is another example of an overlay display of location and type information. In Figure 10 In the example shown, the display control unit 112 displays information where the notification power (second notification power) of the type information of the type of the area of ​​interest that was not actually identified within the area of ​​interest that can be identified through the area of ​​interest identification processing is lower than the notification power (first notification power) of the type information of the type of the area of ​​interest that was actually identified. Specifically, the display control unit 112 displays “Pancreas” and “SV” corresponding to the pancreas 530 and splenic vein 540 (the area of ​​interest that was actually identified) using character 562A, and displays other areas of interest (areas of interest that were not actually identified) using character 562B, which is smaller than character 562A, so that the notification power (second notification power) of character 562B is lower than the notification power (first notification power) of character 562A.

[0094] Figure 11This is another example of overlapping display of location and type information (another method to make the second notification less powerful than the first). Figure 11 In the example shown, the display control unit 112 displays “Pancreas” and “SV” corresponding to the pancreas 530 and splenic vein 540 using character 562A, and displays other areas of interest (areas of interest that are not actually recognized) using grayed-out character 562C, so that the notification power (second notification power) of character 562C is lower than that of character 562A (first notification power).

[0095] Regarding the display control unit 112, as a method to reduce notification power, besides Figure 10 , Figure 11 In addition to the examples shown, methods such as thinning the characters, increasing the transparency of the character colors, and decreasing the brightness of the character colors can also be used. The display control unit 112 can also combine these methods. In the endoscope system 2, such notifications can improve the visual recognition of the actually identified area of ​​interest. For example, the user can access this information via... Figure 4 The screen displays a reduced notification level, and the display control unit 112 can determine the display mode based on the user's operation.

[0096] [Changes in display corresponding to the characteristics of the anatomical region]

[0097] Depending on the anatomical region (area of ​​interest), there are anatomical regions that require constant display of characters on the endoscopic ultrasound image. For example, the pancreas, besides being the primary organ observed in endoscopic ultrasound, is also an organ with indistinct boundaries with surrounding tissues, making it easy to miss. Therefore, it is preferable to display characters on the image for the pancreas so that the user can always easily grasp its location. On the other hand, the splenic vein (SV) is the vessel used to determine where the currently viewed endoscopic ultrasound image is located, so it does not need to be constantly indicated to the user. Furthermore, since the splenic vein is sometimes captured very small due to the scene, it is preferable to avoid hiding the area through graphic overlay.

[0098] Therefore, in the first embodiment, the display mode can also be set according to the characteristics of the anatomical region. Specifically, the display control unit 112 can also set the type of information to be displayed and whether to link the display position of the information with the positional changes of the area of ​​interest caused by the passage of time, according to the type of the area of ​​interest (the characteristics of the anatomical region). Figure 12This is an example of such a screen display. The display control unit 112 overlays character 534 (location information, type information) onto the pancreas 530 and overlays graphic 542 (location information) onto the splenic vein 540. Additionally, the display control unit 112 displays example 552 (type information) and character 552B (“SV”; type information) side-by-side in area 520. Figure 12 In this example, the display control unit 112 links the display positions of the character 534 and the graphic 542 to the temporal position changes of the region of interest (pancreas 530, splenic vein 540) in the ultrasound image, while the display positions of the example 552 and the character 552B remain unchanged regardless of the positional changes of the region of interest. That is, in the first embodiment, the display control unit 112 controls the display positions of a portion of the region of interest (in the ultrasound image)... Figure 12 In the example of the pancreas (530), not only is the location information linked to the temporal changes in the display position of the area of ​​interest, but the display position of the type information is also linked to the temporal changes in the display position of the area of ​​interest. For other areas of interest (in... Figure 12 In the example of the splenic vein (540), the display position of the type information can be maintained regardless of the temporal changes in the display position of the area of ​​interest. Therefore, appropriate notifications can be provided corresponding to the type of the area of ​​interest (the characteristics of the anatomical region).

[0099] In addition, users can conduct transactions via, for example Figure 4 The display control unit 112 can determine the display mode based on the user operation by setting the screen to correspond to the characteristics of the anatomical area.

[0100] As described above, the endoscope system according to the first embodiment can appropriately notify the identification results of the area of ​​interest.

[0101] [Applications to other medical images]

[0102] In the first embodiment described above, the case of recognizing an ultrasound endoscope image as a type of medical image (medical image) was explained. However, the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program of the present invention can also be applied to cases using ultrasound images acquired by an ultrasound device other than an endoscope (such as a surface endoscope device), or endoscope images acquired by an optical endoscope device that captures the subject with white light and / or special light, and other medical images other than ultrasound endoscope images.

[0103] The embodiments and other examples of the present invention have been described above, but the present invention is not limited to the above-described manner and various modifications can be made.

[0104] Symbol Explanation

[0105] 2. Endoscopic System

[0106] 10. Ultrasonic Observer

[0107] 12. Ultrasonic processing device

[0108] 14. Processor device for endoscopes

[0109] 16 Light source device

[0110] 18 monitors

[0111] 20 Insertion section

[0112] 20a Long Axis

[0113] 22 Hands-on Operations Department

[0114] 24 Universal Rope

[0115] 26 Connectors for Ultrasonic Applications

[0116] 28. Connector for endoscopes

[0117] 30 Connector for Light Source

[0118] 32 tubes

[0119] 34 tubes

[0120] 36 Gas and water supply buttons

[0121] 38. Attraction Button

[0122] 42. Bent corner button

[0123] 44. Treatment device insertion port

[0124] 50 Front end body

[0125] 52. Bend

[0126] 54. Soft parts

[0127] 62 Ultrasonic probe

[0128] 64 cysts

[0129] 70 Water Supply Tank

[0130] 72 Suction Pump

[0131] 100 Receiving and Dispatch Department

[0132] 102 Image Generation Unit

[0133] 104 CPU

[0134] 106. Area Identification Department

[0135] 110 Communications Control Department

[0136] 112 Display Control Unit

[0137] 118 Memory

[0138] 120 Recording Department

[0139] 500 display screen

[0140] 510 Image signal display area

[0141] Area 520

[0142] 530 Pancreas

[0143] 532 Graphics

[0144] 533 Graphics

[0145] 534 characters

[0146] 540 Splenic vein

[0147] 542 graphics

[0148] 543 Graphics

[0149] 550B character

[0150] 551B character

[0151] 552B character

[0152] 553B character

[0153] 560 characters

[0154] 562A character

[0155] 562B character

[0156] 562C character

[0157] 570 line segment

[0158] 572 line segments

[0159] S100~S170 Steps of the medical image processing method.

Claims

1. A medical image processing device, comprising a processor, wherein, The processor executes: Image acquisition and processing: sequentially acquire time-series medical images; Region of interest identification processing: Identifying the location and type of the region of interest from the medical image; as well as The display control process causes the location information representing the position of the region of interest and the type information representing the type of the region of interest to be displayed superimposed on the medical image on the display device. In the display control process, the processor causes the position of the displayed location information to change according to the change in the position of the area of ​​interest caused by the passage of time, and sets the display mode of the type information and whether to cause the position of the displayed type information to change according to the change in the position of the area of ​​interest caused by the passage of time, based on the type of the area of ​​interest.

2. The medical image processing apparatus according to claim 1, wherein, In the display control process, the processor displays graphics or characters at the location of the area of ​​interest in the medical image as the location information.

3. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor displays characters representing the type of the region of interest as the type information.

4. The medical image processing apparatus according to claim 1 or 2, wherein, The processor displays a graphic or character set according to the type of the region of interest as the location information.

5. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor displays the location information in association with the type information.

6. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor displays a line segment with one endpoint representing the location of the area of ​​interest and the other endpoint representing the location of the type information as the location information.

7. The medical image processing apparatus according to claim 6, wherein, The processor displays the line segment as the position information by changing at least one of the line type and the color of the line segment according to the type information.

8. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor displays the type of the region of interest identified by the region of interest identification process as the type information.

9. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor displays the type of the region of interest that can be identified by the region of interest identification process as the type information.

10. The medical image processing apparatus according to claim 9, wherein, The processor displays the type information of the interest region that was not actually identified in the interest region that can be identified by the interest region identification process with a second notification power that is lower than the first notification power for the type information representing the interest region actually identified by the interest region identification process.

11. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, the processor causes the position information to be displayed inside the image signal display area in the medical image, and causes the type information to be displayed outside the image signal display area in the medical image.

12. An endoscope system comprising: The medical image processing apparatus according to any one of claims 1 to 11; An endoscopic observer, inserted into the body of the patient, and equipped with a camera unit for sequentially capturing the medical images; and The display device.

13. The endoscopic system according to claim 12, wherein, The endoscopic observer is an ultrasonic endoscopic observer that acquires ultrasound images of the subject as medical images.

14. A medical image processing method, executed by a medical image processing device equipped with a processor, wherein, The medical image processing method is executed by the processor and includes: The image acquisition process involves sequentially acquiring time-series medical images; The region of interest identification process identifies the location and type of the region of interest from the medical image; and The display control process involves overlaying the location information representing the position of the region of interest and the type information representing the type of the region of interest onto the medical image on the display device. In the display control process, the processor causes the position of the displayed position information to change according to the change in the position of the area of ​​interest caused by the passage of time, and sets the display mode of the type information and whether to cause the position of the displayed type information to change according to the change in the position of the area of ​​interest caused by the passage of time, based on the type of the area of ​​interest.

15. A recording medium that is non-transitory and computer-readable, wherein, The recording medium contains a program that causes a computer to execute the medical image processing method of claim 14.

Images