Image processing apparatus, image processing system, image processing method

Abstract

An object of the present application is to provide an image processing apparatus, an image processing system, an image processing method, and an image processing program capable of appropriately notifying an identification result of a region of interest. An image processing apparatus according to an aspect of the present application includes a processor configured to execute: an image acquisition process configured to acquire an image; an identification process configured to identify a region of interest from the image; a notification information determination process configured to determine first notification information indicating a position of the region of interest in the image and second notification information indicating a category of the region of interest; and a notification position determination process configured to determine a notification position of the second notification information in the image based on a pixel value in a region surrounding the region of interest in the image.

Description

Technical Field

[0001] This invention relates to image processing apparatus, image processing system, image processing method, and image processing program, and particularly to a technique for notifying the recognition results of a region of interest. Background Technology

[0002] As an aid for users to observe and diagnose images, there are known techniques for notifying users of the identification results of regions of interest. For example, Patent Document 1 describes a technique for notifying users of regions of interest identified in an image using bounding boxes and icons. Patent Document 2 also describes a technique for overlaying bounding boxes.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: WO18 / 221033 Publication

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-146202 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] Notification of identification results is intended for flexible application in observation and diagnosis; however, depending on the notification method, it may also become an obstacle for users during observation and diagnosis. Therefore, it is preferable to notify of identification results in an appropriate location or with appropriate content. However, the prior art described in Patent Documents 1 and 2 above has not adequately considered this situation.

[0009] The present invention was made in view of the following circumstances, and its object is to provide an image processing apparatus, image processing system, image processing method and image processing program that can appropriately notify the recognition results of the region of interest.

[0010] means for solving technical problems

[0011] To achieve the above objectives, the image processing apparatus according to the first aspect of the present invention includes a processor, wherein the processor performs: image acquisition processing to acquire an image; recognition processing to identify a region of interest from the image; notification information determination processing to determine a first notification information indicating the location of the region of interest in the image and a second notification information indicating the type of the region of interest; and notification location determination processing to determine a notification location in the image that notifies the second notification information based on pixel values ​​in the region surrounding the region of interest in the image.

[0012] Typically, when notifying users of recognition results, methods have been studied such as filling the entire object (region of interest) with any color or surrounding it with a rectangle or circle. Simultaneously, methods using text or icons to notify users what was identified have also been investigated. However, with such notifications, the amount of information increases as the number of objects being identified grows, potentially hindering user comprehension. Furthermore, in situations requiring precise observation, it is preferable to notify the user of information (such as rectangles or circles, text, or icons) in an area that has minimal impact on observation.

[0013] Considering this situation, in the image processing apparatus involved in the first approach, the recognition result of the region of interest can be notified at an appropriate notification location (e.g., a notification location that does not hinder the user's understanding and has little impact on observation) based on the pixel values ​​in the region surrounding the region of interest in the image. Furthermore, "based on pixel values" can mean directly using pixel values ​​or obtaining values ​​obtained by statistically processing multiple pixel values ​​(maximum, minimum, average, etc. within the determined region).

[0014] In the first approach, the "Region of Interest (ROI)" refers to a specific type of subject projected onto the image or a region within the subject that possesses specific characteristics. What constitutes the ROI can vary depending on the type of image or its intended use. The ROI is also known as the region of interest.

[0015] The "region surrounding the region of interest" can be a polygonal region such as a rectangle, or a circular or elliptical region. Furthermore, for the identification of the region of interest, for example, a detector based on machine learning, such as deep learning, can be used.

[0016] Furthermore, the image processing device involved in the first approach can be implemented as, for example, as the processor part of an image processing system, but is not limited to this approach.

[0017] The image processing apparatus involved in the second method, based on the first method, determines the notification location based on the type of region of interest during the notification location determination process. Since the appropriate notification location sometimes differs depending on the type of region of interest, the second method determines the notification location based on the type of region of interest.

[0018] The image processing device involved in the third method is based on the first or second method, in which the processor determines the center of gravity of the region of interest as the notification location during the notification location determination process.

[0019] The image processing apparatus involved in the fourth approach, based on any of the first to third approaches, calculates probability information representing the probability of recognizing a region of interest in the recognition process, and determines the notification location based on pixel values ​​and the probability information in the notification location determination process. In the fourth approach, the notification location can be determined in a region where the probability of recognizing the region of interest is within a specific range (e.g., above a threshold). Furthermore, the probability information can be calculated, for example, by a recognizer constructed using machine learning such as deep learning.

[0020] The image processing apparatus involved in the fifth method, based on the fourth method, acquires time-series images in the image acquisition process and determines the notification location based on the temporal changes of probability information in the time-series images in the notification location determination process. The probability information (the probability of identifying the region of interest) may change due to changes in ambient light, changes in observation position or direction, movement or deformation of the region of interest itself, etc., but according to the fifth method, an appropriate notification location can be determined based on the temporal changes of the probability information.

[0021] Furthermore, in the fifth method and the other methods described below, "acquiring time-series images" includes acquiring multiple images captured at a determined frame rate. Acquisition may be real-time or not. For example, images that have been pre-captured and recorded may also be acquired.

[0022] The image processing apparatus involved in the sixth method is based on any of the first to fourth methods, wherein, in terms of the processor, it acquires a time-series image in the image acquisition process and determines the notification location based on the time change of pixel values ​​in the time-series image in the notification location determination process.

[0023] The image processing apparatus involved in the seventh method is based on the fifth method, and in terms of the processor, in the notification location determination process, the notification location is determined based on the temporal changes of pixel values ​​in the time series image.

[0024] The image processing apparatus involved in the eighth method is based on any of the first to seventh methods, whereby the processor further executes a notification method determination process to determine the notification method of the second notification information. The notification method may include, for example, which of the following is used for notification: text, graphics, or symbols; the color scheme; and whether overlapping display is allowed.

[0025] The image processing apparatus involved in the ninth method is based on the eighth method. In terms of the processor, it acquires a time-series image in the image acquisition process and determines the notification method based on the time change of pixel values ​​in the time-series image in the notification method determination process.

[0026] The image processing apparatus involved in the tenth method, based on the eighth method, involves, for the processor, acquiring a time-series image in the image acquisition process, determining the notification location based on the temporal change in the size of the region of interest in the time-series image in the notification location determination process, and determining the notification method based on the temporal change in the size of the region of interest in the time-series image in the notification method determination process. The size of the region of interest in the image may change due to changes in the observation position or orientation, or deformation of the region of interest itself. If the size changes, the appropriate notification location or notification method of the second notification information (category information) may also change. Therefore, in the tenth method, the notification location and notification method are determined based on the temporal change in the size of the region of interest.

[0027] The image processing apparatus involved in the eleventh method is based on the ninth method. For the processor, in the notification location determination process, the notification location is determined based on the temporal change in the size of the region of interest in the time-series image; and in the notification method determination process, the notification method is determined based on the temporal change in the size of the region of interest in the time-series image. In the eleventh method, similarly to the tenth method, the notification location and notification method are determined based on the temporal change in the size of the region of interest.

[0028] The image processing apparatus involved in the twelfth method is based on any of the first to eleventh methods, wherein the processor overlays the first notification information and the second notification information on the image and records them on the recording device.

[0029] The image processing apparatus involved in the thirteenth method is based on any one of the first to twelfth methods, wherein the processor records the second notification information and / or notification location in the recording device.

[0030] The image processing apparatus involved in the fourteenth method is based on any of the first to thirteenth methods, wherein in the image acquisition processing, the processor acquires a medical image of the subject. In this invention, "medical image" refers to an image obtained as a result of photographing or measuring a living organism such as a human body for purposes such as diagnosis, treatment, or measurement. Examples include endoscopic images, ultrasound images, CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Medical images are also called medical images. Furthermore, in the case of medical images, the area of ​​interest can be a lesion area or a candidate lesion area, an organ or blood vessel, a treated area, or a treatment device, etc.

[0031] To achieve the above objectives, the image processing system according to the fifteenth aspect of the present invention includes: an image processing apparatus according to any one of the first to fourteenth aspects; and a camera device for capturing images. Since the image processing system according to the fifteenth aspect includes the image processing apparatus described in any one of the first to fourteenth aspects, it is able to appropriately display the recognition results of the region of interest.

[0032] The image processing system involved in the sixteenth method is based on the fifteenth method, and the imaging device is an endoscope. According to the sixteenth method, images of the interior of a tubular object can be acquired through the endoscope.

[0033] The image processing system involved in the seventeenth method is based on the sixteenth method, and the endoscopic observer is an ultrasonic endoscopic observer. According to the seventeenth method, ultrasonic images of an object can be acquired through the ultrasonic endoscopic observer.

[0034] The image processing system involved in the eighteenth method is based on any one of the fifteenth to seventeenth methods, wherein the processor overlays and displays the image, the first notification information, and the second notification information on the display device. According to the eighteenth method, the user can easily grasp the location and type information of the area of ​​interest through visual means.

[0035] The image processing system involved in the nineteenth method is based on any of the eighteenth methods and also includes a display device.

[0036] To achieve the above objectives, the image processing method according to the twentieth aspect of the present invention is executed by an image processing apparatus equipped with a processor, wherein the processor executes: an image acquisition step, acquiring an image; an identification step, identifying a region of interest from the image; a notification information determination step, determining first notification information indicating the location of the region of interest in the image and second notification information indicating the type of the region of interest; and a notification location determination step, determining a notification location in the image that notifies the second notification information based on pixel values ​​in the region surrounding the region of interest in the image.

[0037] According to the twentieth method, similarly to the first method, the recognition result of the region of interest can be appropriately notified. Furthermore, the image processing method involved in the twentieth method can further perform the same processing as the second to fourteenth methods.

[0038] To achieve the above objectives, the image processing program according to the twenty-first aspect of the present invention causes an image processing apparatus equipped with a processor to execute an image processing method, wherein the image processing method includes: an image acquisition step of acquiring an image; an identification step of identifying a region of interest from the image; a notification information determination step of determining first notification information indicating the location of the region of interest in the image and second notification information indicating the type of the region of interest; and a notification location determination step of determining a notification location in the image that provides the second notification information based on pixel values ​​in the region surrounding the region of interest in the image.

[0039] According to the twenty-first method, similarly to the first and twenty methods, the identification result of the region of interest can be appropriately notified. Furthermore, the image processing program involved in the twenty-first method can also be a program that further performs the same processing as the second to fourteenth methods. Additionally, a non-volatile recording medium containing computer-readable code of the programs of these methods can also be cited as a method of the present invention.

[0040] Invention Effects

[0041] As described above, the image processing apparatus, image processing system, image processing method, and image processing program according to the present invention can appropriately notify the recognition results of the region of interest. Attached Figure Description

[0042] Figure 1 This is an external view of the endoscope system according to the first embodiment.

[0043] Figure 2 This is a block diagram showing the main structural components of an ultrasonic processor device.

[0044] Figure 3 This is a flowchart illustrating the steps of the image processing method according to the first embodiment.

[0045] Figure 4 This is an example image showing the settings screen for notification methods.

[0046] Figure 5 This is a diagram showing a notification example (display example) of the first and second notification messages.

[0047] Figure 6 This is a diagram representing another notification example (display example) that shows the first and second notification information.

[0048] Figure 7 This is a diagram representing a further notification example (display example) between the first and second notification information.

[0049] Figure 8 This is a diagram showing the location of a notification where the type of information (second notification information) has been determined based on the type of area of ​​interest. Detailed Implementation

[0050] Hereinafter, with reference to the accompanying drawings, embodiments of the image processing apparatus, image processing system, image processing method, and image processing program involved in the present invention will be described in detail.

[0051] [First Implementation Method]

[0052] [The overall structure of the endoscope system, including the image processing device]

[0053] Figure 1 This is an external view of the endoscope system according to the first embodiment. (Example) Figure 1 As shown, the endoscope system 2 (image processing system, endoscope system) includes an ultrasound observer 10 (endoscope observer, ultrasound endoscope observer, camera device), an ultrasound processor device 12 (image processing device, processor, camera device) for generating ultrasound images (medical images), an endoscope processor device 14 (image processing device) for generating endoscopic images (medical images), a light source device 16 for supplying illumination light (observation light) to the ultrasound observer 10 for illuminating the body cavity, and a monitor 18 (display device) for displaying ultrasound images and endoscopic images.

[0054] The ultrasound observer 10 includes an insertion part 20 that is inserted into the body cavity of the subject, a hand-operated part 22 connected to the base of the insertion part 20 for operation by the operator, and a universal plug 24 connected at one end to the hand-operated part 22. The other end of the universal plug 24 is provided with an ultrasound connector 26 connected to the ultrasound processor device 12, an endoscope connector 28 connected to the endoscope processor device 14, and a light source connector 30 connected to the light source device 16.

[0055] The ultrasonic observer 10 is detachably connected to the ultrasonic processor 12, the endoscope processor 14, and the light source device 16 via these connectors. Additionally, a gas / water supply tube 32 and a suction tube 34 are connected to the light source connector 30.

[0056] The light source device 16 consists of a light source for illumination (e.g., a red light source, a green light source, a blue light source, and a purple light source that respectively illuminate narrow band light of red, green, blue, and purple), an aperture, a condenser lens, and a light source control unit, etc. These light sources can be used to set ordinary light (white light), special light (narrow band light, etc.), and combinations thereof as observation light.

[0057] The monitor 18 receives the image signals generated by the ultrasound processor 12 and the endoscope processor 14 and displays the ultrasound image and the endoscope image. Regarding the display of the ultrasound image and the endoscope image, it is possible to switch to display only one image on the monitor 18, or to display both images simultaneously.

[0058] The hand-held control unit 22 is provided with a gas supply and water supply button 36 and a suction button 38, and is also provided with a pair of angle buttons 42 and a treatment device insertion port 44.

[0059] The insertion part 20 has a front end, a base end, and a long axis 20a. Starting from the front end side, it consists of a front end body 50 made of a rigid component, a curved part 52 connected to the base end side of the front end body 50, and a flexible part 54. The flexible part 54 connects the base end side of the curved part 52 and the front end side of the hand-operated part 22, and is elongated and thin. That is, the front end body 50 is located at the front end side in the direction of the long axis 20a of the insertion part 20. In addition, the curved part 52 can be bent by rotating a pair of bend knobs 42 provided on the hand-operated part 22. Thus, the user can orient the front end body 50 in a desired direction.

[0060] An ultrasonic probe 62 (camera device, camera unit) and a bag-shaped capsule 64 covering the ultrasonic probe 62 are mounted on the front end body 50. The capsule 64 can expand or contract by supplying water from the water tank 70 or by using a suction pump 72 to draw water from the capsule 64. During ultrasonic observation, the capsule 64 expands to abut against the inner wall of the body cavity to prevent the attenuation of ultrasonic waves and ultrasonic echoes (echo signals).

[0061] Additionally, an endoscopic observation section (not shown) with an observation section and an illumination section is mounted on the front end body 50. The observation section includes an objective lens and an imaging element. The endoscopic observation section is located behind the ultrasonic probe 62 (on the side of the hand-operated section 22).

[0062] Endoscopic system 2 is capable of acquiring (capturing) endoscopic images (optical images) and ultrasound images through the aforementioned structure. Furthermore, endoscopic system 2 can also acquire endoscopic images or ultrasound images from recording unit 120, a server (not shown), or a database.

[0063] [Image processing device]

[0064] Figure 2 This is a block diagram showing the main structural components of an ultrasonic processor device.

[0065] Figure 2The ultrasound processor device 12 (image processing device, processor, camera device) shown is a device that identifies regions of interest (objects) within medical images based on acquired time-series medical images and displays (notifies) the identification results on a display device. It comprises a transceiver unit 100 (processor, image acquisition unit), an image generation unit 102 (processor, image acquisition unit), a CPU 104 (processor, CPU: Central Processing Unit), a region of interest identification unit 106 (processor, identification unit), a recording control unit 108 (processor, recording control unit), a communication control unit 110 (processor), a display control unit 112 (processor, display control unit), a memory 118, and a recording unit 120 (recording device, memory). The processing of these units, as described below, is implemented by one or more processors.

[0066] The CPU 104 operates based on various programs, including the image processing program according to the present invention, stored in the memory 118 or the recording unit 120. It uniformly controls the region of interest identification unit 106, the recording control unit 108, the communication control unit 110, the display control unit 112, the notification information determination unit 114, the notification location determination unit 116, and the notification method determination unit 117, and functions as a part of each of these units. The memory 118 includes a non-volatile recording medium such as a ROM (Read Only Memory) for recording the image processing program according to the present invention, and a temporary recording medium such as RAM (Random Access Memory) used as a temporary storage area.

[0067] The transceiver unit 100 and the image generation unit 102, which function as image acquisition units, acquire time-series medical images (image acquisition and processing, image acquisition process).

[0068] The transmitting unit of the transceiver unit 100 generates multiple drive signals to be applied to multiple ultrasonic transducers of the ultrasonic probe 62 of the ultrasonic observer 10. Based on the transmission delay mode selected by the scanning control unit (not shown), the multiple drive signals are given their respective delay times and applied to the multiple ultrasonic transducers.

[0069] The receiving unit of the transceiver unit 100 amplifies multiple detection signals output from the multiple ultrasonic transducers of the ultrasonic probe 62, converting the analog detection signals into digital detection signals (also known as RF (Radio Frequency) data). This RF data is then input to the image generation unit 102.

[0070] The image generation unit 102 assigns a delay time to each of the multiple detection signals represented by RF data based on the reception delay mode selected by the scanning control unit, and performs reception focusing processing by adding these detection signals together. Through this reception focusing processing, acoustic ray data with a reduced focus of the ultrasonic echo is formed.

[0071] The image generation unit 102 corrects for attenuation due to distance in the acoustic data based on the depth of the ultrasonic wave reflection location using STC (Sensitivity Time Control). Then, it performs envelope detection processing using a low-pass filter or similar means to generate envelope data. One frame, or more preferably multiple frames, of envelope data is stored in a movie memory (not shown). The image generation unit 102 performs preprocessing on the envelope data stored in the movie memory, such as Logarithmic compression or gain adjustment, to generate a B-mode image.

[0072] In this way, the transceiver unit 100 and the image generation unit 102 acquire time-series B-mode images (hereinafter referred to as "medical images").

[0073] The region of interest identification unit 106 is a component that performs processing (detection processing / detection procedure, identification processing / identification procedure) based on medical image recognition and information related to the location of the region of interest within the medical image, and processing (classification processing / classification procedure, identification processing / identification procedure) based on the medical image to classify the region of interest into any one of multiple categories. For example, it can be constructed using a learned model (a model learned using an image set composed of images obtained from photographs of an organism) constructed through machine learning, such as CNN (Convolutional Neural Network), SVM (Support Vector Machine), or U-net (a type of FCN (Fully Convolutional Network)). In this embodiment, the region of interest is, for example, an organ or blood vessel within the medical image (a tomographic image of a B-mode image), such as the pancreas, main pancreatic duct, spleen, splenic vein, splenic artery, or bile duct.

[0074] An example of the layer structure of the region of interest identification unit 106 composed of a CNN will be explained. The CNN includes an input layer, intermediate layers, and an output layer. The input layer takes into account the medical image generated by the image generation unit 102 and outputs feature values. The intermediate layers include convolutional layers and pooling layers, which take into account the feature values ​​output by the input layer and calculate other feature values. These layers form a structure of multiple "nodes" connected by "edges," maintaining multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The output layer identifies the region of interest appearing in the input medical image based on the feature values ​​output from the intermediate layers and outputs the result.

[0075] In this example, when a time-series medical image is input, the region of interest identification unit 106 identifies (detects) the location of a region of interest for each input medical image, outputs information indicating its location (location information, first notification information), and identifies (classifies) which category (type) the region of interest belongs to, outputting information indicating the identified category (type) (category information, type information, second notification information). Furthermore, in these identifications, the region of interest identification unit 106 can output information indicating the probability of identifying the region of interest (probability information). For example, it can calculate the probability that each pixel of the medical image is a region of interest, and the probability that the region of interest belongs to a predetermined category (e.g., pancreas, main pancreatic duct, spleen, splenic vein, splenic artery, bile duct, others). The region of interest identification unit 106 can calculate such probabilities, for example, based on the output of the output layer of a CNN.

[0076] The display control unit 112 displays time-series medical images (endoscopic images, ultrasound images) acquired by the transceiver unit 100 and the image generation unit 102 on the monitor 18 (display device). In this example, a dynamic image representing an ultrasound tomography image is displayed on the monitor 18. Furthermore, the display control unit 112 displays the area of ​​interest in the medical image on the monitor 18, determined by the notification information determination unit 114, the notification location determination unit 116, and the notification method determination unit 117, respectively, along with their location, type, and method. The notification information determination unit 114 (processor, notification information determination unit) determines notification information (first notification information, second notification information) related to the area of ​​interest; the notification location determination unit 116 (processor, notification location determination unit) determines the notification location of the second notification information; and the notification method determination unit 117 (processor, notification method determination unit) determines the notification method of the second notification information.

[0077] The details of image processing using the above functions will be described later.

[0078] [Implementation of features based on various processors]

[0079] The functions of the ultrasonic processor device 12 described above can be implemented using various processors and recording media. Among these various processors are, for example, general-purpose processors that execute software (programs) to implement various functions, namely CPUs (Central Processing Units). Additionally, among the various processors described above, there are processors specifically designed for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and other processors whose circuit structures can be changed after manufacturing, i.e., Programmable Logic Devices (PLDs). When performing image processing as described in this invention, using a GPU structure is effective. Furthermore, processors with circuit structures specifically designed for performing specific processing, such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits, are also included among the "various processors" described above.

[0080] The functions of each part can be implemented by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, a single processor can implement multiple functions. As examples of a single processor performing multiple functions, firstly, there are forms such as computers, where a single processor is constructed using a combination of one or more CPUs and software, and that processor performs multiple functions. Secondly, there are forms such as System-on-Chip (SoC), where a single IC (Integrated Circuit) chip is used to implement the overall system functions. In this way, one or more of the above-mentioned processors are used as hardware structures to construct various functions. More specifically, the hardware structures of these various processors are circuits composed of circuit elements such as semiconductor components. These circuits can also be circuits that implement the above functions using logical operations such as AND, product, NOT, XOR, and combinations thereof.

[0081] When the aforementioned processor or circuit executes the software (program), the code that can be read by the computer executing the software (e.g., various processors, circuits, and / or combinations thereof constituting the ultrasonic processor device 12) is stored in a non-volatile recording medium such as ROM (Read Only Memory), and the computer refers to the software. The software pre-stored in the non-volatile recording medium includes an image processing program for executing the image processing method involved in this invention and data used during execution (data used in setting the display mode and notification mode, weight parameters used in the region of interest identification unit 106, etc.). Alternatively, the code may not be recorded in ROM, but in a non-volatile recording medium such as various optical-magnetic recording devices or semiconductor memories. When processing the software, for example, RAM (Random Access Memory) may be used as a temporary storage area, or, for example, data stored in EEPROM (Electrically Erasable and Programmable Read Only Memory) not shown may be referenced. As a "non-volatile recording medium", memory 118 or recording unit 120 may be used.

[0082] The recording unit 120 records ultrasound images and endoscopic images (medical images), detection results of the area of ​​interest, and processing conditions (conditions for detection or notification). Other information may also be recorded simultaneously. The communication control unit 110 controls the acquisition of medical images from other medical image imaging devices connected to the endoscope system 2, external servers, or databases. The recording control unit 108 controls the recording of the recording unit 120. Recording control includes recording notification information (first notification information, second notification information), notification location, notification method, ultrasound images, or images obtained by overlaying notification information onto endoscopic images.

[0083] [Image processing order]

[0084] The image processing (execution of the image processing method and image processing program involved in the present invention) in the endoscope system 2 with the above structure will be described. Figure 3 This is a flowchart illustrating the order of the image processing methods according to the first embodiment. Furthermore, the order described below can be changed as needed.

[0085] [1. Setting processing conditions]

[0086] The display control unit 112 (processor) sets the conditions required for the execution of the image processing method / image processing program based on user operations via an operation unit (keyboard, mouse, display panel, microphone, etc., not shown) and / or pre-set processing conditions (e.g., default processing conditions) (step S100: processing condition setting process). The display control unit 112 sets, for example, the display mode of position information and type information (type of text or symbol, color, etc.), the presence or absence of lead lines for type information, the presence or absence of block segmentation, and the time-averaged frame rate of pixel values. For example, the user can... Figure 4 In such a screen (not a screen showing all processing condition settings), processing conditions are set by turning radio buttons on / off, selecting from drop-down menus, and inputting values ​​via the operation unit. The display control unit 112 can display such a screen on a display device such as a monitor 18. This setting includes, for example, specifying how to display position and type information using graphics or text (type, color, etc.). Furthermore, the display control unit 112 can set processing conditions not only at the start of processing but also during the execution of subsequent steps.

[0087] [2. Acquisition of ultrasound images and identification of regions of interest]

[0088] The transceiver unit 100 and the image generation unit 102 acquire time-series ultrasound images (medical images) (step S110: image acquisition processing, image acquisition process), and the display control unit 112 causes the monitor 18 to display the acquired ultrasound images (step S120: display control processing, display control process). The region of interest identification unit 106 identifies the location and type of the region of interest in the ultrasound image (step S130: identification processing, identification process). The region of interest identification unit 106 can, for example, identify rectangles, circles, or ellipses surrounding the region of interest (which can be...). Figure 4 The center position of the area (set in the "display format of location information") is designated as the "location of the area of ​​interest", and the information representing this location (coordinates within the image, etc.) is designated as the "first notification information". Additionally, in this embodiment, information representing the type of area of ​​interest, such as organs or blood vessels (category information, type information), can be designated as the "second notification information". Hereinafter, the "first notification information" will sometimes be referred to as "location information", and the "second notification information" as "type information".

[0089] [3. Determining the notification information, location, and method]

[0090] When the region of interest identification unit 106 detects a region of interest (YES in step S140), it determines the notification information, notification location, and notification method as follows (step S150: notification information determination process / notification information determination procedure, notification location determination process / notification location determination procedure). These processes or procedures can be performed on each frame of the ultrasound image, or they can be performed based on the time changes of various parameters in the time series image, as described later. Alternatively, the image can be divided into multiple regions and the process can be performed based on the pixel values ​​in these regions.

[0091] [3.1 Determination of Notification Information]

[0092] The notification information determination unit 114 determines first notification information indicating the location of a region of interest in an ultrasound image (medical image) and second notification information indicating the type of region of interest (step S150: notification information determination processing, notification information determination step). The notification information determination unit 114 can set the centroid of the pixels identified as "regions of interest" by the region of interest identification unit 106, or the minimum / maximum value of the X and Y coordinates, etc., as the "location of the region of interest (information indicating it)". In addition, the notification information determination unit 114 can set the classification result of the region of interest by the region of interest identification unit 106 (name of organ or blood vessel (pancreas, splenic vein, portal vein, etc.), whether it is a lesion area, or a treated area, etc.) as the "type of region of interest".

[0093] [3.2(1) Determining the Notification Location of Category Information]

[0094] The notification location determination unit 116 determines the notification location (notification location determination process, notification location determination step) of the type information (second notification information) of the region of interest in the image based on the pixel values ​​within the region surrounding the region of interest in the ultrasound image (medical image). For example, the notification location determination unit 116 can determine the centroid of a region with a pixel value above a threshold, or the centroid of a region with a pixel value below a threshold, as the "notification location of the type information". Furthermore, the notification location determination unit 116 can determine a location other than the centroid as the notification location, and can also notify of the type information of a region of interest outside that region of interest (see reference to...). Figures 5-8 (See the example shown later). Furthermore, the "region surrounding the area of ​​interest" can be any shape, such as a rectangle, polygon, circle, or ellipse (in...). Figure 4 In the example, it is illustrated as a "display format of location information" (see diagram).

[0095] The notification location determination unit 116 can also determine the notification location of type information based on the type of the region of interest. For example, when the region of interest is the pancreas, the notification location determination unit 116 can determine the notification location by avoiding regions with low echo values ​​(pixel values), such as the vascular system. Conversely, when the region of interest is the vascular system, it can explore regions with low echo values ​​to determine the notification location.

[0096] [3.2(2) Determination of Notification Location Based on Probability Information]

[0097] The notification location determination unit 116 (processor) can also calculate probability information representing the probability of identifying a region of interest, and determine the notification location based on the pixel value and the probability information. For example, the notification location determination unit 116 can determine the centroid of pixels with pixel values ​​such as echo values ​​above a reference value and a probability above a reference value that belongs to a specific type of region of interest as the notification location. Here, the region of interest identification unit 106 can, for example, calculate the probability information based on the output of the output layer of a CNN.

[0098] [3.3 Determining the Notification Method]

[0099] The notification method determination unit 117 (processor) determines the notification method (notification method determination process, notification method determination step) of the type information (second notification information). The "notification method" includes, for example, whether to use text, graphics, or symbols for the notification, the color scheme, and whether to use overlapping displays. The notification method determination unit 117 determines the notification method based on the user's request, such as... Figure 4 The operation shown on the settings screen determines the notification method.

[0100] [Processing Time Series Images]

[0101] In step S150, the notification information determination unit 114, the notification location determination unit 116, and the notification method determination unit 117 can also process and notify based on the temporal changes of various conditions (pixel values, size of the region of interest, probability information, etc.) in the time-series image acquired in the image acquisition process (image acquisition step). Specifically, the notification information determination unit 114 and the notification location determination unit 116 can determine the notification information (first notification information, second notification information) and the notification location based on the temporal changes of the pixel values, respectively. The notification location determination unit 116 and the notification method determination unit 117 can determine the notification location and the notification method based on the temporal changes of the size of the region of interest in the ultrasonic image, respectively.

[0102] For example, when the shape and pixel values ​​of the area of ​​interest change over time, the notification location shifts significantly, potentially weakening the user's concentration. Therefore, it's advisable to place the notification at a location where the time-varying changes are minimal. The benchmark for considering time-varying changes could be the difference or change in pixel values ​​over time, or it could be based on the shape invariance obtained from recognition scores, etc.

[0103] The notification location determination unit 116 can also determine the notification location based on the time-varying probability information. Here, "based on time-varying probability" includes, for example, considering the average, maximum, or minimum values ​​of various conditions within a determined period (number of frames) for the notification information determination unit 114, notification location determination unit 116, and notification method determination unit 117. Furthermore, the notification information determination unit 114, notification location determination unit 116, and notification method determination unit 117 can also consider whether frames with values ​​above a threshold for various conditions are consecutive within a determined period (number of frames).

[0104] In this way, by considering the temporal changes of various conditions, it is possible to reduce the possibility that frequent changes in notification positions due to temporal variations in pixel values, sizes, etc., may hinder observation; the possibility of inappropriate notifications due to misidentification of areas of interest; or the possibility of no notification due to missed identification. Furthermore, the period (frame number) for considering temporal changes can be, for example, via... Figure 4 Use the screen shown to set it up.

[0105] [4. Display and Recording]

[0106] The display control unit 112 (processor) determines the notification method of the first notification information and the second notification information based on the conditions set in step S100, overlays the first notification information and the second notification information on the ultrasonic image, and displays them on the monitor 18 (display device) (step S160: display control processing, display control process). Additionally, the recording control unit 108 (processor) overlays the first notification information and the second notification information on the ultrasonic image and records them in the recording unit 120 (recording device) (step S160: recording control processing, recording control process). Furthermore, the recording control unit 108 records the type information (second notification information) and / or the notification position of the type information in the recording unit 120 (step S170: recording control processing, recording control process). Furthermore, the processing of steps S110 to S170 is repeated until it becomes YES in step S180 (for example, when the user performs a recording end operation or when processing of all recorded images is finished, this determination becomes YES).

[0107] [Notification Example (Display Example)]

[0108] Figure 5This is a diagram showing an example (display example) of the first and second notification messages. In Figure 5 In the example shown, an ultrasound image 502 is displayed on the screen 500 of monitor 18. The display control unit 112 overlays a rectangle 520 (boundary box, first notification information) indicating the location of the region of interest 510 and a graphic 530 (second notification information) indicating the type of the region of interest 510 on the ultrasound image 502. The location of the region of interest 510 is the center of the rectangle 520. Furthermore, the type of the region of interest 510 is indicated by the graphic 530, which uses the initial letter "P" of the word "pancreas". The notification position of the graphic 530 (second notification information), i.e., the notification position of the type of the region of interest 510, is the center of gravity of the area surrounding the region of interest 510, i.e., the area inside the rectangle 520, where the pixel value is above a reference value (shown in dark gray). In this way, the user's gaze can be confirmed without significantly moving away from the region of interest. Furthermore, the notification position of the type of the region of interest can also be determined using the pixel values ​​within the area surrounding the region of interest and the camera device's setting information. Examples of camera device setting information include the measurement mode (B mode, M mode, etc.) of the ultrasonic transducer 10 or the illumination mode and magnification of the endoscope's light source. For instance, by changing the aforementioned reference values ​​according to the measurement mode of the ultrasonic transducer 10 and the illumination mode of the light source device 16 of the endoscope system 2, the notification location of the type of area of ​​interest can be determined based on the camera device setting information. By determining the notification location of the type of area of ​​interest based on the pixel values ​​within the area surrounding the area of ​​interest and the camera device setting information, the user can visually identify the type information even when brightness and contrast change significantly due to the camera device settings. In addition to the pixel values ​​within the area surrounding the area of ​​interest, the notification location of the type of area of ​​interest can also be determined using the monitor 18 setting information.

[0109] Figure 6 This is a diagram representing another notification example (display example) of the first and second notification messages. In Figure 6 In the example shown, the control unit 112 displays a symbol 532 overlapping the center of an area illustrated in dark gray, and text 534 (Pancreas) representing category information (second notification information) overlapping the other end of a line with the symbol 532 as one end. By displaying the text 534 outside the area of ​​interest 510, the area of ​​interest 510 is not obscured.

[0110] When a notification of recognition results is sent, the placement of the first and second notification messages is crucial. If the notification messages overlap with the observation area within the region of interest, it can hinder detailed observation. Conversely, if the notification extends beyond the region of interest, the user may be confused about which part of the recognition result the notification message represents. Therefore, it is preferable that the notification location determination unit 116 determines the location of the notification recognition result based on pixel values, specifying whether it is an observation area or a non-observation area. Alternatively, the notification location determination unit 116 can directly use pixel values, as shown below. Figure 7 The region surrounding the area of ​​interest is divided into multiple segments, and the notification location for category information is determined based on the pixel values ​​within the segmented regions. Alternatively, the most frequent value in the distribution of pixel values ​​within the area of ​​interest can be used. The notification location determination unit 116 can also set a reference value for the pixel value used to specify whether it is an observed or non-observed region based on the user's operation. Alternatively, for example, the notification location determination unit 116 can set a reference value for the pixel value based on the user's operation, and within the region surrounding the area of ​​interest, determine the notification location for the category of the area of ​​interest in the region where the pixel value is higher than or equal to the set reference value. Furthermore, the notification location determination unit 116 can also determine the notification location for the category of the area of ​​interest based on the pixel values ​​within the region surrounding the area of ​​interest and the detected category of the area of ​​interest. For example, a reference range of pixel values ​​can be preset according to the category of the area of ​​interest, and the notification location for the category of the area of ​​interest can be determined within the region surrounding the area of ​​interest in the region where the pixel values ​​are within the reference range. Thus, notification can be performed at an appropriate notification location based on the category of the area of ​​interest. Furthermore, the notification location determination unit 116 can also preset a reference value according to the color of the pixel, and determine the notification location for the category of the area of ​​interest based on the reference value of a specific color.

[0111] Figure 7 This is a diagram illustrating a further notification (display example) that represents the first and second notification messages. In Figure 7 In the example shown, the notification location determination unit 116 divides the region of rectangle 520 into six columns and four rows along the horizontal and vertical directions of the image, respectively, and determines the location of the category information in the lower left segmented area where the average pixel value is high. Furthermore, the display control unit 112 and... Figure 6 Similarly, at the other end of the line with symbol 532 as one end, text 534 (Pancreas) representing category information (second notification information) is superimposed. Furthermore, the notification information determination unit 114, the notification location determination unit 116, and the notification method determination unit 117 (processor) can determine the notification method based on the user's... Figure 4 The settings screen shown can be used to configure the settings as follows: Figure 7 The rectangle 520 shown is divided into several patterns.

[0112] [Determining, displaying, and recording notification locations based on the type of area of ​​interest]

[0113] Figure 8 This is a diagram showing the location of a notification where category information (secondary notification information) has been determined based on the type of area of ​​interest. Figure 8 In the example, the ultrasound image 502 shows a region of interest 510 (pancreas) and a region of interest 512 (splenic vein). Region 510 is indicated by a rectangle 522 (boundary box, first notification information) representing its location and text 536 (an abbreviation of "Panc.") representing its type. Text 536 overlaps the center (center of gravity) of region 510. On the other hand, region 512 is indicated by a rectangle 524 (boundary box, first notification information) representing its location and text 540 (an abbreviation of "splenic vein," SV) representing its type. Here, text 540 does not overlap with region 512, but is displayed at the other end of a line that overlaps with the center (center of gravity) of region 512, ensuring that text 540 does not obscure region 512. Thus, in the endoscope system 2, type information (second notification information) can be notified at an appropriate notification location based on the type of region of interest.

[0114] As described above, the endoscope system 2 according to the first embodiment can appropriately notify the identification results of the area of ​​interest.

[0115] [Applications to other medical images]

[0116] In the first embodiment described above, the identification and notification of ultrasound endoscopic images as a type of medical image were explained. However, the image processing apparatus, image processing system, image processing method, and image processing program of the present invention can also be applied to cases where ultrasound images are acquired by ultrasound devices other than endoscopes (such as surface endoscopes), or endoscopic images are acquired by optical endoscopes that capture images of the subject using ordinary light (white light) and / or special light (narrow band light, etc.), or medical images other than ultrasound endoscopic images are acquired by CT devices, MRI devices, mammography devices, etc.

[0117] For example, when using endoscopic images acquired by an optical endoscope, the endoscopic processor device 14 (image processing device, processor) of the endoscopic system 2 described above can perform the same processing as the ultrasonic processor device 12 in the first embodiment. Furthermore, when using endoscopic images acquired by an optical endoscope, one or more image signals such as red, green, and blue can be used when determining and notifying notification information, notification location, and notification method. For example, sometimes the normal portion of the inner wall of the lumen appears red while the diseased portion (area of ​​interest) appears black. In this case, the weight of the value (pixel value) of the image signal of a specific color can be increased (or decreased) to notify the part of that specific color, or to notify the part of the specific color that is avoided.

[0118] [Applications of images beyond medical imaging]

[0119] The image processing apparatus, image processing system, image processing method, and image processing program of the present invention can also be applied to images other than medical images. For example, the present invention can also be applied to images obtained by industrial endoscopes used to inspect the internal damage and deformation of tubular structures such as piping. Furthermore, the present invention can also be applied to inspecting damage and deformation of structures such as bridges, roads, and tunnels from images obtained from photographs of such structures. In these cases, the type of damage or deformation can be considered as "type information."

[0120] In addition, the present invention can also be applied to the identification and notification of the location and type of the subject (road, person, vehicle, building, etc.) in images captured by vehicle-mounted cameras or surveillance cameras, such as the case where the type information is overlaid to avoid human faces (in which case the subject is a person).

[0121] The embodiments and other examples of the present invention have been described above, but the present invention is not limited to the above-described manner and various modifications can be made.

[0122] Symbol Explanation

[0123] 2. Endoscopic System

[0124] 10. Ultrasonic Observation Device

[0125] 12. Ultrasonic processing device

[0126] 14. Processor device for endoscopes

[0127] 16 Light source device

[0128] 18 monitors

[0129] 20 Insertion section

[0130] 20a Long Axis

[0131] 22 Hands-on Operations Department

[0132] 24 Universal Rope

[0133] 26 Connectors for Ultrasonic Applications

[0134] 28. Connector for endoscopes

[0135] 30 Connector for Light Source

[0136] 32 tubes

[0137] 34 tubes

[0138] 36 Gas and water supply buttons

[0139] 38. Attraction Button

[0140] 42 Bent-angle button

[0141] 44. Treatment device insertion port

[0142] 50 Front end body

[0143] 52. Bend

[0144] 54. Soft parts

[0145] 62 Ultrasonic probe

[0146] 64 cysts

[0147] 70 Water Supply Tank

[0148] 72 Suction Pump

[0149] 100 Receiving and Dispatch Department

[0150] 102 Image Generation Unit

[0151] 104 CPU

[0152] 106. Area of ​​Concern Identification Department

[0153] 108 Record Control Department

[0154] 110 Communications Control Department

[0155] 112 Display Control Unit

[0156] 114 Notification Information Confirmation Department

[0157] 116 Notification of Location Determination Department

[0158] 117 Notification Method Determination Department

[0159] 118 Memory

[0160] 120 Recording Department

[0161] 500 frames

[0162] 502 Ultrasound Image

[0163] 510 Area of ​​Concern

[0164] 512 Areas of Concern

[0165] 520 Rectangle

[0166] 522 Rectangle

[0167] 524 Rectangles

[0168] 530 graphics

[0169] 534 text

[0170] 536 text

[0171] 540 text

[0172] Steps of the S100~S180 image processing method

Claims

1. An image processing apparatus comprising a processor, wherein, The processor executes: Image acquisition and processing; acquiring images. The recognition process identifies regions of interest from the image and calculates probability information representing the probability of recognizing the regions of interest. The notification information determination process determines a first notification message indicating the location of the region of interest in the image and a second notification message indicating the type of the region of interest. as well as The notification location determination process, based on the probability information and the pixel values ​​within the region surrounding the region of interest in the image, determines the notification location in the image where the second notification information is sent. The notification location is within the area of ​​interest, and the second notification information is superimposed on the area of ​​interest.

2. The image processing apparatus according to claim 1, wherein, In the notification location determination process, the processor further determines the notification location based on the type of the region of interest.

3. The image processing apparatus according to claim 1 or 2, wherein, In the notification location determination process, the processor determines the center of gravity of the region of interest as the notification location.

4. The image processing apparatus according to claim 1, wherein, The processor acquires time-series images during the image acquisition process. In the notification location determination process, the processor determines the notification location based on the temporal changes of the probability information in the time-series image.

5. The image processing apparatus according to claim 4, wherein, In the notification location determination process, the processor determines the notification location based on the temporal changes of the pixel values ​​in the time-series image.

6. The image processing apparatus according to claim 1 or 2, wherein, The processor further executes a notification method determination process to determine the notification method of the second notification information.

7. The image processing apparatus according to claim 6, wherein, The processor acquires time-series images during the image acquisition process. In the notification method determination process, the processor determines the notification method based on the temporal changes of the pixel values ​​in the time-series image.

8. The image processing apparatus according to claim 6, wherein, The processor acquires time-series images during the image acquisition process. In the notification location determination process, the processor determines the notification location based on the temporal change in the size of the region of interest in the time-series image. In the notification method determination process, the processor determines the notification method based on the temporal change in the size of the region of interest in the time-series image.

9. The image processing apparatus according to claim 7, wherein, In the notification location determination process, the processor determines the notification location based on the temporal change in the size of the region of interest in the time-series image. In the notification method determination process, the processor determines the notification method based on the temporal change in the size of the region of interest in the time-series image.

10. The image processing apparatus according to claim 1 or 2, wherein, The processor overlays the first notification information and the second notification information onto the image and records them in the recording device.

11. The image processing apparatus according to claim 1 or 2, wherein, The processor records the second notification information and / or the notification location in the recording device.

12. The image processing apparatus according to claim 1 or 2, wherein, In the image acquisition process, the processor acquires medical images of the subject.

13. An image processing system comprising: The image processing apparatus according to any one of claims 1 to 12; and A camera device that captures the image.

14. The image processing system according to claim 13, wherein, The camera device is an endoscopic observer.

15. The image processing system according to claim 14, wherein, The endoscopic observer is an ultrasonic endoscopic observer.

16. The image processing system according to any one of claims 13 to 15, wherein, The processor overlays and displays the image, the first notification information, and the second notification information on the display device.

17. The image processing system according to claim 16, wherein, It also includes the aforementioned display device.

18. An image processing method, performed by an image processing apparatus equipped with a processor, wherein, The processor executes: The image acquisition process involves acquiring an image. The identification process involves identifying regions of interest from the image and calculating probability information representing the probability of identifying the regions of interest. The notification information determination process involves determining first notification information indicating the location of the region of interest in the image and second notification information indicating the type of the region of interest. as well as The notification location determination step, based on the probability information and the pixel values ​​within the region surrounding the region of interest in the image, determines the notification location in the image where the second notification information is to be sent. The notification location is within the area of ​​interest, and the second notification information is superimposed on the area of ​​interest.

19. A recording medium that is non-volatile and computer-readable, wherein, The document contains a program that causes a computer to perform the image processing method of claim 18.

Images