Medical image processing device, medical image processing method, diagnosis auxiliary device and recording medium

Abstract

The present invention provides a medical image processing device, a medical image processing method and program, and a diagnosis assisting device that appropriately control the notification of the presence or absence of notification information of a medical image without relying on user operation. The above-mentioned problem is solved by a medical image processing device, which has a notification control unit, and the notification control unit controls the notification information contained in the medical image to any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. The notification control unit sets the notification information to the non-notification state when the medical image fully satisfies the non-notification condition, and sets it to the notification state after a non-notification maintenance time has passed since the medical image does not fully satisfy the non-notification condition.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, a medical image processing method and program, and a diagnosis assistance device, and in particular to a technology for controlling the notification of notification information of the presence or absence of medical images. Background Art

[0002] Technology that automatically detects areas of interest, such as lesions, from endoscopic images is expected to prevent missed lesions. Ultimately, the purpose of automatic detection technology is to prevent missed lesions. Continuing to report detection results even after the user has recognized the presence of a lesion is undesirable, as this would hinder observation. Therefore, a non-notification mechanism could be considered, whereby notification information is intentionally withheld after user recognition.

[0003] However, the user may continue to perform the examination without noticing that the notification is set to non-notification. In this case, the user may miss the lesion because the notification information is not notified.

[0004] To address this issue, Patent Document 1 discloses a technology that, by clicking a hidden icon, hides the markers (equivalent to notification information) attached to a CT (Computed Tomography) image for a candidate lesion shadow. After a predetermined time has passed, the image returns to its original marked state. This technology prevents the examination from continuing without the markers, even when the markers are hidden.

[0005] Previous technical literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 4393016 Summary of the Invention

[0008] Technical issues to be solved by the invention

[0009] However, the technology described in Patent Document 1 has a disadvantage in that a user operation is required to hide the mark.

[0010] The present invention is completed in view of this situation, and its purpose is to provide a medical image processing device, medical image processing method and program, and diagnosis assistance device that appropriately control the notification of the presence or absence of medical image notification information without relying on user operation.

[0011] Means for solving technical problems

[0012] In order to achieve the above-mentioned purpose, one aspect of the medical image processing device is a medical image processing device as follows, which has a notification control unit, which controls the notification information contained in the medical image to any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. The notification control unit sets the notification information to the non-notification state when the medical image fully satisfies the non-notification condition, and sets the notification information to the notification state after the non-notification maintenance time has passed since the medical image becomes insufficient to satisfy the non-notification condition.

[0013] According to this aspect, the notification information is set to the non-notification state when the medical image fully satisfies the non-notification condition, and is set to the notification state after the non-notification maintenance time has passed since the medical image no longer satisfies the non-notification condition. Therefore, the notification state is set when the non-notification condition is not fully satisfied, and the non-notification state is set when the non-notification condition is fully satisfied. The non-notification state is maintained from the time the non-notification condition is not fully satisfied until the non-notification maintenance time has passed, and the notification state is set after the non-notification maintenance time has passed. Therefore, the presence or absence of notification information of the medical image can be appropriately controlled without relying on user operation.

[0014] The apparatus preferably includes: an image acquisition unit that sequentially acquires medical images; a notification information acquisition unit that acquires notification information from the medical images; a determination unit that determines whether the medical images sufficiently satisfy the non-notification condition; and a timing unit that measures the time elapsed since the medical image ceased to sufficiently satisfy the non-notification condition. This allows for appropriate acquisition of medical images, acquisition of notification information, determination of the non-notification condition, and determination of the elapsed time.

[0015] Preferably, the non-notification condition is a condition for determining an image feature value of a medical image, and the medical image processing apparatus includes an image feature value acquisition unit for acquiring the image feature value from the medical image.

[0016] Preferably, the image feature includes at least one of brightness, color information, time variation, and frequency information of the medical image.

[0017] Preferably, the non-notification condition is a condition for determining the treatment state of the subject in the medical image, and the medical image processing apparatus includes a treatment state estimating unit for estimating the treatment state based on the medical image.

[0018] It is preferable to include a non-notification maintenance time setting unit for setting the non-notification maintenance time according to the treatment state.

[0019] Preferably, the non-notification condition is a condition for determining a region of interest feature value, and the medical image processing apparatus includes: a region of interest detection unit for detecting a region of interest from a medical image; and a region of interest feature value acquisition unit for acquiring a region of interest feature value from the region of interest. Thus, notification can be stopped based on the region of interest feature value.

[0020] Preferably, the feature value of the region of interest includes at least one of the area of ​​the region of interest, the position in the medical image, and the temporal change.

[0021] Preferably, the notification unit includes a sound output unit that outputs sound, and the notification control unit includes a sound control unit that causes the sound output unit to output sound.

[0022] Preferably, the notification unit includes a first display unit, and the notification control unit includes a display control unit that causes the first display unit to display the notification information.

[0023] Preferably, the notification unit includes a second display unit different from the first display unit, and the display control unit causes the second display unit to display the notification information.

[0024] Preferably, the display control unit changes the display mode of the notification information on the second display unit according to the notification state and the non-notification state.

[0025] Preferably, the display control unit causes the second display unit to display the notification information when the notification state is not in progress. Thus, medical images can be appropriately observed on the first display unit, and notification information can be appropriately displayed on the second display unit.

[0026] Preferably, the display control unit causes a third display unit different from the first display unit to display information indicating that the notification state is non-notification state.

[0027] Preferably, the display control unit causes a fourth display unit different from the first display unit to display information related to the time elapsed since the medical image became insufficient to satisfy the non-notification condition. Thus, the user can know the information related to the elapsed time.

[0028] To achieve the above object, one aspect of a diagnosis support device is a diagnosis support device comprising: the above-mentioned medical image processing device; and a first display unit. According to this aspect, the display of notification information regarding the presence or absence of medical images can be appropriately controlled without relying on user operation.

[0029] In order to achieve the above-mentioned purpose, one method of medical image processing is a medical image processing method as follows: it has a notification control process, which controls the notification information contained in the medical image to be any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. In the notification control process, the notification information is set to the non-notification state when the medical image fully meets the non-notification condition, and the notification information is set to the notification state after the non-notification maintenance time has passed since the medical image becomes insufficient to meet the non-notification condition.

[0030] According to this aspect, the presence or absence of notification information of medical images can be appropriately controlled without depending on user operations. A program for causing a computer to execute the above-mentioned medical image processing method is also included in this aspect.

[0031] Effects of the Invention

[0032] According to the present invention, the presence or absence of notification information of medical images can be appropriately controlled without depending on user operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram showing the overall structure of an endoscope system including a medical image processing device.

[0034] Figure 2 This is a block diagram showing an example of the electrical structure of a medical image processing device.

[0035] Figure 3 This is a flowchart showing an example of each process of the medical image processing method.

[0036] Figure 4 This is a diagram showing display transitions on a display.

[0037] Figure 5 This is a diagram showing an example of display on the display in the notification state.

[0038] Figure 6 This is a diagram showing an example of display on the display in the notification state.

[0039] Figure 7 This is a diagram showing an example of display on the display in the notification state.

[0040] Figure 8 This is a diagram showing an example of display on the display in the notification state.

[0041] Figure 9 This is a diagram showing display transitions on a display.

[0042] Figure 10 This is a diagram showing display transitions on a display.

[0043] Figure 11 This is a diagram showing display transitions on a display.

[0044] Figure 12 This is a diagram showing display transitions on a display.

[0045] Figure 13 This is a block diagram showing the electrical structure of a medical image processing device.

[0046] Figure 14 This is a diagram showing an example of display transition between the first display and the second display.

[0047] Figure 15 This is a diagram showing another example of display transition between the first display and the second display.

[0048] Figure 16 This is a diagram showing another example of display transition between the first display and the third display.

[0049] Figure 17 This is a diagram showing another example of display transition between the first display and the fourth display.

[0050] Figure 18 This is a block diagram showing an example of the electrical structure of a medical image processing device. DETAILED DESCRIPTION

[0051] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0052] <Overall Structure of Endoscope System>

[0053] Figure 1 1 is a schematic diagram showing the overall structure of an endoscope system 9 including a medical image processing device according to this embodiment. Figure 1 As shown, the endoscope system 9 includes an endoscope 10 as an electronic endoscope, a light source device 11 , an endoscope processor device 12 , a display device 13 , a medical image processing device 14 , an operation unit 15 , and a display 16 .

[0054] An endoscope 10 is used to capture time-series medical images and is, for example, a flexible endoscope. The endoscope 10 comprises an insertion portion 20, which is inserted into a subject and has a distal end and a proximal end; a hand-operated control unit 21, which is connected to the proximal end of the insertion portion 20 and is held by a user (physician) to perform various operations; and a universal cord 22, which is connected to the hand-operated control unit 21.

[0055] The insertion portion 20 is thin and elongated. It is composed of a flexible portion 25, a curved portion 26, and a distal portion 27, arranged in sequence from its base end to its distal end. The flexible portion 25 is flexible, the curved portion 26 is bendable by operating the hand-operated operating portion 21, and the distal portion 27 houses an imaging optical system (objective lens) and an imaging element 28 (not shown).

[0056] Image sensor 28 is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type image sensor. Image light from the observed area is incident on the imaging surface of image sensor 28 via an observation window (not shown) opened at the top surface of tip portion 27 and an objective lens (not shown) positioned behind the observation window. Image sensor 28 captures the image light from the observed area incident on the imaging surface (converting it into an electrical signal) and outputs an imaging signal.

[0057] The hand-side operation section 21 is equipped with various operating components operated by the user. Specifically, the hand-side operation section 21 includes two bending operation buttons 29 for bending the bending portion 26, an air and water supply button 30 for operating the air and water supply, and a suction button 31 for suctioning. Furthermore, the hand-side operation section 21 is equipped with a still image capture instruction section 32 and a treatment instrument inlet 33. The still image capture instruction section 32 is used to instruct the user to capture a still image 39 of the observed area. A treatment instrument (not shown) is inserted through the treatment instrument inlet 33 into a treatment instrument insertion passage (not shown) extending through the insertion section 20.

[0058] The universal cord 22 is a connection cord used to connect the endoscope 10 to the light source device 11. The universal cord 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown), which are inserted through the insertion portion 20. Furthermore, a connector 37a is provided at the end of the universal cord 22, which connects to the light source device 11. A connector 37b branches from the connector 37a and connects to the endoscope processor device 12.

[0059] By connecting the connector 37a to the light source device 11, the light guide 35 and the fluid tube (not shown) are inserted into the light source device 11. This allows the light source device 11 to supply the required illumination light, water, and gas to the endoscope 10 via the light guide 35 and the fluid tube (not shown). As a result, illumination light can be irradiated onto the observed area from the illumination window (not shown) on the top surface of the tip portion 27. Furthermore, by pressing the air and water supply button 30, air or water can be sprayed from the air and water supply nozzle (not shown) on the top surface of the tip portion 27 toward the observation window (not shown).

[0060] By connecting the connector 37b to the endoscope processor 12, the signal cable 36 can be electrically connected to the endoscope processor 12. Thus, an image signal of the observed part can be output from the imaging element 28 of the endoscope 10 to the endoscope processor 12 via the signal cable 36, and a control signal can be output from the endoscope processor 12 to the endoscope 10.

[0061] The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. The illumination light can be selected from various wavelength bands depending on the purpose of observation, such as white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or a combination of these. The specific wavelength band is a band narrower than the white wavelength band.

[0062] A first example of a specific wavelength band is, for example, a blue band or a green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm to 450 nm or a wavelength band of 530 nm to 550 nm, and the light of the first example has a peak wavelength in the wavelength band of 390 nm to 450 nm or a wavelength band of 530 nm to 550 nm.

[0063] A second example of a specific wavelength band is, for example, a red band in the visible range. The wavelength band in this second example includes a wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm, and the light in this second example has a peak wavelength within the wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm.

[0064] A third example of the specific wavelength band includes a wavelength band in which the absorption coefficients of oxyhemoglobin and deoxyhemoglobin differ, and the light of the third example has a peak wavelength in the wavelength band in which the absorption coefficients of oxyhemoglobin and deoxyhemoglobin differ. The wavelength band of this third example includes 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm, and the light of the third example has a peak wavelength in any of the above wavelength bands.

[0065] A fourth example of the specific wavelength band is a wavelength band (390 nm to 470 nm) of excitation light used for observing fluorescence emitted by a fluorescent substance in a living body (fluorescence observation) and for exciting the fluorescent substance.

[0066] A fifth example of the specific wavelength band is a wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm.

[0067] The endoscope processor 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. In addition, the endoscope processor 12 generates a time-series frame image 38a (see FIG. 1 ) based on the imaging signal acquired from the imaging element 28 of the endoscope 10 via the connector 37b and the signal cable 36. Figure 2 ) is a time series of medical images composed of a dynamic picture 38. The frame rate of the dynamic picture 38 is, for example, 30fps (frame per second).

[0068] When the still picture capturing instruction unit 32 is operated using the handheld operation unit 21 of the endoscope 10 , the endoscope processor device 12 acquires one frame image 38 a of the dynamic picture 38 as a still picture 39 in accordance with the timing of the capturing instruction in parallel with the generation of the dynamic picture 38 .

[0069] The moving image 38 and the still image 39 are medical images captured from within the subject, i.e., the living body. Furthermore, if the moving image 38 and the still image 39 are images obtained using light in the specific wavelength band (special light), both are special light images. The endoscope processor 12 then outputs the generated moving image 38 and still image 39 to the display device 13 and the medical image processing device 14, respectively.

[0070] Alternatively, the endoscope processor 12 may generate (acquire) a special light image containing information about the specific wavelength band based on the normal light image obtained using the white light. In this case, the endoscope processor 12 functions as a special light image acquisition unit. The endoscope processor 12 then acquires a signal in the specific wavelength band by performing calculations based on the RGB color information (red, green, and blue) or the CMY color information (cyan, magenta, and yellow) contained in the normal light image.

[0071] Furthermore, the endoscope processor 12 may also generate a known characteristic value image, such as an oxygen saturation image, based on at least one of the normal light image obtained using the white light and the special light image obtained using the light in the specific wavelength band (special light). In this case, the endoscope processor 12 functions as a characteristic value image generation unit. Furthermore, each of the dynamic image 38 and the still image 39, including the in-vivo image, normal light image, special light image, and characteristic value image, is a medical image obtained by imaging the human body or visualizing measurement results for the purpose of image diagnosis or examination.

[0072] The display device 13 is connected to the endoscope processor unit 12 and displays a moving image 38 and a still image 39 input from the endoscope processor unit 12. The user performs operations such as inserting and removing the insertion portion 20 while checking the moving image 38 displayed on the display device 13. When a lesion is found in the observed area, the user operates the still image capture instruction unit 32 to capture a still image of the observed area, and also performs diagnosis, biopsy, etc.

[0073] The medical image processing device 14 automatically identifies and distinguishes lesions serving as regions of interest, and notifies the user of notification information contained in medical images. For example, a personal computer is used as the medical image processing device 14. The operating unit 15 utilizes a keyboard and mouse connected to the personal computer by wire or wireless connection, and the display 16 (an example of a notification unit) utilizes various monitors such as an LCD monitor that can be connected to the personal computer.

[0074] The medical image processing device 14 and the display 16 (an example of a first display unit) function as a diagnosis support device that displays a dynamic image 38 and notification information on the display 16 .

[0075] <First embodiment>

[0076] [Structure of medical image processing device]

[0077] Figure 2 It is a block diagram showing an example of the electrical structure of the medical image processing device 14 involved in the first embodiment. Figure 2 The medical image processing device 14 shown is mainly composed of a time series image acquisition unit 40, a region of interest detection unit 42, a coordinate calculation unit 44, an image information acquisition unit 46, a non-notification maintenance time setting unit 54, a judgment unit 56, a timing unit 58, a control unit 60, a display control unit 62, and a storage unit 64.

[0078] The control unit 60 uniformly controls the time series image acquisition unit 40, the area of ​​interest detection unit 42, the coordinate calculation unit 44, the image information acquisition unit 46, the non-notification maintenance time setting unit 54, the judgment unit 56, the timing unit 58, and the display control unit 62 based on the program (medical image processing program) 68 stored in the storage unit 64, and also functions as a part of these units.

[0079] The storage unit 64 is a storage device such as a hard disk device, and stores the detection results of the region of interest detector 42 and the captured still image 39 , as well as the program 68 and information related to various controls of the medical image processing device 14 .

[0080] Furthermore, the storage unit 64 includes a graphic storage unit 66. The graphic storage unit 66 stores graphics for notifying the user of notification information.

[0081] The time series image acquisition unit 40 sequentially acquires endoscopic images as an example of medical images. Here, the time series image acquisition unit 40 is connected to the endoscope processor device 12 (see Figure 1 ) uses an image input / output interface (not shown) of the endoscope processor 12 to acquire a dynamic image 38 (in this example, the dynamic image 38 captured by the endoscope 10) consisting of time-series frame images 38a. Furthermore, when the endoscope 10 captures the still image 39 as described above while the dynamic image 38 is being captured, the time-series image acquisition unit 40 acquires the dynamic image 38 and the still image 39 from the endoscope processor 12.

[0082] Furthermore, instead of directly acquiring the dynamic images 38 from the endoscope processor 12, the time-series image acquisition unit 40 may acquire the dynamic images 38 via various information storage media such as a memory card or a hard disk drive. Alternatively, the time-series image acquisition unit 40 may acquire the dynamic images 38 uploaded to a server or database on the Internet via the Internet.

[0083] The region of interest detection unit 42 is an example of a notification information acquisition unit that acquires notification information contained in a medical image from a medical image, and detects a region of interest from a dynamic image 38 captured during observation inside a subject. The region of interest detection unit 42 includes a convolutional neural network (CNN) that calculates feature values ​​of each frame image 38a of the dynamic image 38 (or frame images 38a that have been thinned out at a certain interval) and performs recognition processing of the region of interest within the image.

[0084] Examples of the area of ​​interest include polyps, cancer, colorectal diverticula, inflammation, EMR (Endoscopic Mucosal Resection) scars, ESD (Endoscopic Submucosal Dissection) scars, clamping sites, bleeding points, perforations, vascular atypia, and treatment instruments.

[0085] The region of interest detection unit 42 can also obtain a classification result such as a classification result of whether the detected region of interest belongs to one of a plurality of categories related to lesions such as “tumorous,” “non-tumorous,” and “other.”

[0086] Furthermore, the attention area detection unit 42 is not limited to detecting the attention area using CNN, and may also detect the attention area by analyzing feature quantities such as color, pixel value gradient, shape, and size within the image using image processing.

[0087] The coordinate calculation unit 44 is an example of a notification information acquisition unit and calculates coordinate information indicating the position of the region of interest detected by the region of interest detection unit 42 within the image. The coordinate calculation unit 44 calculates, for example, one or more coordinate information on the contour line of a polygon or circle surrounding the region of interest. The coordinate calculation unit 44 may calculate the coordinate information for the vertices of the polygon or the midpoints of the sides of the polygon, or may calculate the coordinate information for points that divide the circumference of the circle into multiple equal parts.

[0088] Image information acquisition unit 46 is an image processing unit that acquires image information from dynamic image 38. This image information is used to estimate the timing at which notification information is no longer necessary or the timing at which the reliability of the detection result of region of interest detection unit 42 decreases. Image information acquisition unit 46 includes image feature acquisition unit 48, treatment state estimation unit 50, and region of interest feature acquisition unit 52.

[0089] The image feature quantity acquisition unit 48 acquires image feature quantities, which are feature quantities of each frame image 38a of the dynamic image 38. The treatment state estimation unit 50 analyzes each frame image 38a of the dynamic image 38 through image processing and estimates the treatment state of the subject. The region of interest feature quantity acquisition unit 52 acquires the region of interest feature quantity, which is the feature quantity of the region of interest detected by the region of interest detection unit 42.

[0090] The non-notification maintenance time setting unit 54 sets the non-notification maintenance time, described later. The non-notification maintenance time is stored, for example, in the storage unit 64. The non-notification maintenance time setting unit 54 reads the non-notification maintenance time from the storage unit 64 and sets the read non-notification maintenance time. The non-notification maintenance time setting unit 54 may set the non-notification maintenance time based on the treatment status estimated by the treatment status estimation unit 50, or may set the non-notification maintenance time based on a value input by the user.

[0091] The determination unit 56 determines whether the dynamic image 38 satisfies the non-notification condition. Alternatively, the determination unit 56 may determine whether the dynamic image 38 satisfies the non-notification condition based on the image information acquired by the image information acquisition unit 46. Specifically, the determination unit 56 may determine whether the dynamic image 38 satisfies the non-notification condition based on at least one of the image feature acquired by the image feature acquisition unit 48, the treatment state estimated by the treatment state estimation unit 50, and the region of interest feature acquired by the region of interest feature acquisition unit 52.

[0092] The timer 58 measures the time elapsed since the moving picture 38 became insufficient to satisfy the non-notification condition. The timer 58 may measure the time elapsed by counting the number of frame images 38a (number of frames) since the moving picture 38 became insufficient to satisfy the non-notification condition.

[0093] The display control unit 62 controls the display on the display 16. The display control unit 62 includes an image display control unit 62A and a notification information display control unit 62B. The image display control unit 62A outputs the dynamic image 38 acquired by the time-series image acquisition unit 40 to the display 16, causing the display 16 to display the dynamic image 38. Specifically, the display 16 sequentially displays a plurality of frame images 38a.

[0094] The notification information display control unit 62B is an example of a notification control unit that notifies the notification information included in the medical image in the notification unit. The notification information display control unit 62B controls the notification information to be in any one of a notification state in which notification is performed and a non-notification state in which notification is not performed. In the notification state, the notification information display control unit 62B causes the display 16 to display information related to the area of ​​interest detected by the area of ​​interest detection unit 42 as notification information. Here, the notification information display control unit 62B causes the graphics read from the graphics storage unit 66 to overlap the position of the coordinate information calculated by the coordinate calculation unit 44 on the frame image 38a displayed on the display 16. As a result, the graphics are overlapped and displayed at the position of the area of ​​interest on the dynamic screen 38 displayed on the display 16, and the area of ​​interest is emphasized by the graphics.

[0095] Furthermore, the notification information display control unit 62B sets the notification information to a non-notification state when the dynamic image 38 satisfies the non-notification condition. Specifically, when the dynamic image 38 satisfies the non-notification condition, the notification information display control unit 62B sets the notification information to a non-notification state without displaying the graphics read from the graphics storage unit 66, thereby stopping the notification of the notification information on the display 16.

[0096] Furthermore, the notification information display control unit 62B maintains the non-notification state from the time the dynamic image 38 becomes insufficient to satisfy the non-notification condition until the non-notification maintenance time set by the non-notification maintenance time setting unit 54 has elapsed, and then switches to the notification state where the notification information is displayed after the non-notification maintenance time has elapsed. In other words, the non-notification maintenance time is the time the non-notification state is maintained from the time the notification condition becomes insufficient to satisfy.

[0097] The notification information display control unit 62B determines whether the time elapsed since the dynamic screen 38 becomes insufficient to satisfy the non-notification condition exceeds the non-notification maintenance time, and maintains the notification information in the non-notification state if the time elapsed since the condition becomes insufficient to satisfy the condition does not exceed the non-notification maintenance time.

[0098] Furthermore, if the time elapsed since the condition became insufficiently satisfied exceeds the non-notification maintenance time, the notification information display control unit 62B sets the notification information to the notification state. Here, the notification information display control unit 62B superimposes a graphic read from the graphic storage unit 66 on the position of the coordinate information calculated by the coordinate calculation unit 44 within the frame image 38a displayed on the display 16.

[0099] Furthermore, if a condition opposite to the non-notification condition is set as the notification condition, it can be interpreted that the notification information display control unit 62B sets the notification information to the notification state when the dynamic image 38 fully satisfies the notification condition, and sets the notification information to the non-notification state when the dynamic image 38 does not fully satisfy the notification condition. In this case, the notification information display control unit 62B maintains the non-notification state from the time the dynamic image 38 fully satisfies the notification condition until the non-notification maintenance time has elapsed. Furthermore, the notification information display control unit 62B sets the notification state after the non-notification maintenance time has elapsed from the time the dynamic image 38 fully satisfies the notification condition.

[0100] [Medical Image Processing Methods]

[0101] Next, a description will be given of a medical image processing method using the medical image processing apparatus 14. The medical image processing method is implemented by the control unit 60 executing the program 68 stored in the storage unit 64.

[0102] In this embodiment, the notification information is the location of the region of interest, and the medical image processing apparatus 14 displays a graphic indicating the location of the region of interest on the display 16. The notification information may be the presence or absence of the region of interest or the result of classification of the detected region of interest.

[0103] Figure 3This is a flowchart showing an example of each process of the medical image processing method involved in the first embodiment. The medical image processing method includes an image acquisition process (step S1), an image display process (step S2), a region of interest detection process (step S3), a determination process (step S5), a timing process (step S7), and a notification control process (steps S9 and S10).

[0104] In step S1, the time-series image acquisition unit 40 acquires the frame image 38a of the dynamic picture 38. In step S2, the image display control unit 62A causes the display 16 to display the frame image 38a acquired in step S1.

[0105] In step S3, the attention area detection unit 42 detects an attention area from the frame image 38a acquired in step S1. In step S4, the control unit 60 determines whether an attention area has been detected from the frame image 38a in step S3.

[0106] If it is determined in step S4 that the region of interest is not detected, the process of this flowchart is terminated. In this case, since there is no notification information, the notification information is not notified (displayed). Therefore, the graphic indicating the position of the region of interest is not displayed on the display 16.

[0107] On the other hand, if it is determined in step S4 that the attention area is detected, the process proceeds to step S5. In step S5, the determination unit 56 determines whether the frame image 38a acquired in step S1 sufficiently satisfies the non-notification condition.

[0108] If it is determined in step S5 that frame image 38a satisfies the non-notification condition, the process proceeds to step S9. In step S9, the notification information display control unit 62B sets the notification information to the non-notification state (non-display state), terminating the process of this flowchart. Therefore, the graphic indicating the position of the region of interest is not displayed on the display 16.

[0109] On the other hand, if it is determined in step S5 that the frame image 38a does not sufficiently satisfy the non-notification condition, the process proceeds to step S6. In step S6, the determination unit 56 determines whether the frame image 38a preceding the frame image 38a acquired in step S1 sufficiently satisfies the non-notification condition.

[0110] If it is determined in step S6 that the previous frame image 38a does not sufficiently satisfy the non-notification condition, the process proceeds to step S10. In step S10, the notification information display control unit 62B sets the notification information to the notification state (display state), terminating the process of this flowchart. Consequently, a graphic is superimposed on the position of the target area of ​​the frame image 38a displayed on the display 16.

[0111] If it is determined in step S6 that the previous frame image 38a satisfies the non-notification condition, the process proceeds to step S7. In step S7, the timer unit 58 measures the time elapsed since the dynamic image 38 no longer satisfies the non-notification condition.

[0112] In the next step S8, the notification information display control unit 62B determines whether the elapsed time measured by the timer unit 58 exceeds the non-notification maintenance time. Here, the non-notification maintenance time is set by the non-notification maintenance time setting unit 54 to a predetermined time.

[0113] The non-notification maintenance time is set to a time greater than 0. For example, the non-notification maintenance time is 0.5 seconds to 30 seconds. The non-notification maintenance time is preferably 1 second to 7 seconds, and more preferably 2 seconds to 4 seconds.

[0114] If, in step S8, the elapsed time is determined to be less than the non-notification maintenance time, the process proceeds to step S9. In step S9, similar to the process following the transition from step S5, the notification information display control unit 62B sets the notification information to the non-notification state, terminating the process of this flowchart. Therefore, the graphic indicating the location of the area of ​​interest is not displayed on the display 16.

[0115] If it is determined in step S8 that the elapsed time has exceeded the non-notification maintenance time, the process proceeds to step S10. In step S10, similar to the process of proceeding from step S6, the notification information display control unit 62B sets the notification information to the notification state, terminating the process of this flowchart. Consequently, a graphic is superimposed on the position of the target area of ​​the frame image 38a displayed on the display 16.

[0116] As described above, according to the medical image processing method involved in the first embodiment, the notification (display) of the presence or absence of notification information of the medical image can be appropriately controlled without relying on user operation.

[0117] In this flowchart, the attention area detection unit 42 detects the attention area on the frame image 38 a regardless of the notification state and the non-notification state. However, the attention area detection unit 42 may detect the attention area only in the notification state.

[0118] Figure 4 It is a diagram showing the display transition of the display 16. Figure 4 The F4A shown shows the display of the display 16 at a certain moment. Figure 4 F4B shown shows the display of the display 16 after a certain time has passed since F4A. Figure 4F4C shown shows the display of the display 16 after a certain time has passed since F4B. The frame images 38a displayed on the display 16 in F4A, F4B, and F4C are different frame images 38a constituting one dynamic picture 38 of a certain frame rate.

[0119] F4A indicates that the notification information is in the notification state because frame image 38a does not fully meet the non-notification conditions. In F4A, a frame-shaped graphic F1 surrounding the target region R1 detected in frame image 38a is displayed on display 16, superimposed on frame image 38a. This notification of the target region's location prevents the user from overlooking the target region.

[0120] F4B shows a case where the notification information is in a non-notification state because the frame image 38a satisfies the non-notification condition. In F4B, the frame image 38a including the target region R1 is displayed on the display 16, but the notification information is not displayed, so the graphic F1 is not displayed.

[0121] If graphic F1 is displayed on display 16, it may hinder observation of frame image 38a depending on the situation. For example, it may only hinder observation after the user has discovered region of interest R1. Therefore, as shown in F4B, when the non-notification conditions are fully met at a time when notification information is deemed unnecessary, the user can easily observe the medical image by not displaying the notification information.

[0122] Furthermore, if the graphic F1 is displayed based on an erroneous detection result by the region of interest detection unit 42, observation of the frame image 38a is also hindered. Therefore, even when the non-notification condition, which is considered to be a timing at which the reliability of the detection result by the region of interest detection unit 42 decreases, is fully satisfied, the user can easily observe the medical image by not displaying the notification information.

[0123] Furthermore, since the non-notification state of the notification information is maintained from the time when the frame image 38a does not sufficiently satisfy the non-notification condition until the non-notification maintaining time elapses, the graphic F1 is not displayed similarly to F4B.

[0124] F4C shows a situation where the notification information returns to the notification state after the non-notification maintenance time has passed since frame image 38a ceased to fully meet the non-notification conditions. As in F4A, a frame-shaped graphic F1 surrounding the target region R1 detected in frame image 38a is displayed on display 16 overlaid with frame image 38a.

[0125] If notification information is left in the non-notification state, even if the selection of the focus area is restarted and the notification information needs to be displayed, the notification information will not be displayed, and the user may miss the focus area. Therefore, if the non-notification conditions are no longer fully met, the notification information is set to the notification state. This prevents the display of notification information from being stopped, regardless of the user's operation to restart the display.

[0126] If the notification information is displayed immediately after the non-notification conditions are no longer fully met, the display and non-display will be repeated at short intervals depending on the image information of the dynamic image 38, which would be inconvenient for the user. Therefore, the notification state is returned to after the non-notification maintenance time has passed since the non-notification conditions are no longer fully met. By transitioning from non-display to display in this way, it is possible to prevent repeated display and non-display at short intervals.

[0127] <Display Modification Example>

[0128] [How to report areas of interest]

[0129] The method of notifying the region of interest on the display 16 is not limited to the example of surrounding the region of interest with a frame-shaped graphic.

[0130] Figures 5 to 8 1 and 2 are diagrams showing an example of the display on the display 16 in the notification state, in which a frame image 38 a including the region of interest R1 is displayed.

[0131] exist Figure 5 In the case shown, the notification information display control unit 62B displays information IN1 that fills in the attention area R1, thereby notifying the attention area R1. The color of the fill is not limited as long as it can be notified to the user. In this way, by filling in the attention area, the scope of the attention area can be easily identified and notified.

[0132] Figure 6 and Figure 7 An example is shown in which whether or not the attention area is detected is notified regardless of the position of the attention area in the image.

[0133] exist Figure 6 In the case shown, the notification information display control unit 62B notifies the user that the region of interest R1 has been detected by displaying information IN2 that colors the periphery of the frame image 38a. By using the color of the periphery of the medical image for notification, graphics that would obstruct observation are not superimposed on the medical image, and notification can be performed with minimal eye movement on the part of the user.

[0134] exist Figure 7In the example shown, the notification information display control unit 62B notifies the user that the region of interest R1 has been detected by displaying information IN3 of a star-shaped graphic at the upper right position outside the frame image 38a. The shape and configuration position of the graphic are not limited to this example. In this way, by displaying a specific graphic at a specific position outside the medical image, there is no overlap of graphics that would obstruct observation on the medical image, and notification can be performed while reducing the user's line of sight movement. In addition, since the notification only affects a small range within the display screen, it has the advantage of being troublesome for the user.

[0135] exist Figure 8 In the case shown, the notification information display control unit 62B notifies the approximate position of the region of interest R1 by displaying information IN4 that colors only the area near the region of interest (in this case, the upper right) in the periphery of the frame image 38a. Thus, by using color to indicate the area near the region of interest in the periphery of the medical image, graphics that would obstruct observation are not superimposed on the medical image, and the approximate position of the subject can be grasped and notified.

[0136] [Display of other information]

[0137] Figure 9 It is a diagram showing the display transition of the display 16. Figure 9 F9A shown in FIG represents the display of the display 16 at a certain moment. Figure 9 F9B shown in FIG. 1 shows the display of the display 16 3 seconds after F9A. Figure 9 F9C shown shows the display on the display 16 two seconds after F9B. The frame images 38a displayed on the display 16 in F9A, F9B, and F9C are different frame images 38a constituting one dynamic picture 38 at a constant frame rate.

[0138] In F9A, F9B and F9C, information IN5 (an example of information indicating a non-notification state) indicating a notification state or a non-notification state and information IN6 (an example of information related to the elapsed time from when the medical image becomes insufficient to satisfy the non-notification condition) indicating the remaining time for setting the notification information to the non-notification state, that is, the difference between the non-notification maintenance time and the elapsed time, are displayed in a display area (an example of a third display unit) of the display 16 that is different from the display area (an example of a first display unit) of the frame image 38a.

[0139] F9A and F9B show a non-notification state. In F9A and F9B, the frame image 38a including the attention region R1 is displayed on the display 16, but the notification information display control unit 62B does not perform notification of the attention region R1.

[0140] In F9A, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays "5 sec" as information IN6, indicating that the remaining time is 5 seconds. Furthermore, in F9B, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays "2 sec" as information IN6, indicating that the remaining time is 2 seconds.

[0141] F9C shows the state in which the notification state has been reached after the non-notification maintenance time has elapsed. In F9C, the display 16 displays a frame image 38a including the region of interest R1, and the notification information display control unit 62B superimposes a frame-shaped graphic F1 surrounding the region of interest R1 on the frame image 38a. Furthermore, in F9C, the notification information display control unit 62B displays "on" as information IN5, indicating the notification state, and "0 sec," indicating the non-notification maintenance time has elapsed, as information IN6. The notification information display control unit 62B may also not display information IN6 when the notification state is in place.

[0142] in addition, Figure 10 and Figure 11 Each of these diagrams shows the display transition of the display 16 . Figure 10 The timing of the transition of F10A, F10B and F10C is shown. Figure 11 The transition timings of F11A, F11B, and F11C are shown as follows: Figure 9 The transition timings of F9A, F9B, and F9C are the same. That is, the frame images 38a displayed in F10A, F10B, and F10C and the frame images 38a displayed in F11A, F11B, and F11C are the same as the frame images displayed in F9A, F9B, and F9C, respectively.

[0143] In F10A, F10B and F10C, information IN5 of the notification state or the non-notification state and information IN7 of the remaining time for setting the notification information to the non-notification state (an example of information related to the elapsed time from when the medical image becomes insufficient to satisfy the non-notification condition) are displayed in a display area of ​​the display 16 that is different from the display area of ​​the frame image 38a.

[0144] Information IN7 is information that expresses the remaining time in the non-notification state using a bar (progress bar), and is set so that the length of the filled portion of the bar is correlated with the elapsed time.

[0145] F10A and F10B illustrate the non-notification state. In F10A and F10B, the display 16 displays the frame image 38a including the target region R1, but the notification information display control unit 62B does not display the target region R1. In F10A, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays a bar with a filled-in time of 5 seconds as information IN7. In contrast, in F10B, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays a bar with a filled-in time of 2 seconds as information IN7.

[0146] F10C shows the state in which the notification state is reached after the non-notification maintenance time has elapsed. In F10C, the display 16 displays a frame image 38a including the region of interest R1, and the notification information display control unit 62B superimposes a frame-shaped graphic F1 surrounding the region of interest R1 on the frame image 38a. Furthermore, in F10C, the notification information display control unit 62B displays "on" as information IN5, indicating the notification state, and displays a bar with a remaining time of 0 seconds, i.e., an unfilled bar, as information IN7. The notification information display control unit 62B may not display information IN7 when the notification state is in place.

[0147] In this way, by displaying a bar in which the length of the filled portion has a correlation with the elapsed time, the user can interpret the remaining time graphically.

[0148] In F11A, F11B and F11C, information IN5 of the notification state or the non-notification state and information IN8 of the remaining time for setting the notification information to the non-notification state (an example of information related to the elapsed time from when the medical image becomes insufficient to satisfy the non-notification condition) are displayed in a display area of ​​the display 16 that is different from the display area of ​​the frame image 38a.

[0149] The information IN8 represents the remaining time by color information of a portion of the display area. The shorter the remaining time, the darker the color.

[0150] F11A and F11B illustrate the non-notification state. In F11A and F11B, the display 16 displays the frame image 38a including the target region R1, but the notification information display control unit 62B does not display the target region R1. In F11A, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays a color with a density equivalent to 5 seconds for the remaining time as information IN8. In contrast, in F11B, the notification information display control unit 62B displays "off" as information IN5, indicating the non-notification state, and displays a color with a density equivalent to 2 seconds for the remaining time as information IN8.

[0151] F11C shows the state in which the notification state is reached after the non-notification maintenance time has elapsed. In F11C, the display 16 displays a frame image 38a including the region of interest R1, and the notification information display control unit 62B superimposes a frame-shaped graphic F1 surrounding the region of interest R1 on the frame image 38a. Furthermore, in F11C, the notification information display control unit 62B displays "on" as information IN5, indicating the notification state, and displays a dark color corresponding to a remaining time of 0 seconds as information IN8.

[0152] Here, the color density of information IN8 of F11B is higher than that of information IN8 of F11A, and the color density of information IN8 of F11C is higher than that of information IN8 of F11B. Alternatively, the color may be darker as the remaining time is shorter.

[0153] In this way, by using the color density around the display area of ​​the medical image to express the remaining time, the user can grasp the remaining time without having to look at a specific area, which has the advantage of not hindering observation.

[0154] Figure 12 It is a diagram showing the display transition of the display 16. Figure 12 F12A shown in FIG. 12A represents the display of the display 16 at a certain moment. Figure 12 F12B shown in FIG. 12A represents the display of the display 16 after a certain time has passed. Figure 12 F12C shown shows the display of the display 16 after a certain time has passed since F12B. The frame images 38a displayed on the display 16 in F12A, F12B, and F12C are different frame images 38a constituting one dynamic picture 38 at a certain frame rate.

[0155] In F12A, F12B, and F12C, information IN9 is displayed in a display area of ​​the display 16 different from the display area of ​​the frame image 38a. This information IN9 collectively shows the information of the notification state or the non-notification state and the remaining time for the notification information to be set to the non-notification state.

[0156] Information IN9 indicates a non-notification state where the non-notification conditions are fully met when the text "AI" turns gray. Furthermore, a constant red intensity indicates a notification state. Furthermore, during the non-notification state, from the time the non-notification conditions are no longer fully met until the non-notification maintenance time has elapsed, the red intensity changes continuously based on the remaining time.

[0157] F12A shows a non-notification state where all non-notification conditions are met. In F12A, while the display 16 displays frame image 38a including the region of interest R1, the notification information display control unit 62B does not notify the region of interest R1. Furthermore, the notification information display control unit 62B displays the grayed-out text "AI" as information IN9. This indicates that the AI ​​(Artificial Intelligence) function is not functioning to detect and notify the region of interest.

[0158] F12B shows the non-notification state from the time the non-notification conditions are no longer fully met until the non-notification maintenance time has elapsed. In F12B, while the display 16 displays the frame image 38a including the region of interest R1, the notification information display control unit 62B does not issue notification for the region of interest R1. Furthermore, the notification information display control unit 62B displays the red text "AI" in a density corresponding to the remaining time as information IN9.

[0159] F12C shows the notification state. In F12C, the display 16 displays a frame image 38a including the region of interest R1. The notification information display control unit 62B superimposes a frame-shaped graphic F1 surrounding the region of interest R1 on the frame image 38a. Furthermore, the notification information display control unit 62B displays the text "AI" in red at a constant density corresponding to the notification state as information IN9. This indicates that the detection notification function for the region of interest is functioning.

[0160] Here, the red density of information IN9 of F12C is relatively higher than the red density of information IN9 of F12B. Note that the color in the non-notification state, the color in the notification state, and the density of information IN9 that fully satisfy the non-notification condition are not limited to this example.

[0161] By unifying the display of notification and non-notification information with the remaining time, the screen display is less complex. Furthermore, by changing the color of the text between notification and non-notification states, users can easily distinguish between the two. Furthermore, by continuously changing the color density of the text according to the remaining time, users can understand the remaining time. The text in the non-notification state can be displayed without graying out. Furthermore, the remaining time can be displayed not through text but through the shading of an icon or other graphic.

[0162] <Details of non-notification conditions>

[0163] [Determining image feature quantities]

[0164] The non-notification condition determined by the determination unit 56 is, for example, a condition for determining an image feature value of a medical image. The image feature value includes at least one of brightness, color information, time variation, and frequency information of the medical image.

[0165] For example, it is considered that there is no need to notify information while the user finds the region of interest in the dynamic image 38 and is observing the region of interest. When the user is observing the region of interest, the temporal change of the dynamic image 38 generally becomes relatively small.

[0166] Therefore, the image feature acquisition unit 48 detects temporal changes in the plurality of frame images 38a of the dynamic image 38 as image features. The determination unit 56 compares the detected temporal changes with the temporal changes set as the non-notification condition. If the detected temporal changes are determined to be relatively small, the notification information display control unit 62B determines that the user is observing after discovering the area of ​​interest and sets the notification information to the non-notification state.

[0167] Furthermore, it is considered that there is no need to notify information while the endoscope system 9 is supplying or absorbing water. During water supply and absorption, the frame image 38a generally contains more yellow components and high-frequency components.

[0168] Therefore, the image feature acquisition unit 48 detects the color components of the frame image 38a as image features. The determination unit 56 compares the detected yellow component with the yellow component set as the non-notification condition. If the detected yellow component is relatively high, the notification information display control unit 62B determines that the endoscope system 9 is supplying or absorbing water and sets the notification information to the non-notification state.

[0169] Similarly, the image feature acquisition unit 48 detects the frequency components of the frame image 38a as image features. The determination unit 56 compares the detected high-frequency components with the high-frequency components set as the non-notification condition. If the detected high-frequency components are relatively high, the notification information display control unit 62B determines that the endoscope system 9 is supplying or absorbing water and sets the notification information to the non-notification state.

[0170] When the brightness of the frame image 38 a is too high or too low, or when the blur of the frame image 38 a is too large, it is considered that the reliability of the detection result of the attention area detection unit 42 is reduced.

[0171] Therefore, the image feature acquisition unit 48 detects the brightness of the frame image 38a as an image feature. The determination unit 56 compares the detected brightness with the brightness range set as a non-notification condition. If the detected brightness is not within the set brightness range, the notification information display control unit 62B determines that the accuracy of the detection result of the attention area detection unit 42 has deteriorated and sets the notification information to the non-notification state.

[0172] Similarly, the image feature acquisition unit 48 detects blur in the frame image 38a as an image feature. The determination unit 56 compares the detected blur with the blur set as the non-notification condition. If the detected blur is determined to be relatively large, the notification information display control unit 62B determines that the accuracy of the detection result of the attention area detection unit 42 has deteriorated and sets the notification information to the non-notification state.

[0173] In this way, by setting the non-notification condition to the image feature value at a timing when the notification information is not needed, or the timing when the reliability of the detection result of the area of ​​interest is reduced, the judgment unit 56 compares the set image feature value with the image feature value obtained by the image feature value acquisition unit 48, and can automatically set the notification information to the non-notification state at an appropriate timing.

[0174] When the non-notification conditions for determining the image feature value are not fully met, the notification state of the notification information should not be restarted immediately. For example, when the time change of frame image 38a is small, due to the influence of body movement, the time change temporarily becomes larger, and the amount of change immediately returns to a smaller state. This temporary large time change does not mean that the screening has been restarted, and there is no need to display the notification information. In addition, if the display and non-display are repeated at short intervals, there is a problem that it is more troublesome for the user. Therefore, even if the non-notification conditions are not fully met, the non-notification state is maintained until the non-notification maintenance time has passed, and the notification state is set after the non-notification maintenance time has passed, thereby avoiding this problem.

[0175] [Determination of disposal status]

[0176] The non-notification condition may also be a condition for determining the treatment status of the subject in the medical image. The period during which the user is performing treatment on the subject is considered a time when no notification information is required. Examples of treatment status include the presence of a treatment instrument in the frame image 38a, the presence of bleeding, or the endoscope system 9 being set to a magnification mode for magnifying and capturing images at a predetermined magnification.

[0177] For example, the treatment status estimation unit 50 detects a treatment instrument based on the frame image 38a. The determination unit 56 compares the detected treatment instrument with the treatment instrument set as a non-notification condition, namely, a biopsy forceps or a snare. If it is determined that the detected treatment status satisfies the non-notification condition, the notification information display control unit 62B determines that the user is currently undergoing treatment and sets the notification information to the non-notification state.

[0178] Furthermore, if a treatment instrument is detected based on the frame image 38a, it is possible to determine that the user is currently undergoing treatment, regardless of the type of treatment instrument, and set the notification information to a non-notification state. In this case, the determination unit 56 may not determine whether the non-notification condition is satisfied, or may perform the determination by setting all treatment instruments to the non-notification condition.

[0179] Furthermore, the treatment state estimation unit 50 detects the amount of bleeding based on the frame image 38a. The determination unit 56 compares the detected amount of bleeding with the amount of bleeding set as the non-notification condition. If it is determined that the detected amount of bleeding sufficiently satisfies the non-notification condition, the notification information display control unit 62B determines that the user is undergoing treatment and sets the notification information to the non-notification state.

[0180] Furthermore, the treatment state estimation unit 50 detects the mode set for the endoscope system 9. The determination unit 56 compares the detected mode with the magnification mode set as a non-notification condition. If it is determined that the detected mode satisfies the non-notification condition, the notification information display control unit 62B determines that the user is currently undergoing treatment and sets the notification information to a non-notification state.

[0181] Furthermore, the treatment state estimation unit 50 detects color materials dispersed within the subject. The determination unit 56 compares the detected color materials with the color materials set as non-notification conditions. If it is determined that the detected color materials sufficiently meet the non-notification conditions, the notification information display control unit 62B determines that the user is currently undergoing a diagnosis and sets the notification information to the non-notification state.

[0182] Furthermore, when a color material is detected, it may be determined that the user is currently processing the color material, regardless of the type of color material, and the notification information may be set to a non-notification state. In this case, the determination unit 56 may not determine whether the non-notification condition is met, or may set all color materials to the non-notification condition for determination.

[0183] By setting the non-notification condition to a treatment state at a timing that does not require notification information, the determination unit 56 compares the set treatment state with the treatment state acquired by the treatment state estimation unit 50 and can automatically set the notification information to the non-notification state at an appropriate timing.

[0184] If the non-notification conditions for determining the treatment status are no longer fully met, the display of notification information should not be immediately resumed. For example, if frame image 38a contains biopsy forceps and the notification is set to non-notification, there are many cases where the forceps is temporarily covered by mucous membrane or residual fluid and is not captured, and then the display is immediately resumed. For the same reasons as when determining image features, it is not preferable to immediately resume the display when the forceps are temporarily not captured. Therefore, even if the non-notification conditions are no longer fully met, the non-notification state is maintained until the non-notification maintenance time has expired, and the notification state is resumed after the non-notification maintenance time has expired, thereby avoiding this problem.

[0185] Furthermore, the non-notification maintenance time can be changed depending on the treatment status. For example, if the biopsy forceps are included in the frame image 38a, the biopsy treatment can be determined to be complete if the biopsy forceps are not extended from the forceps opening. Therefore, the non-notification maintenance time setting unit 54 can set the non-notification maintenance time after the biopsy forceps are not extended from the forceps opening to a relatively short time.

[0186] On the other hand, in procedures such as EMR (Endoscopic Mucosal Resection), a series of procedures involves local injection, excision with a snare, and hemostasis with clips. For example, during the period from the local injection until the snare is removed from the forceps, the treatment instrument cannot be detected based on frame image 38a. However, the treatment is still ongoing during this period, and the notification information should not be re-displayed. Therefore, if a local injection is detected based on frame image 38a, the non-notification maintenance time setting unit 54 sets the non-notification maintenance time after the local injection is not detected to a relatively long time, thereby preventing the notification information from being re-displayed during the series of procedures.

[0187] [Determining the Feature Quantity of the Region of Interest]

[0188] The non-notification condition may also be a condition for determining a feature value of a region of interest of a medical image. The feature value of the region of interest includes at least one of an area of ​​the region of interest, a position in the medical image, and a temporal change.

[0189] When observing the region of interest, generally, the area ratio of the region of interest in the frame image 38 a is large, the position of the region of interest in the frame image 38 a is located in the center, and the temporal change of the region of interest is small.

[0190] Therefore, the attention area feature value acquisition unit 52 detects the area of ​​the attention area within the frame image 38a as the attention area feature value. The determination unit 56 compares the detected area with the area set as the non-notification condition. If the detected area is determined to be relatively large, the notification information display control unit 62B determines that the user is observing the attention area after discovering it and sets the notification information to the non-notification state.

[0191] Furthermore, the region of interest feature acquisition unit 52 detects the position of the region of interest within the frame image 38a as the region of interest feature. The determination unit 56 compares the detected position with the position set as the non-notification condition. If it is determined that the detected region of interest position is closer to the center than the position set as the non-notification condition, the notification information display control unit 62B determines that the user is observing the region of interest after discovering it and sets the notification information to the non-notification state.

[0192] Similarly, the attention area feature acquisition unit 52 detects temporal changes in the attention area of ​​the dynamic image 38 as the attention area feature. The determination unit 56 compares the detected temporal changes with the temporal changes set as the non-notification condition. If the detected temporal changes are determined to be relatively small, the notification information display control unit 62B determines that the user is observing the attention area after discovering it and sets the notification information to the non-notification state.

[0193] In this way, by setting the non-notification condition to the timed attention area feature value that does not require notification information, the judgment unit 56 compares the set attention area feature value with the attention area feature value obtained by the attention area feature value acquisition unit 52, and can automatically set the notification information to the non-notification state at an appropriate timing.

[0194] As before, if the non-notification conditions for the region of interest feature value are determined to be insufficiently met, the display of notification information should not be immediately resumed. For example, even if the region of interest is large, a portion of it may be temporarily covered by residual fluid, etc., making it appear smaller. Or, even if it is located in the center of frame image 38a, it may temporarily move to the edge of frame image 38a due to body movement. Therefore, even if the non-notification conditions are insufficiently met, the non-notification state is maintained until the non-notification maintenance time has expired, and the notification state is restored after the non-notification maintenance time has expired, thus avoiding such problems.

[0195] <Second embodiment>

[0196] [Structure of medical image processing device]

[0197] Although an example in which a single display 16 is connected to the medical image processing apparatus 14 has been described so far, the medical image processing apparatus 14 may also be connected to a plurality of display apparatuses.

[0198] Figure 13 This is a block diagram showing the electrical structure of the medical image processing device 14 according to the second embodiment. Figure 2 The display control unit 62 is connected to the first display 16A, the second display 16B, the third display 16C, and the fourth display 16D, and controls the display of each display.

[0199] The first display 16A, the second display 16B, the third display 16C, and the fourth display 16D are, for example, provided in the endoscope system 9. The first display 16A, the second display 16B, the third display 16C, and the fourth display 16D are each a separate display device such as an LCD monitor. Here, four display devices are connected, but the number of display devices can be determined as needed. Alternatively, the first display 16A, the second display 16B, the third display 16C, and the fourth display 16D may be separate areas within the screen of a single display device.

[0200] Figure 14 16A and 16B are diagrams showing an example of display transition between the first display 16A and the second display 16B.

[0201] The notification information display control unit 62B controls the display of the first display 16A (an example of a first display unit) in the same manner as the display of the display 16 according to the first embodiment. Figure 14 The F14A, F14B and F14C shown in FIG. 14A represent the display of the first display 16A. Figure 4 The F4A, F4B and F4C shown are identical.

[0202] Specifically, F14A and F14C show the notification state. In F14A and F14C, a frame-shaped graphic F1, which encloses the target region R1 detected in frame image 38a, is superimposed on frame image 38a. Furthermore, F14B shows the non-notification state. In F14B, graphic F1 is not displayed, and only frame image 38a is shown.

[0203] Furthermore, the notification information display control unit 62B changes the display mode of the notification information on the second display 16B (an example of a second display unit) according to the notification state and the non-notification state of the first display 16A. Here, when the notification information is in the non-notification state, the notification information display control unit 62B displays the notification information on the second display 16B.

[0204] Figure 14F14D, F14E, and F14F respectively represent the display of the second display 16B at the same timing as F14A, F14B, and F14C. The frame images 38a displayed in F14D, F14E, and F14F are the same as the frame images 38a displayed in F14A, F14B, and F14C.

[0205] In F14D and F14F, when the notification information is in the notification state, the graphic F1 is not displayed. On the other hand, in F14E, when the notification information is in the non-notification state, the frame-shaped graphic F1 surrounding the target region R1 is superimposed on the frame image 38a.

[0206] Thus, the second display 16B displays the notification information only when the notification information is in the non-notification state. This prevents the first display 16A from being obstructed by the user's viewing when the notification information is not needed. Furthermore, the second display 16B allows the user to view the notification information when the user needs it.

[0207] Here, the frame image 38a is displayed on the second display 16B, but only the graphic F1 serving as notification information may be displayed on the second display 16B.

[0208] Figure 15 This figure shows another example of the display transition between first display 16A and second display 16B. In this example, when the notification information is in the non-notification state, notification information display control unit 62B changes the notification information displayed on second display 16B to a higher emphasis level. Here, emphasis level refers to the user's ease of identifying the notification information; the higher the emphasis level, the easier it is for the user to identify the notification information.

[0209] Figure 15 The F15A, F15B and F15C shown in FIG. 15A represent the display of the first display 16A, and Figure 14 The F14A, F14B and F14C shown are the same. Figure 15 F15D, F15E, and F15F respectively represent the display of the second display 16B at the same timing as F15A, F15B, and F15C. The frame images 38a displayed in F15D, F15E, and F15F are the same as the frame images 38a displayed in F15A, F15B, and F15C.

[0210] In F15D and F15F, which are in the notification state, similar to F15A and F15C, a frame-shaped graphic F1 enclosing the target region R1 detected in frame image 38a is superimposed on frame image 38a. Furthermore, in F15E, which is in the non-notification state, a frame-shaped graphic F2 enclosing the target region R1 detected in frame image 38a is superimposed on frame image 38a. The line thickness of graphic F2 is relatively thicker than that of graphic F1. Therefore, the emphasis level of graphic F2 is relatively higher than that of graphic F1.

[0211] By controlling the display of the second display 16B in this manner, when the user needs notification information when the notification information is in the non-notification state, the user can more easily confirm the notification information.

[0212] The notification information display control unit 62B can also make the second display 16B display Figures 5 to 8 The information IN1 to IN4 for notifying the region of interest is shown. Furthermore, the notification information display control unit 62B may cause the second display 16B to display the information IN1 to IN4 only when the notification information is in the non-notification state. In this case, the image display control unit 62A may or may not display the medical image on the second display 16B.

[0213] Figure 16 This figure shows another example of the display transition between the first display 16A and the third display 16C. In this example, when the notification information is in the non-notification state, the notification information display control unit 62B displays information indicating that the notification information is in the non-notification state on the third display 16C (an example of a third display unit).

[0214] Figure 16 The F16A, F16B and F16C shown in FIG. 16A represent the display of the first display 16A. Figure 14 The F14A, F14B and F14C shown are the same. Figure 16 F16D, F16E, and F16F shown respectively represent the displays of the third display 16C at the same timing as F16A, F16B, and F16C.

[0215] The notification information display control unit 62B does not display on the third display 16C in F16D and F16F, which are the notification state. In addition, the notification information display control unit 62B displays information IN10 indicating the non-notification state on the third display 16C in F16E, which is the non-notification state.

[0216] Here, text is used as the information IN10, but it may also be a symbol or color, etc. In addition, the notification information display control unit 62B may display information indicating the notification state on the third display 16C at F16D and F16F, which are timings of the notification state.

[0217] By controlling the display of the third display 16C in this manner, the user can recognize whether the notification state is in the non-notification state. Alternatively, in the non-notification state, the user may operate the operation unit 15 to immediately set the first display 16A to the notification state.

[0218] Figure 17 This figure shows another example of the display transition between the first display 16A and the fourth display 16D. In this example, the notification information display control unit 62B displays information related to the elapsed time since the non-notification condition became insufficiently satisfied on the fourth display 16D (an example of a fourth display unit).

[0219] Figure 17 The F17A, F17B and F17C shown in FIG. 17A represent the display of the first display 16A. Figure 14 F14A, F14B, and F14C are the same as those shown. In addition, F17B shows that the non-notification state of the notification information is maintained from the time when the frame image 38a does not fully satisfy the non-notification condition until the non-notification maintenance time elapses. Figure 17 F17D, F17E, and F17F shown respectively represent the displays of the fourth display 16D at the same timing as F17A, F17B, and F17C.

[0220] The notification information display control unit 62B does not display information on the fourth display 16D during F17D and F17F, which are timings when the non-notification condition is not fully satisfied. Furthermore, during F17E, which is the timing from when the frame image 38a becomes insufficiently satisfied for the non-notification condition until the non-notification maintenance time has elapsed, the notification information display control unit 62B displays information IN11 on the fourth display 16D, indicating the remaining time until the notification information is in the non-notification state, that is, the difference between the non-notification maintenance time and the elapsed time.

[0221] Here, as the information IN11, text information indicating the remaining time until the notification information is set to the non-notification state and a graphic indicating the remaining time indirectly, that is, a graphic in which the color of the area changes sequentially, are displayed. However, the time elapsed from when the non-notification condition becomes insufficient may also be displayed. In addition, the notification information display control unit 62B may also cause the second display 16B to display Figures 9 to 12 The remaining time information IN6 to IN9 is shown.

[0222] By controlling the display of the fourth display 16D in this manner, the user can recognize information related to the time elapsed since the non-notification condition became insufficiently satisfied. Alternatively, the user can immediately switch the first display 16A to the notification state by operating the operation unit 15 while the information IN11 is displayed on the fourth display 16D.

[0223] <Third embodiment>

[0224] [Structure of medical image processing device]

[0225] Figure 18 1 is a block diagram showing an example of the electrical structure of the medical image processing device 14 according to the third embodiment. Figure 2 The common parts of the blocks shown are given the same symbols, and their detailed description is omitted.

[0226] The medical image processing device 14 includes a sound control unit 70. The sound control unit 70 is an example of a notification control unit that notifies notification information included in a medical image obtained by photographing a subject. The sound control unit 70 is connected to the sound output unit 17. The sound output unit 17 functions as a notification unit that notifies notification information, and is, for example, provided in the endoscope system 9 (refer to Figure 1 ). The sound output unit 17 is configured to include a component that outputs sound, such as a buzzer or a speaker. The sound control unit 70 causes the sound output unit 17 to output sound in the notification state to notify the notification information. The output sound may be an alarm sound or a sound related to the notification information.

[0227] Here, the configuration is described in which the notification unit for notifying the notification information includes the display 16 and the sound output unit 17, but it is also possible to include only the sound output unit 17. Furthermore, the notification unit for notifying the notification information may use a light that emits light or a vibrator that vibrates.

[0228] <Other>

[0229] The above-mentioned medical image processing method can be configured as a program for causing a computer to implement each step, or can be configured as a non-transitory recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program.

[0230] In the embodiments described so far, the endoscope processor device 12 and the medical image processing device 14 are described as different devices, but the endoscope processor device 12 and the medical image processing device 14 can also be configured as an integrated device, as an endoscope processor device 12 having the functions of the medical image processing device 14.

[0231] The hardware configuration of the processing units that perform various processes in the endoscope processor device 12 and the medical image processing device 14 is composed of various processors as shown below. These processors include general-purpose processors (CPUs) that execute software (programs) and function as various processing units; processors specialized for image processing (GPUs) (GPUs); processors (PLDs) whose circuit configurations can be modified after manufacturing, such as FPGAs (Field Programmable Gate Arrays); and processors (ASICs) with circuit configurations specifically designed to perform specific processes, such as dedicated circuits.

[0232] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, multiple processing units can be composed of one processor. As an example of using one processor to constitute multiple processing units, first, there is a form such as represented by computers such as servers and clients, which uses a combination of one or more CPUs and software to constitute a processor, and the processor functions as multiple processing units. Secondly, there is a form such as represented by a system on chip (System On Chip: SoC), which uses a processor that uses an IC (Integrated Circuit) chip to realize the functions of the entire system including multiple processing units. In this way, various processing units are composed of one or more various processors as a hardware structure.

[0233] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

[0234] The technical scope of the present invention is not limited to the scope described in the above embodiments, and the structures and the like in each embodiment can be appropriately combined between the embodiments without departing from the gist of the present invention.

[0235] Explanation of symbols

[0236] 9 Endoscopic system

[0237] 10 Endoscope

[0238] 11 Light source device

[0239] 12 Endoscope processor device

[0240] 13 Display device

[0241] 14Medical image processing device

[0242] 15 Operation Department

[0243] 16 displays

[0244] 16A First Display

[0245] 16B Second Display

[0246] 16C third display

[0247] 16D fourth display

[0248] 17 Sound output unit

[0249] 20 Insertion

[0250] 21 Hand operation department

[0251] 22 Universal Cord

[0252] 25 Soft part

[0253] 26 bending part

[0254] 27 Top

[0255] 28 camera elements

[0256] 29 curved operating button

[0257] 30 Gas and water supply buttons

[0258] 31 Attract Button

[0259] 32 Still picture shooting instruction section

[0260] 33 Treatment instrument introduction port

[0261] 35 light guide

[0262] 36 signal cable

[0263] 37a connector

[0264] 37b connector

[0265] 38 dynamic images

[0266] 38a frame image

[0267] 39 still images

[0268] 40 Time series image acquisition unit

[0269] 42 Focus Area Detection Department

[0270] 44 Coordinate calculation unit

[0271] 46 Image Information Acquisition Unit

[0272] 48 Image feature acquisition unit

[0273] 50 Disposal Status Estimation Department

[0274] 52 Attention area feature acquisition unit

[0275] 54 non-notification maintenance time setting unit

[0276] 56 Judgment Department

[0277] 58 Timing Department

[0278] 60 Control Department

[0279] 62 display control unit

[0280] 62A Image display control unit

[0281] 62B notification information display control unit

[0282] 64 storage unit

[0283] 66 Graphics Storage Unit

[0284] 68 Programs

[0285] 70 Sound Control Unit

[0286] F1, F2 graphics

[0287] IN1~IN11 information

[0288] R1 Focus Area

[0289] S1~S10 Steps of medical image processing method

Claims

1. A medical image processing device, wherein: The medical image processing device includes a notification control unit, which controls the notification information contained in the medical image to be in any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. The notification control unit sets the notification information to the non-notification state when the medical image fully satisfies the non-notification condition, and sets the notification information to the notification state after a non-notification maintenance time has passed since the medical image ceased to fully satisfy the non-notification condition. The non-notification condition is a condition for determining the treatment status of the subject by the user identified from the medical image. The medical image processing device includes a treatment state estimating unit that estimates the treatment state based on the medical image.

2. The medical image processing apparatus according to claim 1, wherein: The medical image processing apparatus includes a non-notification maintaining time setting unit for setting the non-notification maintaining time according to the treatment state.

3. A medical image processing device, wherein: The medical image processing device includes a notification control unit, which controls the notification information contained in the medical image to be in any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. The notification control unit sets the notification information to the non-notification state when the medical image fully satisfies the non-notification condition, and sets the notification information to the notification state after a non-notification maintenance time has passed since the medical image ceased to fully satisfy the non-notification condition. The non-notification condition is a condition for determining the feature value of the attention area. The medical image processing device comprises: a region of interest detecting unit that detects a region of interest from the medical image; and The attention area feature quantity acquisition unit acquires the attention area feature quantity from the attention area.

4. The medical image processing apparatus according to claim 3, wherein: The feature value of the region of interest includes at least one of the area of ​​the region of interest, the position in the medical image, and the time change.

5. The medical image processing apparatus according to any one of claims 1 to 4, wherein: The medical image processing device comprises: An image acquisition unit, which sequentially acquires the medical images; a notification information acquisition unit, which acquires the notification information from the medical image; a determination unit configured to determine whether the medical image satisfies the non-notification condition; and A timing unit is configured to count the time elapsed from when the medical image ceases to fully satisfy the non-notification condition.

6. The medical image processing apparatus according to any one of claims 1 to 4, wherein: The notification unit includes a sound output unit that outputs sound. The notification control unit includes a sound control unit that causes the sound output unit to output the sound.

7. The medical image processing apparatus according to any one of claims 1 to 4, wherein: The notification unit includes a first display unit, The notification control unit includes a display control unit that causes the first display unit to display the notification information.

8. The medical image processing apparatus according to claim 7, wherein: The notification unit includes a second display unit different from the first display unit, The display control unit causes the second display unit to display the notification information.

9. The medical image processing apparatus according to claim 8, wherein: The display control unit changes a display mode of the notification information on the second display unit according to the notification state and the non-notification state.

10. The medical image processing apparatus according to claim 9, wherein: The display control unit causes the second display unit to display the notification information in the non-notification state.

11. The medical image processing apparatus according to claim 7, wherein: The display control unit causes a third display unit different from the first display unit to display information indicating the non-notification state.

12. The medical image processing apparatus according to claim 7, wherein: The display control unit causes a fourth display unit different from the first display unit to display information related to the elapsed time from when the medical image becomes insufficient to satisfy the non-notification condition.

13. A diagnostic aid, wherein: have: The medical image processing device according to any one of claims 7 to 12; and The first display unit.

14. A medical image processing method, wherein: A notification control process is provided to control the notification information contained in the medical image to be in any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. In the notification control process, the notification information is set to the non-notification state when the medical image fully satisfies the non-notification condition, and the notification information is set to the notification state after the non-notification maintenance time has passed since the medical image no longer fully satisfies the non-notification condition. The non-notification condition is a condition for determining the treatment status of the subject by the user identified from the medical image. The medical image processing method also has: The treatment state estimation step estimates the treatment state based on the medical image.

15. A medical image processing method, wherein: A notification control process is provided to control the notification information contained in the medical image to be in any one of a notification state in which notification is performed in the notification unit and a non-notification state in which notification is not performed. In the notification control process, the notification information is set to the non-notification state when the medical image fully satisfies the non-notification condition, and the notification information is set to the notification state after the non-notification maintenance time has passed since the medical image no longer fully satisfies the non-notification condition. The non-notification condition is a condition for determining the feature value of the attention area. The medical image processing method further comprises: a region of interest detection step of detecting a region of interest from the medical image; and The attention area feature value acquisition step acquires the attention area feature value from the attention area.

16. A recording medium which is a non-transitory computer-readable recording medium, which enables the computer to execute the medical image processing method according to claim 14 or 15 when the instructions stored in the recording medium are read by the computer.

Images