Medical image processing apparatus, medical image processing method, and recording medium

Abstract

The present application provides a medical image processing apparatus, a medical image processing method, and a recording medium capable of outputting a notification sound without causing discomfort to a user and without damaging the effectiveness of the notification sound. The medical image processing apparatus (14) includes a processor and a voice notifier (17) that outputs a notification sound, wherein the processor performs: image reception processing that receives images that are continuous in a time series; attention region detection processing that detects an attention region from the images; and sound control processing that, when an attention region is currently detected by the attention region detection processing, performs control that causes the voice notifier to output the notification sound for a certain period, and outputs the notification sound from the voice notifier according to a first time from a point in time at which detection of an attention region ended in the past by the attention region detection processing to a point in time at which detection of an attention region currently exists by the attention region detection processing.

Description

Technical Field

[0001] This invention relates to medical image processing apparatus, medical image processing method and program, and particularly to the technology of medical image processing apparatus, medical image processing method and program that outputs notification sounds. Background Technology

[0002] During an endoscopy, a user (e.g., a doctor) inserts an endoscope with a camera at the tip into the patient's body cavity to perform the examination. The endoscopic images captured by the camera are displayed on a monitor screen, which the user then observes. Generally, during an endoscopy, the user observes the endoscopic images to identify lesions or perform procedures such as lesion removal.

[0003] Endoscopic examinations, which involve simultaneous imaging and examination using a viewing device, can be burdensome for users and may result in missed lesions.

[0004] In recent years, high-precision automatic image recognition has been achieved through deep learning (Non-Patent Document 1). A system has been proposed that automatically detects lesions from endoscopic images during endoscopic examinations and notifies the user. For example, a known technique displays a notification marker on a display (monitor) or emphasizes the detected region of interest by surrounding it with a square when a region of interest is detected in an endoscopic image using a detector learned through deep learning (learned model), thus notifying the user of the detected region of interest. However, since the user performs the endoscopic examination visually, they may sometimes even miss the notification or emphasis display.

[0005] In response, a technology has been proposed that, when an area of ​​interest is detected, a beep is emitted simultaneously with a notification display, thereby engaging not only the user's vision but also their hearing to draw attention (Patent Document 1).

[0006] Previous technical documents

[0007] Non-patent literature

[0008] Non-patent literature 1: A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent Application Publication No. 2006-129950 Summary of the Invention

[0011] The technical problem to be solved by the invention

[0012] However, continuously outputting notification sounds while an area of ​​interest is detected can sometimes cause discomfort to the user. Therefore, outputting a notification sound when an area of ​​interest is detected can suppress this discomfort by stopping the sound output even if the area of ​​interest is subsequently detected again.

[0013] In endoscopic examinations, because the observation device is moved, the area of ​​interest may temporarily disappear from the camera's view. Furthermore, when treating the area of ​​interest (lesion) while observing the endoscopic image, the lesion may temporarily disappear from the image due to obstruction by the treatment instruments. Additionally, the presence of water during endoscopic image observation, as well as blurring or bokeh of the endoscopic image, may cause the detector to temporarily fail to detect the area of ​​interest. If a notification sound is output when the detection of the area of ​​interest is temporarily interrupted and then re-detected, the notification sound is often repeated within a short period, causing discomfort to the user. Moreover, because the notification sound is repeated many times in a short period, the user becomes accustomed to it, reducing its effectiveness in drawing attention.

[0014] Patent document 1 mentioned above does not address the issue of repeatedly outputting notification sounds within a short period of time.

[0015] The present invention was made in view of the following circumstances, and its object is to provide a medical image processing device, medical image processing method and program that can output notification sounds without causing discomfort to the user or impairing the effectiveness of the notification sound in reminding the user.

[0016] means for solving technical problems

[0017] To achieve the above objectives, a medical image processing apparatus according to one aspect of the present invention includes a processor and a voice notification device that outputs a notification sound. The processor performs: image receiving processing, receiving images in a time sequence; region of interest detection processing, detecting regions of interest from the images; and sound control processing, controlling the voice notification device to output a notification sound for a certain period of time when a region of interest is currently detected by the region of interest detection processing. The notification sound is output from the voice notification device based on a first time from the point when the detection of a region of interest ends by the region of interest detection processing to the point when the detection of a region of interest currently exists by the region of interest detection processing.

[0018] According to this method, a notification sound is output from the voice notification device based on the immediate time between the point when the detection of a previously observed region of interest ended and the point when the detection of a currently observed region of interest begins. Therefore, the output of the notification sound can be controlled based on the timing of past region of interest detection, ensuring that the notification sound is output without causing discomfort to the user or compromising its effectiveness in alerting the user.

[0019] Preferably, in the sound control processing, the region of interest is detected intermittently through the region of interest detection processing, and the output of the notification sound is stopped if the region of interest is less than a first threshold in the first instant.

[0020] According to this method, even when the area of ​​interest is detected intermittently, a notification sound can be output without repeating the notification sound, without causing discomfort to the user, and without compromising the effectiveness of the notification sound in attracting attention.

[0021] Preferably, in the voice control processing, if the first time is above a first threshold, a notification sound is output through the voice notification device; if the first time is below the first threshold, the output of the notification sound is stopped through the voice notification device.

[0022] Preferably, the processor performs: the second time detection processing, detecting a second time equivalent to the first time when the region of interest was detected in the past through the region of interest detection processing; and a threshold change processing, changing the first threshold according to the second time.

[0023] According to this method, a first threshold is changed based on a second time corresponding to a first time when the area of ​​interest was detected in the past, and the output of the notification sound is controlled based on this first threshold. Therefore, the notification sound can be output at appropriate timing, without causing discomfort to the user or compromising the effectiveness of the notification sound in drawing attention.

[0024] Preferably, the processor performs a count of interest regions that is detected in the image by the region of interest detection process, and in the threshold change process, if the number of interest regions is greater than or equal to a second threshold, the first threshold is increased.

[0025] According to this method, the number of detected areas of interest is counted, and if the number exceeds a second threshold, the first threshold is increased. This allows for the output of notification sounds at appropriate timings, ensuring that the notification sounds do not cause user discomfort or diminish their effectiveness in drawing attention.

[0026] Preferably, the processor performs a count of interest regions processing to count the number of interest regions detected in the image through the interest region detection processing, and in the threshold change processing, if the number of interest regions is less than the second threshold, the first threshold is reduced.

[0027] According to this method, the number of detected areas of interest is counted, and if the number is less than a second threshold, the first threshold is reduced. This allows for the output of notification sounds at appropriate timings, ensuring that the notification sounds do not cause user discomfort or compromise their effectiveness in drawing attention.

[0028] Preferably, the first threshold is set in the range of more than 1.5 seconds to less than 2.5 seconds.

[0029] Preferably, the first threshold is varied within the range of more than 0.5 seconds to less than 3.5 seconds.

[0030] Preferably, the processor performs area calculation processing on the area of ​​the region of interest detected in the region of interest detection processing. In the voice control processing, if the area calculated in the area calculation processing is less than a third threshold, a notification sound is output through the voice notification device. If the area calculated in the area calculation processing is above the third threshold and is less than the first threshold for a first time, the output of the notification sound based on the voice notification device is stopped. If the area calculated in the area calculation processing is above the third threshold and is above the first threshold for a first time, a notification sound is output through the voice notification device.

[0031] According to this method, the output of the notification sound is controlled based on the area of ​​the detected area of ​​interest, thus enabling the output of the notification sound without causing discomfort to the user or compromising the effectiveness of the notification sound in reminding them to pay attention.

[0032] Preferably, the processor performs classification processing to classify the regions of interest detected in the region of interest detection processing. In the voice control processing, if the classification result in the classification processing is a specific type, a notification sound is output through the voice notification device. If the classification result in the classification processing is an unspecific type and the first time is less than a first threshold, the output of the notification sound is stopped through the voice notification device. If the classification result in the classification processing is an unspecific type and the first time is greater than or equal to the first threshold, a notification sound is output through the voice notification device.

[0033] According to this method, the output of the notification sound is controlled based on the classification of the detected area of ​​interest, thus enabling the output of the notification sound without causing discomfort to the user or compromising the effectiveness of the notification sound in attracting attention.

[0034] Preferably, in the sound control processing, the first time is detected based on the number of images in a time-series sequence.

[0035] Preferably, the display unit is provided to display the image received in the image receiving process, and the current region of interest detected by the region of interest detection process is highlighted and displayed on the display unit.

[0036] As another aspect of the present invention, a medical image processing method uses a medical image processing apparatus equipped with a processor and a voice notification device that outputs a notification sound. The processor executes: an image receiving step, receiving images in a time sequence; a region of interest detection step, detecting a region of interest from the image; and a sound control step, controlling the voice notification device to output a notification sound for a certain period of time when a region of interest is currently detected by the region of interest detection step. The notification sound is output from the voice notification device based on a first time from the point when the detection of a region of interest ends by the region of interest detection step to the point when the detection of a region of interest currently exists by the region of interest detection step.

[0037] As another aspect of the present invention, a medical image processing apparatus equipped with a processor and a voice notification device that outputs a notification sound performs an image processing method, wherein the processor performs: an image receiving step, receiving images in a time sequence; a region of interest detection step, detecting a region of interest from the image; and a sound control step, controlling the voice notification device to output a notification sound for a certain period of time when a region of interest is currently detected by the region of interest detection step, and outputting a notification sound from the voice notification device based on a first time from the point when the detection of a region of interest ends by the region of interest detection step to the point when the detection of a region of interest currently exists by the region of interest detection step.

[0038] Invention Effects

[0039] According to the present invention, a notification sound is output from the voice notification device based on a first time interval from the point when the detection of a previously concluded region of interest ended to the point when the detection of a currently existing region of interest occurs. Therefore, the output of the notification sound can be controlled based on the timing of past region of interest detection, and the notification sound can be output without causing discomfort to the user or compromising its effectiveness in alerting the user. Attached Figure Description

[0040] Figure 1 It is a schematic diagram showing the overall structure of the endoscope system.

[0041] Figure 2 This is a block diagram representing the structure of a medical image processing device.

[0042] Figure 3 This diagram illustrates an example of repeatedly outputting notification sounds within a short period of time.

[0043] Figure 4 This diagram illustrates the situation where a notification sound is output based on the continuous undetected time.

[0044] Figure 5 This is a graph illustrating the continuous period of no detection.

[0045] Figure 6 This diagram illustrates the intermittent detection of the area of ​​interest.

[0046] Figure 7 It is a diagram showing the flow of an image processing method.

[0047] Figure 8 This is a block diagram representing the structure of a medical image processing device.

[0048] Figure 9 This is a graph illustrating the continuous periods of no detection in the past.

[0049] Figure 10 This is a block diagram representing the structure of a medical image processing device.

[0050] Figure 11 This diagram illustrates the timing of the output notification sound.

[0051] Figure 12 This is a block diagram representing the structure of a medical image processing device.

[0052] Figure 13 This diagram illustrates the timing of the output notification sound.

[0053] Figure 14 This is a block diagram representing the structure of a medical image processing device.

[0054] Figure 15 This diagram illustrates the timing of the output notification sound. Detailed Implementation

[0055] Preferred embodiments of the medical image processing apparatus, medical image processing method, and program of the present invention will now be described with reference to the accompanying drawings.

[0056] [The overall structure of an endoscope system, including medical image processing equipment]

[0057] Figure 1 This is a schematic diagram showing the overall structure of an endoscope system including the medical image processing device involved in this invention.

[0058] like Figure 1 As shown, the endoscope system 9 includes an endoscope observer 10 as an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a medical image processing device 14, an operation unit 15, and a display 16.

[0059] The endoscope observer 10 captures time-series medical images including images of the subject, such as an observer for the lower or upper digestive tract. The endoscope observer 10 has an insertion part 20 that is inserted into the subject (e.g., stomach, large intestine) and has a front end and a base end; a hand-held operating part 21 connected to the base end of the insertion part 20 and for the operator, i.e., the physician, to hold and perform various operations; and a universal tether 22 connected to the hand-held operating part 21.

[0060] The insertion part 20 is narrow in diameter and elongated. The insertion part 20 is constructed by sequentially connecting a flexible soft part 25, a bending part 26 that can be bent by the operation of the hand operation part 21, and a front end part 27 that houses a camera optical system (objective lens) and camera element 28 (not shown) from its base end side to the front end side.

[0061] The imaging element 28 is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type imaging element. The image light from the observed area is incident on the imaging surface of the imaging element 28 via an observation window (not shown) opened on the front end face of the front end 27 and an objective lens (not shown) disposed behind the observation window. The imaging element 28 captures the image light (converted into an electrical signal) incident on its imaging surface and outputs an imaging signal. That is, medical images are sequentially captured by the imaging element 28. Furthermore, medical images are acquired as dynamic images 38 and still images 39, which will be described later.

[0062] The handheld operating unit 21 is equipped with various operating components operated by the user (doctor). Specifically, the handheld operating unit 21 is equipped with two bending operation knobs 29 for bending operation of the bending section 26, an air and water supply button 30 for air and water supply operation, and a suction button 31 for suction operation. In addition, the handheld operating unit 21 is equipped with a still image photography indicator 32 for photographing the observed area in a still image 39, and a treatment device inlet 33 for inserting a treatment device (not shown) into the treatment device insertion passage (not shown) that is inserted into the insertion section 20.

[0063] The universal plug 22 is a connecting plug used to connect the endoscope observer 10 to the light source device 11. The universal plug 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) that pass through the insertion part 20. Additionally, at the end of the universal plug 22 are a connector 37a that connects to the light source device 11 and a connector 37b that branches off from the connector 37a and connects to the endoscope processor device 12.

[0064] By connecting connector 37a to the light source device 11, light guide 35 and fluid tube (not shown) are inserted into the light source device 11. Thus, necessary illumination light, water, and gas are supplied to the endoscope observation device 10 from the light source device 11 via light guide 35 and fluid tube (not shown). As a result, illumination light is shone from the illumination window (not shown) on the front end face of the front end 27 toward the observed area. Furthermore, according to the pressing operation of the aforementioned air / water supply button 30, gas or water is sprayed from the air / water supply nozzle (not shown) on the front end face of the front end 27 toward the observation window (not shown) on the front end face.

[0065] By connecting connector 37b to endoscope processor device 12, signal cable 36 is electrically connected to endoscope processor device 12. Thus, via signal cable 36, image signal of the observed part is output from camera element 28 of endoscope observer 10 to endoscope processor device 12, and control signal is output from endoscope processor device 12 to endoscope observer 10.

[0066] The light source device 11 supplies illumination light to the light guide 35 of the endoscope observer 10 via connector 37a. The illumination light is selected from various wavelength bands corresponding to the observation purpose, such as white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or combinations thereof.

[0067] The endoscope processor 12 controls the operation of the endoscope observer 10 via connector 37b and signal cable 36. Furthermore, based on the imaging signal acquired from the imaging element 28 of the endoscope observer 10 via connector 37b and signal cable 36, the endoscope processor 12 generates an image (also referred to as "moving image 38") consisting of a time-series of frame images 38a including the subject image. Moreover, when the still image photography instruction unit 32 is operated using the handheld operation unit 21 of the endoscope observer 10, the endoscope processor 12 simultaneously sets one frame image 38a from the moving image 38 as a still image 39 corresponding to the timing of the photography instruction, in conjunction with the generation of the moving image 38.

[0068] The dynamic image 38 and the static image 39 are medical images obtained by capturing images inside the body of the patient, i.e., inside a biological organ. Furthermore, when the dynamic image 38 and the static image 39 are images obtained using light (special light) in the aforementioned specific wavelength band, they are special light images. Then, the endoscope processor device 12 outputs the generated dynamic image 38 and static image 39 to the display device 13 and the medical image processing device 14. Moreover, the dynamic image 38 is acquired at 30fps (frames per second), 60fps, or 120fps.

[0069] Furthermore, the endoscope processor device 12 can also generate (acquire) a special light image with information of the specific wavelength band described above, based on a normal light image obtained through the aforementioned white light. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. Moreover, the endoscope processor device 12 obtains a signal of the specific wavelength band by performing calculations based on the color information of red, green, and blue [RGB (Red, Green, Blue)] or cyan, magenta, and yellow [CMY (Cyan, Magenta, Yellow)] contained in the normal light image.

[0070] Alternatively, the endoscope processor device 12 can generate a known characteristic image, such as an oxygen saturation image, based on at least one of a normal light image obtained through the aforementioned white light and a special light image obtained through the aforementioned light of a specific wavelength band (special light). In this case, the endoscope processor device 12 functions as a characteristic image generation unit. Furthermore, the dynamic image 38 or static image 39, including the aforementioned in vivo images, normal light images, special light images, and characteristic images, are medical images obtained by capturing images of a human body or by visualizing measurement results for image-based diagnosis and examination.

[0071] The display device 13 is connected to the endoscope processor device 12 and functions as a display unit that displays the dynamic image 38 and static image 39 input from the endoscope processor device 12. While checking the dynamic image 38 displayed on the display device 13, the user performs operations such as advancing and retracting the insertion unit 20. When a lesion is found at the observed site, the user operates the static image imaging instruction unit 32 to perform static image imaging of the observed site, and also performs diagnostic procedures, biopsies, etc. Furthermore, the dynamic image 38 and static image 39 are also displayed on the monitor 16, which is connected to the medical image processing device 14 described later. Additionally, when the dynamic image 38 and static image 39 are displayed on the monitor 16, a notification sound, described later, is also output. Therefore, it is preferable for the user to observe the display on the monitor 16 for diagnostic purposes.

[0072] <First Implementation Method>

[0073] [Medical Image Processing Device]

[0074] Figure 2This is a block diagram showing the structure of the medical image processing device 14. The medical image processing device 14 sequentially acquires time-series medical images, detects regions of interest, and outputs a notification sound. The medical image processing device 14 is, for example, a computer. The operation unit 15 includes, in addition to a keyboard and mouse that are wired or wirelessly connected to the computer, buttons located on the hand-held operation unit 21 of the endoscope observer 10. The display (display unit) 16 uses various monitors, such as an LCD monitor that can be connected to a computer.

[0075] The medical image processing apparatus 14 comprises a medical image acquisition unit 40, a CPU (Central Processing Unit) 41, a region of interest detection unit 42, a display control unit 46, a sound control unit 47, and a memory 48. Processing in each unit is performed by one or more processors. Here, the processor may be CPU 41, or it may be one or more CPUs (not shown).

[0076] The CPU 41 operates based on the operating system stored in the memory 48 and various programs, including medical image processing programs, to uniformly control the medical image acquisition unit 40, the area of ​​interest detection unit 42, the display control unit 46, and the sound control unit 47. In addition, it functions as a part of these units.

[0077] The medical image acquisition unit 40 performs image reception processing on medical images (pictures) received in a time-series sequence. The medical image acquisition unit 40 is used in conjunction with the endoscope processor device 12 ( Figure 1 The medical image acquisition unit 40 acquires a continuous time-series medical image, including the image of the subject, from the endoscope processor device 12 via a wired or wireless image input / output interface (not shown). In this example, a dynamic image 38 captured by the endoscope observer 10 is acquired. Furthermore, when a static image 39 (described above) is captured during the capture of the dynamic image 38 using the endoscope observer 10, the medical image acquisition unit 40 acquires both the dynamic image 38 and the static image 39 from the endoscope processor device 12.

[0078] The region of interest detection unit 42 performs region of interest detection processing on the medical images acquired from the medical image acquisition unit 40. Here, a region of interest refers to a lesion, various parts of the organs being examined, etc. Specific examples of regions of interest (lesions) include cancer lesions, adenoma lesions, and hyperplastic lesions. The region of interest detection unit 42 is, for example, composed of a detector based on a Convolutional Neural Network model. The region of interest detection unit 42 is pre-trained using deep learning. Specifically, it is trained by inputting medical images into the Convolutional Neural Network model to detect regions of interest. Furthermore, the trained model constitutes the region of interest detection unit 42.

[0079] Furthermore, the region of interest detection unit 42 includes a continuous detection count recording unit 43 and a continuous non-detection count recording unit 44. The continuous detection count recording unit 43 records the number of frames in which the region of interest detection unit 42 continuously detects the region of interest. The continuous non-detection count recording unit 44 records the number of frames in which the region of interest detection unit 42 continuously fails to detect the region of interest. For example, if the region of interest detection unit 42 continuously detects the region of interest at a predetermined number of frames based on the records in the continuous detection count recording unit 43, it determines the detection of the region of interest. Additionally, for example, the region of interest detection unit 42 can calculate the continuous non-detection time (first time), which will be described later, based on the records in the continuous non-detection count recording unit 44.

[0080] The display control unit 46 causes the display 16 to display medical images. Additionally, the display control unit 46 causes the display 16 to display an emphasis display of the detected area of ​​interest. The display control unit 46 generates display image data based on the medical images (moving image 38) acquired by the medical image acquisition unit 40 and outputs it to the display 16. Furthermore, the display control unit 46 causes the display 16 to display, for example, an emphasis display of the detected area of ​​interest enclosed by a rectangle.

[0081] The voice control unit 47 performs voice control processing, and when the area of ​​interest detection unit 42 detects an area of ​​interest, it outputs a notification sound from the voice notification device 17. The voice notification device 17 is, for example, a speaker. The notification sound is output for a certain period of time. For example, the notification sound consists of a one-second "pong" sound followed by a two-second echo. Furthermore, the length of the notification sound can be changed by setting. For example, the notification sound (the "pong" sound) can be set by the user within a range that is not perceived as long by human senses (between 1 second and 2 seconds). In addition, the echo can be set by the user within a range that is not perceived as a continuous notification sound by human senses (between 2 seconds and 3 seconds).

[0082] Furthermore, the sound control unit 47 outputs a notification sound based on the continuous undetected time. The sound control unit 47 compares the continuous undetected time with a first threshold. If a region of interest is detected when the continuous undetected time is equal to the first threshold, a notification sound is output. Even if a region of interest is detected when the continuous undetected time is less than the first threshold, the sound control unit 47 stops outputting the notification sound. Here, the first threshold is set based on the number of frames 38a or the time. For example, if the first threshold is set based on the number of frames 38a, it is set to 30 frames per second for a 60fps motion picture. Alternatively, if the first threshold is set based on the time, it is set to 0.5 seconds. Furthermore, regarding the first threshold, for example, when considering the time required for colon examination and the number of observation sites, it is preferable to set it in the range of 0.5 seconds to 3.5 seconds, and more preferably in the range of 1.5 seconds to 2.5 seconds. Moreover, even when the first threshold is changed (in the second and third embodiments described below), it is preferable to change the first threshold within the range of 0.5 seconds to 3.5 seconds.

[0083] The memory 48 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), and a hard disk drive. Flash memory, ROM, and the hard disk drive are non-volatile memories that store various programs such as the operating system and medical image processing programs, as well as captured still images 39. RAM, on the other hand, is a volatile memory that functions as a temporary storage area for various programs stored in the non-volatile memory and as the operating area of ​​the CPU 41, capable of high-speed data read and write operations.

[0084] Next, the output of the notification sound based on the sound control unit 47 will be explained.

[0085] Figure 3 This diagram illustrates a previous example of repeatedly outputting notification sounds within a short period of time. Furthermore, the medical images 101A to 101H shown are frame images 38a constituting the dynamic image 38, but they do not correspond to the actual frame rate.

[0086] exist Figure 3In the example shown, medical images 101A to 101H, presented as a time sequence, are received by the medical image acquisition unit 40. Then, the region of interest detection unit 42 detects lesion F in medical images 101B to 101D, and also in medical images 101G to 101H. When the region of interest detection unit 42 detects lesion F in medical image 101C, the voice control unit 47 outputs a notification sound "Pong" via the voice notification device 17. In this case, the region of interest detection unit 42 is set to determine detection when the continuous detection count recording unit 43 records "2". Therefore, the timing of the detection of lesion F in medical image 101C by the voice control unit 47 is determined by the voice notification device 17 outputting a notification sound. Similarly, when lesion F is detected in medical image 101H, the voice control unit 47 also outputs a notification sound. The interval between the notification sound in medical image 101C and the notification sound in medical image 101H is relatively short (4 frames in the illustrated case (medical images 101D to 101G)). Thus, when the notification sound is repeatedly output at short intervals, it can sometimes cause discomfort to the user and impair the effectiveness of the notification sound in drawing attention. Therefore, in this embodiment, as explained below, the sound control unit 47 outputs the notification sound based on the continuous undetected time.

[0087] Figure 4 This diagram illustrates the situation where a notification sound is output based on the continuous period of no detection. In Figure 4 In the case shown, with Figure 3 Similarly, medical image acquisition unit 40 receives medical images 101A to 101H in a time sequence. Then, region of interest detection unit 42 detects regions of interest in medical images 101B to 101D and medical images 101G to 101H. In this example, the continuous non-detection time is set to, for example, two seconds. If the continuous non-detection time has not elapsed for two seconds, the sound control unit 47 stops outputting the notification sound even if a lesion F is detected. In the illustrated case, a region of interest is detected in medical image 101D. Afterward, during the period before the continuous non-detection time has elapsed, lesion F is detected in medical images 101G and 101H. Therefore, even if lesion F is detected in medical images 101G and 101H, the sound control unit 47 stops outputting the notification sound. In this way, by controlling the output of the notification sound according to the continuous non-detection time, the repeated output of the notification sound in a short period of time is suppressed, and the effectiveness of the notification sound in attracting attention can be prevented.

[0088] Next, the setting of the continuous non-detection time will be explained. The voice control unit 47 outputs a notification sound from the voice notification unit 17 based on the continuous non-detection time. Here, the continuous non-detection time refers to the time from the point when the detection of the region of interest ended in the past to the point when the detection of the region of interest is currently underway.

[0089] Figure 5 This diagram illustrates the continuous undetected time. Furthermore, frames H1 to H10 in the diagram are frame images 38a constituting the dynamic scene 38, but they do not correspond to the actual frame rate. Figure 5 The diagram shows frames H1 to H10 in a time sequence. It also shows the timing of the output notification sound (circles are shown in the frames where the notification sound is output), and whether a region of interest is detected in each frame (if a region of interest is detected, "detected" is highlighted; if no region of interest is detected, "not detected" is highlighted). In frames H1 and H2, the region of interest is detected by the region of interest detection unit 42. Subsequently, no region of interest is detected in frames H3 to H6, but a region of interest is detected in frames H7 to H10. Furthermore, a notification sound is output in frame H8. Here, the continuous non-detection time T is the period from frame H3 to frame H6. Moreover, because the continuous non-detection time T is above a set first threshold, the sound control unit 47 outputs a notification sound in frame H8. In this case, the region of interest detection unit 42 is set to determine detection when the continuous detection count recording unit 43 records "2", therefore, a notification sound is output at the timing of the detection of the region of interest in frame H8.

[0090] Figure 6 This diagram illustrates the intermittent detection of the area of ​​interest.

[0091] Figure 6 and Figure 5 Similarly, frames H1 to H10, arranged sequentially in time, are shown. Figure 6 In the case shown, regions of interest are detected in frames H2, H4, H6, and H7. The continuous non-detection time T is the period of frames H3 and H5. Moreover, the continuous non-detection time T is less than the set first threshold, so even when regions of interest are detected, the sound control unit 47 stops outputting the notification sound. In this way, by controlling the output of the notification sound according to the continuous non-detection time, no notification sound is output when regions of interest are intermittently detected, thus suppressing the repeated output of the notification sound within a short period of time.

[0092] Next, the image processing method using the medical image processing device 14 will be described.

[0093] Figure 7 This is a flowchart illustrating the image processing method. Furthermore, each step of the image processing method is performed by the processor of the medical image processing device 14 executing the program.

[0094] First, the medical image acquisition unit 40 receives a medical image (image receiving process: step S101). Then, the region of interest detection unit 42 detects a region of interest in the received medical image (region of interest detection process: step S102). If the region of interest detection unit 42 does not detect a region of interest in the received medical image, the consecutive undetected count recording unit 44 counts the number of undetected regions (step S109). Then, the consecutive detection count recording unit 43 initializes the consecutive detection count (returns to 0) (step S110).

[0095] On the other hand, when the region of interest detection unit 42 detects a region of interest in the received medical image, the continuous detection count recording unit 43 counts the number of detections (step S103). Then, the continuous detection count recording unit 43 determines whether the count of frames in which the region of interest was detected is above a predetermined value (sound control process: step S104). In this case, the region of interest detection unit 42 is set to determine the detection of the region of interest when it is above the predetermined value. Next, when the count of detections is above the predetermined value, the continuous non-detection count recording unit 44 determines whether the count of non-detections is above a first threshold (step S105). In this case, the continuous non-detection time is measured based on the number recorded by the continuous non-detection count recording unit 44. If the number of non-detections is less than the first threshold, the continuous non-detection count recording unit 44 initializes the number of non-detections (step S111), and the medical image acquisition unit 40 determines whether there is a new medical image (step S108). If there is a new medical image, the medical image is re-acquired (step S101). On the other hand, if the number of undetected items is above a first threshold, the continuous undetected item recording unit 44 initializes the count of undetected items (step S106). After that, the sound control unit 47 outputs a notification sound (sound control process: step S107). Then, the medical image acquisition unit 40 determines whether there is a new image (step S108), and if there is a new image, acquires the image (step S101).

[0096] As described above, according to this embodiment, a notification sound is output based on the continuous non-detection time from the time when the detection of the area of ​​interest ended in the past to the time when the area of ​​interest is currently being detected. Therefore, this embodiment can control the output of the notification sound based on the elapsed time since the detection of the area of ​​interest in the past, and can output the notification sound without causing discomfort to the user or compromising the effectiveness of the notification sound in alerting the user.

[0097] <Second Implementation Method>

[0098] Next, a second embodiment of the present invention will be described. In this embodiment, a past continuous non-detection time (second time) equivalent to the continuous non-detection time when the region of interest was previously detected is detected, and the first threshold is changed based on this past continuous non-detection time.

[0099] Figure 8 This is a block diagram illustrating the structure of the medical image processing apparatus 14 according to this embodiment. Furthermore, regarding... Figure 2 The parts that have already been explained are marked with the same symbol and the explanation is omitted.

[0100] The second time detection unit 51 performs second time detection processing, which detects the past continuous undetected time equivalent to the continuous undetected time when the area of ​​interest was detected by the area of ​​interest detection unit 42 in the past.

[0101] Figure 9 This is a graph illustrating the continuous periods of no detection in the past. In Figure 9 In, with Figure 5 Similarly, the example shown depicts frames H1 to H14 arranged sequentially in time. Figure 9 In the scenario shown, regions of interest are detected in frames H1, H7, H8, H12, H13, and H14. Furthermore, a notification sound is output when regions of interest are detected in frames H8 and H13. Here, when frame H13 is set as the current frame and the notification sound output in frame H13 is considered, the continuous undetected time T corresponds to the time in frames H9 to H11, and the past continuous undetected time TP corresponds to the time in frames H2 to H6. Thus, the second time detection unit 51 detects the past continuous undetected time TP. By pre-recording the continuous undetected time T, the second time detection unit 51 detects the past continuous undetected time TP.

[0102] Threshold change unit 52 ( Figure 8 The first threshold is changed based on the past continuous undetected time TP detected by the second time detection unit 51. The threshold changing unit 52 can change the first threshold in various ways.

[0103] For example, if the threshold changing unit 52 increases the first threshold when the past continuous undetected time TP is greater than or equal to a predetermined value, and the sound control unit 47 outputs a notification sound when a longer continuous undetected time is detected. Alternatively, if the threshold changing unit 52 decreases the first threshold when the past continuous undetected time TP is less than the predetermined value, and the sound control unit 47 outputs a notification sound when a shorter continuous undetected time is detected.

[0104] Alternatively, for example, the threshold changing unit 52 may increase the first threshold if the past continuous undetected time TP is less than a predetermined value, or decrease the first threshold if the past continuous undetected time TP is greater than or equal to a predetermined value.

[0105] As described above, according to this embodiment, a first threshold is changed based on the past continuous undetected time TP, and the output of the notification sound is controlled based on the changed first threshold. Therefore, this embodiment can output the notification sound at appropriate timing, without causing discomfort to the user or compromising the effectiveness of the notification sound in alerting the user.

[0106] <Third Implementation Method>

[0107] Next, a third embodiment of the present invention will be described. In this embodiment, the number of regions of interest detected by the region of interest detection unit 42 is counted, and a first threshold is changed according to the number of regions of interest.

[0108] Figure 10 This is a block diagram illustrating the structure of the medical image processing apparatus 14 according to this embodiment. Furthermore, regarding... Figure 2 The same symbols are used to mark the parts that have already been explained, and the explanations are omitted.

[0109] The region of interest counting unit 61 performs region of interest counting processing and records the number of regions of interest detected by the region of interest detection unit 42 in each (frame) of medical image.

[0110] The threshold changing unit 52 performs threshold changing processing to change the first threshold based on the count of the number of areas of interest count unit 61. For example, if the number of areas of interest is greater than or equal to a second threshold, the threshold changing unit 52 increases the first threshold. For example, the sound control unit 47 controls the output of a notification sound with a long continuous undetected time when many lesions are detected. This prevents the frequent output of notification sounds due to the detection of many lesions. Alternatively, if the number of areas of interest is less than the second threshold, the threshold changing unit 52 decreases the first threshold. This allows the output of a notification sound when only a few lesions are detected, preventing lesions from being missed.

[0111] Figure 11 This diagram illustrates the timing of the output notification sound in this embodiment.

[0112] Figure 11 (A) is a diagram illustrating the situation where the number of detected lesions is less than the second threshold. Figure 11 (B) is a diagram illustrating the case where the number of detected lesions is above the second threshold.

[0113] exist Figure 11In (A), the region of interest detection unit 42 detects lesion F1 in medical images 101B-101D, 101G, and 101H. Furthermore, the region of interest counting unit 61 counts the number of regions of interest detected in each medical image as "1". Because the number of detected regions of interest is insufficient for the second threshold (in this case, the second threshold is set to 2), the threshold changing unit 52 decreases the first threshold. Therefore, the sound control unit 47 outputs a notification sound at the timing of the detection of lesion F1 in medical image 101H based on the passage of a short, continuous period of non-detection (2 frames in the illustrated case).

[0114] exist Figure 11 In (B), the region of interest detection unit 42 detects lesions F1 and F2 in medical images 101B-101D, 101G, and 101H. Furthermore, the region of interest counting unit 61 counts the number of regions of interest detected in each medical image as "2". Because the number of detected regions of interest is above a second threshold (in this case, the second threshold is set to 2), the threshold changing unit 52 increases the first threshold. Therefore, the sound control unit 47 outputs a timing stop notification sound for the detection of lesions F1 and F2 in medical image 101H based on the passage of a short continuous undetected time (2 frames in the illustrated case). Furthermore, in the illustrated case, the region of interest detection unit 42 is set to detect when the number of records by the continuous detection count recording unit 43 is 2.

[0115] According to this embodiment, the number of detected areas of interest is counted. If the number is above a second threshold, the first threshold is increased; if the number is below the second threshold, the first threshold is decreased. Therefore, this embodiment can output a notification sound at an appropriate time, thus outputting the notification sound without causing discomfort to the user or compromising its effectiveness in alerting the user.

[0116] <Fourth Implementation Method>

[0117] Next, the fourth embodiment will be described. In this embodiment, the output of the notification sound is controlled based on the size of the detected area of ​​interest and the continuous period of no detection.

[0118] Figure 12 This is a block diagram illustrating the structure of the medical image processing apparatus 14 according to this embodiment. Furthermore, regarding... Figure 2 The same symbols are used to mark the parts that have already been explained, and the explanations are omitted.

[0119] The area calculation unit 71 performs area calculation processing to calculate the area of ​​the region of interest detected by the region of interest detection unit 42. The area calculation unit 71 can calculate the area of ​​the detected region of interest by performing image processing.

[0120] The sound control unit 47 controls the output of the notification sound based on the size of the area of ​​interest and the continuous non-detection time. For example, the sound control unit 47 outputs a notification sound when the area of ​​the area of ​​interest is less than a third threshold. Conversely, the sound control unit 47 stops outputting the notification sound when the area of ​​the area of ​​interest is greater than the third threshold and the continuous non-detection time is less than a first threshold. Additionally, the sound control unit 47 outputs a notification sound when the area of ​​the area of ​​interest is greater than the third threshold and the continuous non-detection time is greater than or equal to the first threshold.

[0121] Figure 13 This diagram illustrates the timing of the output notification sound in this embodiment. Figure 13 (A) is a graph representing the case where the area of ​​the detected lesion F is less than the third threshold. Figure 13 (B) is a graph representing the case where the area of ​​the detected lesion F is above the third threshold.

[0122] exist Figure 13 In (A), the region of interest detection unit 42 detects lesion F in medical images 101B-101D, 101G, and 101H. Furthermore, the area calculation unit 71 calculates the area of ​​the lesion F detected in each of the medical images 101B-101D, 101G, and 101H. Figure 13 In the case shown in (A), because the area of ​​lesion F detected in each of medical images 101B-101D, 101G, and 101H is less than the third threshold, a timed notification sound is output for detecting lesion F in each of medical images 101B-101D, 101G, and 101H, regardless of the continuous undetected time. Thus, when the area of ​​interest is small, a notification sound is output based on the detection of the area of ​​interest to remind the user to pay attention, thereby preventing the user from missing the area of ​​interest.

[0123] exist Figure 13 In the case shown in (B), the area of ​​lesion F is above the third threshold. Therefore, the sound control unit 47 outputs a notification sound based on the continuous undetected time. Here, in Figure 13 In the case shown in (B), because the continuous undetected time is below the first threshold, the sound control unit 47 stops outputting the notification sound even though it detects the lesion F in the medical images 101G and 101H. Thus, by stopping the output of the notification sound, even in a large area of ​​interest, the user can still make sufficient visual identification, thereby preventing the output of the notification sound beyond what is needed.

[0124] As described above, according to this embodiment, the output of the notification sound is controlled based on the area of ​​the detected area of ​​interest, thus enabling the output of the notification sound without causing discomfort to the user or compromising the effectiveness of the notification sound in attracting attention.

[0125] <Fifth Implementation Method>

[0126] Next, the fifth embodiment of the present invention will be described. In this embodiment, the output of the notification sound is controlled according to the type of the region of interest detected by the region of interest detection unit 42.

[0127] Figure 14 This is a block diagram illustrating the structure of the medical image processing apparatus 14 according to this embodiment. Furthermore, regarding... Figure 2 The same symbols are used to mark the parts that have already been explained, and the explanations are omitted.

[0128] The classification unit 81 performs classification processing on the regions of interest detected by the region of interest detection unit 42. For example, if the region of interest is a lesion, the classification unit 81 classifies it according to the type or severity of the lesion. Alternatively, if the region of interest is a site of examination, the classification unit 81 classifies it according to the type of site. The classification unit 81 can classify the detected regions of interest using various methods. For example, the classification unit 81 can use a classifier composed of a learned CNN model to classify the detected regions of interest.

[0129] The voice control unit 47 controls the output of the notification sound according to the type of the area of ​​interest. For example, the voice control unit 47 outputs a notification sound when the classification result of the classification by the classification unit 81 is a specific type. Here, the specific type refers to a serious lesion, a severely injured lesion, a lesion that is particularly noteworthy and cannot be missed, or a landmark site, etc. Furthermore, the voice control unit 47 stops outputting the notification sound when the classification result of the classification by the classification unit 81 is an unspecific type and the continuous non-detection time is less than a first threshold. Conversely, the voice control unit 47 outputs a notification sound via the voice notification device 17 when the classification result of the classification by the classification unit 81 is an unspecific type and the continuous non-detection time is greater than or equal to the first threshold.

[0130] Figure 15 This diagram illustrates the timing of the output notification sound in this embodiment. Figure 15 (A) is a graph representing the case where the detected lesion F is classified as a specific type. Figure 15 (B) is a graph representing cases where the detected lesion F is classified as not belonging to a specific type.

[0131] exist Figure 15In (A), the region of interest detection unit 42 detects lesion F in medical images 101B-101D, 101G, and 101H. Furthermore, the classification unit 81 classifies the lesion F detected in each of the medical images 101B-101D, 101G, and 101H. Figure 15 In the case shown in (A), lesion F is of a specific type. Therefore, regardless of the continuous undetected time, a notification sound is output when lesion F is detected in each of the medical images 101B-101D, 101G, and 101H. In this way, when lesion F is of a specific type (e.g., a severe lesion), a notification sound is output for each detection, preventing the user from missing areas of interest.

[0132] exist Figure 15 In the case shown in (B), lesion F is of an unusual type. Therefore, the sound control unit 47 outputs a notification sound based on the continuous undetected time. Here, in Figure 15 In the case shown in (B), the continuous undetected time is below the first threshold, so the sound control unit 47 stops the output of the notification sound even though it detects the lesion F in the medical images 101G and 101H.

[0133] As described above, according to this embodiment, since the output of the notification sound is controlled based on the classification of the detected area of ​​interest, the notification sound can be output without causing discomfort to the user or compromising the effectiveness of the notification sound in attracting attention.

[0134] <Other>

[0135] In the above embodiments, the hardware structure of the processing unit (medical image acquisition unit 40, region of interest detection unit 42, second time detection unit 51, threshold changing unit 52, region of interest counting unit 61, area calculation unit 71, classification unit 81) of the medical image processing apparatus 14 is a variety of processors as shown below. Among the various processors, there are general-purpose processors that execute software (programs) and function as various processing units, i.e., CPUs (Central Processing Units); processors such as FPGAs (Field Programmable Gate Arrays) whose circuit structures can be changed after manufacturing, i.e., Programmable Logic Devices (PLDs); and processors such as ASICs (Application Specific Integrated Circuits) that have circuit structures specifically designed for performing specific processes, i.e., dedicated circuits.

[0136] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of CPU and FPGA). Alternatively, a single processor can constitute multiple processing units. Examples of a single processor constituting multiple processing units include, firstly, client or server computers, which use a combination of one or more CPUs and software to form a single processor that functions as multiple processing units. Secondly, there are systems-on-a-chip (SoCs), which use a single integrated circuit (IC) chip to implement the overall functionality of a system containing multiple processing units. In this way, various processing units can be constructed using one or more of the aforementioned processors as hardware structures.

[0137] Furthermore, more specifically, the hardware architecture of these various processors is a circuit composed of circuit elements such as semiconductor components.

[0138] The aforementioned structures and functions can be suitably implemented by any hardware, software, or a combination of both. For example, the present invention can also be applied to a program that causes a computer to perform the above-described processing steps (processing sequence), a computer-readable recording medium (non-temporary recording medium) containing such a program, or a computer on which such a program can be installed.

[0139] The examples of the present invention have been described above, but the present invention is not limited to the embodiments described above. Of course, various modifications can be made without departing from the spirit of the present invention.

[0140] Symbol Explanation

[0141] 9: Endoscopic System

[0142] 10: Endoscopic Observation Device

[0143] 11: Light source device

[0144] 12: Endoscopic processor device

[0145] 13: Display device

[0146] 14: Medical image processing device

[0147] 15: Operations Department

[0148] 16: Monitor

[0149] 17: Voice Notification Device

[0150] 20: Insertion section

[0151] 21: Hands-on Operations Department

[0152] 22: Universal Rope

[0153] 25: Soft parts

[0154] 26: Curved section

[0155] 27: Front end

[0156] 28: Camera components

[0157] 29: Bend the operating knob

[0158] 30: Gas and water supply buttons

[0159] 31: Attraction Button

[0160] 32: Still Image Photography Instruction Department

[0161] 33: Treatment device inlet

[0162] 35: Optical guide

[0163] 36: Signal cable

[0164] 37a: Connector

[0165] 37b: Connector

[0166] 38: Dynamic Images

[0167] 38a: Frame Image

[0168] 39: Still Images

[0169] 40: Medical Image Acquisition Unit

[0170] 41: CPU

[0171] 42: Focus on the regional testing department

[0172] 43: Continuous Detection Recording Department

[0173] 44: Recording Department for Continuous Undetected Counts

[0174] 46: Display Control Unit

[0175] 47: Sound Control Department

[0176] 48: Memory.

Claims

1. A medical image processing device, comprising a processor and a voice notification device for outputting notification sounds, wherein, The processor performs: Image receiving and processing: receiving images in a continuous time sequence; Region of interest detection processing is performed to detect regions of interest from the image; as well as The voice control processing, when a region of interest is detected by the region of interest detection processing, controls the voice notification device to output the notification sound for a certain period of time. The notification sound is output from the voice notification device based on a first time interval from the point when the detection of a region of interest ended by the region of interest detection processing to the point when the detection of a region of interest currently exists. In the aforementioned sound control processing. If the first time period is above a first threshold, the notification sound is output through the voice notification device. If the first time interval is less than the first threshold, the output of the notification sound is stopped by the voice notification device. The processor performs: The second time-based detection process is equivalent to the second time of the first time when the region of interest was detected in the past through the region of interest detection process. as well as Threshold change processing: The first threshold is changed according to the second time.

2. The medical image processing device according to claim 1, wherein, The processor performs: The number of interest regions detected in the image by the interest region detection process is counted. In the threshold change process, If the number of regions of interest is greater than or equal to a second threshold, the first threshold is increased.

3. The medical image processing device according to claim 1, wherein, The processor performs: A region of interest counting process is used to count the number of regions of interest detected in the image by the region of interest detection process. In the threshold change process, If the number of regions of interest is less than the second threshold, the first threshold is reduced.

4. The medical image processing device according to claim 1, wherein, The first threshold is set in the range of more than 1.5 seconds to less than 2.5 seconds.

5. The medical image processing apparatus according to any one of claims 1 to 3, wherein, The first threshold changes within the range of more than 0.5 seconds to less than 3.5 seconds.

6. The medical image processing apparatus according to any one of claims 1 to 4, wherein, The processor performs: The area calculation process involves calculating the area of ​​the region of interest detected in the region of interest detection process. In the aforementioned sound control processing. If the area calculated in the area calculation process is less than the third threshold, the notification sound is output through the voice notification device. If the area calculated in the area calculation process is greater than or equal to a third threshold, and the first time is less than the first threshold, then the output of the notification sound based on the voice notification device is stopped. If the area calculated in the area calculation process is above the third threshold and the first time is above the first threshold, the notification sound is output through the voice notification device.

7. The medical image processing apparatus according to any one of claims 1 to 4, wherein, The processor performs classification processing, classifying the regions of interest detected in the region of interest detection process. In the aforementioned sound control processing. If the classification result in the classification process is a specific type, the notification sound is output through the voice notification device. If the classification result in the classification process is an unspecified type and the first time is less than the first threshold, the output of the notification sound based on the voice notification device shall be stopped. If the classification result in the classification process is an unspecified type and the first time is above the first threshold, the notification sound is output through the voice notification device.

8. The medical image processing apparatus according to any one of claims 1 to 4, wherein, In the sound control processing, the first time is detected based on the number of images that are consecutive in time sequence.

9. The medical image processing apparatus according to any one of claims 1 to 4, wherein, It includes a display unit that displays the image received in the image receiving process. The region of interest detected through the region of interest detection process is highlighted and displayed on the display unit.

10. A medical image processing method, comprising a medical image processing apparatus equipped with a processor and a voice notification device for outputting notification sounds, wherein, The processor executes: The image receiving process receives images in a continuous time sequence. The region of interest detection process detects the region of interest from the image; as well as In the voice control process, when a region of interest is detected by the region of interest detection process, control is performed to cause the voice notification device to output the notification sound for a certain period of time. The notification sound is output from the voice notification device based on a first time interval from the point when the region of interest detection process ended to the point when the region of interest detection process currently detects a region of interest. In the sound control process. If the first time period is above a first threshold, the notification sound is output through the voice notification device. If the first time interval is less than the first threshold, the output of the notification sound is stopped by the voice notification device. The processor performs: The second time detection process detects a second time equivalent to the first time when the region of interest was detected in the past through the region of interest detection process. as well as The threshold change process involves changing the first threshold according to the second time.

11. A recording medium that is non-transitory and computer-readable, wherein, The system contains a program that causes a computer to perform the medical image processing method of claim 10.

Images