Medical image processing apparatus, medical image processing method, program product, and storage medium

Abstract

The invention provides a medical image processing device, a medical image processing method, a program product, and a storage medium, by which a user can correctly recognize a plurality of recognition results. A medical image processing device according to the present invention is provided with a processor that performs: a process for acquiring a plurality of time-series medical images; performing a first recognition process from a first medical image among the plurality of time-series medical images; performing a second recognition process different from the first recognition process from a second medical image that follows the first medical image in the time sequence among the first medical image or the plurality of medical images in the time sequence; notifying a first recognition result of the first recognition process; notifying a second recognition result of the second recognition process; and a notification method for changing the second recognition result in accordance with the notification state of the first recognition result.

Description

Technical Field

[0001] This invention relates to a medical image processing apparatus, a medical image processing method, a program product, and a storage medium, and particularly to a technology for controlling the notification of recognition results of medical images. Background Technology

[0002] It is known that there are AI-assisted functions that use artificial intelligence (AI) to detect lesions based on images taken by medical imaging diagnostic devices such as endoscopes or ultrasound diagnostic devices.

[0003] For example, Patent Document 1 describes a computer program that causes a computer to perform the following processing: acquiring an image taken by an endoscope; inputting the image taken by the endoscope to a first recognizer and a second recognizer, wherein the first recognizer identifies a lesion based on the image, and the second recognizer identifies the lesion based on the image with a higher recognition accuracy than the first recognizer; acquiring provisional information containing the recognition result identified by the first recognizer; outputting an image containing the acquired provisional information; acquiring definitive information regarding the provisional information containing the recognition result identified by the second recognizer; and outputting an image containing the acquired definitive information.

[0004] Patent Document 2 describes an endoscope system for examining a patient's lumen. The endoscope system includes: an insertion unit for insertion into the lumen; a camera for capturing images of the lumen to obtain endoscopic images; a region of interest detection unit for detecting a region of interest from the endoscopic images; a detection result notification unit for notifying the detection result of the region of interest; an insertion / removal discrimination unit for determining whether the examination process is an insertion or removal process; and a notification control unit for notifying the detection result notification unit based on the process determined by the insertion / removal discrimination unit.

[0005] Furthermore, Patent Document 3 describes a medical image processing apparatus that includes a notification control unit. The notification control unit controls either a notification state that notifies the notification information contained in the medical image or a non-notification state that sets the notification information to a non-notification state when the medical image meets the non-notification conditions, and sets the notification information to a notification state after a non-notification maintenance time after the medical image no longer meets the non-notification conditions.

[0006] Patent Document 1: International Publication No. 2021 / 010225

[0007] Patent Document 2: International Publication No. 2020 / 017212

[0008] Patent Document 3: International Publication No. 2020 / 090729

[0009] These accessibility features need to notify users of AI results in real time during the inspection process. However, when multiple features are implemented, the timing of these notifications may overlap. If multiple notifications overlap, users may not be able to correctly identify the notification information due to ignoring one or confusion. Summary of the Invention

[0010] The present invention was made in view of this situation, and its object is to provide a medical image processing device, medical image processing method, program product and storage medium that enable a user to correctly identify multiple recognition results.

[0011] To achieve the above objectives, the medical image processing apparatus according to the first aspect of the present invention includes a processor, wherein the processor performs the following processing: acquiring a plurality of medical images in a time series; performing a first recognition process on a first medical image from the plurality of medical images in the time series; performing a second recognition process different from the first recognition process on a second medical image from the first medical image or a second medical image in the time series after the first medical image in the plurality of medical images in the time series; notifying the first recognition result of the first recognition process; notifying the second recognition result of the second recognition process; and a notification method for changing the second recognition result according to the notification status of the first recognition result.

[0012] The medical image processing apparatus according to the second aspect of the present invention is preferably, in the medical image processing apparatus according to the first aspect, wherein the processor notifies at least one of the first recognition result and the second recognition result through image information.

[0013] The medical image processing apparatus according to the third aspect of the present invention is preferably, in the medical image processing apparatus according to the first or second aspect, wherein the processor notifies at least one of the first recognition result and the second recognition result via sound information.

[0014] The medical image processing apparatus according to the fourth aspect of the present invention is preferably a medical image processing apparatus according to any one of the first to third aspects, wherein the processor performs the following processing: notifying the second recognition result by a plurality of information including at least image information and sound information; and a notification method for changing any one of the plurality of information according to the notification status.

[0015] The medical image processing apparatus according to the fifth aspect of the present invention is preferably the medical image processing apparatus according to the fourth aspect, wherein the processor changes the notification method of the sound information according to the notification status.

[0016] The medical image processing apparatus according to the sixth aspect of the present invention preferably includes the medical image processing apparatus according to any one of the second to fifth aspects, wherein the processor performs the following processing: notifying the first recognition result by means of first image information in a first display unit displaying a medical image; and notifying the second recognition result by means of second image information in a second display unit different from the first display unit.

[0017] The medical image processing apparatus according to the seventh aspect of the present invention is preferably a medical image processing apparatus according to any one of the first to sixth aspects, wherein the first identification process is a region of interest detection process for detecting a region of interest contained in a first medical image or a region of interest classification process for classifying a region of interest contained in a first medical image.

[0018] The medical image processing apparatus according to the eighth aspect of the present invention is preferably the same as that according to the seventh aspect, wherein the processor performs the following processing: notifying the first recognition result for a specified period of time; and notifying the second recognition result after the specified period of time.

[0019] The medical image processing apparatus according to the ninth aspect of the present invention preferably includes the medical image processing apparatus according to any one of the first to eighth aspects, wherein the processor changes the notification method according to whether or not a notification of the first recognition result is received.

[0020] The medical image processing apparatus according to the 10th aspect of the present invention preferably includes the medical image processing apparatus according to any one of the 1st to 9th aspects, wherein the processor changes the notification method according to whether or not a notification of the first recognition result is received within a preset time period or whether or not a change in the notification status is received.

[0021] The medical image processing apparatus according to the 11th aspect of the present invention preferably includes the second recognition process in any of the 1st to 10th aspects of the medical image processing apparatus, wherein the second recognition process is a location recognition process for recognizing the observation area of ​​the first medical image or the second medical image.

[0022] The medical image processing apparatus according to the 12th aspect of the present invention is preferably, in the medical image processing apparatus according to the 11th aspect, capable of visually recognizing the parts captured and the parts not captured by multiple medical images in a time series as a second recognition result.

[0023] The medical image processing apparatus according to the 13th aspect of the present invention is preferably a medical image processing apparatus according to any one of the 1st to 12th aspects, wherein the processor, as a notification method, changes at least one of the notification time, notification moment, and notification level of the second recognition result.

[0024] The medical image processing apparatus according to the 14th aspect of the present invention is preferably, in the medical image processing apparatus according to the 13th aspect, wherein the processor changes the notification time of the second recognition result to after the notification time of the first recognition result.

[0025] To achieve the above objectives, in the medical image processing method according to the 15th aspect of the present invention, the processor performs the following processing: acquiring a plurality of medical images in a time series; performing a first recognition process on a first medical image in the plurality of medical images in the time series; performing a second recognition process different from the first recognition process on a second medical image in the time series after the first medical image or a second medical image in the time series after the first medical image; notifying the first recognition result of the first recognition process; notifying the second recognition result of the second recognition process; and a notification method for changing the second recognition result according to the notification status of the first recognition result.

[0026] To achieve the above objectives, the program involved in the 16th aspect of the present invention is a program that causes a computer to execute the medical image processing method involved in the 15th aspect. The present invention also includes a non-transitory, computer-readable storage medium storing the program involved in the 16th aspect.

[0027] In the medical image processing method of the 15th aspect and the procedure involved in the 16th aspect, it is possible to set up a structure that includes the same specific method as the medical image processing apparatus described above.

[0028] Invention Effects

[0029] According to the present invention, the user is able to correctly identify multiple identification results. Attached Figure Description

[0030] Figure 1 It is a schematic diagram showing the overall structure of the endoscope system.

[0031] Figure 2 This is a block diagram illustrating an example of the electrical structure of a medical image processing device.

[0032] Figure 3 This is a flowchart illustrating a medical image processing method.

[0033] Figure 4 This is an example table that indicates the time of output of lesion detection results and observation site identification results.

[0034] Figure 5 This is a diagram showing an example of a display device.

[0035] Figure 6 This is a block diagram illustrating an example of the electrical structure of a medical image processing device.

[0036] Figure 7This is a flowchart illustrating a medical image processing method.

[0037] Figure 8 This is a flowchart illustrating a medical image processing method. Detailed Implementation

[0038] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, the same reference numerals are used to denote the same constituent elements, and repeated descriptions are omitted.

[0039] <Overall Structure of the Endoscopic System>

[0040] Figure 1 This is a schematic diagram showing the overall structure of the endoscope system 9, which includes the medical image processing apparatus according to the present invention. Figure 1 As shown, the endoscope system 9 includes an endoscope observer 10 as an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a medical image processing device 14, an input device 15, and a display device 16.

[0041] The endoscope observer 10 is used to capture multiple endoscopic images in a time sequence, such as with a flexible endoscope. The endoscope observer 10 has: an insertion part 20, which is inserted into the body of the patient and has a front end and a base end; a handheld operating part 21, which is connected to the base end of the insertion part 20 and is held by the user (doctor) to perform various operations; and a universal lanyard 22, which is connected to the handheld operating part 21.

[0042] The insertion part 20 is generally formed into a narrow and elongated shape. The insertion part 20 is composed of the following components connected in sequence from its base end to its front end: a flexible soft part 25, a bending part 26 that can be bent by the operation of the hand-held operation part 21, and a front end part 27 that houses a camera optical system (objective lens) and imaging element 28 (not shown).

[0043] Imaging element 28 is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type imaging element. Image light from the observation area enters the imaging surface of imaging element 28 through an observation window (not shown) open at the front end face of the front end 27 and an objective lens (not shown) disposed behind the observation window. Imaging element 28 captures (converts) the image light incident on its imaging surface from the observation area into an electrical signal and outputs an image signal.

[0044] The handheld operating unit 21 is equipped with various operating components operated by the user. Specifically, the handheld operating unit 21 is equipped with two bending operation knobs 29 for bending the bending section 26, an air / water supply button 30 for air / water supply operation, and a suction button 31 for suction operation. Furthermore, the handheld operating unit 21 is equipped with: a still image camera indicator 32 for camera indication of a still image 39 of the observation area; and a treatment device inlet 33 for inserting a treatment device (not shown) into an insertion channel (not shown) that passes through the insertion section 20.

[0045] The universal plug 22 is a connecting plug for connecting the endoscope observer 10 to the light source device 11. The universal plug 22 contains a light guide 35, a signal cable 36, and a fluid hose (not shown) inserted into the insertion portion 20. Furthermore, at one end of the universal plug 22 is a connector 37a for connecting to the light source device 11, and a connector 37b branching from the connector 37a and connecting to the endoscope processor device 12.

[0046] By connecting connector 37a to the light source device 11, the light guide 35 and fluid hose are inserted into the light source device 11. Thus, the required illumination light, water, and gas are supplied to the endoscope observer 10 from the light source device 11 via the light guide 35 and fluid hose. As a result, the illumination light shines from an illumination window (not shown) on the front end face of the front end 27 toward the observation area. Furthermore, upon pressing the aforementioned air / water supply button 30, gas or water is sprayed from an air / water supply nozzle (not shown) on the front end face of the front end 27 toward the observation window (not shown) on the front end face.

[0047] By connecting connector 37b to endoscope processor device 12, signal cable 36 is electrically connected to endoscope processor device 12. Thus, via signal cable 36, imaging signal of the observed site is output from imaging element 28 of endoscope observer 10 to endoscope processor device 12, and control signal is output from endoscope processor device 12 to endoscope observer 10.

[0048] The light source device 11 supplies illumination light to the light guide 35 of the endoscope observer 10 via connector 37a. The illumination light can be selected from white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or various wavelength bands corresponding to the observation purpose such as combinations thereof. In addition, the specific wavelength band is a band narrower than the white wavelength band.

[0049] The first example of a specific wavelength band is, for example, the blue or green band in the visible light region. The wavelength band of this first example includes wavelengths above 390 nm and below 450 nm or above 530 nm and below 550 nm, and the light of the first example has a peak wavelength within the wavelength band above 390 nm and below 450 nm or above 530 nm and below 550 nm.

[0050] A second example of a specific wavelength band is, for example, the red band in the visible light region. This second example's wavelength band includes wavelengths above 585 nm and below 615 nm or above 610 nm and below 730 nm, and the light of the second example has a peak wavelength within the wavelength bands above 585 nm and below 615 nm or above 610 nm and below 730 nm.

[0051] The third example of a specific wavelength band includes wavelength bands where oxyhemoglobin and deoxyhemoglobin have different absorption coefficients, and the light of the third example has a peak wavelength in the wavelength bands where oxyhemoglobin and deoxyhemoglobin have different absorption coefficients. The wavelength band of this third example includes wavelength bands of 400±10 nm, 440±10 nm, 470±10 nm, or wavelengths above 600 nm and below 750 nm, and the light of the third example has a peak wavelength in the aforementioned wavelength bands of 400±10 nm, 440±10 nm, 470±10 nm, or wavelengths above 600 nm and below 750 nm.

[0052] The fourth example of a specific wavelength band is the wavelength band (390 nm to 470 nm) used to observe the fluorescence emitted by a fluorescent substance in vivo (fluorescence observation) and to excite the excitation light of the fluorescent substance.

[0053] The fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of this fifth example includes wavelength bands of 790 nm and above and 820 nm or 905 nm and above and 970 nm, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm and above and 820 nm or 905 nm and above and 970 nm.

[0054] The endoscope processor 12 controls the operation of the endoscope observer 10 via connector 37b and signal cable 36. Furthermore, the endoscope processor 12 generates a time-series frame image 38a (reference) based on the imaging signals acquired from the imaging element 28 of the endoscope observer 10 via connector 37b and signal cable 36. Figure 2 The time sequence of the moving images 38 is composed of, for example, 30 fps (frames per second).

[0055] Furthermore, when the still image camera instruction unit 32 is operated by the handheld operation unit 21 of the endoscope observer 10, the endoscope processor device 12 acquires one frame image 38a from the moving image 38 according to the time indicated by the camera instruction, and sets it as a still image 39. Then, the endoscope processor device 12 outputs the generated moving image 38 and still image 39 to the display device 13 and the medical image processing device 14, respectively.

[0056] Furthermore, the endoscope processor device 12 can generate (acquire) a special light image with information having the specific wavelength band described above, based on the ordinary light image obtained through the aforementioned white light. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. Then, the endoscope processor device 12 performs calculations based on the RGB color information of red, green, and blue, or the CMY color information of cyan, magenta, and yellow contained in the ordinary light image, thereby obtaining a signal with the specific wavelength band.

[0057] Furthermore, the endoscope processor device 12 can generate known characteristic images such as oxygen saturation images based on at least one of a normal light image obtained through the aforementioned white light and a special light image obtained through light of the aforementioned specific wavelength band (special light). In this case, the endoscope processor device 12 functions as a characteristic image generation unit. Additionally, the dynamic image 38 or still image 39, including the aforementioned in vivo image, normal light image, special light image, and characteristic image, are examples of medical images obtained by photographing the human body or visualizing measurement results for the purpose of image diagnosis and examination. The dynamic image 38 is an example of multiple medical images in a time sequence.

[0058] The display device 13 is connected to the endoscope processor device 12 and displays dynamic images 38 and still images 39 input from the endoscope processor device 12. While the user confirms the dynamic image 38 displayed on the display device 13, the user performs operations such as advancing and retracting the insertion unit 20. If a lesion is found at the observation site, the user operates the still image imaging instruction unit 32 to perform still image imaging of the observation site and perform diagnosis and biopsy.

[0059] The medical image processing device 14 functions as a diagnostic aid. It notifies the user of information contained in the medical image. This information may include, for example, the result of recognition processing of the medical image. The medical image processing device 14 may be, for example, a personal computer equipped with a processor.

[0060] Input device 15 uses a keyboard and mouse, etc., that are connected to a personal computer via wired or wireless connection. Display device 16 uses various displays, such as an LCD monitor, that can be connected to a personal computer. Medical image processing device 14 can be connected to sound output device 18 (see reference). Figure 6 The sound output device 18 uses a buzzer or a speaker.

[0061] <First Embodiment>

[0062] [Structure of a medical image processing device]

[0063] Figure 2 This is a block diagram illustrating an example of the electrical structure of the medical image processing device 14. Figure 2 The medical image processing apparatus 14 shown includes a processor 40 and a memory 42. The processor 40 is composed of an arithmetic unit such as a CPU (Central Processing Unit). The memory 42 is composed of storage elements such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 42 stores a program 44. The program 44 includes instructions executed by the processor 40. The program 44 can be provided by a computer-readable non-transitory storage medium. In this case, the processor 40 can read the program 44 from the non-transitory storage medium and store it in the memory 42.

[0064] The processor 40 includes an image acquisition unit 50, a first recognition processing unit 52, a second recognition processing unit 54, a first notification unit 56, a notification status detection unit 58, a second notification unit 60, and a display control unit 62. The functions of each of the image acquisition unit 50, the first recognition processing unit 52, the second recognition processing unit 54, the first notification unit 56, the notification status detection unit 58, the second notification unit 60, and the display control unit 62 are implemented by the processor 40 executing a program 44.

[0065] The image acquisition unit 50 sequentially acquires endoscopic images captured by the endoscope observer 10. Here, the image acquisition unit 50 is used in conjunction with the endoscope processor device 12 (reference 12). Figure 1 The image acquisition unit 50 acquires a dynamic image 38 composed of time-series frame images 38a from the endoscope processor device 12 via a wired or wireless image input / output interface (not shown). Furthermore, when a still image 39 is captured during the capture of the dynamic image 38 by the endoscope observer 10, the image acquisition unit 50 acquires both the dynamic image 38 and the still image 39 from the endoscope processor device 12.

[0066] Alternatively, the image acquisition unit 50 can acquire the moving image 38 via a non-temporary storage medium such as a memory card or hard disk, instead of directly acquiring the moving image 38 from the endoscope processor device 12. Furthermore, the image acquisition unit 50 can acquire the moving image 38 from a server or database uploaded to the Internet via the Internet.

[0067] The first recognition processing unit 52 performs a first recognition process on the endoscopic image acquired by the image acquisition unit 50 to obtain a first recognition result. The first recognition process is, for example, a region of interest detection process that detects regions of interest contained in the endoscopic image. The first recognition processing unit 52 includes a convolutional neural network (CNN) that calculates feature quantities of the endoscopic image and performs region of interest recognition processing within the endoscopic image.

[0068] Areas of concern may include lesions. Areas of concern may include at least one of the following: polyps, cancer, colonic diverticulum, inflammation, EMR (Endoscopic Mucosal Resection) scars, ESD (Endoscopic Submucosal Dissection) scars, clip sites, bleeding points, perforations, vascular atypia, and treatment instruments.

[0069] The first identification processing unit 52 can perform identification processing of the region of interest, such as classifying the detected region of interest into one of several categories related to the lesion, such as "tumorous", "non-tumorous", or "other".

[0070] Furthermore, the first recognition processing unit 52 is not limited to detecting regions of interest through CNN, but can also detect regions of interest by analyzing features such as color, pixel value gradient, shape, and size within the image through image processing.

[0071] The second recognition processing unit 54 performs a second recognition process, different from the first recognition process, on the endoscopic image acquired by the image acquisition unit 50, and obtains a second recognition result. The second recognition process may be, for example, an observation site recognition process for identifying the observation site in the endoscopic image. The second recognition processing unit 54 includes a CNN that calculates feature quantities of the endoscopic image and performs observation site recognition processing on the endoscopic image. The second recognition processing unit 54 may include processing to determine whether the endoscopic image acquired by the image acquisition unit 50 satisfies predetermined conditions such as jitter, composition, and location.

[0072] When the endoscope 10 is a lower endoscope inserted through the anus of the patient for observing the rectum and large intestine, the observation site may include at least one of the following: rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, ileum, and jejunum. When the endoscope 10 is an upper endoscope inserted through the mouth or nose of the patient for observing the esophagus and stomach, the observation site may include at least one of the following: pharynx, esophagus, stomach, and duodenum.

[0073] In addition, the second recognition processing unit 54 is not limited to detecting the region of interest through CNN, but can also identify the observation area by analyzing the color, pixel value gradient, shape, size and other features in the image through image processing.

[0074] The first notification unit 56 notifies the first recognition processing unit 52 of the first recognition result. Here, the first notification unit 56 displays the first recognition result on the display device 16 via the display control unit 62, thereby notifying the first recognition result through image information.

[0075] The notification status detection unit 58 detects the notification status of the first notification unit 56. The notification status of the first notification unit 56 includes, for example, the presence or absence of a notification based on the first identification result.

[0076] The second notification unit 60 notifies the second recognition processing unit 54 of the second recognition result. Here, the second notification unit 60 displays the second recognition result on the display device 16 via the display control unit 62, thereby notifying the second recognition result through image information.

[0077] Furthermore, the second notification unit 60 changes the notification method for the second identification result based on the detection result of the notification status detection unit 58, that is, based on the notification status of the first identification result. For example, the second notification unit 60 changes at least one of the notification time, notification moment, and notification level of the second identification result as the notification method.

[0078] The display control unit 62 controls the display of the display device 16 based on signals from the first notification unit 56 and the second notification unit 60.

[0079] Thus, the medical image processing device 14 notifies at least one of the first recognition result and the second recognition result via image information in the display device 16. The user can recognize the image information by visually recognizing the screen displayed on the display device 16.

[0080] [Medical Image Processing Methods]

[0081] Figure 3This is a flowchart illustrating a medical image processing method using the medical image processing apparatus 14. The medical image processing method is implemented by the processor 40 executing program 44. Here, the processing of the frame image 38a of the Nth frame of the moving image 38 will be described.

[0082] In step S1, the image acquisition unit 50 acquires the frame image 38a of the Nth frame of the moving image 38. The frame image 38a acquired in step S1 can be displayed on the display device 16.

[0083] In step S2, the first recognition processing unit 52 performs the first recognition processing, namely the region of interest detection processing, on the frame image 38a obtained in step S1, and obtains the first recognition result, namely the region of interest detection result.

[0084] In step S3, the first identification processing unit 52 determines whether the detection result of the region of interest obtained in step S2 should be notified. For example, if a region of interest is detected, the first identification processing unit 52 determines that the detection result of the region of interest should be notified, and if no region of interest is detected, it determines that the detection result of the region of interest should not be notified.

[0085] If the determination result in step S3 is "yes", that is, if the detection result of the region of interest should be notified, the first notification unit 56 performs the first notification process in step S4. The first notification process is the process of displaying the detection result of the region of interest as image information on the display device 16 via the display control unit 62.

[0086] On the other hand, if the determination result in step S3 is "no", that is, if the detection result of the area of ​​interest should not be notified, the first notification unit 56 will not perform the first notification process.

[0087] The processing steps S5 to S10 are performed in parallel with the processing steps S2 to S4. The processing steps S5 to S10 can be performed after the processing steps S2 to S4 are completed. In step S5, the notification status detection unit 58 determines whether it has received the second notification information based on the processing of the (N-1)th frame, which is the preceding frame of the Nth frame. The second notification information includes, for example, the observation part identification result of the observation part identification processing of the frame image 38a of the (N-1)th frame or a frame preceding it.

[0088] If the determination result in step S5 is "no", that is, if the second notification information is not received, in step S6, the second recognition processing unit 54 performs the second recognition processing, namely the observation part recognition processing, on the frame image 38a obtained in step S1, and obtains the second recognition result, namely the observation part recognition result.

[0089] In step S7, the second identification processing unit 54 determines whether the observation location identification result obtained in step S6 should be notified. For example, if the observation location is identified, the second identification processing unit 54 determines that the observation location identification result should be notified, and if the observation location is not identified, it determines that the observation location identification result should not be notified.

[0090] If the determination result in step S7 is "no", that is, if the observation site identification result should not be notified, the second notification unit 60 will not perform the second notification process.

[0091] On the other hand, if the determination result in step S7 is "yes," that is, if the observation part identification result should be notified, in step S8, the status detection unit 58 is notified to determine whether the first notification unit 56 is performing the first notification processing. Similarly, if the determination result in step S5 is "yes," that is, if the second notification information is received, in step S8, the status detection unit 58 is notified to determine whether the first notification unit 56 is performing the first notification processing.

[0092] If the determination result in step S8 is "no", that is, if the first notification unit 56 does not perform the first notification processing, the second notification unit 60 performs the second notification processing in step S9.

[0093] If it is determined in step S5 that the second notification information has not been received, the second notification process involves displaying the observation part identification result obtained in step S6 as image information on the display device 16 via the display control unit 62. Conversely, if it is determined in step S5 that the second notification information has been received, the second notification process involves displaying the observation part identification result as image information on the display device 16 based on the second notification information of the (N-1)th frame.

[0094] On the other hand, if the determination result in step S8 is "yes", that is, if the first notification unit 56 is performing the first notification processing, in step S10, the second notification unit 60 does not perform the second notification processing, but sends the second notification information to the processing of the (N+1)th frame, which is the next frame.

[0095] If it is determined in step S5 that the second notification information has not been received, the second notification information sent to the processing of the (N+1)th frame is the second notification information of the Nth frame, which includes the observation part identification result obtained in step S6. If it is determined in step S5 that the second notification information has been received, the second notification information is the second notification information received from the processing of the (N-1)th frame, which is the previous frame.

[0096] Thus, according to the medical image processing method, while the first notification processing is in progress, the notification time of the second notification processing is changed to after the notification time of the first notification processing. By performing this processing on a per-frame basis, the overlap of the notification times of the first and second notification processing can be avoided. This prevents the user from ignoring a recognition result, and enables the user to correctly identify multiple recognition results. The change in notification time is an example of the "notification method for changing the second recognition result" in this invention.

[0097] Here, the notification time of the second notification processing was changed while the first notification processing was in progress. However, the notification method can also be changed based on the presence or absence of a notification from the first notification processing that is temporally adjacent, or based on the presence or absence of a change in the notification status of the first notification processing. Temporally adjacent can be a pre-set time period tracing back from the current time.

[0098] For example, if the first notification is processed within a preset time period from the current time, the notification time of the second notification can be changed. Furthermore, if the notification status of the first notification changes within a preset time period from the current time, the notification time of the second notification can also be changed.

[0099] Furthermore, the first notification unit 56 can continue processing the first notification for a specified period of time. The second notification unit 60 can process the second notification after the specified period of time has elapsed.

[0100] For example, in step S3, the first identification processing unit 52 may determine that the detection result of the region of interest should be notified if a predetermined time has not been maintained since the last detection of the region of interest, and determine that the detection result of the region of interest should not be notified if a predetermined time has been maintained since the last detection of the region of interest.

[0101] The first notification process can be, for example, the display of a closed rectangular bounding box surrounding the detected region of interest. The bounding box may not be closed. Furthermore, the first notification process can be performed along a medical image (e.g., reference image). Figure 5 In the endoscopic image 102 shown, among the arc-shaped auxiliary circles displayed along the left edge and the arc-shaped auxiliary circles displayed along the right edge, the auxiliary circles that are close to the detected area of ​​interest are highlighted with a specified color. The first notification process can be a combination of the processing of displaying the rectangular box and the processing of the auxiliary circles.

[0102] Furthermore, the first notification unit 56, with the aim of not hindering the user's observation of the medical image, performs auxiliary circle processing, and keeps the auxiliary circle lit for a predetermined time (e.g., less than 1 second) after a lesion is detected. If the second recognition processing unit 54 performs second recognition processing on the medical image captured while the auxiliary circle is lit after the lesion detection, the second notification unit 60 can perform second notification processing after the predetermined time (after the auxiliary circle is turned off).

[0103] [Example of screen display]

[0104] Figure 4 This table represents an example of the output times for lesion detection based on the first recognition processing unit 52 and observation site identification based on the second recognition processing unit 54. Here, a lesion is detected at time t, but no lesion is detected at times before t (t-1) and after t (t+1). Similarly, for the observation site, it is identified at time t, but no observation site is identified at times before t (t-1) and after t (t+1).

[0105] Figure 5 This indicates that the display device 16 is relative to Figure 4 The example shown is a diagram illustrating the moment. Figure 5 In the diagram, 1000 represents time (t-1), 1002 represents time t, and 1004 represents the screen of display device 16 at time (t+1).

[0106] like Figure 5 As shown in Figure 1000, at time (t-1), in display device 16, at... Figure 5 The left-center region, i.e., the first display unit 100, displays an endoscopic image 102. Figure 5 The right-hand side of the middle section, i.e., the second display unit 110, shows a schematic diagram 112. At time (t-1), no lesion detection results or observation site identification results are displayed.

[0107] like Figure 5 As shown in 1002, at time t, in the display device 16, the endoscopic image 102 is displayed in the first display unit 100, and the schematic diagram 112 is displayed in the second display unit 110. Furthermore, in the display device 16, the frame 106 (an example of "first image information") surrounding the lesion 104 ("first identification result") as a lesion detection result) is superimposed on the endoscopic image 102. On the other hand, at time t, the observation site identification result is not displayed.

[0108] like Figure 5As shown in 1004, at time (t+1), in the display device 16, the first display unit 100 displays the endoscopic image 102, and the second display unit 110 displays the schematic diagram 112. Furthermore, in the display device 16, at the position of the observation site (an example of "second identification result") which is the observation site identification result in the schematic diagram 112, an observation position mark 114 (an example of "second image information") is superimposed on the schematic diagram 112. Through the observation position mark 114, the areas that have been captured by the dynamic image 38 and the areas that have not been captured can be displayed visually. Additionally, at time (t+1), no lesion detection results are displayed.

[0109] like Figure 4 As shown, a lesion is detected and the observation site is identified at time t. However, if the results of both the lesion detection and the observation site identification are displayed on the display device 16 at time t, the user may overlook one of the identification results. Therefore, as... Figure 5 As shown, the results of the observation site identification are displayed at the end of the lesion detection result display (t+1), avoiding the situation where the identification results are displayed simultaneously. Thus, when performing lesion detection and observation site identification, priority is given to lesion detection, which has a higher probability of occurrence and is performed in real time.

[0110] Here, in Figure 5 In this example, the left side of the display device 16 is designated as the first display unit 100, and the right side is designated as the second display unit 110. However, the configuration of the first display unit 100 and the second display unit 110 is not limited to this example. Furthermore, by using the display device 16 and a display device different from the display device 16, one of them can be designated as the first display unit 100, and the other as the second display unit 110.

[0111] <Second Implementation>

[0112] [Structure of a medical image processing device]

[0113] Figure 6 This is a block diagram illustrating an example of the electrical structure of the medical image processing device 14A. Figure 6 The medical image processing apparatus 14A shown includes a sound output control unit 64. The sound output control unit 64 controls the sound output of the sound output device 18 of the endoscope system 9 based on a signal from the second notification unit 60.

[0114] The second notification unit 60 displays the second recognition result on the display device 16 via the display control unit 62, thereby notifying the second recognition result via image information. Furthermore, the second notification unit 60 outputs the sound of the second recognition result to the sound output device 18 via the sound output control unit 64, thereby notifying the second recognition result via sound information.

[0115] Thus, the medical image processing device 14A notifies the second recognition result of multiple pieces of information, including at least image information and sound information. Additionally, the medical image processing device 14A can notify at least one of the first and second recognition results via sound information through the sound output device 18. The user can recognize the sound information by listening to the sound output by the sound output device 18.

[0116] [Medical Image Processing Methods]

[0117] The second notification unit 60 can change the notification method of any one of the multiple pieces of information, including image information and sound information, based on the detection result of the notification status detection unit 58, that is, based on the notification status of the first recognition result. The changed notification method can be the notification method for sound information.

[0118] <Third Implementation>

[0119] [Medical Image Processing Methods]

[0120] Figure 7 This is a flowchart illustrating a medical image processing method according to the third embodiment using medical image processing apparatus 14 or medical image processing apparatus 14A.

[0121] The processing in steps S1 to S4 is the same as that in the medical image processing method of the first embodiment. In step S11, the second recognition processing unit 54 performs observation site recognition processing as the second recognition process on the frame image 38a acquired in step S1, and obtains the second recognition result, i.e., the observation site recognition result.

[0122] In step S12, the second identification processing unit 54 determines whether the identification result of the observation part obtained in step S11 should be notified.

[0123] If the determination result in step S12 is "yes", that is, if the observation part identification result should be notified, the second notification unit 60 performs a second notification process in step S13. The second notification process is, for example, the process of displaying the observation part identification result obtained in step S11 as image information on the display device 16 via the display control unit 62.

[0124] Next, in step S14, the notification status detection unit 58 determines whether the first notification unit 56 is performing first notification processing. If the determination result in step S14 is "yes," that is, if the first notification unit 56 is performing first notification processing, in step S15, the second notification unit 60 sends the second notification information of the Nth frame to the next frame, i.e., the (N+1)th frame, for processing. The second notification information of the Nth frame includes the observation part identification result obtained in step S11.

[0125] On the other hand, if the determination result in step S14 is "no", that is, if the first notification unit 56 does not perform the first notification processing, the second notification unit 60 will not perform the processing in step S15.

[0126] Furthermore, if the determination result in step S12 is "no", that is, if the observation part identification result should not be notified, in step S16, the status detection unit 58 is notified to determine whether the second notification information of the (N-1)th frame has been received from the processing of the previous frame of the Nth frame, i.e., the (N-1)th frame.

[0127] If the determination result in step S16 is "yes", that is, if the second notification information is received from the processing of the (N-1)th frame, the second notification unit 60 performs the second notification processing of the (N-1)th frame in step S17. The second notification processing is the process of displaying the observation part identification result as image information on the display device 16 based on the second notification information received from the processing of the (N-1)th frame.

[0128] On the other hand, if the determination result in step S16 is "no", that is, if the second notification information is not received, the second notification unit 60 will not perform the processing in step S17.

[0129] Thus, by extending the notification time for the second notification during the first notification process, it is possible to prevent the user from ignoring the second identification result. This extension of the notification time is an example of the "notification method for changing the second identification result" in this invention.

[0130] <Fourth Implementation>

[0131] [Medical Image Processing Methods]

[0132] Figure 8 This is a flowchart illustrating a medical image processing method according to the fourth embodiment using medical image processing apparatus 14 or medical image processing apparatus 14A. Here, an example is described where the first recognition processing unit 52 performs observation area recognition processing as the first recognition processing, and the second recognition processing unit 54 performs region of interest detection processing as the second recognition processing.

[0133] The processing in steps S1 to S4 is the same as that in the medical image processing method of the first embodiment. In step S11, the second recognition processing unit 54 performs region of interest detection processing as a second recognition process on the frame image 38a acquired in step S1, and obtains the second recognition result, i.e., the region of interest detection result.

[0134] In step S12, the second identification processing unit 54 determines whether the detection results of the region of interest obtained in step S11 should be notified.

[0135] If the determination result in step S12 is "no", that is, if the detection result of the area of ​​interest should not be notified, the second notification unit 60 will not make a notification.

[0136] On the other hand, if the determination result in step S12 is "yes", that is, if the detection result of the area of ​​interest should be notified, in step S21, the status detection unit 58 is notified to determine whether the first notification unit 56 is performing the first notification processing.

[0137] If the determination result in step S21 is "No," that is, if the first notification unit 56 does not perform the first notification processing, then in step S22, the second notification unit 60 performs the second notification processing at the first notification level. The second notification processing is the processing of the notification of the detection result of the area of ​​interest obtained in step S11. The notification level can be a certain emphasis or a certain volume in the screen display, etc.

[0138] On the other hand, if the determination result in step S21 is "yes," that is, if the first notification unit 56 is performing the first notification processing, in step S23, the second notification unit 60 performs the second notification processing at a second notification level that is relatively higher than the first notification level. For example, the thickness of the frame surrounding the area of ​​interest in the display screen may be thicker than the first notification level, the brightness of the color of the frame surrounding the area of ​​interest in the display screen may be higher than the first notification level, and the volume of the sound in the notification area of ​​interest in the sound output may be higher than the first notification level.

[0139] Thus, when performing the first notification processing, by increasing the notification level of the second notification processing, it is possible to prevent the second identification result from being ignored. This change in notification level is an example of the "notification method for changing the second identification result" in this invention.

[0140] <Variation Example>

[0141] So far, an example of performing the first recognition process and the second recognition process on the frame image 38a of the Nth frame of the dynamic image 38 has been described. However, the first recognition process can also be performed on the frame image 38a of the Nth frame (an example of the "first medical image"), and the second recognition process can be performed on the frame image 38a of the time series after the (N+1)th frame after the Nth frame (an example of the "second medical image").

[0142] The first and second identification processes can be any one of the following: area of ​​interest detection processing, observation site identification processing, lesion identification processing, and examination status identification processing. Lesion identification processing involves identifying at least one piece of information related to the size, type, and shape of the lesion. Examination status identification processing involves identifying the examination status, such as the treatment status.

[0143] In embodiments 1 to 3, the first identification process is used as the primary identification process, and the second identification process is used as the auxiliary identification process. However, in embodiment 4, the second identification process is designated as the primary identification process, and the first identification process is designated as the auxiliary identification process. Preferably, functions with high ignore risk or requiring higher response speed, such as lesion detection, are designated as the primary identification process.

[0144] As a medical image, an endoscopic image has been used as an example, but the medical image processing device 14 and the medical image processing device 14A can be applied to medical images such as capsule endoscopy images and ultrasound images.

[0145] <Structure of Medical Image Processing Device>

[0146] In this embodiment, each process is executed by any computer. Furthermore, any computer can execute these processes via a processor, a program, or a combination thereof. This arbitrary computer can be a general-purpose computer, a special-purpose computer, a workstation, or other hardware capable of executing programs.

[0147] The processor can be composed of one or more hardware components, and the type of hardware is not limited. For example, the processor can be composed of programmable logic devices such as CPUs (Central Processing Units), MPUs (Micro Processing Units), FPGAs (Field Programmable Gate Arrays), dedicated circuits for performing specific processes such as ASICs (Application Specific Integrated Circuits), GPUs (Graphics Processing Units), or NPUs (Neural Processing Units). Furthermore, the processor has units or means that execute the various processes described in this embodiment. The type of hardware can be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a certain processor, these multiple hardware components can exist in physically separate devices or in the same device. Furthermore, in any embodiment, the order of the processor's processes is not limited to the above order and can be appropriately varied. Additionally, the hardware can be composed of circuits composed of semiconductor elements and other circuit elements.

[0148] Furthermore, this embodiment can also be implemented using hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode consist of programs. Furthermore, a program can be, for example, a group of program modules, each of which can be implemented by a processor configured to perform its respective function. A program can be program code or multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storage devices). A program can be divided and stored in multiple non-transitory computer-readable media existing in physically separate devices. Program code or code segments can represent steps, functions, subroutines, routines, subroutines, modules, software packages, classes, or commands, data structures, or any combination of program statements. Program code or code segments can be connected to other code segments or hardware circuitry by sending and receiving information, data, arguments, parameters, or the contents of memory.

[0149] <Other>

[0150] The scope of the present invention is not limited to the scope described in the above embodiments. The structures and other components in each embodiment can be appropriately combined among the embodiments without departing from the spirit of the present invention.

[0151] Symbol Explanation

[0152] 9-Endoscope system, 10-Endoscope observer, 11-Light source device, 12-Endoscope processor device, 13-Display device, 14, 14A-Medical image processing device, 15-Input device, 16-Display device, 18-Sound output device, 20-Insertion part, 21-Handheld operating part, 22-Universal plug cord, 25-Soft part, 26-Bend part, 27-Front end, 28-Imaging element, 29-Bend operating knob, 30-Air / water supply button, 31-Suction button, 32-Still image camera indicator, 33-Treatment instrument inlet, 35-Light guide, 36-Signal cable, 37a-Connector, 37b-Connector, 38-Dynamic image, 38 a-Frame image, 39-Still image, 40-Processor, 42-Memory, 44-Program, 50-Image acquisition unit, 52-First recognition processing unit, 54-Second recognition processing unit, 56-First notification unit, 58-Notification status detection unit, 60-Second notification unit, 62-Display control unit, 64-Sound output control unit, 100-First display unit, 102-Endoscopic image, 104-Lysfunction, 106-Frame, 110-Second display unit, 112-Schematic diagram, 114-Observation position marker, S1~S10-Steps of the medical image processing method, S11~S17-Steps of the medical image processing method, S21~S23-Steps of the medical image processing method.

Claims

1. A medical image processing device, comprising a processor, wherein, The processor performs the following processing: Acquire multiple medical images in a time series; A first identification process is performed on a first medical image from a plurality of medical images in the time series; A second recognition process, different from the first recognition process, is performed on a second medical image following the first medical image from among the first medical image or the time series of multiple medical images; The first identification result of the first identification process is notified; The notification indicates the second identification result of the second identification process; and The notification method for the second identification result is changed based on the notification status of the first identification result.

2. The medical image processing apparatus according to claim 1, wherein, The processor notifies at least one of the first recognition result and the second recognition result via image information.

3. The medical image processing apparatus according to claim 1, wherein, The processor notifies at least one of the first recognition result and the second recognition result via voice information.

4. The medical image processing apparatus according to claim 1, wherein, The processor performs the following processing: The second recognition result is notified by multiple pieces of information, including at least image and sound information; and The notification method for changing any one of the multiple pieces of information based on the notification status.

5. The medical image processing apparatus according to claim 4, wherein, The notification method by which the processor changes the sound information according to the notification status.

6. The medical image processing apparatus according to claim 2, wherein, The processor performs the following processing: The first recognition result is notified via first image information in the first display unit showing the medical image; and The second recognition result is communicated via second image information in a second display unit that is different from the first display unit.

7. The medical image processing apparatus according to claim 1, wherein, The first identification process is either a region of interest detection process or a region of interest classification process that classifies the regions of interest contained in the first medical image.

8. The medical image processing apparatus according to claim 7, wherein, The processor performs the following processing: The first identification result will be notified continuously for a specified period of time; and The second identification result is notified after the specified time has elapsed.

9. The medical image processing apparatus according to claim 1, wherein, The processor changes the notification method based on whether or not a notification of the first identification result is received.

10. The medical image processing apparatus according to claim 1, wherein, The processor changes the notification method based on whether or not a notification of the first identification result is received within a preset time period, or whether or not the notification status changes.

11. The medical image processing apparatus according to claim 1, wherein, The second identification process is a location identification process that identifies the observation area of ​​the first medical image or the second medical image.

12. The medical image processing apparatus according to claim 11, wherein, The processor is capable of visually recognizing the areas captured and uncaptured by multiple medical images from the time series, as the second recognition result.

13. The medical image processing apparatus according to any one of claims 1 to 12, wherein, The processor, as the notification method, changes at least one of the notification time, notification moment, and notification level of the second identification result.

14. The medical image processing apparatus according to claim 13, wherein, The processor changes the notification time of the second identification result to after the notification time of the first identification result.

15. A medical image processing method, wherein, The processor performs the following processing: Acquire multiple medical images in a time series; A first identification process is performed on a first medical image from a plurality of medical images in the time series; A second recognition process, different from the first recognition process, is performed on a second medical image following the first medical image from among the first medical image or the time series of multiple medical images; The first identification result of the first identification process is notified; The notification indicates the second identification result of the second identification process; and The notification method for the second identification result is changed based on the notification status of the first identification result.

16. A program product, when read by a computer, causes the computer to perform the medical image processing method of claim 15.

17. A non-volatile and computer-readable storage medium, which, when read by a computer, causes the computer to perform the medical image processing method of claim 15.

Images