Medical image processing device, medical image processing method, and program

Abstract

This invention provides a medical image processing device, a medical image processing method, and a program that enable the user to correctly recognize multiple recognition results. [Solution] The medical image processing device of the present disclosure is a medical image processing device equipped with a processor, the processor acquires a plurality of medical images in time series, performs a first recognition process from a first medical image among the plurality of medical images in time series, performs a second recognition process different from the first recognition process from the first medical image or a second medical image among the plurality of medical images in time series that is later in the time series than the first medical image, reports the first recognition result of the first recognition process, reports the second recognition result of the second recognition process, and changes the method of reporting the second recognition result according to the reporting state of the first recognition result.

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program, and particularly relates to a technique for controlling notification of recognition results of medical images.

Background Art

[0002] There is known an AI support function for detecting lesions by artificial intelligence (AI) from an image captured by a medical image diagnostic apparatus such as an endoscope or an ultrasonic diagnostic apparatus.

[0003] For example, Patent Document 1 describes a computer program that causes a computer to execute a process of acquiring an image captured by an endoscope, inputting the image to a first recognizer that recognizes a lesion part based on the image and a second recognizer that recognizes the lesion part with a higher recognition accuracy than the first recognizer based on the image, acquiring provisional information including the recognition result recognized by the first recognizer, outputting an image including the acquired provisional information, acquiring confirmation information for the provisional information including the recognition result recognized by the second recognizer, and outputting an image including the acquired confirmation information.

[0004] Patent Document 2 describes an endoscope system for examining the lumen of a patient, which includes an insertion part inserted into the lumen, a camera that captures an image of the lumen to acquire an endoscope image, a target area detection part that detects a target area from the endoscope image, a detection result notification part that notifies the detection result of the target area, an insertion / removal discrimination part that discriminates whether the inspection process is an insertion process or a removal process, and a notification control part that causes the detection result notification part to perform notification according to the process discriminated by the insertion / removal discrimination part.

[0005] Furthermore, Patent Document 3 describes a medical image processing device that includes a notification control unit that controls the notification information contained in a medical image to either a notification state in which notification information is notified or a non-notification state in which notification information is not notified, wherein the notification control unit sets the notification information to a non-notification state when the medical image satisfies the non-notification conditions, and sets the medical image to a notification state after a non-notification maintenance period has elapsed since the medical image no longer satisfies the non-notification conditions. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2021 / 010225 [Patent Document 2] International Publication No. 2020 / 017212 [Patent Document 3] International Publication No. 2020 / 090729 [Overview of the project] [Problems that the invention aims to solve]

[0007] These support functions need to inform the user of AI results in real time during the examination, but if multiple functions are installed, the timing of multiple notifications may overlap. When the timing of multiple notifications overlaps, the user may miss one or become confused, which can lead to problems in correctly understanding the notification information.

[0008] This invention has been made in view of these circumstances, and aims to provide a medical image processing device, a medical image processing method, and a program that enable a user to correctly recognize multiple recognition results. [Means for solving the problem]

[0009] To achieve the above objective, the medical image processing apparatus according to the first aspect of this disclosure is a medical image processing apparatus equipped with a processor, wherein the processor acquires a plurality of medical images in a time series, performs a first recognition process from a first medical image among the plurality of medical images in the time series, performs a second recognition process different from the first recognition process from the first medical image or a second medical image among the plurality of medical images in the time series that is later in the time series than the first medical image, reports the first recognition result of the first recognition process, reports the second recognition result of the second recognition process, and changes the method of reporting the second recognition result according to the reporting state of the first recognition result.

[0010] In the medical image processing apparatus according to the second aspect of this disclosure, it is preferable that the processor, in the medical image processing apparatus according to the first aspect, notifies at least one of the first recognition result and the second recognition result as image information.

[0011] In the medical image processing apparatus according to the third aspect of this disclosure, in the medical image processing apparatus according to the first or second aspect, it is preferable that the processor notifies at least one of the first recognition result and the second recognition result by voice information.

[0012] In the medical image processing apparatus according to the fourth aspect of this disclosure, in the medical image processing apparatus according to any of the first to third aspects, the processor preferably notifies the second recognition result with a plurality of pieces of information including at least image information and sound information, and changes the notification method of any one of the plurality of pieces of information according to the notification state.

[0013] In the fifth aspect of this disclosure, the medical image processing apparatus is preferable in which the processor changes the method of notifying audio information according to the notification state, as is the case in the medical image processing apparatus according to the fourth aspect.

[0014] In the sixth aspect of the present disclosure, the medical image processing apparatus according to any of the second to fifth aspects preferably has a processor that notifies a first recognition result to a first display unit that displays a medical image using first image information, and notifies a second recognition result to a second display unit different from the first display unit using second image information.

[0015] In the seventh aspect of this disclosure, the medical image processing apparatus is a medical image processing apparatus according to any of the first to sixth aspects, wherein the first recognition process is preferably a focus region detection process for detecting a focus region included in a first medical image, or a focus region classification process for classifying a focus region included in a first medical image.

[0016] In the medical image processing apparatus according to the eighth aspect of this disclosure, in the medical image processing apparatus according to the seventh aspect, it is preferable that the processor continuously notifies the first recognition result for a predetermined time and notifies the second recognition result after the predetermined time has elapsed.

[0017] In the medical image processing apparatus according to the ninth aspect of this disclosure, in the medical image processing apparatus according to any of the first to eighth aspects, it is preferable that the processor changes the notification method depending on whether or not the first recognition result is notified.

[0018] In the medical image processing apparatus according to the tenth aspect of this disclosure, in the medical image processing apparatus according to any of the first to ninth aspects, it is preferable that the processor changes the notification method according to whether or not the first recognition result is notified within a predetermined time, or whether or not there is a change in the notification state.

[0019] In the eleventh aspect of this disclosure, the medical image processing device is preferably a medical image processing device according to any of the first to tenth aspects, in which the second recognition process is a site recognition process that recognizes an observed site in the first medical image or the second medical image.

[0020] In the medical image processing apparatus according to the twelfth aspect of this disclosure, it is preferable that, in the medical image processing apparatus according to the eleventh aspect, the second recognition result displays in a way that allows visual identification of areas captured by multiple medical images in a time series and areas that have not been captured.

[0021] The medical image processing apparatus according to the 13th aspect of the present disclosure is the medical image processing apparatus according to any one of the 1st to 12th aspects, and it is preferable that the processor changes at least one of the notification time, notification timing, and notification level of the second recognition result as the notification method.

[0022] The medical image processing apparatus according to the 14th aspect of the present disclosure is the medical image processing apparatus according to the 13th aspect, and it is preferable that the processor changes the notification timing of the second recognition result to be later than the notification timing of the first recognition result.

[0023] To achieve the above object, the medical image processing method according to the 15th aspect of the present disclosure is such that a processor acquires a plurality of medical images in time series, performs a first recognition process on a first medical image among the plurality of medical images in time series, and performs a second recognition process different from the first recognition process on a second medical image whose time series is later than the first medical image among the first medical image or the plurality of medical images in time series, notifies the first recognition result of the first recognition process, notifies the second recognition result of the second recognition process, and changes the notification method of the second recognition result according to the notification state of the first recognition result.

[0024] To achieve the above object, the program according to the 16th aspect of the present disclosure is a program that causes a computer to execute the medical image processing method according to the 15th aspect. A non-temporary and computer-readable storage medium on which the program according to the 16th aspect is stored is also included in the present disclosure.

[0025] In the medical image processing method according to the 15th aspect and the program according to the 16th aspect, a configuration including the same specific aspects as the above-described medical image processing apparatus can be adopted.

Advantages of the Invention

[0026] According to the present invention, a user can correctly recognize a plurality of recognition results.

Brief Description of the Drawings

[0027] [Figure 1]Figure 1 is a schematic diagram showing the overall configuration of the endoscope system. [Figure 2] Figure 2 is a block diagram showing an example of the electrical configuration of a medical image processing device. [Figure 3] Figure 3 is a flowchart showing a medical image processing method. [Figure 4] Figure 4 is a table showing an example of the timing of output for lesion detection results and observation site recognition results. [Figure 5] Figure 5 shows an example of a display on a display device. [Figure 6] Figure 6 is a block diagram showing an example of the electrical configuration of a medical image processing device. [Figure 7] Figure 7 is a flowchart showing a medical image processing method. [Figure 8] Figure 8 is a flowchart showing a medical image processing method. [Modes for carrying out the invention]

[0028] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. The same reference numerals are used for identical components, and redundant descriptions are omitted.

[0029] <Overall configuration of the endoscopy system> Figure 1 is a schematic diagram showing the overall configuration of an endoscope system 9 including a medical image processing device according to this disclosure. As shown in Figure 1, the endoscope system 9 comprises an endoscope scope 10 which is an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a medical image processing device 14, an input device 15, and a display device 16.

[0030] The endoscope scope 10 is used to capture multiple endoscopic images in a time series, and is, for example, a flexible endoscope. This endoscope scope 10 has an insertion section 20 that is inserted into the subject and has a tip and a base, a handheld control section 21 that is connected to the base end of the insertion section 20 and is grasped by the user (physician) to perform various operations, and a universal cord 22 connected to the handheld control section 21.

[0031] The insertion section 20 is formed in a long, slender shape with a small diameter throughout. The insertion section 20 is composed of a flexible section 25 that is flexible from the base end to the tip end, a curved section 26 that can be bent by the operation of the handheld operation section 21, and a tip section 27 that houses an imaging optical system (objective lens) and an image sensor 28, etc. (not shown).

[0032] The image sensor 28 is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type image sensor. Image light from the observation area is incident on the imaging surface of the image sensor 28 through an observation window (not shown) opened on the tip surface of the tip portion 27, and an objective lens (not shown) positioned behind this observation window. The image sensor 28 captures (converts into an electrical signal) the image light from the observation area incident on its imaging surface and outputs an imaging signal.

[0033] The user-operated control unit 21 is equipped with various operating components that are operated by the user. Specifically, the user-operated control unit 21 is equipped with two types of bending operation knobs 29 used for bending the bending section 26, an air supply / water supply button 30 for air supply / water supply operation, and a suction button 31 for suction operation. The user-operated control unit 21 is also equipped with a still image acquisition instruction unit 32 for instructing the acquisition of a still image 39 of the observation area, and a treatment instrument introduction port 33 for inserting a treatment instrument (not shown) into a treatment instrument insertion passage (not shown) that runs through the insertion section 20.

[0034] The universal cord 22 is a connecting cord for connecting the endoscope scope 10 to the light source device 11. This universal cord 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) that are inserted through the insertion section 20. The end of the universal cord 22 is provided with a connector 37a that connects to the light source device 11, and a connector 37b that branches off from this connector 37a and connects to the endoscope processor device 12.

[0035] By connecting connector 37a to light source device 11, the light guide 35 and fluid tube are inserted into light source device 11. As a result, the necessary illumination light, water, and gas are supplied from light source device 11 to endoscope scope 10 via the light guide 35 and fluid tube. Consequently, illumination light is shone from an illumination window (not shown) on the tip surface of tip 27 toward the observation site. In addition, in response to the pressing of the air / water supply button 30 described above, gas or water is sprayed from an air / water supply nozzle (not shown) on the tip surface of tip 27 toward an observation window (not shown) on the tip surface.

[0036] By connecting connector 37b to the endoscope processor device 12, the signal cable 36 and the endoscope processor device 12 are electrically connected. As a result, imaging signals of the observation site are output from the image sensor 28 of the endoscope scope 10 to the endoscope processor device 12 via the signal cable 36, and control signals are output from the endoscope processor device 12 to the endoscope scope 10.

[0037] The light source device 11 supplies illumination light to the light guide 35 of the endoscope scope 10 via the connector 37a. The illumination light is selected from various wavelength bands depending on the observation purpose, such as white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or a combination thereof. Note that the specific wavelength band is narrower than the white wavelength band.

[0038] A first example of a specific wavelength band is, for example, the blue or green band in the visible spectrum. This first example wavelength band includes the wavelength bands of 390 nm to 450 nm or 530 nm to 550 nm, and the light in this first example has a peak wavelength within the wavelength bands of 390 nm to 450 nm or 530 nm to 550 nm.

[0039] A second example of a specific wavelength band is, for example, the red band in the visible spectrum. This second example wavelength band includes wavelengths between 585 nm and 615 nm or between 610 nm and 730 nm, and the light in this second example has a peak wavelength within the wavelength bands between 585 nm and 615 nm or between 610 nm and 730 nm.

[0040] A third example of a specific wavelength band includes a wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients, and the light of the third example has a peak wavelength in the wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients. This wavelength band of the third example includes wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light of the third example has a peak wavelength in the above wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0041] A fourth example of a specific wavelength band is the wavelength band (390 nm to 470 nm) used for observing the fluorescence emitted by fluorescent substances in living organisms (fluorescence observation) and for the excitation light that excites these fluorescent substances.

[0042] The fifth example of a specific wavelength band is the infrared light wavelength band. This fifth example's wavelength band includes the wavelength bands of 790 nm to 820 nm or 905 nm to 970 nm, and the light of this fifth example has a peak wavelength in the wavelength bands of 790 nm to 820 nm or 905 nm to 970 nm.

[0043] The endoscope processor device 12 controls the operation of the endoscope scope 10 via the connector 37b and the signal cable 36. The endoscope processor device 12 also generates a time-series video 38 consisting of time-series frame images 38a (see Figure 2) based on the imaging signal acquired from the image sensor 28 of the endoscope scope 10 via the connector 37b and the signal cable 36. The frame rate of the video 38 is, for example, 30 fps (frames per second).

[0044] Furthermore, when the still image acquisition instruction unit 32 is operated on the hand-held control unit 21 of the endoscope scope 10, the endoscope processor device 12 acquires one frame image 38a from the video 38 in parallel with the generation of the video 38, according to the timing of the acquisition instruction, and converts it into a still image 39. The endoscope processor device 12 then outputs the generated video 38 and still image 39 to the display device 13 and the medical image processing device 14, respectively.

[0045] Furthermore, the endoscope processor device 12 may generate (acquire) a special light image having information of the specific wavelength band described above, based on the normal light image obtained with the white light described above. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. The endoscope processor device 12 then obtains a signal of the specific wavelength band by performing calculations based on the RGB color information of red, green, and blue, or the CMY color information of cyan, magenta, and yellow, contained in the normal light image.

[0046] Furthermore, the endoscope processor device 12 may generate a feature image, such as a known oxygen saturation image, based on, for example, a normal light image obtained with the white light described above and a special light image obtained with the light of a specific wavelength band (special light) described above. In this case, the endoscope processor device 12 functions as a feature image generation unit. The video 38 or still image 39, which include the in vivo image, normal light image, special light image, and feature image, are all examples of medical images obtained by imaging or measuring the human body for the purpose of image-based diagnosis and examination. The video 38 is an example of multiple medical images in time series.

[0047] The display device 13 is connected to the endoscope processor device 12 and displays video 38 and still images 39 input from the endoscope processor device 12. The user performs operations such as advancing and retracting the insertion unit 20 while checking the video 38 displayed on the display device 13, and if a lesion or other abnormality is found in the observation area, the user operates the still image acquisition instruction unit 32 to perform still image acquisition of the observation area, and then performs diagnosis and biopsy, etc.

[0048] The medical image processing device 14 functions as a diagnostic support device. The medical image processing device 14 informs the user of information contained in the medical image. The information contained in the medical image is, for example, the result of recognition processing on the medical image. The medical image processing device 14 is, for example, a personal computer equipped with a processor.

[0049] The input device 15 uses a keyboard and mouse, etc., which are connected to the personal computer by wire or wirelessly. The display device 16 uses various monitors, such as LCD monitors, that can be connected to the personal computer. An audio output device 18 (see Figure 6) may be connected to the medical image processing device 14. The audio output device 18 uses a buzzer or a speaker.

[0050] <First Embodiment> [Configuration of medical image processing equipment] Figure 2 is a block diagram showing an example of the electrical configuration of a medical image processing device 14. The medical image processing device 14 shown in Figure 2 includes a processor 40 and a memory 42. The processor 40 is composed of an arithmetic unit such as a CPU (Central Processing Unit). The memory 42 is composed of storage elements such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 42 stores a program 44. The program 44 includes instructions to be executed by the processor 40. The program 44 may be provided by a non-temporary storage medium that is readable by a computer. In this case, the processor 40 may read the program 44 from the non-temporary storage medium and store it in the memory 42.

[0051] The processor 40 includes an image acquisition unit 50, a first recognition processing unit 52, a second recognition processing unit 54, a first notification unit 56, a notification state detection unit 58, a second notification unit 60, and a display control unit 62. The functions of the image acquisition unit 50, the first recognition processing unit 52, the second recognition processing unit 54, the first notification unit 56, the notification state detection unit 58, the second notification unit 60, and the display control unit 62 are realized when the processor 40 executes the program 44.

[0052] The image acquisition unit 50 sequentially acquires endoscopic images captured by the endoscope scope 10. Here, the image acquisition unit 50 acquires a video 38 consisting of time-series frame images 38a from the endoscope processor device 12 (see Figure 1) using an image input / output interface (not shown) that is connected to the endoscope processor device 12 by wire or wireless connection. Furthermore, if a still image 39 is captured while the endoscope scope 10 is capturing the video 38, the image acquisition unit 50 acquires both the video 38 and the still image 39 from the endoscope processor device 12.

[0053] The image acquisition unit 50 may acquire the video 38 via a non-temporary storage medium such as a memory card or hard disk drive, instead of directly acquiring the video 38 from the endoscope processor device 12. Alternatively, the image acquisition unit 50 may acquire the video 38 uploaded to a server or database on the Internet via the Internet.

[0054] The first recognition processing unit 52 performs a first recognition process on the endoscopic image acquired by the image acquisition unit 50 and obtains a first recognition result. The first recognition process is, for example, a focus region detection process that detects a focus region included in the endoscopic image. The first recognition processing unit 52 includes a convolutional neural network (CNN) that calculates feature quantities of the endoscopic image and performs recognition processing of the focus region within the endoscopic image.

[0055] The area of ​​focus may include lesions. The area of ​​focus may include at least one of the following: polyps, cancer, colonic diverticulum, inflammation, EMR (Endoscopic Mucosal Resection) scars, ESD (Endoscopic Submucosal Dissection) scars, clipping sites, bleeding points, perforations, vascular dysplasia, and treatment instruments.

[0056] The first recognition processing unit 52 may perform classification processing of the detected area of ​​interest, such as categorization (differentiation), to determine which of several categories related to lesions, such as "neoplastic," "non-neoplastic," or "other," the detected area of ​​interest belongs to.

[0057] Furthermore, the first recognition processing unit 52 is not limited to detecting the region of interest using a CNN; it may also detect the region of interest by analyzing features such as color, pixel value gradient, shape, and size within the image through image processing.

[0058] The second recognition processing unit 54 performs a second recognition process different from the first recognition process on the endoscopic image acquired by the image acquisition unit 50 and obtains a second recognition result. The second recognition process is, for example, an observation area recognition process that recognizes the observation area of ​​the endoscopic image. The second recognition processing unit 54 includes a CNN that calculates feature quantities of the endoscopic image and performs the observation area recognition process of the endoscopic image. The second recognition processing unit 54 may also include a process to determine whether the endoscopic image acquired by the image acquisition unit 50 is an endoscopic image that satisfies predetermined conditions such as blur amount, composition, and area.

[0059] If the endoscope scope 10 is a lower endoscope inserted through the anus of a subject and used to observe the rectum and large intestine, the observation area may include at least one of the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, ileum, and jejunum. If the endoscope scope 10 is an upper endoscope inserted through the mouth or nose of a subject and used to observe the esophagus and stomach, the observation area may include at least one of the pharynx, esophagus, stomach, and duodenum.

[0060] Furthermore, the second recognition processing unit 54 is not limited to detecting the region of interest using a CNN; it may also recognize the observed area by analyzing features such as color, pixel value gradient, shape, and size within the image through image processing.

[0061] The first notification unit 56 notifies the first recognition result of the first recognition processing unit 52. Here, the first notification unit 56 notifies the first recognition result as image information by displaying the first recognition result on the display device 16 via the display control unit 62.

[0062] The notification state detection unit 58 detects the notification state of the first notification unit 56. The notification state of the first notification unit 56 includes, for example, whether or not the first recognition result is being notified.

[0063] The second notification unit 60 notifies the second recognition result of the second recognition processing unit 54. Here, the second notification unit 60 notifies the second recognition result as image information by displaying the second recognition result on the display device 16 via the display control unit 62.

[0064] Furthermore, the second notification unit 60 changes the notification method for the second recognition result according to the detection result of the notification state detection unit 58, that is, according to the notification state of the first recognition result. For example, the second notification unit 60 changes at least one of the notification time, notification timing, and notification level of the second recognition result as a notification method.

[0065] The display control unit 62 controls the display of the display device 16 based on signals from the first notification unit 56 and the second notification unit 60.

[0066] In this manner, the medical image processing device 14 notifies the user of at least one of the first recognition result and the second recognition result as image information on the display device 16. The user can recognize the image information by viewing the screen display of the display device 16.

[0067] [Medical Image Processing Methods] Figure 3 is a flowchart showing a medical image processing method using the medical image processing device 14. The medical image processing method is realized when the processor 40 executes the program 44. Here, we will explain the processing of the frame image 38a of the Nth frame of the video 38.

[0068] In step S1, the image acquisition unit 50 acquires a frame image 38a of the Nth frame of the video 38. The frame image 38a acquired in step S1 may be displayed on the display device 16.

[0069] In step S2, the first recognition processing unit 52 performs a first recognition process, which is a focus region detection process, from the frame image 38a acquired in step S1, and obtains a first recognition result, which is a focus region detection result.

[0070] In step S3, the first recognition processing unit 52 determines whether or not to report the results of the area of ​​interest detection acquired in step S2. For example, the first recognition processing unit 52 determines that the results of the area of ​​interest detection should be reported if an area of ​​interest is detected, and that the results of the area of ​​interest detection should not be reported if an area of ​​interest is not detected.

[0071] If the determination result in step S3 is a Yes determination, that is, if the detection result of the area of ​​interest should be notified, in step S4, the first notification unit 56 performs the first notification process. The first notification process is a process that displays the detection result of the area of ​​interest as image information on the display device 16 via the display control unit 62.

[0072] On the other hand, if the determination result in step S3 is a No determination, that is, if the detection result of the area of ​​interest should not be reported, the first notification unit 56 does not perform the first notification process.

[0073] The processes in steps S5 to S10 are performed in parallel with the processes in steps S2 to S4. The processes in steps S5 to S10 may also be performed after the processes in steps S2 to S4 have been completed. In step S5, the notification state detection unit 58 determines whether or not it has received second notification information from the processing of the (N-1) frame, which is the frame preceding the N frame. The second notification information includes, for example, the observation area recognition result of the observation area recognition process for the frame image 38a of the (N-1) frame or an earlier frame.

[0074] If the determination result in step S5 is No, that is, if the second notification information has not been received, in step S6, the second recognition processing unit 54 performs the observation area recognition process, which is the second recognition process, from the frame image 38a acquired in step S1, and obtains the observation area recognition result, which is the second recognition result.

[0075] In step S7, the second recognition processing unit 54 determines whether or not to report the observed area recognition result obtained in step S6. For example, the second recognition processing unit 54 determines that the observed area recognition result should be reported if an observed area is recognized, and determines that the observed area recognition result should not be reported if an observed area is not recognized.

[0076] If the determination result in step S7 is No, that is, if the observation site recognition result should not be reported, the second notification unit 60 does not perform the second notification process.

[0077] On the other hand, if the determination result in step S7 is a Yes determination, that is, if the observed area recognition result should be reported, in step S8 the notification state detection unit 58 determines whether or not the first notification unit 56 is performing the first notification process. Similarly, if the determination result in step S5 is a Yes determination, that is, if second notification information has been received, in step S8 the notification state detection unit 58 determines whether or not the first notification unit 56 is performing the first notification process.

[0078] If the determination result in step S8 is No, that is, if the first notification unit 56 has not performed the first notification process, then in step S9, the second notification unit 60 performs the second notification process.

[0079] The second notification process, if it is determined in step S5 that the second notification information has not been received, is the process of displaying the observation area recognition result obtained in step S6 as image information on the display device 16 via the display control unit 62. If it is determined in step S5 that the second notification information has been received, the second notification process is the process of displaying the observation area recognition result as image information on the display device 16 based on the second notification information of the (N-1)th frame.

[0080] On the other hand, if the determination result in step S8 is a Yes determination, that is, if the first notification unit 56 is performing the first notification process, then in step S10, the second notification unit 60 does not perform the second notification process and instead sends the second notification information to the processing of the next frame, the (N+1)th frame.

[0081] The second notification information sent to the processing of the (N+1)th frame is the second notification information of the Nth frame, and includes the observed area recognition result obtained in step S6, if it was determined in step S5 that the second notification information had not been received. If it was determined in step S5 that the second notification information had been received, the second notification information is the second notification information received from the processing of the previous frame, the (N-1)th frame.

[0082] Thus, according to the medical image processing method, when the first notification process is being performed, the notification timing of the second notification process is changed to be later than the notification timing of the first notification process. By performing this process frame by frame, it is possible to avoid overlapping notification timings between the first and second notification processes. This prevents the user from missing one of the recognition results, and allows the user to correctly recognize multiple recognition results. Changing the notification timing is an example of "changing the notification method for the second recognition result" in this disclosure.

[0083] Here, the timing of the second notification process is changed when the first notification process is being performed. However, the notification method may be changed depending on whether or not the first notification process has been notified in the temporal vicinity, or depending on whether or not there has been a change in the notification status of the first notification process. The temporal vicinity may be a predetermined time period counting backward from the present moment.

[0084] For example, if the first notification process was performed within a predetermined time period retrospectively from the present moment, the notification timing of the second notification process may be changed. Also, if there has been a change in the notification status of the first notification process within a predetermined time period retrospectively from the present moment, the notification timing of the second notification process may be changed.

[0085] Furthermore, the first notification unit 56 may continue to perform the first notification process for a predetermined period of time. The second notification unit 60 may perform the second notification process after the predetermined period of time has elapsed.

[0086] For example, in step S3, the first recognition processing unit 52 may determine that it should notify the results of the detection of the area of ​​interest if a predetermined amount of time has not elapsed since the last detection of the area of ​​interest, and that it should not notify the results of the detection of the area of ​​interest if a predetermined amount of time has elapsed since the last detection of the area of ​​interest.

[0087] The first notification process may be, for example, a process that displays a closed rectangular frame surrounding the detected area of ​​interest. The lines of the frame do not have to be closed. Alternatively, the first notification process may be an assist circle process that lights up the assist circle that is closer to the detected area of ​​interest among the arc-shaped assist circles displayed along the left edge and the arc-shaped assist circles displayed along the right edge of the medical image (see, for example, the endoscopic image 102 shown in Figure 5) in a predetermined color. The first notification process may also be a process that combines the process of displaying a rectangular frame and the assist circle process.

[0088] Furthermore, the first notification unit 56, with the aim of not interfering with the user's observation of medical images, performs assist circle processing by continuously illuminating the assist circle for a predetermined time (for example, within 1 second) after detecting a lesion. If the second recognition processing unit 54 performs second recognition processing on a medical image taken while the assist circle is lit after lesion detection, the second notification unit 60 may perform second notification processing after the predetermined time has elapsed (after the assist circle has turned off).

[0089] [Screen display example] Figure 4 is a table showing an example of the timing of the output for lesion detection by the first recognition processing unit 52 and observation site recognition by the second recognition processing unit 54. Here, a lesion is detected at time t, but not at times before time t (t-1) or after time t (t+1). Similarly, the observation site is recognized at time t, but not at times before time t (t-1) or after time t (t+1).

[0090] Figure 5 shows an example of the display on the display device 16 for the timing example shown in Figure 4. In Figure 5, 1000 shows the screen of the display device 16 at time (t-1), 1002 shows the screen at time t, and 1004 shows the screen at time (t+1).

[0091] As shown at 1000 in Figure 5, at time (t-1), the display device 16 shows the endoscopic image 102 in the first display unit 100, which is the left region in Figure 5, and the schematic diagram 112 in the second display unit 110, which is the right region in Figure 5. At time (t-1), the lesion detection result and the observation site recognition result are not displayed.

[0092] As shown in Figure 5, 1002, at time t, the display device 16 displays the endoscopic image 102 on the first display unit 100 and the schematic diagram 112 on the second display unit 110. In addition, the display device 16 displays a frame 106 (an example of "first image information") surrounding the lesion 104 (an example of "first recognition result"), which is the lesion detection result, superimposed on the endoscopic image 102. On the other hand, at time t, the observation site recognition result is not displayed.

[0093] As shown in Figure 5, at time (t+1), the display device 16 displays the endoscopic image 102 on the first display unit 100 and the schematic diagram 112 on the second display unit 110. The display device 16 also displays observation position marks 114 (an example of "second image information") superimposed on the schematic diagram 112 at the locations of the observation areas (an example of "second recognition results") which are the observation area recognition results of the schematic diagram 112. The observation position marks 114 make it possible to visually distinguish between areas already captured by the video 38 and areas that have not yet been captured. Note that at time (t+1), the lesion detection results are not displayed.

[0094] As shown in Figure 4, a lesion is detected and the observation site is recognized at time t. However, if both the lesion detection result and the observation site recognition result are displayed on the display device 16 at time t, the user risks missing one of the recognition results. Therefore, as shown in Figure 5, the observation site recognition result is displayed at the time (t+1) when the display of the lesion detection result has finished, thereby avoiding a situation where both recognition results are displayed simultaneously. In this way, when performing lesion detection and observation site recognition, priority is given to lesion detection, which has a high necessity to be performed in real time.

[0095] In this example, Figure 5 shows the left side of the display device 16 as the first display unit 100 and the right side as the second display unit 110, but the arrangement of the first display unit 100 and the second display unit 110 is not limited to this example. Alternatively, display device 16 and a different display device may be used, with one serving as the first display unit 100 and the other as the second display unit 110.

[0096] <Second Embodiment> [Configuration of medical image processing equipment] Figure 6 is a block diagram showing an example of the electrical configuration of the medical image processing device 14A. The medical image processing device 14A shown in Figure 6 includes an audio output control unit 64. The audio output control unit 64 controls the audio output of the audio output device 18 of the endoscope system 9 based on a signal from the second notification unit 60.

[0097] The second notification unit 60 notifies the second recognition result as image information by displaying the second recognition result on the display device 16 via the display control unit 62. Furthermore, the second notification unit 60 notifies the second recognition result as audio information by outputting the second recognition result as audio to the audio output device 18 via the audio output control unit 64.

[0098] In this manner, the medical image processing device 14A notifies the user of the second recognition result using multiple pieces of information, including at least image information and audio information. The medical image processing device 14A may also notify the user of at least one of the first and second recognition results as audio information using the audio output device 18. The user can recognize the audio information by listening to the audio output from the audio output device 18.

[0099] [Medical Image Processing Methods] The second notification unit 60 may change the notification method for one of the information items, including image information and audio information, according to the detection result of the notification state detection unit 58, that is, according to the notification state of the first recognition result. The notification method to be changed may be the notification method for audio information.

[0100] <Third Embodiment> [Medical Image Processing Methods] Figure 7 is a flowchart illustrating a medical image processing method according to the third embodiment using the medical image processing device 14 or the medical image processing device 14A.

[0101] The processing in steps S1 to S4 is the same as in the medical image processing method according to the first embodiment. In step S11, the second recognition processing unit 54 performs observation area recognition processing as a second recognition process from the frame image 38a acquired in step S1, and obtains the observation area recognition result, which is the second recognition result.

[0102] In step S12, the second recognition processing unit 54 determines whether or not to report the observed area recognition result obtained in step S11.

[0103] If the determination result in step S12 is a Yes determination, that is, if the observed area recognition result should be reported, then in step S13, the second reporting unit 60 performs a second reporting process. The second reporting process is, for example, the process of displaying the observed area recognition result acquired in step S11 as image information on the display device 16 via the display control unit 62.

[0104] Next, in step S14, the notification state detection unit 58 determines whether the first notification unit 56 is performing the first notification process. If the determination result in step S14 is Yes, that is, if the first notification unit 56 is performing the first notification process, then in step S15, the second notification unit 60 sends the second notification information for the Nth frame to the next frame, the (N+1)th frame processing. The second notification information for the Nth frame includes the observation area recognition result obtained in step S11.

[0105] On the other hand, if the determination result in step S14 is No, that is, if the first notification unit 56 has not performed the first notification process, the second notification unit 60 will not perform the process in step S15.

[0106] Furthermore, if the determination result in step S12 is a No determination, that is, if the observed area recognition result should not be reported, in step S16, the notification state detection unit 58 determines whether or not it has received the second notification information for the (N-1) frame from the processing of the (N-1) frame, which is the frame preceding the N frame.

[0107] If the determination result in step S16 is a Yes determination, that is, if the second notification information is received from the processing of the (N-1)th frame, the second notification unit 60 performs the second notification processing for the (N-1)th frame in step S17. The second notification processing is a process that displays the observed area recognition result as image information on the display device 16 based on the second notification information received from the processing of the (N-1)th frame.

[0108] On the other hand, if the determination result in step S16 is No, that is, if the second notification information has not been received, the second notification unit 60 does not perform the processing in step S17.

[0109] In this way, by extending the notification time of the second notification process while the first notification process is being performed, it is possible to prevent the user from missing the second recognition result. Extending the notification time is one example of "changing the notification method for the second recognition result" in this disclosure.

[0110] <Fourth Embodiment> [Medical Image Processing Methods] Figure 8 is a flowchart illustrating a medical image processing method according to the fourth embodiment using the medical image processing device 14 or the medical image processing device 14A. Here, we will explain an example in which the first recognition processing unit 52 performs observation area recognition processing as the first recognition processing, and the second recognition processing unit 54 performs focus area detection processing as the second recognition processing.

[0111] The processing in steps S1 to S4 is the same as in the medical image processing method according to the first embodiment. In step S11, the second recognition processing unit 54 performs a focus region detection process as a second recognition process from the frame image 38a acquired in step S1, and obtains a focus region detection result, which is the second recognition result.

[0112] In step S12, the second recognition processing unit 54 determines whether or not to report the detection result of the area of ​​interest acquired in step S11.

[0113] If the result of step S12 is a NO judgment, that is, if the detection result of the area of ​​interest should not be reported, the second notification unit 60 will not provide any notification.

[0114] On the other hand, if the determination result in step S12 is a Yes determination, that is, if the detection result of the area of ​​interest should be notified, in step S21, the notification state detection unit 58 determines whether or not the first notification unit 56 is performing the first notification process.

[0115] If the determination result in step S21 is No, that is, if the first notification unit 56 has not performed the first notification process, then in step S22, the second notification unit 60 performs the second notification process at the first notification level. The second notification process is a process that notifies the detection result of the area of ​​interest acquired in step S11. The notification level may be a certain degree of emphasis on the screen display, or a certain volume of sound, etc.

[0116] On the other hand, if the determination result in step S21 is a Yes determination, that is, if the first notification unit 56 is performing the first notification process, then in step S23, the second notification unit 60 performs the second notification process at a second notification level that is relatively higher than the first notification level. For example, the thickness of the frame surrounding the area of ​​interest on the screen display may be thicker than the first notification level, the brightness of the color of the frame surrounding the area of ​​interest on the screen display may be higher than the first notification level, or the volume of the audio output notifying the area of ​​interest may be higher than the first notification level, etc.

[0117] In this way, by increasing the notification level of the second notification process when the first notification process is being performed, it is possible to prevent missing the second recognition result. Changing the notification level is an example of "changing the notification method for the second recognition result" in this disclosure.

[0118] <Variation> Up to this point, we have explained an example in which the first and second recognition processes are performed on the frame image 38a of the Nth frame of video 38. However, the first recognition process may be performed on the frame image 38a of the Nth frame (an example of the "first medical image"), and the second recognition process may be performed on the frame images 38a of the (N+1)th frame or later (an example of the "second medical image"), which are in a later time series than the Nth frame.

[0119] The first and second recognition processes may be any of the following: area of ​​interest detection, observation site recognition, lesion recognition, and examination status recognition. The lesion recognition process is a process that recognizes information regarding at least one of the size, type, and shape of a lesion. The examination status recognition process is a process that recognizes the examination status, such as the treatment status.

[0120] In the first to third embodiments, the first recognition process is the main recognition process, and the second recognition process is the sub-recognition process. On the other hand, in the fourth embodiment, the second recognition process is the main recognition process, and the first recognition process is the sub-recognition process. It is preferable to make functions with a high risk of being missed, such as lesion detection, or functions requiring a faster reaction speed, the main recognition process.

[0121] Although endoscopic images were used as an example of medical images in this explanation, the medical image processing device 14 and the medical image processing device 14A can be applied to other medical images such as capsule endoscopy images and ultrasound images.

[0122] <Configuration of medical image processing equipment> In this embodiment, each process is performed on any computer. Furthermore, any computer may perform these processes using a processor, a program, or a combination thereof. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other hardware element capable of running a program.

[0123] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing specific processing such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the processor has various units or means that execute the various processes described in this embodiment. The type of hardware may also be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of the processes performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is composed of electrical circuits (circuitry) combining circuit elements such as semiconductor elements.

[0124] Furthermore, this embodiment may be implemented by hardware, software, firmware, microcode, or a combination thereof. The software, firmware, and microcode are composed of a program. The program may also be, for example, a group of program modules, each of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. The program code or code segment may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. The program code or code segment may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0125] <Other> The technical scope of the present invention is not limited to the scope described in the embodiments above. The configurations and other elements in each embodiment can be appropriately combined with those in each embodiment without departing from the spirit of the present invention. [Explanation of symbols]

[0126] 9… Endoscopy system 10… Endoscope 11...Light source device 12… Endoscope processor device 13…Display device 14, 14A…Medical imaging processing equipment 15…Input device 16...Display device 18…Audio output device 20... Insertion part 21...Handheld control unit 22…Universal Code 25…Soft part 26... Curved section 27...Tip 28…Image sensor 29…Curved control knob 30...Air / Water Supply Button 31... Suction button 32...Still image capture instruction section 33…Instrument entry port 35...Light Guide 36… Signal Cable 37a… Connector 37b… Connector 38…video 38a... Frame image 39…Still images 40…Processor 42...memory 44…Program 50...Image acquisition unit 52...First Recognition Processing Unit 54...Second Recognition Processing Unit 56... First News Department 58... Notification status detection unit 60...2nd Hochi Department 62...Display Control Unit 64…Audio Output Control Unit 100...1st display section 102... Endoscopic image 104... Lesion 106...frame 110…Second display section 112...Schema diagram 114... Observation position marker S1-S10...Steps in medical image processing methods S11-S17…Steps in medical image processing methods S21-S23...Steps in medical image processing methods

Claims

1. A medical image processing device equipped with a processor, The aforementioned processor, By acquiring multiple medical images in a time series, First recognition processing is performed on the first medical image from among the multiple medical images in the aforementioned time series. A second recognition process different from the first recognition process is performed on the first medical image or a second medical image from among the multiple medical images in the time series that is later in the time series than the first medical image. The first recognition result of the first recognition process is reported, The second recognition result of the second recognition process is reported, The method for notifying the second recognition result is changed according to the notification status of the first recognition result. Medical image processing equipment.

2. The processor provides at least one of the first recognition result and the second recognition result as image information. The medical image processing apparatus according to claim 1.

3. The processor provides at least one of the first recognition result and the second recognition result as audio information. The medical image processing apparatus according to claim 1.

4. The aforementioned processor, The second recognition result is communicated using multiple pieces of information, including at least image information and audio information. Depending on the notification state, the notification method for any one of the multiple pieces of information is changed. The medical image processing apparatus according to claim 1.

5. The aforementioned processor, The method of notifying the voice information is changed according to the notification state. The medical image processing apparatus according to claim 4.

6. The aforementioned processor, The first recognition result is communicated to the first display unit that displays the medical image using first image information. The second recognition result is communicated to a second display unit, which is different from the first display unit, using second image information. The medical image processing apparatus according to claim 2.

7. The first recognition process is a focus region detection process that detects a focus region included in the first medical image, or a focus region classification process that classifies a focus region included in the first medical image. The medical image processing apparatus according to claim 1.

8. The aforementioned processor, The first recognition result is continuously reported for a predetermined period of time. After the predetermined period of time has elapsed, the second recognition result is reported. The medical image processing apparatus according to claim 7.

9. The processor changes the notification method depending on whether or not the first recognition result is notified. The medical image processing apparatus according to claim 1.

10. The processor changes the notification method depending on whether or not the first recognition result is notified within a predetermined time, or whether or not the notification state changes. The medical image processing apparatus according to claim 1.

11. The second recognition process is a region recognition process that recognizes the observed region of the first medical image or the second medical image. The medical image processing apparatus according to claim 1.

12. The aforementioned processor, As a result of the second recognition, the areas captured by the multiple medical images in the time series and the areas that were not captured are displayed in a visually discernible manner. The medical image processing apparatus according to claim 11.

13. The processor modifies at least one of the notification time, notification timing, and notification level of the second recognition result as the notification method. A medical image processing apparatus according to any one of claims 1 to 12.

14. The processor changes the notification timing of the second recognition result to a later timing than the notification timing of the first recognition result. The medical image processing apparatus according to claim 13.

15. The processor, By acquiring multiple medical images in a time series, First recognition processing is performed on the first medical image from among the multiple medical images in the aforementioned time series. A second recognition process different from the first recognition process is performed on the first medical image or a second medical image from among the multiple medical images in the time series that is later in the time series than the first medical image. The first recognition result of the first recognition process is reported, The second recognition result of the second recognition process is reported, The method for notifying the second recognition result is changed according to the notification status of the first recognition result. Medical image processing methods.

16. A program that causes a computer to execute the medical image processing method described in claim 15.

Images