Endoscopy assistance device, endoscopy assistance method, and recording medium

Abstract

In this endoscopy assistance device, an acquisition means acquires endoscopic images in chronological order. A detection means detects a lesion from the endoscopic images. A tracking means tracks the lesion in the endoscopic images arranged in chronological order. An output control means performs control so as to output a notification indicating that a new lesion has been found upon initial detection of the lesion and not to output the notification while the lesion is being tracked. The endoscopy assistance device can optimize information output in medical fields.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium ,

[0008]

[0001] The present disclosure relates to a technology for detecting lesions.

[0002] A system that supports endoscopic examinations by detecting and differentiating lesions is known. For example, Patent Document 1 describes a method of recognizing medical images and notifying the recognition results. Patent Document 1 also describes a method of making the user recognize the operating state of the support function (switching from the notification state to the non-notification state).

[0003] International Publication WO2021 / 029293

[0004] In an endoscopic examination support system, since lesions are detected for each frame image, the same lesion may be notified multiple times. Such excessive notifications may inhibit the doctor's concentration. Note that the method of Patent Document 1 does not always enable appropriate notification for the same lesion.

[0005] One object of the present disclosure is to provide an endoscopic examination support device capable of appropriately outputting the detection result of a lesion in an endoscopic examination.

[0006] In one aspect of the present disclosure, an endoscopic examination support device includes: acquisition means for acquiring endoscopic images in chronological order; detection means for detecting lesions from the endoscopic images; tracking means for tracking the lesions in the endoscopic images arranged in chronological order; and output control means for outputting a notification indicating that a new lesion has been found at the first detection of the lesion and controlling not to output the notification while tracking the lesion.

[0007] In another aspect of the present disclosure, an endoscopic examination support method includes: acquiring endoscopic images in chronological order; detecting lesions from the endoscopic images; tracking the lesions in the endoscopic images arranged in chronological order; outputting a notification indicating that a new lesion has been found at the first detection of the lesion; and controlling not to output the notification while tracking the lesion.

[0008] In yet another aspect of this disclosure, the recording medium records a program that causes a computer to execute a process that acquires endoscopic images in chronological order, detects lesions from the endoscopic images, tracks the lesions in the endoscopic images arranged in chronological order, outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the computer not to output the notification while the lesion is being tracked.

[0009] According to this disclosure, it becomes possible to appropriately output the results of lesion detection during endoscopic examinations.

[0010] This is a block diagram showing the schematic configuration of an endoscopic examination system. This is a block diagram showing the hardware configuration of an endoscopic examination support device. This is a block diagram showing the functional configuration of an endoscopic examination support device. This is a diagram illustrating the processing by the identical lesion determination unit. This is an example of output from the sound output unit. This is an example of display from the display device. This is a flowchart of the processing by the endoscopic examination support device. This is a block diagram showing the functional configuration of another endoscopic examination support device related to this disclosure. This is a diagram illustrating differential diagnosis by the lesion differentiation unit. This is another example of display from the display device. This is a block diagram showing the functional configuration of another endoscopic examination support device related to this disclosure. This is a flowchart of the processing by another endoscopic examination support device related to this disclosure.

[0011] Preferred embodiments of this disclosure will be described below with reference to the drawings.

[0012] <First Embodiment> [System Configuration] Figure 1 shows a schematic configuration of the endoscopic examination system 100. When the endoscopic examination system 100 detects a lesion during an examination (including treatment) using an endoscope, it outputs a notification indicating that a lesion has been found, such as an alert sound or an alert display. In particular, the endoscopic examination system 100 of this embodiment is characterized by controlling the system so as not to output multiple alerts for the same lesion. As a result, excessive notifications (alerts) are suppressed, allowing the doctor to concentrate on the examination.

[0013] As shown in Figure 1, the endoscopic examination system 100 mainly comprises an endoscopic examination support device 1, a display device 2, and an endoscope scope 3 connected to the endoscopic examination support device 1.

[0014] The endoscopic examination support device 1 acquires images (hereinafter also referred to as "endoscopic image Ic") taken by the endoscope scope 3 during the endoscopic examination and displays the data on the display device 2 for the examiner (physician) to review. Specifically, the endoscopic examination support device 1 acquires the video of the inside of the organs captured by the endoscope scope 3 as endoscopic image Ic during the endoscopic examination.

[0015] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the endoscopy support device 1.

[0016] The endoscope scope 3 mainly consists of an operating unit 36 ​​for the physician to input information such as air insufflation, water insufflation, angle adjustment, and imaging instructions; a flexible shaft 37 that is inserted into the organ being examined by the patient; a tip 38 that incorporates an endoscope camera such as a miniature image sensor; and a connection unit 39 for connecting to the endoscopy support device 1.

[0017] The following explanation will primarily focus on procedures performed during colonoscopy, but the examination may be limited to the large intestine, and may also apply to other parts of the digestive tract, such as the stomach, esophagus, small intestine, and duodenum.

[0018] [Hardware Configuration] Figure 2 shows the hardware configuration of the endoscopic examination support device 1. The endoscopic examination support device 1 mainly includes a processor 11, memory 12, interface 13, input unit 14, light source unit 15, sound output unit 16, and database (hereinafter referred to as "DB") 17. Each of these elements is connected via a data bus 19.

[0019] The processor 11 executes predetermined processes by running programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). Note that the processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0020] Memory 12 is composed of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing the endoscopy support device 1. Memory 12 may also include an external storage device such as a hard disk connected to or built into the endoscopy support device 1, or it may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores a program for the endoscopy support device 1 to execute each of the processes in this embodiment.

[0021] Furthermore, the memory 12 temporarily stores a series of endoscopic images Ic taken by the endoscope scope 3 during an endoscopic examination, based on the control of the processor 11.

[0022] Interface 13 performs interface operations between the endoscopic examination support device 1 and an external device. For example, interface 13 supplies display data Id generated by processor 11 to display device 2. Interface 13 also supplies illumination light generated by light source unit 15 to endoscope scope 3. Interface 13 also supplies an electrical signal indicating the endoscopic image Ic supplied from endoscope scope 3 to processor 11. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

[0023] The input unit 14 generates input signals based on the physician's actions. The input unit 14 can be, for example, a button, touch panel, remote controller, or voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 15 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0024] DB17 stores information about the patient, etc. DB17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or it may include a storage medium such as a removable flash memory. Alternatively, instead of having DB17 within the endoscopic examination system 100, DB17 may be provided on an external server, and relevant information may be obtained from the server via communication.

[0025] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the endoscopic examination support device 1. Functionally, the endoscopic examination support device 1 includes an image preprocessing unit 101, a lesion detection unit 102, a lesion tracking initialization unit 103, a lesion tracking unit 104, a same lesion determination unit 105, and an output unit 106.

[0026] The endoscopic examination support device 1 receives the endoscopic image Ic from the endoscope scope 3. The endoscopic image Ic is sequentially input to the image preprocessing unit 101 and the lesion detection unit 102.

[0027] The image preprocessing unit 101 performs multiple preprocessing operations on each frame image included in the endoscopic image Ic. These preprocessing operations include smoothing, contrast adjustment, resizing, and resolution conversion.

[0028] For example, the image preprocessing unit 101 applies Gaussian blur to the frame image to smooth it. The image preprocessing unit 101 also processes the frame image using the CLAHE method (Contrast Limited Adaptive Histogram Equalization) to enhance the image contrast. The image preprocessing unit 101 also resizes the frame image to a size that allows for tracking by the lesion tracking unit 104, which will be described later. The image preprocessing unit 101 also converts the resolution of the frame image by a predetermined magnification to create multiple images with different resolutions. For example, the image preprocessing unit 101 can use magnifications such as 1.0x, 0.8x, or 0.5x as the predetermined magnification.

[0029] The image preprocessing unit 101 outputs the preprocessed frame images to the lesion tracking initialization unit 103 and the lesion tracking unit 104.

[0030] The lesion detection unit 102 uses a pre-prepared lesion detection AI (Artificial Intelligence) to detect lesions from each frame image included in the endoscopic image Ic. The lesion detection AI is, for example, an AI model that has been pre-trained to detect areas that appear to be lesions in the endoscopic image. The lesion detection unit 102 surrounds the detected lesion with a rectangle or the like and generates its coordinate information. This rectangle will also be referred to as the "detection rectangle" below. The lesion detection unit 102 outputs the coordinate information of the detection rectangle in the frame image to the lesion tracking initialization unit 103 and the output unit 106.

[0031] The lesion tracking initialization unit 103 and the lesion tracking unit 104 track lesions detected by the lesion detection unit 102. The lesion tracking initialization unit 103 and the lesion tracking unit 104 track lesions using a tracking method (tracker) based on a correlation filter such as MOSSE.

[0032] Specifically, the lesion tracking initialization unit 103 sets the information necessary for tracking lesions (hereinafter also referred to as "parameters"). The lesion tracking initialization unit 103 sets the input image information and the location information of the lesion to be tracked (hereinafter also referred to as "target lesion") as parameters. The input image information is the image information that is sequentially input to the lesion tracking unit 104, and includes information such as image size and resolution.

[0033] The lesion tracking initialization unit 103 sets the information of the input image based on the pre-processed frame image input from the image pre-processing unit 101. The lesion tracking initialization unit 103 also sets the position information of the target lesion in the pre-processed frame image based on the coordinate information of the detection rectangle input from the lesion detection unit 102. The pre-processed frame image in which the position information of the target lesion has been set is the reference image for tracking, and will hereinafter also be called the "reference frame image".

[0034] The lesion tracking initialization unit 103 outputs the set parameters to the lesion tracking unit 104.

[0035] The lesion tracking unit 104 receives pre-processed frame images sequentially from the image pre-processing unit 101. Parameters are also input to the lesion tracking unit 104 from the lesion tracking initialization unit 103. The lesion tracking unit 104 tracks the target lesion for a series of pre-processed frame images taken after the reference frame image. The lesion tracking unit 104 then outputs the tracking results to the identical lesion determination unit 105. Since the lesion tracking unit 104 tracks the target lesion for each of the frame images at multiple resolutions, the identical lesion determination unit 105 will output tracking results based on each resolution.

[0036] The identical lesion determination unit 105 determines whether the tracking of the target lesion has been successful and outputs the determination result to the output unit 106. If the identical lesion determination unit 105 determines that the tracking of the target lesion has failed, it outputs an initialization instruction to the lesion tracking initialization unit 103.

[0037] Figure 4 is a diagram illustrating the processing performed by the identical lesion determination unit 105. Figure 4(A) includes a frame image 41, a target lesion 42, a tracking result 43a, a tracking result 43b, and an average 44.

[0038] Frame image 41 is a frame image at a certain point in time. Target lesion 42 is the lesion being tracked. Tracking results 43a and 43b are the tracking results by the lesion tracking unit 104 and are represented by solid rectangular areas. Tracking result 43a is the result of tracking processing on an image obtained by converting the resolution of frame image 41 at a predetermined magnification A. Tracking result 43b is the result of tracking processing on an image obtained by converting the resolution of frame image 41 at a predetermined magnification B. The average 44 is the average position of tracking results 43a and 43b and is represented by a dotted rectangular area.

[0039] The identical lesion determination unit 105 determines whether the tracking of the target lesion has been successful based on the overlap between tracking result 43a and tracking result 43b. Specifically, the identical lesion determination unit 105 calculates the IoU (Intersection over Union) index between tracking result 43a and tracking result 43b. If the IoU is above a predetermined threshold, the identical lesion determination unit 105 determines that the tracking of the target lesion by the lesion tracking unit 104 has been successful. On the other hand, if the IoU is below a predetermined threshold, the identical lesion determination unit 105 determines that the tracking of the target lesion has failed.

[0040] Figure 4(B) shows an example of when tracking of the target lesion fails. For example, if the endoscope camera moves significantly from Figure 4(A) to Figure 4(B), and the position of the target lesion changes significantly, a large discrepancy will occur in the respective tracking results (i.e., the IoU falls below a predetermined threshold). In such cases, the same lesion determination unit 105 determines that tracking of the target lesion has failed and outputs an initialization instruction to the lesion tracking initialization unit 103.

[0041] Note that FIG. 4(B) includes a frame image 45, a target lesion 46, a tracking result 47a, and a tracking result 47b. The frame image 45 is a frame image taken after the frame image 41 in FIG. 4(A). The target lesion 46 is a lesion to be tracked and is the same lesion as the target lesion 42 in FIG. 4(A). The tracking results 47a and 47b are the tracking results by the lesion tracking unit 104.

[0042] Returning to FIG. 3, when the lesion tracking initialization unit 103 receives an initialization instruction from the same lesion determination unit 105, it resets the parameters. Specifically, the lesion tracking initialization unit 103 sets the position information of a new lesion as a parameter. The new lesion is the lesion detected by the lesion detection unit 102 immediately after the initialization instruction is input (i.e., immediately after the tracking of the target lesion fails). Then, the lesion tracking initialization unit 103 outputs the reset parameters to the lesion tracking unit 104, and the lesion tracking unit 104 starts tracking the new lesion.

[0043] The output unit 106 controls the sound output by the sound output unit 16 and the display content by the display device 2 based on the lesion detection result by the lesion detection unit 102 and the determination result by the same lesion determination unit 105.

[0044] For example, when a lesion is detected, the output unit 106 causes the sound output unit 16 or the display device 2 to output an alert. At this time, the output unit 106 controls the sound output unit 16 or the display device 2 so as not to output an alert while the above-mentioned lesion is a lesion being tracked and the tracking is successful. Note that the alert is a sound or display that attracts the attention of a doctor. Examples of the alert output by the sound output unit 16 include a detection sound. Examples of the alert output by the display device 2 include the display of a predetermined text and the blinking of the screen.

[0045] In addition, the output unit 106 may assign a number each time a lesion is detected and cause the display device 2 to display the number. At this time, the output unit 106 assigns the same number while the above-mentioned lesion is a lesion being tracked and the tracking is successful.

[0046] [Output Example] Next, an output example by the sound output unit 16 will be described. FIG. 5 is an output example by the sound output unit 16. As shown in FIG. 5, the sound output unit 16 does not output an alert for each frame image, but outputs an alert only once for each detected lesion. In this way, excessive alerts are suppressed, so that the doctor can concentrate on the examination.

[0047] [Display Example] Next, a display example by the display device 2 will be described. FIG. 6 is a display example by the display device 2. In FIG. 6(A), an endoscope image 61 and a lesion history 62 are displayed on the display device 2. The endoscope image 61 is the endoscope image Ic during the examination, and is updated as the endoscope camera moves. The endoscope image 61 includes a lesion 61a. The lesion 61a indicates a lesion detected during the endoscope examination. In FIG. 6(A), the lesion 61a is assigned the number "1". "1" indicates that it is the first detected lesion in the endoscope examination. In the lesion history 62, an image of the most recently detected lesion is displayed. In FIG. 6(A), an image obtained by cutting out the area including the lesion 61a is displayed.

[0048] FIG. 6(B) is another display example by the display device 2. In this example, an endoscope image 65 and a lesion history 66 are displayed on the display device 2. The endoscope image 65 is the endoscope image Ic during the examination, and is an endoscope image taken later than the endoscope image 61 in FIG. 6(A). The endoscope image 65 includes a lesion � and a lesion 65b. The lesion � is the same lesion as the lesion 61a in FIG. 6(A). The lesion � is assigned the number "1" in the same manner as the lesion 61a. The lesion 65b indicates a new lesion. In FIG. 6(B), the lesion 65b is assigned the number "2". "2" indicates that it is the second detected lesion in the endoscope examination. In the lesion history 66 in FIG. 6(B), an image of the most recently detected lesion is displayed. In FIG. 6(B), an image obtained by cutting out the area including the lesion 65b is displayed.

[0049] Furthermore, the display device 2 will not update lesion history 62 and lesion history 66 if the detected lesion is a lesion being tracked and tracking is successful. This suppresses excessive screen updates, allowing the doctor to concentrate on the examination. Updating the lesion history is an example of an alert.

[0050] Furthermore, if the lesion being observed goes out of scope, the endoscopic examination support device 1 may display a guide to re-scope the lesion. For example, if the lesion being observed goes out of scope to the lower left, the endoscopic examination support device 1 may display a guide instructing the user to return the endoscope scope to the upper right.

[0051] Furthermore, the endoscopic examination support device 1 may assign numbers "1", "2", "3", ... to lesions detected in the order they are detected when the endoscope is inserted, and display the assigned numbers when the endoscope is withdrawn. When the endoscope is withdrawn, the numbers will be displayed in the order of ..., "3", "2", "1". By seeing the display of "1", the physician can understand that the endoscope is close to the exit and that the possibility of further lesions is small.

[0052] In the above configuration, the image preprocessing unit 101 is an example of an acquisition means, the lesion detection unit 102 is an example of a detection means, the lesion tracking initialization unit 103, the lesion tracking unit 104, and the identical lesion determination unit 105 are examples of tracking means, and the output unit 106 is an example of an output control means.

[0053] [Processing Flow] Next, the tracking process described above will be explained. Figure 7 is a flowchart of the processing performed by the endoscopic examination support device 1. This process is achieved when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 3.

[0054] The endoscopic examination support device 1 receives the endoscopic image Ic from the endoscope 3. The endoscopic image Ic is sequentially input to the image preprocessing unit 101 and the lesion detection unit 102 (step S101).

[0055] Next, the image preprocessing unit 101 performs multiple preprocessing operations on each frame image included in the endoscopic image Ic (step S102). The image preprocessing unit 101 outputs the preprocessed frame images to the lesion tracking initialization unit 103 and the lesion tracking unit 104. Next, the lesion detection unit 102 detects lesions from each frame image included in the endoscopic image Ic (step S103). The lesion detection unit 102 outputs the lesion detection results to the lesion tracking initialization unit 103 and the output unit 106.

[0056] Next, the lesion tracking initialization unit 103 sets parameters based on the pre-processed frame images and the lesion detection results (step S104). Then, the lesion tracking unit 104 tracks the target lesion (step S105). The lesion tracking unit 104 outputs the tracking results to the same lesion determination unit 105.

[0057] Next, the identical lesion determination unit 105 determines whether the tracking of the target lesion has been successful (step S106). If the identical lesion determination unit 105 determines that the tracking of the target lesion has been successful (step S106: Yes), it outputs the determination result to the output unit 106. Then, the process returns to step S105. If the identical lesion determination unit 105 determines that the tracking of the target lesion has failed (step S106: No), it outputs the determination result to the output unit 106. Then, the process proceeds to step S107.

[0058] Next, the endoscopic examination support device 1 determines whether the examination is complete or not (step S107). The endoscopic examination support device 1 determines that the examination is complete, for example, when a physician performs an operation to end the examination on the endoscopic examination support device 1 or the endoscope scope 3. Alternatively, the endoscopic examination support device 1 may automatically determine that the examination is complete by analyzing the images captured by the endoscope scope 3 when the captured image becomes an image of an area outside the organ. If it is determined that the examination is not complete (step S107: No), the process returns to step S101. On the other hand, if it is determined that the examination is complete (step S107: Yes), the process ends.

[0059] Step S103 may be executed before step S102, or it may be executed simultaneously with step S102.

[0060] As described above, the endoscopic examination support device according to the first embodiment tracks the same lesion by combining lesion detection and object tracking technologies. Furthermore, the endoscopic examination support device controls itself so as not to output alerts while tracking is successful. This method makes it possible for the endoscopic examination support device to suppress excessive alerts.

[0061] <Second Embodiment> Next, a second embodiment will be described.

[0062] In AI-based lesion differential diagnosis, diagnosis is sometimes performed based on a single frame image. However, in this case, even for the same lesion, the diagnosis may change if the imaging position or other factors differ. Therefore, the endoscopic examination support device 1a according to the second embodiment stores multiple frame images of the same lesion and outputs a final diagnosis result by taking into account the diagnostic results of multiple frame images. This makes it possible to improve the accuracy of lesion differential diagnosis.

[0063] The endoscopic examination support device 1a according to the second embodiment has the same system configuration and hardware configuration as the endoscopic examination support device 1 according to the first embodiment, so a description will be omitted.

[0064] [Functional Configuration] Figure 8 is a block diagram showing the functional configuration of the endoscopic examination support device 1a according to the second embodiment. The endoscopic examination support device 1a is based on the endoscopic examination support device 1 according to the first embodiment, but further includes a lesion image storage unit 107 and a lesion identification unit 108.

[0065] The lesion image storage unit 107 stores the endoscopic image Ic input from the endoscope scope 3 in the memory 12. At this time, the lesion image storage unit 107 stores the endoscopic image Ic, including the lesion detection result, in the memory 12.

[0066] Specifically, the lesion image storage unit 107 uses lesion detection AI to detect lesions from each frame image included in the endoscopic image Ic. The lesion image storage unit 107 surrounds the detected lesion with a rectangle or the like and stores the cropped image (hereinafter also called the "cropped image") around the detection rectangle in the memory 12. At this time, in order to identify which frame image generated the cropped image, the lesion image storage unit 107 stores the cropped image in the memory 12 in association with, for example, the frame number. The lesion image storage unit 107 may also store the frame image including the detection rectangle in the memory 12.

[0067] The lesion differentiation unit 108 performs differential diagnosis of lesions using a pre-prepared differential diagnosis AI. Differential diagnosis includes determining whether a lesion is benign or malignant, and determining its size. The differential diagnosis AI in this embodiment is assumed to be an AI model that has been pre-trained to output the degree of malignancy for lesions included in endoscopic images.

[0068] Figure 9 is a diagram illustrating the differential diagnosis performed by the lesion differentiation unit 108. As shown in Figure 9, the lesion differentiation unit 108 performs differential diagnosis of lesions on multiple images and outputs a final result by taking into account the results of each diagnosis.

[0069] For example, when a doctor finds a lesion during an endoscopic examination, they operate the endoscope scope 3 to input instructions for imaging the lesion. The endoscopic examination support device 1a generates still images from the endoscopic images Ic, which are video, based on the doctor's imaging instructions. The lesion identification unit 108 acquires a predetermined number of frame images from memory 12 that were taken before the still images and that contain the same lesion. In Figure 9, the lesion identification unit 108 acquires three frame images taken before the still images from memory 12. The lesion identification unit 108 then performs a differential diagnosis for each image, calculates a weighted average of the diagnostic results, and outputs the final diagnostic result. In Figure 9, the lesion identification unit 108 diagnoses the degree of malignancy for each image and outputs the probability of malignancy. The lesion identification unit 108 also calculates the average value of each probability and outputs the final diagnostic result (average degree of malignancy ≈ 58%).

[0070] Although Figure 9 uses frame images as an example, the lesion differentiation unit 108 may acquire crop images from memory 12 and perform differential diagnosis based on the crop images.

[0071] Furthermore, while the lesion differentiation unit 108 performs differential diagnosis at the time of the physician's instruction to take images, the timing of differential diagnosis is not limited to this. For example, the lesion differentiation unit 108 may generate a still image and perform differential diagnosis at the time of the physician's voice input, such as "differentiate it."

[0072] The lesion identification unit 108 outputs the differential diagnosis results to the output unit 106. The output unit 106 outputs the differential diagnosis results to the display device 2.

[0073] [Display Example] Next, an example of display by the display device 2 will be explained. Figure 10 is an example of display by the display device 2. In Figure 10, the display device 2 displays the endoscopic image 91, the lesion history 92, and the differential diagnosis result 93. The endoscopic image 91 is the endoscopic image Ic during the examination and is updated as the endoscopic camera moves. The endoscopic image 91 includes lesions 91a and 91b. The lesion history 92 displays images of the most recently detected lesions. The differential diagnosis result 93 displays the result of the differential diagnosis by the lesion differentiation unit 108.

[0074] Furthermore, as shown in Figure 10, if the endoscopic image 91 contains one or more lesions, the physician may give instructions via voice input, such as "Differentiate lesion 2." The lesion differentiation unit 108 performs a differential diagnosis of the relevant lesions based on the physician's instructions, and the display device 2 displays the results of the differential diagnosis.

[0075] In the above configuration, the lesion image storage unit 107 and the lesion identification unit 108 are examples of a reception means and a malignancy prediction means.

[0076] <Third Embodiment> Figure 11 is a block diagram showing the functional configuration of the endoscopic examination support device of the third embodiment. The endoscopic examination support device 300 includes an acquisition means 301, a detection means 302, a tracking means 303, and an output control means 304.

[0077] Figure 12 is a flowchart of the processing performed by the endoscopic examination support device of the third embodiment. The acquisition means 301 acquires endoscopic images in chronological order (step S301). The detection means 302 detects lesions from the endoscopic images (step S302). The tracking means 303 tracks the lesions in the endoscopic images arranged in chronological order (step S303). The output control means 304 outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the output of the notification while tracking the lesion (step S304).

[0078] According to the endoscopic examination support device 300 of the third embodiment, it becomes possible to appropriately output the detection results of lesions during endoscopic examinations. In this way, the endoscopic examination support device 300 can optimize information output in the medical field.

[0079] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0080] (Note 1) An endoscopy support device comprising: an acquisition means for acquiring endoscopic images in chronological order; a detection means for detecting lesions from the endoscopic images; a tracking means for tracking the lesions in the endoscopic images arranged in chronological order; and an output control means for outputting a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controlling the output of the notification while the lesion is being tracked.

[0081] (Note 2) The endoscopic examination support device according to Note 1, comprising: a receiving means for receiving instructions to photograph a lesion from an endoscope operator; and a malignancy prediction means for predicting the malignancy of a lesion using a still image generated based on the photography instructions and a predetermined number of endoscopic images that include the lesion and were taken before the still image.

[0082] (Note 3) The notification is an audio notification, provided by the endoscopic examination support device described in Note 1.

[0083] (Note 4) The notification is an endoscopic examination support device as described in Note 1, which is a notification displayed on a screen.

[0084] (Note 5) The endoscopy support device according to Note 1, wherein the acquisition means converts the endoscopic image to multiple resolutions, and the tracking means tracks the lesion in the endoscopic image at each of the multiple resolutions, and determines whether the tracking is successful based on the respective tracking results.

[0085] (Note 6) The endoscopic examination support device according to Note 5, wherein the tracking result is a prediction of the area of ​​the lesion in an endoscopic image taken after the endoscopic image in which the lesion was detected, and the tracking means determines whether the tracking is successful based on the IoU between each tracking result in the endoscopic image at a certain point in time.

[0086] (Note 7) The endoscopic examination support device according to Note 6, wherein the tracking means determines that the tracking is successful if the IoU is above a predetermined threshold, and determines that the tracking is not successful if the IoU is below a predetermined threshold.

[0087] (Note 8) The endoscopic examination support device according to Note 5, wherein the tracking means continues tracking the lesion if the tracking is successful, and terminates tracking the lesion if the tracking is not successful.

[0088] (Note 9) The endoscopic examination support device according to Note 8, wherein the tracking means starts tracking the next lesion if the detection means detects the next lesion after the tracking of the previous lesion has been completed.

[0089] (Note 10) The endoscopic examination support device according to Note 1, wherein the detection means takes the endoscopic image as input and detects the lesion using an AI model that has been trained to detect lesions contained in the endoscopic image.

[0090] (Note 11) The endoscopic examination support device according to Note 2, wherein the malignancy prediction means predicts the malignancy of the still image and a predetermined number of endoscopic images, and outputs a weighted average of the prediction results as the malignancy of the lesion.

[0091] (Note 12) The malignancy prediction means is an endoscopic examination support device according to Note 11, which uses an AI model trained to take the endoscopic image as input and output the probability that the lesion contained in the endoscopic image is malignant to predict the degree of malignancy.

[0092] (Note 13) The endoscopic examination support device according to Note 11, wherein the malignancy prediction means extracts an image of the region containing the lesion from the still image and the endoscopic image, and predicts the malignancy based on the extracted image.

[0093] (Note 14) The endoscopic examination support device according to Note 5, wherein the acquisition means performs preprocessing on the endoscopic image, including smoothing, contrast adjustment, and resizing, and the tracking means tracks the lesion in the preprocessed endoscopic image.

[0094] (Note 15) An endoscopy support method that acquires endoscopic images in chronological order, detects lesions from the endoscopic images, tracks the lesions in the endoscopic images arranged in chronological order, outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the system so that the notification is not output while the lesion is being tracked.

[0095] (Note 16) A recording medium that records a program that causes a computer to execute a process that acquires endoscopic images in chronological order, detects lesions from the endoscopic images, tracks the lesions in the endoscopic images arranged in chronological order, outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the computer to not output the notification while the lesion is being tracked.

[0096] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure.

[0097] 1, 1a Endoscopy support device 2 Display device 3 Endoscope scope 11 Processor 12 Memory 13 Interface 100 Endoscopy system 101 Image preprocessing unit 102 Lesion detection unit 103 Lesion tracking initialization unit 104 Lesion tracking unit 105 Identical lesion determination unit 106 Output unit 107 Lesion image storage unit 108 Lesion differentiation unit

Claims

1. An endoscopy support device comprising: an acquisition means for acquiring endoscopic images in chronological order; a detection means for detecting lesions from the endoscopic images; a tracking means for tracking the lesions in the endoscopic images arranged in chronological order; and an output control means for outputting a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controlling the output of the notification while the lesion is being tracked.

2. The endoscopic examination support device according to claim 1, comprising: a receiving means for receiving instructions from an endoscope operator to photograph a lesion; and a malignancy prediction means for predicting the malignancy of a lesion using a still image generated based on the photography instructions and a predetermined number of endoscopic images that include the lesion and were taken before the still image.

3. The endoscopic examination support device according to claim 1, wherein the notification is an audio notification.

4. The endoscopic examination support device according to claim 1, wherein the notification is a notification via screen display.

5. The endoscopy support device according to claim 1, wherein the acquisition means converts the endoscopic image to multiple resolutions, and the tracking means tracks the lesion in the endoscopic image at each of the multiple resolutions, and determines whether the tracking is successful based on the respective tracking results.

6. The endoscopic examination support device according to claim 5, wherein the tracking result is a prediction of the area of ​​the lesion in an endoscopic image taken after the endoscopic image in which the lesion was detected, and the tracking means determines whether the tracking is successful based on the IoU between each tracking result in the endoscopic image at a certain point in time.

7. The endoscopic examination support device according to claim 6, wherein the tracking means determines that the tracking is successful if the IoU is above a predetermined threshold, and determines that the tracking is not successful if the IoU is below a predetermined threshold.

8. The endoscopic examination support device according to claim 5, wherein the tracking means continues tracking the lesion if the tracking is successful, and terminates tracking the lesion if the tracking is not successful.

9. The endoscopic examination support device according to claim 8, wherein the tracking means starts tracking the next lesion if the detection means detects the next lesion after the tracking of the lesion has been completed.

10. The endoscopic examination support device according to claim 1, wherein the detection means takes the endoscopic image as input and detects the lesion using an AI model that has been trained to detect lesions contained in the endoscopic image.

11. The endoscopic examination support device according to claim 2, wherein the malignancy prediction means predicts the malignancy of the still image and a predetermined number of endoscopic images, and outputs a weighted average of the prediction results as the malignancy of the lesion.

12. The endoscopic examination support device according to claim 11, wherein the malignancy prediction means uses an AI model trained to take the endoscopic image as input and output the probability that the lesion contained in the endoscopic image is malignant to predict the degree of malignancy.

13. The endoscopic examination support device according to claim 11, wherein the malignancy prediction means extracts an image of the region including the lesion from the still image and the endoscopic image, and predicts the malignancy based on the extracted image.

14. The endoscopy support device according to claim 5, wherein the acquisition means performs preprocessing on the endoscope image, including smoothing, contrast adjustment, and resizing, and the tracking means tracks the lesion in the preprocessed endoscope image.

15. An endoscopy support method that acquires endoscopic images in chronological order, detects lesions from the endoscopic images, tracks the lesions in the endoscopic images arranged in chronological order, outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the system to not output the notification while tracking the lesion.

16. A recording medium that contains a program that causes a computer to execute a process that acquires endoscopic images in chronological order, detects lesions from the endoscopic images, tracks the lesions in the endoscopic images arranged in chronological order, outputs a notification indicating that a new lesion has been found when the lesion is detected for the first time, and controls the computer to not output the notification while the lesion is being tracked.

Images