Medical support device, endoscope apparatus, medical support method, and program

Abstract

To provide a medical support device, an endoscope apparatus, a medical support method, and a program that allow a user to grasp a presence position of an internal characteristic region outside an image obtained by imaging the inside of the body with a camera even if a positional relationship between the camera and the internal characteristic region changes.SOLUTION: A medical support device includes a processor. The processor acquires an image obtained by imaging the inside of the body with a camera. The processor outputs screen generation information used for generating a screen in which the image and presence position information that allows a presence position of an internal characteristic region recognized through execution of object recognition processing using the image to be identified outside the image are displayed. A display position of the presence position information relative to the image in the screen is changed according to the change in a positional relationship between the camera and the internal characteristic region.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a medical support device, an endoscope device, a medical support method, and a program. [Background technology]

[0002] Patent Document 1 discloses an information processing device having a detection means, a determination means, and a notification means. In the information processing device described in Patent Document 1, the detection means detects an abnormal area inside the body from a video of the inside of the body. The determination means determines whether or not a predetermined condition is satisfied when an abnormal area is detected within a predetermined range of a first video frame of the video, and the abnormal area is not detected within a predetermined range of a second video frame of the video generated after the first video frame. The notification means issues a first notification when it is determined that the predetermined condition is not satisfied.

[0003] Patent Document 2 discloses an endoscopic observation support device that supports the observation of a tubular organ by an endoscope. The endoscopic observation support device described in Patent Document 2 includes an image information acquisition unit, a lesion information acquisition unit, a determination unit, and a notification unit. In the endoscopic observation support device described in Patent Document 2, the image information acquisition unit acquires an image of the tubular organ photographed by an endoscope and displays it on a display unit. The lesion information acquisition unit detects a predetermined lesion based on the captured image and acquires lesion information related to the lesion. The determination unit tracks the lesion based on the captured image and the lesion information, and determines whether the lesion has disappeared from the captured image. The notification unit notifies the determination result when the determination unit determines that the lesion has disappeared from the captured image.

[0004] Patent Document 3 discloses an endoscope system including an endoscope, an endoscope control device, and a medical image processing device. In the endoscope system described in Patent Document 3, the medical image processing device includes an image acquisition unit, an attention area detection unit, an unobserved condition determination unit, an unobserved image storage unit, and a display signal transmission unit. The image acquisition unit acquires an observation image of a subject. The attention area detection unit detects an attention area from a frame image constituting the observation image. The unobserved condition determination unit determines whether or not the frame image in which the attention area is detected satisfies an unobserved condition indicating that an attention area overlooked by the user is included. The unobserved image storage unit stores an unobserved image that satisfies the unobserved condition. The display signal transmission unit transmits a first display signal representing the observation image and a second display signal representing the unobserved image to the display device. The attention area detection unit determines whether the attention area of ​​the unobserved image and the attention area of ​​the observation image are identical. The display signal transmission unit stops transmission of the second display signal when a determination result is obtained that the attention area of ​​the unobserved image and the attention area of ​​the observation image are identical.

[0005] In the endoscopic system described in Patent Document 3, the unobserved condition determination unit determines that the unobserved condition is met when the number of frame images containing the same area of ​​interest within a specified period is less than or equal to a specified number, determines that the unobserved condition is met when the amount of change between frame images is greater than or equal to a specified threshold, or determines that the unobserved condition is met when the same area of ​​interest remains in any area within the screen within a specified period.

[0006] Patent Document 4 discloses a medical image processing system including a medical image acquisition unit, an attention area detection unit, and a display control unit. In the medical image processing system described in Patent Document 4, the medical image acquisition unit acquires a medical image obtained by imaging an observation target. The attention area detection unit detects an attention area from the medical image. The display control unit displays the detection result of the attention area on the display unit in a display mode that differs depending on at least the detection position of the attention area.

[0007] Patent Document 5 discloses a medical image processing system including an image acquisition unit that acquires a medical image, a display unit that displays the medical image, an attention area detection unit that detects an attention area from within the medical image, a motion amount estimation unit that estimates the amount of motion of an apparatus that captures the medical image based on the medical image, and a notification unit that, when an attention area is detected, highlights the attention area in the medical image displayed on the display unit to notify the detection of the attention area, and changes the degree of highlighting of the attention area according to the amount of motion estimated by the motion amount estimation unit after the notification. Here, the notification unit increases the degree of highlighting of the attention area when the amount of motion estimated by the motion amount estimation unit is equal to or greater than a threshold after notifying the detection of the attention area, or decreases the degree of highlighting of the attention area or turns off the highlighting of the attention area when the amount of motion estimated by the motion amount estimation unit is less than the threshold after notifying the detection of the attention area. The notification unit also highlights the attention area by surrounding the attention area with a frame in the medical image displayed on the display unit. Here, the notification unit changes at least one of the thickness, line type, color, shape, blinking degree, and brightness of the frame to change the degree of highlighting of the attention area.

[0008] In the medical image processing system described in Patent Document 5, the notification unit generates a sound when a region of interest is detected to notify the detection of the region of interest, and changes the volume of the sound according to the amount of motion estimated by the motion amount estimation unit after the notification. Here, the notification unit increases the volume of the sound when the amount of motion estimated by the motion amount estimation unit is equal to or greater than a threshold after notifying the detection of the region of interest, and decreases the volume of the sound or turns off the sound when the amount of motion estimated by the motion amount estimation unit is less than the threshold after notifying the detection of the region of interest. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] International Publication No. 2018 / 216618 [Patent Document 2] JP 2019-180966 A [Patent Document 3] Patent No. 7256275 [Patent Document 4] International Publication No. 2020 / 039968 [Patent Document 5] Patent No. 7225417 Summary of the Invention

[0010] One embodiment of the present disclosure provides a medical support device, an endoscopic device, a medical support method, and a program that enable a user to grasp the location of an internal characteristic area outside an image obtained by capturing an image of the inside of the body with a camera, even if the positional relationship between the camera and the internal characteristic area changes. [Means for solving the problem]

[0011] A first aspect of the present disclosure is a medical support device that includes a processor, which acquires images obtained by capturing images of the inside of the body with a camera, and outputs screen generation information used to generate a screen that displays the images and location information that enables the location outside of the image to be identified of internal characteristic areas recognized by executing an object recognition process using the images, and the display position of the location information relative to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic areas.

[0012] A second aspect of the present disclosure is the medical support device according to the first aspect, in which the change is brought about by camera operation and / or internal body movement.

[0013] A third aspect of the present disclosure is the medical support device according to the second aspect, in which the display position is changed in accordance with the change to follow the operation and / or body movement.

[0014] A fourth aspect of the present disclosure is the medical support device according to any one of the first to third aspects, in which a display mode of the location information is changed according to a characteristic of the change.

[0015] A fifth aspect of the present disclosure is the medical support device according to the fourth aspect, wherein the characteristic includes a speed of change, an amount of change, and / or a direction of change.

[0016] A sixth aspect of the present disclosure is a medical support device according to any one of the first to fifth aspects, wherein the presence position information includes within-angle-of-view position information that enables the presence position of the internal characteristic region to be identified externally when the internal characteristic region is within the angle of view of the camera, and outside-angle-of-view position information that enables the presence position of the internal characteristic region to be identified externally when the internal characteristic region is outside the angle of view, and the within-angle-of-view position information is displayed on the screen when the internal characteristic region is within the angle of view, and the outside-angle-of-view position information is displayed on the screen when the internal characteristic region is outside the angle of view.

[0017] A seventh aspect of the present disclosure is a medical support device according to any one of the first to sixth aspects, in which the location information includes outside-of-view position information that enables the location of an internal characteristic region to be identified externally when the internal characteristic region is outside the angle of view of a camera, the outside-of-view position information is displayed on a screen when the internal characteristic region is outside the angle of view, and the display manner of the outside-of-view position information is changed according to the characteristics of the change.

[0018] An eighth aspect of the present disclosure is the medical support device according to the seventh aspect, in which the display aspect includes the presence or absence of display, display intensity, display time, and / or a speed at which the display intensity is changed.

[0019] A ninth aspect of the present disclosure is a medical support device according to any one of the first to eighth aspects, wherein the location information includes outside-angle-of-view location information capable of externally identifying the location of an internal characteristic region when the internal characteristic region is outside the angle of view of a camera, the outside-angle-of-view location information is displayed on a screen when the internal characteristic region is outside the angle of view, the internal characteristic region is a lesion, and the display manner of the outside-angle-of-view location information is changed depending on the malignancy of the lesion, the location where the lesion is located, the type of lesion, the form of the lesion, the boundary between the lesion and its surroundings, and / or the manner in which mucus adheres to the lesion.

[0020] A tenth aspect of the present disclosure is a medical support device according to any one of the first to ninth aspects, in which the location information includes outside-angle-of-view location information that enables the location of an internal characteristic region to be identified externally when the internal characteristic region is outside the angle of view of a camera, and the outside-angle-of-view location information is displayed on the screen under condition that the internal characteristic region is outside the angle of view.

[0021] An eleventh aspect of the present disclosure is a medical support device according to the tenth aspect, in which the display on the screen of outside-angle-of-view position information when the within-angle-of-view time during which the internal characteristic region is within the angle of view is less than a certain value is emphasized more than the display on the screen of outside-angle-of-view position information when the within-angle-of-view time is equal to or greater than a certain value.

[0022] A twelfth aspect of the present disclosure is a medical support device according to any one of the first to ninth aspects, wherein the location information includes outside-angle-of-view location information that enables the location of the internal characteristic region to be identified externally when the internal characteristic region is outside the angle of view of the camera, and the outside-angle-of-view location information is displayed on the screen on condition that a predetermined time has elapsed since the internal characteristic region moved out of the angle of view.

[0023] A thirteenth aspect of the present disclosure is a medical support device according to any one of the first to twelfth aspects, wherein the location information includes outside-angle-of-view location information that enables external identification of the location of an internal characteristic region when the internal characteristic region is outside the angle of view of a camera, and the outside-angle-of-view location information is displayed on the screen under conditions that the internal characteristic region is outside the angle of view and the degree of change exceeds a predetermined degree.

[0024] A fourteenth aspect of the present disclosure is a medical support device according to any one of the first to thirteenth aspects, wherein the location information includes outside-of-view location information capable of externally identifying the location of an internal characteristic region when it is outside the angle of view of a camera, the outside-of-view location information is displayed on a screen when the internal characteristic region is outside the angle of view, and the display on the screen of the outside-of-view location information when the frequency at which the internal characteristic region moves in and out of the angle of view within a unit time exceeds a predetermined frequency is emphasized more than the display on the screen of the outside-of-view location information when the frequency is equal to or less than the predetermined frequency.

[0025] A fifteenth aspect of the present disclosure is a medical support device relating to any one of the first to fourteenth aspects, in which the screen generation information includes an image and position indication information indicating the position of the presence position information within the screen, and the position indication information is updated according to changes.

[0026] A sixteenth aspect of the present disclosure is the medical support device according to the fifteenth aspect, in which the screen generation information includes an image, location information, and location indication information.

[0027] A seventeenth aspect of the present disclosure is a medical support device according to any one of the first to sixteenth aspects, wherein the object recognition processing includes processing for recognizing internal characteristic regions based on an image by using AI.

[0028] An eighteenth aspect of the present disclosure is the medical support device according to any one of the first to seventeenth aspects, in which the internal characteristic region is a lesion.

[0029] A 19th aspect of the present disclosure is a medical support device of any one of the 1st to 18th aspects, in which the image is contained in a plurality of frames obtained in time series by imaging the inside of the body with a camera, and a processor identifies changes based on the plurality of frames and changes the display position in accordance with the identified changes.

[0030] A twentieth aspect of the present disclosure is a medical support device according to any one of the first to nineteenth aspects, in which a processor identifies a change based on a detection result by a sensor capable of detecting behavior of the camera inside the body, and changes the display position according to the identified change.

[0031] A twenty-first aspect of the present disclosure is a medical support device that includes a processor, which acquires images obtained by capturing images of the inside of the body with a camera, and outputs screen generation information used to generate a screen that displays a medical image generated based on the images and location information that enables the location of an internal characteristic area recognized by executing an object recognition process using the images to be identified outside the image, and a display position of the location information relative to the medical image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic area.

[0032] A twenty-second aspect of the present disclosure is an endoscope device including a medical support device according to any one of the first to twenty-first aspects and a camera.

[0033] A 23rd aspect of the present disclosure is a medical support method including obtaining an image obtained by capturing an image of the inside of the body with a camera, and outputting screen generation information used to generate a screen on which the image and location information capable of identifying, outside the image, the location of an internal characteristic area recognized by executing an object recognition process using the image are displayed, and the display position of the location information relative to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic area.

[0034] A 24th aspect of the present disclosure is a program for causing a computer to execute medical support processing, the medical support processing including acquiring an image obtained by capturing an image of the inside of the body with a camera, and outputting screen generation information used to generate a screen on which the image and location information capable of identifying, outside the image, the location of an internal characteristic area recognized by executing an object recognition process using the image are displayed, and the display position of the location information relative to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic area. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope device is used. [Diagram 2] 1 is a conceptual diagram showing an example of an overall configuration of an endoscope apparatus. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope apparatus. FIG. [Figure 4] 2 is a block diagram showing an example of main functions of a processor included in the medical support device according to an embodiment, and an example of information stored in a storage. FIG. [Diagram 5] FIG. 4 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit. [Figure 6]13A and 13B are conceptual diagrams showing an example of the display content of the first display area when a lesion is within the angle of view, and an example of the display content of the first display area when the lesion is outside the angle of view. [Figure 7] 10 is a conceptual diagram showing an example of a mode in which screen information is generated by a control unit and a screen indicated by the screen information is displayed on a display device. FIG. [Figure 8] FIG. 13 is a conceptual diagram showing an example of the display content when the position of a lesion that is out of the frame (i.e., the position of a lesion that is not captured in the frame) is changed. [Figure 9] 13 is a conceptual diagram showing an example of a display mode of a visual assist mark when the distance from the center of the frame to the lesion changes. FIG. [Figure 10] 13 is a flowchart showing an example of the flow of a medical support process. [Figure 11] 13 is a flowchart showing a first modified example of the flow of medical support processing. [Figure 12] 13 is a flowchart showing a second modified example of the flow of medical support processing. [Figure 13] 13 is a flowchart showing a third modified example of the flow of medical support processing. [Figure 14] 13 is a flowchart showing a fourth modified example of the flow of medical support processing. [Figure 15] 13 is a flowchart showing a fifth modified example of the flow of medical support processing. [Figure 16] 13A and 13B are conceptual diagrams showing an example of a screen display when a lesion position is frame-in and an example of a screen display when the lesion position is frame-out. [Figure 17] FIG. 13 is a conceptual diagram showing a first display example of a screen when multiple lesions exist inside and outside the angle of view. [Figure 18] FIG. 13 is a conceptual diagram showing a second display example of the screen when multiple lesions exist inside and outside the angle of view. [Figure 19] FIG. 13 is a conceptual diagram showing a third display example of the screen when multiple lesions exist inside and outside the angle of view. [Figure 20]FIG. 13 is a conceptual diagram showing a fourth display example of the screen when multiple lesions exist inside and outside the angle of view. [Figure 21] 13 is a conceptual diagram showing an example of an aspect in which a lesion location map and a visual assist mark are displayed in the second display area. FIG. [Figure 22] 13 is a conceptual diagram showing a modified example of a visual assist mark displayed on the screen. FIG. [Diagram 23] 13 is a conceptual diagram showing an example of processing content when the display mode of a visual assist mark displayed on the screen is changed according to the characteristics of a lesion. FIG. [Figure 24] 1 is a block diagram showing an example of a hardware configuration of an electrical system of an endoscope apparatus equipped with an endoscope insertion shape observation device. FIG. [Diagram 25] 11 is a conceptual diagram showing an example of a manner in which screen generation information is generated and output by a control unit. FIG. [Figure 26] 1 is a conceptual diagram showing an example of a series of processes in which a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the computer receives a processing result from the external device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, examples of embodiments of a medical support device, an endoscope device, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0037] First, the terms used in the following description will be explained.

[0038] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System." FIFO is an abbreviation for "First In First Out."

[0039] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0040] In the following description, a signed memory is at least one memory (eg, a volatile memory) such as a RAM in which information is temporarily stored and which is used as a working memory by a processor.

[0041] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0042] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0043] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0044] Fig. 1 is a conceptual diagram showing an example of an aspect in which an endoscopic device 10 is used. As shown in Fig. 1, the endoscopic device 10 is used by a doctor 12, who is a user, in an endoscopic examination. The endoscopic examination is assisted by staff such as a nurse 14. In this embodiment, the endoscopic device 10 is an example of an "endoscopic device" according to the present disclosure.

[0045] The endoscope device 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope device 10 is transmitted to the communication device. An example of the communication device is a server and / or a client terminal (e.g., a personal computer and / or a tablet terminal, etc.) that manages various information such as an electronic medical record. The communication device receives the information transmitted from the endoscope device 10 and executes processing using the received information (e.g., processing to store the information in an electronic medical record, etc.).

[0046] The endoscope device 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24.

[0047] The endoscope device 10 is a modality for performing medical examination on a large intestine 28 contained in the body of a subject 26 (e.g., a patient) using an endoscope scope 16. In this embodiment, the large intestine 28 is an object observed by a doctor 12 in an endoscopic examination.

[0048] The endoscope 16 is used by the doctor 12 and inserted into a hollow organ of the subject 26. In this embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. The endoscope device 10 causes the endoscope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and performs various medical procedures on the large intestine 28 as necessary.

[0049] The endoscope device 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In this embodiment, the endoscope device 10 is an endoscope having an optical imaging function for capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0050] Although endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and endoscopic examination of a hollow organ such as the esophagus, stomach, duodenum, or trachea may also be performed.

[0051] The light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of tables arranged vertically, and the medical support device 24, the control device 22, and the light source device 20 are installed from the lower table to the upper table. In addition, the display device 18 is installed on the top table of the wagon 34.

[0052] The control device 22 controls the entire endoscope device 10. The medical support device 24, under the control of the control device 22, performs various image processing on the images obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0053] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, instead of the display device 18 or together with the display device 18, a tablet terminal with a display may be used.

[0054] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas. The size of the first display area 36 is larger than the size of the second display area 38. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub-display area. Note that the size relationship between the first display area 36 and the second display area 38 is not limited thereto, and may be any size relationship that fits within the screen 35.

[0055] An endoscopic moving image 39 is displayed in the first display area 36. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0056] The intestinal wall 32 shown in the endoscopic video 39 includes an internal characteristic region, which is a region having characteristics inside the body (here, as an example, inside the large intestine 28). One example of an internal characteristic region is a region of interest (i.e., a region to be observed) that is gazed upon by the doctor 12. In this embodiment, the doctor 12 can visually recognize the aspect of the intestinal wall 32 including the internal characteristic region through the endoscopic video 39. Note that, in the following, at least one lesion 42 (for example, one lesion 42 in the example shown in FIG. 1) will be described as an example of an internal characteristic region. In this embodiment, the lesion 42 is an example of an "internal characteristic region" and a "lesion" according to the present disclosure.

[0057] There are various types of lesions 42, and examples of the types of lesions 42 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are assumed in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions will differ depending on the organ in which the endoscopic examination is performed.

[0058] The image displayed in the first display area 36 is one frame 40 included in a moving image that includes a plurality of frames 40 (i.e., a plurality of frames 40 in chronological order) obtained by imaging the intestinal wall 32 with the endoscope 16. That is, the first display area 36 displays the plurality of frames 40 in chronological order at a preset frame rate (e.g., several tens of frames per second).

[0059] In this embodiment, the frame 40 is an example of an "image" according to the present disclosure. Also, in this embodiment, the multiple frames 40 obtained in time series by imaging the intestinal wall 32 with the endoscope 16 are an example of a "multiple frames" according to the present disclosure.

[0060] An example of a moving image displayed in the first display area 36 is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post view format, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on the screen 35 (for example, the first display area 36) as an endoscopic moving image 39.

[0061] Within the screen 35, the second display area 38 exists outside the first display area 36. In the example shown in Fig. 1, the second display area 38 is adjacent to the first display area 36, ​​and is displayed on the right side in a front view within the screen 35. The display position of the second display area 38 may be anywhere as long as it is different from the first display area 36, ​​but it is preferable that the second display area 38 is displayed at a position comparable to the endoscopic moving image 39 displayed within the first display area 36.

[0062] In the second display area 38, medical information 44, which is information related to medical care, is displayed. Examples of the medical information 44 include information that assists the medical judgment of the doctor 12. A first example of the information that assists the medical judgment of the doctor 12 includes various visible information (e.g., name, sex, medication, medical history, blood pressure value, and / or heart rate) related to the subject 26 into which the endoscope 16 is inserted. A second example of the information that assists the medical judgment of the doctor 12 includes visible information such as text and / or images (e.g., feature map and / or information obtained by processing the feature map) obtained by processing the endoscopic video 39 using AI.

[0063] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope device 10. As shown in Fig. 2, the endoscope scope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation section 46 by the doctor 12 (see Fig. 1).

[0064] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, this is merely one example, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing scope.

[0065] The camera 52 is inserted into the body cavity of the subject 26 to capture an image of the observation target region. In this embodiment, the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to obtain the endoscopic moving image 39. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. In this embodiment, the camera 52 is an example of a "camera" according to the present disclosure.

[0066] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0067] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0068] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. Examples of the treatment tool 58 include a hemostat, a puncture needle, a high-frequency knife, a snare, a catheter, a guide wire, and a cannula. In the example shown in FIG. 2, a hemostat protruding from the treatment tool opening 56 is shown as the treatment tool 58.

[0069] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0070] In this example, the medical support device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the medical support device 24 may be installed in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as the process executed by the medical support device 24 (for example, the medical support process described later) and to receive and use the results of the process by the server.

[0071] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0072] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0073] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (for example, the medical support device 24).

[0074] The medical support device 24 performs various types of image processing on the endoscopic moving image 39 input from the control device 22 to provide support for medical care (here, as an example, an endoscopic examination). The medical support device 24 outputs the endoscopic moving image 39 that has been subjected to various types of image processing to a predetermined output destination (for example, the display device 18).

[0075] Although an example in which the endoscopic moving image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 has been described here, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic moving image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0076] Fig. 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope device 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0077] The external I / F 70 is responsible for transmitting and receiving various types of information between the processor 72 and one or more devices (hereinafter, also referred to as “first external devices”) that exist outside the control device 22.

[0078] The camera 52 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for transmitting and receiving various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, an endoscopic moving image 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).

[0079] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 20.

[0080] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0081] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In this embodiment, the medical support device 24 is an example of a "medical support device" according to the present disclosure, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0082] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0083] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter, also referred to as “second external devices”) that exist outside the medical support device 24.

[0084] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic moving images 39 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 80, and performs various image processing on the acquired endoscopic moving images 39.

[0085] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80 to cause the display device 18 to display various information (e.g., an endoscopic moving image 39 that has been subjected to various image processing, etc.).

[0086] In the endoscopic examination, the doctor 12 visually recognizes the lesion 42 present in the large intestine 28 while observing the endoscopic video image 39 displayed in the first display area 36. However, if the doctor 12 is unable to operate the camera 52 as desired or if the body movement (for example, the movement of the large intestine 28) is greater than expected, the positional relationship between the camera 52 and the lesion 42 becomes unexpected. In this case, the lesion 42 may fall outside the viewing angle of the camera 52 (hereinafter, also simply referred to as the "viewing angle"). If the lesion 42 falls outside the viewing angle, the doctor 12 cannot observe the lesion 42 through the first display area 36. This causes the doctor 12 to lose sight of the lesion 42 that is outside the viewing angle. If the doctor 12 loses sight of the lesion 42, there is a risk that the doctor 12 will forget to perform medical treatment (for example, differentiation and / or resection, etc.) for the lesion 42 that has been lost. In order to prevent such a situation from occurring, it is important to make the doctor 12 aware of the presence of the lesion 42 that is outside the viewing angle.

[0087] In view of such circumstances, in this embodiment, as an example shown in Fig. 4, medical support processing is performed by a processor 82 of the medical support device 24. Note that Fig. 4 is a block diagram showing an example of the main functions of the processor 82 included in the medical support device 24 and an example of information stored in a storage 86.

[0088] A medical support program 90 is stored in the storage 86. The medical support program 90 is an example of a "program" according to the present disclosure. The processor 82 reads the medical support program 90 from the storage 86 and executes the read medical support program 90 on the memory 84 to perform medical support processing. The medical support processing is realized by the processor 82 operating as a recognition unit 82A and a control unit 82B in accordance with the medical support program 90 executed on the memory 84.

[0089] The storage 86 stores a recognition model 92. As will be described in detail later, the recognition model 92 is used by the recognition unit 82A.

[0090] Fig. 5 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B. As shown in Fig. 5, the recognition unit 82A and the control unit 82B acquire, from the camera 52, each of a plurality of frames 40 in time series included in the endoscopic moving image 39 generated by the camera 52 capturing an image of the inside of the large intestine 28 according to an imaging frame rate (e.g., several tens of frames / second), in a frame-by-frame manner in time series. In the example shown in Fig. 5, a mode in which a plurality of frames 40 showing a lesion 42 are acquired by the recognition unit 82A is shown. Moreover, the frames 40 acquired from the camera 52 by the recognition unit 82A are acquired by the control unit 82B in synchronization with the recognition unit 82A.

[0091] The control unit 82B outputs an endoscopic moving image 39 including a plurality of frames 40 arranged in a time series to the display device 18. For example, the control unit 82B displays the endoscopic moving image 39 as a live view image in the first display area 36. That is, every time the control unit 82B acquires a frame 40 from the camera 52, the control unit 82B displays the acquired frame 40 in order in the first display area 36 according to a display frame rate (for example, several tens of frames per second). The control unit 82B also displays medical information 44 in the second display area 38. For example, the control unit 82B also updates the display content of the second display area 38 (for example, the medical information 44) in accordance with the display content of the first display area 36.

[0092] The recognition unit 82A uses the endoscopic video 39 acquired from the camera 52 to recognize the lesion 42 shown in the endoscopic video 39. That is, the recognition unit 82A recognizes the lesion 42 shown in the frame 40 by sequentially performing a recognition process 96 on each of a plurality of frames 40 in time series included in the endoscopic video 39 acquired from the camera 52. For example, the recognition unit 82A recognizes the presence or absence of the lesion 42, the size of the image area showing the lesion 42 (hereinafter also referred to as the "lesion image area"), the position of the lesion image area within the frame 40, and the characteristics of the lesion 42 (for example, the malignancy of the lesion 42, the site where the lesion 42 exists, the type of the lesion 42, the form of the lesion 42 (for example, the appearance) of the lesion 42, the state of the boundary between the lesion 42 and the periphery of the lesion 42, and the state of mucus adhesion of the lesion 42, etc.).

[0093] The recognition process 96 is an example of "object recognition process using an image" according to the present disclosure. The recognition process 96 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired. The recognition process 96 is a process for recognizing a lesion 42 based on the frame 40 by using AI. Here, the recognition process 96 is a process using a recognition model 92. The recognition model 92 is a trained model for object recognition by AI using a bounding box method.

[0094] The recognition model 92 is optimized by performing machine learning on the neural network using first training data. The first training data is a data set including a plurality of data (i.e., data for a plurality of frames) in which the first example data and the first answer data are associated with each other.

[0095] The first example data is an image assuming the frame 40. A first example of an image assuming the frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. A second example of an image assuming the frame 40 is a virtually created image. The first correct answer data is correct answer data (i.e., annotation) for the first example data. Here, as an example of the first correct answer data, annotations are used that specify the geometric characteristics of the lesion 42 shown in the image used as the first example data, the malignancy of the lesion 42, the location where the lesion 42 exists, the type of the lesion 42, the form of the lesion 42, the state of the boundary between the lesion 42 and the periphery of the lesion 42, and the state of adhesion of mucus to the lesion 42.

[0096] The recognition unit 82A acquires a frame 40 from the camera 52, and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 recognizes a lesion 42 appearing in the input frame 40, and outputs a recognition result 98.

[0097] The recognition result 98 includes lesion presence / absence information 98A. The lesion presence / absence information 98A is information indicating whether or not a lesion 42 is captured in the frame 40 input to the recognition model 92. Furthermore, when a lesion 42 is captured in the frame 40 input to the recognition model 92, the recognition result 98 includes geometric property information 98B, a lesion position map 98C, lesion feature information 98D, and the like.

[0098] The geometric property information 98B is information (e.g., coordinates) capable of identifying the size, shape, and position of the lesion 42 within the frame 40. The lesion position map 98C is a map capable of identifying the position of the lesion 42 within the frame 40. The lesion characteristic information 98D is information capable of identifying the characteristics of the lesion 42 appearing in the frame 40 that is input to the recognition model 92. Here, the characteristics of the lesion 42 refer to the malignancy of the lesion 42, the location where the lesion 42 is present, the type of the lesion 42, the form of the lesion 42, the state of the boundary between the lesion 42 and its periphery, the state of mucus adhesion to the lesion 42, and the like.

[0099] The geometric characteristics of the lesion location map 98C (e.g., the shape and size of the outer contour) correspond to the geometric characteristics of the frame 40 (e.g., the shape and size of the outer contour). The lesion location map 98C includes a bounding box BB. The bounding box BB is a rectangular frame (e.g., a rectangular frame circumscribing the lesion image area) that can identify the position recognized by the recognition model 92 as the position within the frame 40 of the lesion 42 captured in the frame 40. In the example shown in FIG. 5, the bounding box BB is superimposed and displayed in the first display region 36. The display of the bounding box BB is updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display region 36.

[0100] Here, an example in which the bounding box BB is displayed is given, but this is merely an example, and the bounding box BB does not have to be displayed. Also, the bounding box BB may be displayed or hidden depending on various conditions. For example, the bounding box BB may be displayed or hidden depending on an instruction received by the reception device 64, or the bounding box BB may be displayed or hidden depending on the content of the process (e.g., the content of the medical support process) performed by the endoscope device 10. Also, the bounding box BB is merely an example, and an identifier (e.g., a code) may be displayed in place of the bounding box BB. In this case, for example, the control unit 82B may specify the position of the lesion 42 in the frame 40 from the geometric characteristic information 98B, and display the identifier in place of the bounding box BB near the specified position.

[0101] The lesion location map 98C may be displayed in the second display area 38 as a part of the medical information 44, or may be displayed in an area other than the second display area 38 in the screen 35. The lesion location map 98C displayed on the screen 35 is updated according to the display frame rate applied to the first display area 36. The display of the lesion location map 98C is updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display area 36. With this configuration, the doctor 12 can grasp the general position of the lesion 42 in the endoscopic moving image 39 displayed in the first display area 36 by referring to the lesion location map 98C while observing the endoscopic moving image 39 displayed in the first display area 36. The display and non-display of the lesion location map 98C may be switched in the same manner as the display and non-display of the bounding box BB in the first display area 36.

[0102] The control unit 82B displays the lesion characteristic information 98D on the screen 35. In this case, for example, the lesion characteristic information 98D is displayed in the second display area 38 as a part of the medical information 44.

[0103] 6 is a conceptual diagram showing an example of the display content of the first display region 36 when the lesion 42 is within the angle of view, and an example of the display content of the first display region 36 when the lesion 42 is out of the angle of view. As shown in FIG. 6, the region including the frame 40 and the periphery of the frame 40 is divided into first to fourth quadrants by a virtual cross line VL that is not displayed on the screen 35. The cross line VL is formed by a line in the horizontal direction when viewed from the front with respect to the frame 40, and a line in the vertical direction when viewed from the front with respect to the frame 40. The center of the cross line VL (i.e., the intersection of the line in the horizontal direction when viewed from the front with respect to the frame 40 and the line in the vertical direction when viewed from the front with respect to the frame 40) is located at the center of the frame 40. The first to fourth quadrants are set in order counterclockwise around the center of the frame 40 as an axis. Note that, although an example is given here in which the area including frame 40 and the outer periphery of frame 40 is divided into first to fourth quadrants, this is merely one example, and the area including frame 40 and the outer periphery of frame 40 may be divided into three or fewer areas, or may be divided into five or more areas.

[0104] When the control unit 82B acquires the frame 40 from the camera 52, it displays the acquired frame 40 in the first display area 36. Here, when the lesion 42 is within the angle of view, the frame 40 in which the lesion 42 is captured is displayed in the first display area 36. In the example shown in FIG. 6, the lesion 42 is captured in the third quadrant of the frame 40. On the other hand, when the lesion 42 is outside the angle of view, the frame 40 in which the lesion 42 is not captured is displayed in the first display area 36. In the example shown in FIG. 6, since the lesion 42 is outside the angle of view toward the first quadrant, the lesion 42 is not captured in the frame 40 displayed in the first display area 36. Therefore, the doctor 12 cannot visually recognize the presence of the lesion 42 that is outside the angle of view toward the first quadrant.

[0105] Therefore, as shown in FIG. 7 as an example, the control unit 82B uses the recognition result 98 and the like to identify the location of the lesion 42 (hereinafter also referred to as the "lesion location") regardless of whether it is captured in the frame 40 or not, and displays the identification result on the screen 35. To achieve this, the control unit 82B acquires feature location coordinates 102 from the frame 40 acquired from the camera 52 every time the control unit 82B acquires a frame 40 from the camera 52. The feature location coordinates 102 are coordinates capable of identifying a feature location in the frame 40. The feature location in the frame 40 refers to, for example, an edge 100 of a high frequency component region having a feature shape (for example, an edge of a feature curve). For example, the high frequency component region having a feature shape is extracted by using a frequency component filter.

[0106] The memory 84 is provided with a first storage area 84A, and the control unit 82B stores the acquired feature position coordinates 102 in the first storage area 84A in a FIFO manner every time the control unit 82B acquires feature position coordinates 102 from a frame 40. As a result, the first storage area 84A stores a plurality of feature position coordinates 102 acquired from each of the plurality of frames 40 in a time series manner. Note that, for convenience of explanation, an example in which one feature position coordinate 102 is acquired from each frame 40 and stored in the first storage area 84A in a FIFO manner is given here, but this is merely an example, and a plurality of feature position coordinates 102 may be acquired from each frame 40 and stored in the first storage area 84A in a FIFO manner. In this case, for example, the feature position coordinates 102 may be acquired from each of a plurality of high frequency component regions having characteristic shapes, and the feature position coordinates 102 may be stored in the first storage area 84A in a FIFO manner for each high frequency component region having a characteristic shape.

[0107] The control unit 82B calculates an optical flow 104 (i.e., a motion vector between frames 40 along the time series) based on a plurality of feature position coordinates 102 stored in time series in the first storage area 84A. For example, the calculation of the optical flow 104 is realized by a gradient method. Note that the gradient method is merely an example, and the calculation of the optical flow 104 may be realized by a block matching method or the like. Also, here, an example is given in which the edge 100 is used to calculate the optical flow 104, but this is merely an example, and the calculation of the optical flow 104 may use a location that can be tracked between frames 40 along the time series, such as the center of gravity of a high-frequency component region with a characteristic shape.

[0108] The memory 84 is provided with a second storage area 84B, and the control unit 82B calculates an optical flow 104 for each frame 40 and stores it in the second storage area 84B in a FIFO manner. As a result, a plurality of optical flows 104 obtained by calculation for each frame 40 are stored in chronological order in the second storage area 84B.

[0109] The control unit 82B identifies the lesion position based on the recognition result 98 obtained by the recognition unit 82A performing the recognition process 96 (see FIG. 5) on the frame 40 acquired from the camera 52 and one or more optical flows 104 (e.g., the latest optical flow 104 and / or a plurality of optical flows 104 in a time series including the latest optical flow 104) stored in the second storage area 84B. For example, the control unit 82B identifies the lesion position of the lesion 42 captured in the frame 40 from the geometric characteristic information 98B included in the recognition result 98 acquired from the recognition unit 82A. Then, the control unit 82B identifies the lesion position of the lesion 42 not captured in the frame 40 (lesion 42 deviating from the angle of view to the first quadrant side in the example shown in FIG. 7) based on the lesion position of the lesion 42 captured in the frame 40 and one or more optical flows 104 stored in the second storage area 84B. The control unit 82B identifies the lesion position by acquiring two-dimensional coordinates by the control unit 82B. Here, the two-dimensional coordinates acquired by the control unit 82B refer to two-dimensional coordinates with the center of the frame 40 as the origin, and are capable of identifying the lesion position.

[0110] The control unit 82B generates and outputs screen information 105. The screen information 105 is information indicating the screen 35, and is information used to generate the screen 35. The frame 40, medical information 44, and visual assist mark 106 are displayed on the screen 35. Here, the screen information 105 is an example of "screen generation information" according to the present disclosure, and the visual assist mark 106 is an example of "presence position information" and "out-of-view position information" according to the present disclosure.

[0111] In the first display area 36, ​​a frame 40 included in the screen information 105 is displayed. In the second display area 38, medical information 44 included in the screen information 105 is displayed.

[0112] The visual assist mark 106 is displayed on the screen 35 when the lesion 42 is out of the angle of view. The visual assist mark 106 is information that allows the position of the lesion to be identified outside the frame 40 displayed in the first display area 36 when the lesion 42 is out of the angle of view. In the example shown in FIG. 7, as an example of the visual assist mark 106, a mark that is displayed on the screen 35 and assists in visually identifying the position of the lesion outside the frame 40 displayed in the first display area 36 is shown. The shape of the visual assist mark 106 is an arc shape. The visual assist mark 106 is displayed on the screen 35 in quadrant units. The visual assist mark 106 is curved along the outer edge of the first display area 36 in quadrant units. In the example shown in FIG. 7, the lesion 42 is out of the angle of view toward the first quadrant, so that the visual assist mark 106 is displayed in the first quadrant on the screen 35, outside the first display area 36 in which the frame 40 in which the lesion 42 is not shown is displayed.

[0113] Fig. 8 is a conceptual diagram showing an example of the display content when the lesion position out of the frame (i.e., the lesion position for the lesion 42 not shown in the frame 40) is changed. As shown in Fig. 8, when the lesion position out of the frame moves outside the angle of view, the control unit 82B changes the display position of the visual assist mark 106 relative to the frame 40 within the screen 35 by the above-mentioned method (i.e., the method using one or more optical flows 104).

[0114] The display position of the visual assist mark 106 relative to the frame 40 on the screen 35 is changed according to a change in the positional relationship between the camera 52 and the lesion 42. The change in the positional relationship between the camera 52 and the lesion 42 is identified by the controller 82B from one or more optical flows 104 stored in the second storage area 84B.

[0115] Controller 82B identifies a change in the positional relationship between camera 52 and lesion 42 based on one or more optical flows 104 stored in second storage area 84B, and changes the display position of visual assist mark 106 relative to frame 40 on screen 35 in accordance with the identified change. The change in the positional relationship between camera 52 and lesion 42 is brought about by the operation of camera 52 by doctor 12 and / or bodily movement within large intestine 28. Therefore, the display position of visual assist mark 106 is changed in accordance with the change in the positional relationship between camera 52 and lesion 42, thereby tracking the operation of camera 52 by doctor 12 and / or bodily movement within large intestine 28.

[0116] 8 shows an example in which the lesion position moves from the first quadrant to the second quadrant outside the angle of view due to the operation of the camera 52 by the doctor 12 and / or body movement within the large intestine 28. When the lesion position moves from the first quadrant to the second quadrant outside the angle of view, the control unit 82B identifies a path along which the lesion position moves from the first quadrant to the second quadrant outside the angle of view from the optical flow 104. Then, the control unit 82B changes the display position of the visual assist mark 106 on the screen 35 from the first quadrant to the second quadrant outside the first display area 36 in which the frame 40 not showing the lesion 42 is displayed, according to the identified path.

[0117] Fig. 9 is a conceptual diagram showing an example of the display mode of visual assist mark 106 when the distance from the center of frame 40 to lesion 42 changes. As shown in Fig. 9, when the distance from the center of frame 40 to lesion 42 changes, the display mode of visual assist mark 106 displayed on screen 35 changes according to the distance from the center of frame 40 to lesion 42.

[0118] In the example shown in FIG. 9, the thickness of the visual assist mark 106 displayed on the screen 35 is changed according to the distance from the center of the frame 40 to the lesion 42. The farther the lesion position is from the center of the frame 40, the thicker the visual assist mark 106 displayed on the screen 35 is made within a certain range. In this case, for example, the control unit 82B changes the display mode (here, thickness, as an example) of the visual assist mark 106 according to the distance from the center of the frame 40 to the lesion 42. The distance from the center of the frame 40 to the lesion 42 is identified based on one or more optical flows 104 stored in the second storage area 84B. Here, the distance from the center of the frame 40 to the lesion 42 is an example of a "feature amount" and an "amount of change" according to the present disclosure.

[0119] Also, although an example in which the thickness of the visual assist mark 106 is changed has been given here as an example of the display mode of the visual assist mark 106, the display mode of the visual assist mark 106 is not limited to this. For example, the display mode of the visual assist mark 106 may include the size of at least a portion of the visual assist mark 106, the shape of at least a portion of the visual assist mark 106, the color of at least a portion of the visual assist mark 106, the brightness of at least a portion of the visual assist mark 106, the transparency of at least a portion of the visual assist mark 106, the pattern of at least a portion of the visual assist mark 106, and / or the shape of at least a portion of the edge of the visual assist mark 106.

[0120] Even if the display mode of the visual assist mark 106 is a display mode other than the thickness of the visual assist mark 106, the display mode of the visual assist mark 106 may be changed so that the visual assist mark 106 becomes more noticeable as the distance from the center of the frame 40 to the lesion 42 becomes longer. Conversely, the display mode of the visual assist mark 106 may be changed so that the visual assist mark 106 becomes more noticeable as the distance from the center of the frame 40 to the lesion 42 becomes shorter. Note that the display mode in which the visual assist mark 106 becomes more noticeable as the distance from the center of the frame 40 to the lesion 42 becomes longer and the display mode in which the visual assist mark 106 becomes more noticeable as the distance from the center of the frame 40 to the lesion 42 becomes shorter may be switched according to various conditions (e.g., instructions received by the reception device 64).

[0121] Next, the operation of the portion of the endoscope device 10 according to the present disclosure will be described with reference to Fig. 10. The flow of the medical support processing shown in Fig. 10 is an example of a "medical support method" according to the present disclosure. The medical support processing shown in Fig. 10 is executed by the processor 82 when a condition for starting the medical support processing is satisfied. One example of the condition for starting the medical support processing is a condition that an instruction to start the medical support processing has been given to the endoscope device 10 (for example, a condition that an instruction to start the medical support processing has been accepted by the acceptance device 64).

[0122] 10, first, in step ST10, the recognition unit 82A determines whether or not one frame of images has been captured by the camera 52 inside the large intestine 28. If one frame of images has not been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is negative, and the medical support process proceeds to step ST28. If one frame of images has been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0123] In step ST12, the recognition unit 82A and the control unit 82B acquire a frame 40 obtained by imaging the large intestine 28 with the camera 52. Then, the control unit 82B displays the frame 40 in the first display area 36. Here, if the frame 40 obtained by imaging the previous frame is displayed in the first display area 36, ​​the control unit 82B updates the frame 40 displayed in the first display area 36 to the latest frame 40. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0124] In step ST14, the control unit 82B acquires the feature position coordinates 102 from the frame 40 acquired in step ST12. Then, the control unit 82B stores the feature position coordinates 102 acquired from the frame 40 in the first storage area 84A in a FIFO manner. After the process of step ST14 is executed, the medical support process proceeds to step ST16. Note that, for convenience of explanation, the following description will be given on the assumption that a plurality of feature position coordinates 102 are stored in chronological order in the first storage area 84A.

[0125] In step ST16, the recognition unit 82A executes recognition processing 96 on the frame 40 acquired in step ST12. After the processing of step ST14 is executed, the medical support processing proceeds to step ST18.

[0126] In step ST18, the control unit 82B determines whether or not the lesion 42 is captured in the frame 40 acquired in step ST12 (i.e., whether or not the lesion 42 has been recognized by the recognition unit 82A) based on the recognition result 98 obtained by performing the recognition process 96 in step ST14. If the lesion 42 is not captured in the frame 40 acquired in step ST12 in step ST18, the determination is negative, and the medical support process proceeds to step ST22. If the lesion 42 is captured in the frame 40 acquired in step ST12 in step ST16, the determination is positive, and the medical support process proceeds to step ST20.

[0127] In step ST20, the control unit 82B superimposes and displays a bounding box BB on the frame 40 in the first display region 36 based on the recognition result 98 obtained by performing the recognition process 96 in step ST14. As a result, the bounding box BB is superimposed and displayed at the display position of the lesion image region in the first display region 36. Furthermore, if a bounding box BB is displayed in the first display region 36 by performing the previous process of step ST20, the control unit 82B updates the bounding box BB displayed in the first display region 36 based on the recognition result 98 obtained by performing the recognition process 96 in step ST14. After the process of step ST20 is performed, the medical support process proceeds to step ST10.

[0128] In step ST22, the control unit 82B calculates an optical flow 104 based on the multiple feature position coordinates 102 stored in time series in the first storage area 84A. Then, the control unit 82B stores the calculated optical flow 104 in the second storage area 84B in a FIFO manner. After the process of step ST22 is executed, the medical support process proceeds to step ST24.

[0129] In step ST24, the control unit 82B identifies the lesion position based on the recognition result 98 obtained by performing the recognition process 96 in step ST14 and one or more optical flows 104 stored in time series in the second storage area 84B. After the process of step ST24 is performed, the medical support process proceeds to step ST26.

[0130] In step ST26, control unit 82B displays visual assist mark 106 in a display mode according to the positional relationship between frame 40 and the lesion position identified in step ST24 (for example, the distance from the center of frame 40 to lesion 42) in the quadrant corresponding to the lesion position identified in step ST24, among the first to fourth quadrants in screen 35. After the processing of step ST26 is executed, the medical support processing proceeds to step ST28.

[0131] In step ST28, the control unit 82B determines whether or not a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process is given to the endoscope device 10 (for example, that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0132] In step ST28, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST28, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0133] As described above, in this embodiment, the screen information 105 showing the screen 35 is generated by the control unit 82B and output to the display device 18. The screen 35 shown by the screen information 105 includes the frame 40 obtained by imaging the large intestine 28 by the camera 52, and the visual assist mark 106. The frame 40 is displayed in the first display area 36 (see FIG. 7). The visual assist mark 106 is a mark that can identify the position of a lesion that is out of the angle of view outside the first display area 36 in which the frame 40 is displayed (see FIG. 7). The display position of the visual assist mark 106 relative to the frame 40 in the screen 35 is changed according to a change in the positional relationship between the camera 52 and the lesion 42 (see FIGS. 8 and 9).

[0134] This allows the doctor 12 to grasp the lesion position outside the first display area 36 in the screen 35, in which the frame 40 obtained by imaging the large intestine 28 with the camera 52 is displayed, even if the positional relationship between the camera 52 and the lesion 42 changes. As a result, even if the lesion 42 falls outside the angle of view, the doctor 12 can visually grasp that the lesion 42 exists outside the angle of view by visually checking the positional relationship between the visual assist mark 106 and the frame 40 displayed on the screen 35, and can also visually estimate the lesion position outside the angle of view. In addition, since the visual assist mark 106 is displayed outside the first display area 36 in which the frame 40 is displayed on the screen 35, the visibility of the frame 40 displayed in the first display area 36 can be maintained good, compared to a case in which information indicating that the lesion position exists outside the angle of view is superimposed on the frame 40 in the first display area 36.

[0135] Furthermore, in this embodiment, a change in the positional relationship between the camera 52 and the lesion 42 is brought about by the operation of the camera 52 by the doctor 12 and / or a body movement within the large intestine 28. Therefore, even if the positional relationship between the camera 52 and the lesion 42 changes due to the operation of the camera 52 by the doctor 12 and / or a body movement within the large intestine 28, the doctor 12 can visually grasp the position of the lesion outside the first display area 36 on the screen 35 in which the frame 40 obtained by imaging the large intestine 28 by the camera 52 is displayed.

[0136] Furthermore, in this embodiment, the display position of the visual assist mark 106 is changed according to a change in the positional relationship between the camera 52 and the lesion 42, thereby tracking the operation of the camera 52 by the doctor 12 and / or the body movement within the large intestine 28 (see FIGS. 8 and 9). Therefore, even if the positional relationship between the camera 52 and the lesion 42 changes due to the operation of the camera 52 by the doctor 12 and / or the body movement within the large intestine 28, the doctor 12 can visually track the display position of the visual assist mark 106.

[0137] Furthermore, in this embodiment, the display mode of the visual assist mark 106 is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42 (here, as an example, the distance between the center of the frame 40 and the lesion position) (see FIG. 9). Therefore, compared to a case where the display mode of the visual assist mark 106 is always constant regardless of the characteristics of the change in the positional relationship between the camera 52 and the lesion 42, it is possible for the doctor 12 to visually grasp the behavior of the characteristics of the change in the positional relationship between the camera 52 and the lesion 42.

[0138] In this embodiment, the visual assist mark 106 is displayed on the screen 35 on the condition that the lesion 42 is out of the field of view. This allows the doctor 12 to know at an appropriate time that the lesion 42 is out of the field of view.

[0139] Furthermore, in this embodiment, when the lesion 42 is out of the angle of view, a visual assist mark 106 is displayed on the screen 35, and the display mode of the visual assist mark 106 is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42 (here, as an example, the distance between the center of the frame 40 and the lesion position) (see FIG. 9). Therefore, even if the lesion 42 is out of the angle of view, the doctor 12 can visually grasp the behavior of the characteristics of the change in the positional relationship between the camera 52 and the lesion 42.

[0140] In this embodiment, the recognition process 96 is a process for recognizing the lesion 42 based on the frame 40 by using AI. Therefore, the endoscope device 10 can recognize the lesion 42 more quickly and accurately than when the lesion 42 is recognized solely by the intuition and / or experience of the doctor 12 or the like.

[0141] [Variations] In the above embodiment, an example in which the medical support process shown in Fig. 10 is executed by the processor 82 has been given, but this is merely an example, and for example, the medical support process shown in Fig. 11 may be executed by the processor 82. The medical support process shown in Fig. 11 differs from the medical support process shown in Fig. 10 in that it includes the process of step ST100. The process of step ST100 is provided between the process of step ST24 and the process of step ST26.

[0142] In the medical support process shown in FIG. 11, in step ST100, the control unit 82B determines whether the time when the lesion 42 is out of the frame (hereinafter also referred to as the "frame-out time") is equal to or longer than a first preset time (for example, a number of seconds specified in advance within a range of several seconds to several tens of seconds). That is, in step ST100, it is determined whether the time when the lesion 42 is out of the angle of view is equal to or longer than the first preset time. Here, the first preset time is an example of the "preset time" according to the present disclosure.

[0143] In step ST100, if the frame-out time is less than the first preset time, the determination is negative, and the medical support process proceeds to step ST28. In step ST100, if the frame-out time is equal to or greater than the first preset time, the determination is positive, and the medical support process proceeds to step ST26.

[0144] In this manner, if the frame-out time is less than the first preset time, the visual assist mark 106 is not displayed on the screen 35, and if the frame-out time is equal to or greater than the first preset time, the process of step ST28 is executed, thereby displaying the visual assist mark 106 on the screen 35. As a result, the visual assist mark 106 is displayed on the screen 35 when there is a high possibility that the frame-out lesion 42 has been forgotten by the doctor 12, and this makes it possible to prevent the doctor 12 from forgetting about the existence of the lesion 42 that is out of the field of view.

[0145] The first preset time is preferably a time during which the lesion 42 is out of the field of view, and is a time during which the doctor 12 forgets about the existence of the lesion 42 that is out of the field of view, and is derived by a statistical method and / or a computer simulation using a plurality of data collected in advance through a plurality of endoscopic examinations. The first preset time may also be a time determined according to an instruction received by the reception device 64 or the like.

[0146] In the medical support process shown in Fig. 11, the process of step ST26 is executed when the condition that the frame-out time is equal to or longer than the first preset time is satisfied in step ST100, whereas in the medical support process shown in Fig. 12, the process of step ST26 is executed when a plurality of mark display conditions are satisfied. The medical support process shown in Fig. 12 differs from the medical support process shown in Fig. 11 in that the process of step ST200 is applied instead of the process of step ST100.

[0147] In the medical support process shown in Fig. 12, in step ST200, control unit 82B determines whether or not both of a plurality of mark display conditions are satisfied. The plurality of mark display conditions are a first mark display condition and a second mark display condition. The first mark display condition is a condition stipulated in step ST100 included in the medical support process shown in Fig. 11, that is, a condition that the frame-out time is equal to or longer than a first preset time. The second mark display condition is a condition that the degree of change in the positional relationship between camera 52 and lesion 42 exceeds a preset degree.

[0148] Here, a first example of the degree of change in the positional relationship between the camera 52 and the lesion 42 is the distance from the center of the frame 40 to the lesion 42, i.e., the amount of change in the positional relationship between the camera 52 and the lesion 42. A second example of the degree of change in the positional relationship between the camera 52 and the lesion 42 is the speed of change in the positional relationship between the camera 52 and the lesion 42. A third example of the degree of change in the positional relationship between the camera 52 and the lesion 42 is a combination of the amount of change in the positional relationship between the camera 52 and the lesion 42 and the speed of change in the positional relationship between the camera 52 and the lesion 42. The combination of the amount of change in the positional relationship between the camera 52 and the lesion 42 and the speed of change in the positional relationship between the camera 52 and the lesion 42 may be expressed by a score that combines a score indicating the amount of change in the positional relationship between the camera 52 and the lesion 42 and a score indicating the speed of change in the positional relationship between the camera 52 and the lesion 42. An example of an integrated score is the sum of a score indicating the amount of change in the positional relationship between camera 52 and lesion 42 and a score indicating the rate of change in the positional relationship between camera 52 and lesion 42, or the product of a score indicating the amount of change in the positional relationship between camera 52 and lesion 42 and a score indicating the rate of change in the positional relationship between camera 52 and lesion 42.

[0149] The preset degree is the degree of change in the positional relationship between the camera 52 and the lesion 42, and is preferably the degree to which the doctor 12 forgets the presence of the lesion 42 that is out of the field of view, which is derived by a statistical method and / or computer simulation using multiple data collected in advance through multiple endoscopic examinations. The preset degree may also be a degree determined according to an instruction received by the reception device 64, etc.

[0150] In step ST200, if both of the mark display conditions are not satisfied, the determination is negative and the medical support process proceeds to step ST28. In step ST200, if both of the mark display conditions are satisfied, the determination is positive and the medical support process proceeds to step ST26.

[0151] In this manner, if both of the multiple mark display conditions are not satisfied, the visual assist mark 106 is not displayed on the screen 35, and if both of the multiple mark display conditions are satisfied, the processing of step ST26 is executed, thereby displaying the visual assist mark 106 on the screen 35. As a result, the visual assist mark 106 is displayed on the screen 35 when there is a high possibility that the doctor 12 has forgotten about the existence of the lesion 42 that is out of the field of view, thereby preventing the occurrence of a situation in which the doctor 12 forgets about the existence of the lesion 42 that is out of the field of view.

[0152] In step ST200 shown in FIG. 12, it is determined whether or not both of the multiple mark display conditions are satisfied; however, the present disclosure is not limited to this, and in step ST200 shown in FIG. 12, it may be determined only whether or not the second mark display condition is satisfied.

[0153] Incidentally, a lesion 42 that appears in the frame 40 only for an instant and then goes out of frame is likely to be overlooked by the doctor 12. In order to reduce the possibility of overlooking such a lesion 42, a medical support process shown in Fig. 13 is executed by the processor 82. The medical support process shown in Fig. 13 differs from the medical support process shown in Fig. 12 in that it includes a process of step ST300 and a process of step ST302.

[0154] If the determination is positive in step ST200 included in the medical support processing shown in Fig. 13, the medical support processing proceeds to step ST300. In step ST300, the control unit 82B determines whether the time when the lesion 42 comes into the frame (hereinafter also referred to as "frame-in time") is less than a second preset time (e.g., several seconds specified in advance). That is, in step ST300, it is determined whether the time during which the lesion 42 is within the angle of view is less than the second preset time. Here, the frame-in time is an example of the "time within the angle of view" according to the present disclosure, and the second preset time is an example of "fixed" according to the present disclosure.

[0155] In step ST300, if the frame-in time is equal to or longer than the second preset time, the determination is negative, and the medical support process proceeds to step ST26. After the process of step ST26 is executed, the medical support process proceeds to step ST302. In step ST300, if the frame-in time is shorter than the second preset time, the determination is positive, and the medical support process proceeds to step ST302.

[0156] In step ST302, control unit 82B displays visual assist mark 106 in a display mode according to the positional relationship between frame 40 and the lesion position identified in step ST24 (e.g., the distance from the center of frame 40 to lesion 42) in the quadrant corresponding to the lesion position identified in step ST24 (i.e., the quadrant in which the lesion position exists) among the first to fourth quadrants on screen 35. Here, control unit 82B displays visual assist mark 106 more emphasized than when the frame-in time is equal to or longer than the second preset time. After the processing of step ST302 is executed, the medical support processing proceeds to step ST28.

[0157] In this way, by performing the process of step ST300 and the process of step ST302, it is possible to make it easier for the doctor 12 to visually understand the presence of the lesion 42 that was only captured in the frame 40 for an instant and then went out of frame. As a result, it is possible to prevent the doctor 12 from overlooking the lesion 42 that was only captured in the frame 40 for an instant and then went out of frame.

[0158] The second preset time is preferably a time derived by a statistical method and / or a computer simulation using a plurality of data collected in advance through a plurality of endoscopic examinations, as a lower limit of the time during which the doctor 12 can recognize the lesion 42 that is temporarily in frame and the doctor 12 will not forget the existence of the lesion 42 even if the lesion 42 is out of frame. The second preset time may be a time determined according to an instruction received by the reception device 64 or the like.

[0159] Fig. 14 is a modified example of the medical support process shown in Fig. 13. The medical support process shown in Fig. 14 differs from the medical support process shown in Fig. 13 in that it has a process of step ST400 instead of the process of step ST302, and has a process of step ST402 instead of the process of step ST26.

[0160] In step ST400, control unit 82B displays visual assist mark 106 in a display mode according to the positional relationship between frame 40 and the lesion position identified in step ST24 (for example, the distance from the center of frame 40 to lesion 42) and the change characteristics in the quadrant corresponding to the lesion position identified in step ST24 (i.e., the quadrant in which the lesion position exists) among the first to fourth quadrants on screen 35. Control unit 82B also displays visual assist mark 106 more emphatically than when the frame-in time is equal to or longer than the second preset time.

[0161] In step ST402, control unit 82B displays visual assist mark 106 in the first to fourth quadrants on screen 35 that corresponds to the lesion position identified in step ST24 (i.e., the quadrant in which the lesion position is located) in a display mode that corresponds to the positional relationship between frame 40 and the lesion position identified in step ST24 (e.g., the distance from the center of frame 40 to lesion 42) and the change characteristics.

[0162] Here, the change feature refers to, for example, the feature of the change in the positional relationship between the camera 52 and the lesion 42. One example of the change feature is the speed of change in the positional relationship between the camera 52 and the lesion 42 and / or the direction of change in the positional relationship between the camera 52 and the lesion 42. The speed of change in the positional relationship between the camera 52 and the lesion 42 and the direction of change in the positional relationship between the camera 52 and the lesion 42 are identified by the control unit 82B from one or more optical flows 104.

[0163] One example of a display mode for specifying the rate of change in the positional relationship between camera 52 and lesion 42 is a display mode in which visual assist mark 106 flashes and the flashing time interval becomes shorter as the rate of change in the positional relationship between camera 52 and lesion 42 increases, or a display mode in which a numerical value indicating the rate of change in the positional relationship between camera 52 and lesion 42 is included in visual assist mark 106. Note that the display mode exemplified here is merely one example, and any display mode may be used as long as it is a display mode in which the rate of change in the positional relationship between camera 52 and lesion 42 can be specified.

[0164] An example of a display mode for identifying the direction of change in the positional relationship between camera 52 and lesion 42 is a display mode in which an arrow indicating the direction of change in the positional relationship between camera 52 and lesion 42 is used as the pattern of visual assist mark 106, or a display mode in which text or the like indicating the direction of change in the positional relationship between camera 52 and lesion 42 is included in visual assist mark 106. Note that the display mode illustrated here is merely an example, and any display mode may be used as long as it is a display mode in which the direction of change in the positional relationship between camera 52 and lesion 42 can be identified.

[0165] In this manner, by performing the processing of steps ST400 and ST402, doctor 12 can visually grasp the characteristics of the change in the positional relationship between camera 52 and lesion 42 (e.g., the distance from the center of frame 40 to lesion 42, the speed of change in the positional relationship between camera 52 and lesion 42, and the direction of change in the positional relationship between camera 52 and lesion 42).

[0166] In the medical support process shown in Fig. 14, an example has been given in which the visual assist mark 106 is highlighted when the condition that the frame-in time is less than the second preset time is satisfied in step ST300, but the present disclosure is not limited to this. For example, as shown in Fig. 15, the visual assist mark 106 may be highlighted when a plurality of highlighting conditions are satisfied. The medical support process shown in Fig. 15 differs from the medical support process shown in Fig. 14 in that it includes processing of step ST500 and processing of step ST502 between the processing of step ST200 and the processing of step ST28.

[0167] In the medical support process shown in FIG. 14, in step ST500, the control unit 82B determines whether or not both of a plurality of highlighting conditions are satisfied. The plurality of highlighting conditions are a first highlighting condition and a second highlighting condition. The first highlighting condition is a condition stipulated in step ST300 included in the medical support process shown in FIG. 14, that is, a condition that the frame-in time is shorter than a second preset time. The second highlighting condition is a condition that the frequency at which the lesion 42 repeatedly goes in and out of the frame within a unit time, that is, the frequency at which the lesion 42 goes in and out of the angle of view within a unit time (e.g., within one second) (hereinafter also referred to as "going in and out frequency") exceeds a preset frequency.

[0168] The predetermined frequency is the frequency at which the lesion 42 repeatedly goes in and out of frame within a unit time, and is preferably a value derived by a statistical method and / or computer simulation using a plurality of data collected in advance through a plurality of endoscopic examinations as a lower limit of the frequency at which the doctor 12 is likely to overlook the lesion 42. The predetermined frequency may also be a frequency determined according to an instruction received by the reception device 64 or the like.

[0169] In step ST500, if both of the highlighting conditions are not satisfied, the determination is negative and the medical support process proceeds to step ST402. In step ST500, if both of the highlighting conditions are satisfied, the determination is positive and the medical support process proceeds to step ST502.

[0170] In step ST502, control unit 82B displays visual assist mark 106 in a display mode according to the positional relationship between frame 40 and the lesion position identified in step ST24 (e.g., the distance from the center of frame 40 to lesion 42) and the change characteristics in the quadrant corresponding to the lesion position identified in step ST24 (i.e., the quadrant in which the lesion position exists) among the first to fourth quadrants on screen 35. Control unit 82B also displays visual assist mark 106 more emphasized than when both of the multiple highlighting conditions are not satisfied.

[0171] In this manner, if both of the multiple highlighting conditions are not satisfied, the visual assist mark 106 is not highlighted, and if both of the multiple highlighting conditions are satisfied, the process of step ST502 is executed to highlight the visual assist mark 106. This makes it possible to prevent the doctor 12 from overlooking a lesion 42 that frequently moves in and out of the field of view.

[0172] In step ST500 shown in FIG. 15, it is determined whether or not both of the multiple highlighting conditions are satisfied; however, the present disclosure is not limited to this, and in step ST500 shown in FIG. 15, it may be determined only whether or not the second highlighting condition is satisfied.

[0173] In the above embodiment, the distance from the center of the frame 40 to the lesion 42 (i.e., the amount of change in the positional relationship between the camera 52 and the lesion 42) is exemplified as an example of the characteristic of the change in the positional relationship between the camera 52 and the lesion 42, but this is merely one example. Examples of the characteristic of the change in the positional relationship between the camera 52 and the lesion 42 include the speed of the change in the positional relationship between the camera 52 and the lesion 42 and / or the direction of the change in the positional relationship between the camera 52 and the lesion 42. The characteristic of the change in the positional relationship between the camera 52 and the lesion 42 may be a combination of the speed of the change in the positional relationship between the camera 52 and the lesion 42 and the amount of change in the positional relationship between the camera 52 and the lesion 42. The characteristic of the change in the positional relationship between the camera 52 and the lesion 42 may be a combination of the direction of the change in the positional relationship between the camera 52 and the lesion 42 and the amount of change in the positional relationship between the camera 52 and the lesion 42. In addition, the characteristics of the change in the positional relationship between the camera 52 and the lesion 42 may be a combination of the speed of the change in the positional relationship between the camera 52 and the lesion 42, the direction of the change in the positional relationship between the camera 52 and the lesion 42, and the amount of change in the positional relationship between the camera 52 and the lesion 42.

[0174] In the above embodiment, an example was given of a display mode of the visual assist mark 106 in which the thickness of the visual assist mark 106 is changed, but the display mode of the visual assist mark 106 is not limited to this. For example, the display mode of the visual assist mark 106 may be whether or not the visual assist mark 106 is displayed, the display intensity of the visual assist mark 106, the display time of the visual assist mark 106, and / or the speed at which the display intensity of the visual assist mark 106 is changed, etc.

[0175] When the display mode of the visual assist mark 106 is whether or not to display the visual assist mark 106, for example, the visual assist mark 106 is switched between display and non-display depending on the characteristics of the change in the positional relationship between the camera 52 and the lesion 42. To explain in more detail, for example, when the speed of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined speed, the visual assist mark 106 is displayed, and when the speed of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined speed, the visual assist mark 106 is not displayed. Also, for example, when the amount of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined amount, the visual assist mark 106 is displayed, and when the amount of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined amount, the visual assist mark 106 is not displayed. Also, for example, when the direction of change in the positional relationship between the camera 52 and the lesion 42 is a predetermined direction, the visual assist mark 106 is displayed, and when the direction of change in the positional relationship between the camera 52 and the lesion 42 is other than the predetermined direction, the visual assist mark 106 is not displayed.

[0176] When the display mode of the visual assist mark 106 is the display intensity of the visual assist mark 106, for example, the display intensity of the visual assist mark 106 is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42. To explain in more detail, for example, when the rate of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined rate, the display intensity of the visual assist mark 106 is set to equal to or greater than the predetermined intensity, and when the rate of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined rate, the display intensity of the visual assist mark 106 is set to less than the predetermined intensity. Also, for example, when the amount of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined amount, the display intensity of the visual assist mark 106 is set to equal to or greater than the predetermined intensity, and when the amount of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined amount, the display intensity of the visual assist mark 106 is set to less than the predetermined intensity. Also, for example, when the direction of change in the positional relationship between the camera 52 and the lesion 42 is a predetermined direction, the display intensity of the visual assist mark 106 is set to greater than the predetermined intensity, and when the direction of change in the positional relationship between the camera 52 and the lesion 42 is other than the predetermined direction, the display intensity of the visual assist mark 106 is set to less than the predetermined intensity.

[0177] When the display mode of the visual assist mark 106 is the display time of the visual assist mark 106, for example, the display time of the visual assist mark 106 (for example, the time for which the visual assist mark 106 is continuously displayed) is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42. To explain in more detail, for example, when the rate of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined rate, the display time of the visual assist mark 106 is set to a certain time or more, and when the rate of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined rate, the display time of the visual assist mark 106 is set to less than the certain time. Also, for example, when the amount of change in the positional relationship between the camera 52 and the lesion 42 is equal to or greater than a predetermined amount, the display time of the visual assist mark 106 is set to a certain time or more, and when the amount of change in the positional relationship between the camera 52 and the lesion 42 is less than the predetermined amount, the display time of the visual assist mark 106 is set to less than the certain time. Also, for example, when the direction of change in the positional relationship between the camera 52 and the lesion 42 is a predetermined direction, the display time of the visual assist mark 106 is set to a certain period of time or more, and when the direction of change in the positional relationship between the camera 52 and the lesion 42 is other than the predetermined direction, the display time of the visual assist mark 106 is set to less than the certain period of time.

[0178] When the display mode of the visual assist mark 106 is the speed at which the display intensity of the visual assist mark 106 is changed, for example, the speed at which the display intensity of the visual assist mark 106 is changed is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42. To explain in more detail, for example, the faster the change in the positional relationship between the camera 52 and the lesion 42, the faster the speed at which the display intensity of the visual assist mark 106 is changed.

[0179] In this way, by changing the presence or absence of displaying the visual assist mark 106, the display intensity of the visual assist mark 106, the display time of the visual assist mark 106, and / or the speed at which the display intensity of the visual assist mark 106 is changed according to the characteristics of the change in the positional relationship between the camera 52 and the lesion 42, even if the lesion 42 is out of the field of view, the doctor 12 can visually recognize the characteristics of the change in the positional relationship between the camera 52 and the lesion 42.

[0180] In the above embodiment, an example was given in which the visual assist mark 106 is displayed on the screen 35 to allow the doctor 12 to visually grasp the lesion position of the lesion 42 not captured in the frame 40, but the present disclosure is not limited thereto. For example, as shown in FIG. 16, even when the lesion 42 is captured in the frame 40 (i.e., when the lesion 42 is within the angle of view), the visual assist mark 108 may be displayed on the screen 35 outside the first display area 36 in which the frame 40 in which the lesion 42 is captured is displayed. The visual assist mark 108 is information that can specify the location of the lesion 42 when the lesion 42 is within the angle of view, outside the first display area 36 in which the frame 40 in which the lesion 42 is captured is displayed. In the example shown in FIG. 16, the visual assist mark 108 is an example of "position information within the angle of view" according to the present disclosure.

[0181] Here, the visual assist mark 108 is illustrated, but this is merely one example. For example, text or code may be applied instead of the visual assist mark 108, and any information may be used as long as it is information that can identify the location of the lesion 42 when the lesion 42 is within the angle of view outside the first display area 36 in which the frame 40 showing the lesion 42 is displayed.

[0182] When the visual assist mark 108 is displayed outside the first display area 36 in which the frame 40 showing the lesion 42 is displayed, the bounding box BB may be hidden. In this way, the visibility of the frame 40 is not impaired by the presence of the bounding box BB, and it is possible to visually identify in which quadrant of the frame 40 the lesion 42 is shown.

[0183] Moreover, it is preferable that visual assist mark 108 is displayed in a manner that allows it to be distinguished from visual assist mark 106. In the example shown in Fig. 16, visual assist mark 106 is represented by a thin arc-shaped line, and is displayed in a manner that is less noticeable than visual assist mark 108. In this manner, doctor 12 can visually understand which lesions 42 are within the angle of view and which lesions 42 are outside the angle of view.

[0184] In the above embodiment, an example was given in which a single visual assist mark 106 is displayed on the screen 35 when a single lesion 42 falls outside the angle of view, but the present disclosure is not limited thereto. For example, as shown in Fig. 17, when multiple lesions 42 (two lesions 42 in the example shown in Fig. 17) fall outside the angle of view, visual assist marks 108 (two visual assist marks 108 in the example shown in Fig. 17) may be displayed on the screen 35 for the number of lesions 42. In this case, as in the above embodiment, the visual assist marks 108 are displayed in the quadrants corresponding to the positions of the respective lesions outside the first display area 36 in which the frame 40 is displayed, so that the quadrants in which the lesions 42 exist can be visually identified.

[0185] Furthermore, when multiple lesions 42 (two lesions 42 in the example shown in FIG. 17) are shown in frame 40, visual assist marks 106 (two visual assist marks 106 in the example shown in FIG. 17) may be displayed on screen 35 for the number of lesions 42 shown in frame 40. In this case, as in the example shown in FIG. 16, each visual assist mark 108 is displayed in the quadrant corresponding to each lesion position outside first display area 36 in which frame 40 is displayed, so that the quadrant in which each lesion 42 exists can be visually identified.

[0186] Also, in the example shown in Fig. 17, an example is given in which visual assist marks 108 are displayed on screen 35 for the number of lesions 42, but this is merely one example. For example, as shown in Fig. 18, when multiple lesions 42 that are outside the angle of view are present in the same quadrant (the first quadrant in the example shown in Fig. 18), one visual assist mark 108 is displayed in the same quadrant (the first quadrant in the example shown in Fig. 18) on screen 35, and a symbol or text indicating the number of lesions 42 (the text "x2" indicating two in the example shown in Fig. 18) is displayed.

[0187] Furthermore, when multiple lesions 42 that fall within the field of view are present in the same quadrant (the third quadrant in the example shown in FIG. 18), one visual assist mark 106 is displayed in the same quadrant (the third quadrant in the example shown in FIG. 18) on the screen 35, and a symbol or text indicating the number of lesions 42 (the text "×2" indicating two in the example shown in FIG. 18) is displayed.

[0188] In the above embodiment, the visual assist mark 108 is displayed on the screen 35 so that the doctor 12 can visually grasp the presence of the lesion 42 outside the angle of view, but the present disclosure is not limited thereto. For example, as shown in FIG. 19, instead of the visual assist mark 108, a code 107 may be displayed on the screen 35. The code 107 is an identifier uniquely determined for the lesion 42. Therefore, by displaying different codes 107 on the screen 35, it becomes possible to visually grasp that different lesions 42 exist outside the angle of view. In addition, the code 107 may be a code that can specify the characteristics of the lesion 42 (for example, the malignancy of the lesion 42, the site where the lesion 42 exists, the type of the lesion 42, the form of the lesion 42, the state of the boundary between the lesion 42 and the periphery of the lesion 42, and the state of adhesion of mucus to the lesion 42, etc.). In addition, the display mode of the code 107 (for example, color, brightness, thickness, and / or blinking pattern, etc.) may be changed according to the above-mentioned change characteristics.

[0189] In the example shown in FIG. 19, the visual assist mark 106 is displayed on the screen 35, but the code 107 may be displayed on the screen 35 instead of the visual assist mark 106.

[0190] 19 shows an example of a form in which the doctor 12 visually grasps the presence of a lesion 42 outside the angle of view by displaying the code 107 on the screen 35, but the present disclosure is not limited to this. For example, as shown in Fig. 20, when a lesion 42 outside the angle of view is present, an arrow 109 that can identify the presence of the lesion 42 outside the angle of view may be displayed on the screen 35 instead of the code 107.

[0191] For example, the direction of the arrow 109 is determined according to the positional relationship between the camera 52 and the lesion 42 outside the angle of view, and is changed according to the change in the positional relationship between the camera 52 and the lesion 42 outside the angle of view. For example, when the distance between the camera 52 and the lesion 42 outside the angle of view is less than a threshold, the tip of the arrow 109 is directed toward the first display area 36, ​​and when the distance between the camera 52 and the lesion 42 outside the angle of view is equal to or greater than the threshold, the tip of the arrow 109 is directed outside the first display area 36. In addition, the longer the distance between the camera 52 and the lesion 42 outside the angle of view, the longer the length of the arrow 109 may be within a certain range. In addition, the display mode (e.g., color, brightness, thickness, and / or blinking pattern, etc.) of the arrow 109 may be changed according to the above-mentioned change characteristics.

[0192] In the above embodiment, an example in which the visual assist mark 106 is displayed outside the first display area 36 has been given. However, for example, as shown in FIG. 21, the visual assist mark 106 may be displayed in the second display area 38 as part of the medical information 44. In this case, the lesion location map 98C may be displayed in the second display area 38 as part of the medical information 44. When the lesion location map 98C is displayed in the second display area 38, the visual assist mark 106 may be positioned with respect to the lesion location map 98C in the second display area 38 in the same manner as the visual assist mark 106 was positioned with respect to the frame 40 in the above embodiment. That is, the visual assist mark 106 may be displayed outside the lesion location map 98C. Also, instead of the visual assist mark 106, a code 107 or an arrow 109 may be displayed in the second display area 38. Also, the visual assist mark 108 may be displayed in the second display area 38 in the same manner as shown in FIG. 16 to FIG. 18, etc.

[0193] In the example shown in FIG. 21, the lesion location map 98C displayed in the second display area 38 is an example of a "medical image generated based on an image" according to the present disclosure, and the visual assist mark 106 displayed in the second display area 38 is an example of "existence location information" and "out-of-field-of-view location information" according to the present disclosure.

[0194] In the example shown in FIG. 21, an example of a form in which the lesion location map 98C is displayed in the second display area 38 has been given, but the frame 40 may be displayed in the second display area 38 instead of the lesion location map 98C. In this case, the visual assist mark 106 may be positioned relative to the frame 40 in the second display area 38 in the same manner as in the above embodiment. Also, instead of the lesion location map 98C, an image in which the frame 40 has been processed may be displayed in the second display area 38. Examples of the image in which the frame 40 has been processed include an image in which the image quality of the frame 40 has been reduced (e.g., an image in which the number of pixels and / or gradation has been reduced), or an image in which the image quality of the frame 40 has been improved (e.g., an image obtained by removing noise and the like from the frame 40 by a nonlinear filter using AI, or an image newly generated by AI based on the frame 40).

[0195] In this way, frame 40 or an image generated based on frame 40 is displayed in the second display area 38, and visual assist marks 106 and / or 108 are displayed in a manner similar to that of the above embodiment, thereby enabling the same effect as the above embodiment to be achieved.

[0196] 21 shows an example in which lesion location map 98C is displayed in second display area 38 as an example of an image generated based on frame 40, and visual assist marks 106 and / or 108 are displayed in a similar manner to the above embodiment, but this is merely one example. For example, an image generated based on frame 40 (e.g., lesion location map 98C, etc.) may be displayed in first display area 36, ​​and visual assist marks 106 and / or 108 may be displayed outside first display area 36 in a similar manner to the above embodiment.

[0197] In the above embodiment, an example in which the visual assist mark 106 is displayed in quadrant units has been given, but the present disclosure is not limited thereto. For example, as shown in FIG. 22, when the center of the frame 40 is set as the center of a virtual circle VC, the control unit 82B identifies a radial position (i.e., the position of the lesion 42 in the radial direction of the circle VC) corresponding to the position of the lesion 42 outside the angle of view based on one or more optical flows 104. The control unit 82B also identifies the size of the lesion 42 outside the angle of view based on the geometric characteristic information 98B included in the recognition result 98. Then, the control unit 82B displays the visual assist mark 106 of a size corresponding to the size of the lesion 42 at a radial position outside the first display area 36 in which the frame 40 is displayed within the screen 35. Even when the lesion 42 outside the angle of view moves, the control unit 82B identifies the radial position and the size of the lesion 42 in a similar manner, and displays the visual assist mark 106 of a size corresponding to the size of the lesion 42 at the identified radial position. In this case, too, the display position of the visual assist mark 106 is outside the first display area 36 in which the frame 40 is displayed, but since there is a high degree of freedom and it is not restricted to quadrant units as in the above embodiment, the doctor 12 can visually and accurately grasp the location of the lesion 42 that is outside the field of view from the display position of the visual assist mark 106.

[0198] 14 shows an example in which the visual assist mark 106 is displayed on the screen 35 in a display mode corresponding to the change characteristic, but the present disclosure is not limited to this. For example, the display mode of the visual assist mark 106 may be changed according to the characteristics of the lesion 42. In this case, for example, the control unit 82B identifies the characteristics of the lesion 42 based on the lesion characteristic information 98D (see FIG. 5) included in the recognition result 98. Then, the control unit 82B displays the visual assist mark 106 on the screen 35 in a display mode corresponding to the characteristics of the lesion 42.

[0199] 23 as an example, the display mode according to the characteristics of the lesion 42 is defined by a display mode table 110. The display mode table 110 is stored in the storage 86. The display mode table 110 is classified into first to seventh display mode tables 110A to 110G.

[0200] In the first display mode table 110A, the malignancy of the lesion 42 is associated with a display mode. The display mode associated with the malignancy of the lesion 42 is color. In the example shown in FIG. 23, a different color is associated with each of the malignancies of the lesion 42. The control unit 82B identifies the malignancy of the lesion 42 from the lesion characteristic information 98D. Then, the control unit 82B refers to the first display mode table 110A to identify a color corresponding to the identified malignancy from the lesion characteristic information 98D, and displays the visual assist mark 106 on the screen 35 in the identified color.

[0201] In second display mode table 110B, the area where lesion 42 is present is associated with a display mode. The display mode associated with the area where lesion 42 is present is a pattern to be included in visual assist mark 106. In the example shown in FIG. 23, a different pattern is associated with each area where lesion 42 is present. Controller 82B identifies the area where lesion 42 is present from lesion characteristic information 98D. Then, controller 82B refers to second display mode table 110B to identify the pattern corresponding to the identified area from lesion characteristic information 98D, and displays visual assist mark 106 including the identified pattern on screen 35.

[0202] In the third display mode table 110C, the type of lesion 42 is associated with a display mode. The display mode associated with the type of lesion 42 is a blinking pattern that blinks the visual assist mark 106. In the example shown in FIG. 23, a different blinking pattern is associated with each type of lesion 42. The control unit 82B identifies the type of lesion 42 from the lesion characteristic information 98D. Then, by referring to the third display mode table 110C, the control unit 82B identifies a blinking pattern that corresponds to the identified type from the lesion characteristic information 98D, and displays the visual assist mark 106 on the screen 35 in the identified blinking pattern.

[0203] In the fourth display mode table 110D, the types of lesion 42 are associated with display modes. The display mode associated with the types of lesion 42 is the shape of the visual assist mark 106. In the example shown in FIG. 23, a different shape is associated with each type of lesion 42. The control unit 82B identifies the type of lesion 42 from the lesion characteristic information 98D. Then, the control unit 82B refers to the fourth display mode table 110D to identify a shape corresponding to the identified type from the lesion characteristic information 98D, and displays the visual assist mark 106 formed in the identified shape on the screen 35.

[0204] In fifth display mode table 110E, the morphology of lesion 42 is associated with a display mode. The display mode associated with the morphology of lesion 42 is the luminance of the visual assist mark 106. In the example shown in FIG. 23, a different luminance is associated with each of the morphologies of lesion 42. Controller 82B identifies the morphology of lesion 42 from lesion characteristic information 98D. Controller 82B then refers to fifth display mode table 110E to identify the luminance corresponding to the identified morphology from lesion characteristic information 98D, and displays visual assist mark 106 on screen 35 at the identified luminance.

[0205] In the sixth display mode table 110F, the mode of the boundary between the lesion 42 and the periphery of the lesion 42 is associated with the display mode. The display mode associated with the mode of the boundary between the lesion 42 and the periphery of the lesion 42 is the outline of the visual assist mark 106. In the example shown in FIG. 23, a different outline is associated with each of the modes of the boundary between the lesion 42 and the periphery of the lesion 42 (in the example shown in FIG. 23, a mode in which there is a boundary between the lesion 42 and the periphery of the lesion 42, and a mode in which there is no boundary between the lesion 42 and the periphery of the lesion 42). The control unit 82B specifies the mode of the boundary between the lesion 42 and the periphery of the lesion 42 from the lesion characteristic information 98D. The control unit 82B then specifies the outline corresponding to the mode specified from the lesion characteristic information 98D by referring to the sixth display mode table 110F, and displays the visual assist mark 106 of the specified outline on the screen 35.

[0206] In the seventh display mode table 110G, the adhesion mode of the mucus of the lesion 42 is associated with the display mode. The display mode associated with the adhesion mode of the mucus of the lesion 42 is the presence or absence of semi-transparency. In the example shown in FIG. 23, the presence or absence of semi-transparency is associated with each of the adhesion modes of the mucus of the lesion 42 (in the example shown in FIG. 23, the presence or absence of mucus adhesion). The control unit 82B identifies the adhesion mode of the mucus of the lesion 42 from the lesion characteristic information 98D. Then, the control unit 82B refers to the seventh display mode table 110G to identify the presence or absence of semi-transparency corresponding to the adhesion mode identified from the lesion characteristic information 98D, and displays the visual assist mark 106 on the screen 35 according to the identified presence or absence of semi-transparency. In this case, for example, if there is mucus adhering to the lesion 42, the visual assist mark 106 is made semi-transparent and displayed on the screen 35, and if there is no mucus adhering to the lesion 42, the visual assist mark 106 is displayed on the screen 35 without being made semi-transparent.

[0207] In this way, by changing the display mode of the visual assist mark 106 according to the characteristics of the lesion 42 outside the field of view, the doctor 12 can visually grasp the characteristics of the lesion 42 outside the field of view.

[0208] It should be noted that the display mode shown in FIG. 23 is merely an example, and any display mode may be used as long as the characteristics of the lesion 42 can be visually identified.

[0209] In the above embodiment, the positional relationship between the camera 52 and the lesion 42 is specified by the control unit 82B based on one or more optical flows 104, but the present disclosure is not limited to this. For example, the positional relationship between the camera 52 and the lesion 42 may be specified based on a detection result by an endoscope insertion shape observation device (commonly known as Coronabi).

[0210] In this case, as shown in Fig. 24 as an example, the camera 52 is provided with a magnetic field generating coil 112, and a receiver 114 is connected to the external I / F 80. The magnetic field generated from the magnetic field generating coil 112 is received by the receiver 114. The processor 82 detects the behavior (e.g., position and orientation) of the camera 52 within the large intestine 28 based on the reception result by the receiver 114, and specifies the positional relationship between the camera 52 and the lesion 42 based on the detection result. In this way, the behavior of the camera 52 within the large intestine 28 is detected with high accuracy, and therefore it is possible to specify the positional relationship between the camera 52 and the lesion 42 with high accuracy.

[0211] The endoscope insertion shape observation device given here is merely an example, and a sensor other than the endoscope insertion shape observation device may be used. For example, the positional relationship between the camera 52 and the lesion 42 in the large intestine 28 may be specified based on the detection result by a sensor capable of detecting the behavior of the camera 52 in the large intestine 28, such as an acceleration sensor, a gyro sensor, and / or a magnetic sensor. The endoscope insertion shape observation device, acceleration sensor, gyro sensor, and magnetic sensor given here as examples are examples of the "sensor" according to the present disclosure.

[0212] In the above embodiment, an example is given in which the control unit 82B generates the screen information 105 and outputs it to the display device 18, but the present disclosure is not limited to this. For example, as shown in FIG. 25, the control unit 82B may generate screen generation information 116. The screen generation information 116 includes a frame 40, medical information 44, a visual assist mark 106, and layout information 117. Here, the screen generation information 116 is an example of the "screen generation information" according to the present disclosure, and the layout information 117 is an example of the "position instruction information" according to the present disclosure.

[0213] The layout information 117 is information that defines the layout of the frame 40, the medical information 44, and the visual assist mark 106 within the screen 35. An example of the information that defines the layout of the frame 40 within the screen 35 is information that indicates the position at which the frame 40 is displayed within the screen 35 (for example, information including coordinates that can specify the position at which the frame 40 is displayed within the screen 35). An example of the information that defines the layout of the medical information 44 within the screen 35 is information that indicates the position at which the medical information 44 is displayed within the screen 35 (for example, information including coordinates that can specify the position at which the medical information 44 is displayed within the screen 35). An example of the information that defines the layout of the visual assist mark 106 within the screen 35 is information that indicates the position at which the visual assist mark 106 is displayed within the screen 35 (for example, information including coordinates that can specify the position at which the visual assist mark 106 is displayed within the screen 35).

[0214] Here, the position where frame 40 is displayed on screen 35 refers to the position of first display area 36. Moreover, the position where medical information 44 is displayed on screen 35 refers to the position of second display area 38. Moreover, the position where visual assist mark 106 is displayed on screen 35 refers to a position where the lesion position can be identified outside first display area 36 where frame 40 is displayed.

[0215] The position at which visual assist mark 106 is displayed on screen 35 is determined by control unit 82B in a manner similar to that of the above embodiment, depending on the positional relationship between camera 52 and lesion 42. When the positional relationship between camera 52 and lesion 42 changes, optical flow 104 is updated, and accordingly, the position at which visual assist mark 106 is displayed on screen 35 is updated.

[0216] In other words, the control unit 82B updates the optical flow 104 in accordance with changes in the positional relationship between the camera 52 and the lesion 42, and updates a portion of the information contained in the layout information 117 (i.e., information indicating the position where the visual assist mark 106 is to be displayed on the screen 35) in accordance with the updated optical flow 104.

[0217] The control unit 82B outputs the screen generation information 116 to the controller 118. Examples of the controller 118 include the control device 22, a tablet terminal, a personal computer, or a server. The display device 18 is connected to the controller 118. The controller 118 generates the screen 35 based on the screen generation information 116 input from the control unit 82B, and displays it on the display device 18.

[0218] Furthermore, control unit 82B updates screen generation information 116 in response to a change in the positional relationship between camera 52 and lesion 42. In this case, for example, the position at which visual assist mark 106 is displayed on screen 35 is updated in response to a change in the positional relationship between camera 52 and lesion 42. Screen 35 generated by controller 118 based on the updated screen generation information 116 is displayed on display device 18.

[0219] In this way, even when screen generation information 116 including frame 40, medical information 44, visual assist mark 106, and layout information 117 is generated and output by control unit 82B, the same effects as in the above embodiment can be expected.

[0220] In addition, in the example shown in Figure 25, an example is given in which screen generation information 116 including frame 40, medical information 44, visual assist mark 106, and layout information 117 is generated and output by control unit 82B, but if visual assist mark 106 is generated by controller 118, control unit 82B may generate information including frame 40, medical information 44, and layout information 117 as screen generation information 116 and output it to controller 118.

[0221] Although an example in which the screen generating information 116 includes the medical information 44 has been given here, the screen generating information 116 does not necessarily have to include the medical information 44 .

[0222] Furthermore, the control unit 82B may divide the screen generation information 116 and output it to the controller 118. For example, the control unit 82B may output the frames 40, the medical information 44, the visual assist marks 106, and the layout information 117 in a time-division manner.

[0223] In the above embodiment, an example in which the screen information 105 is output to the display device 18 has been given, but this is merely one example, and the screen information 105 may be output to the storage 76 and / or 86. The screen information 105 may also be output to a processing device (e.g., a tablet terminal, a personal computer, and / or a server) that exists outside the endoscope device 10. The screen information 105 may also be output to a printer. In this case, for example, the printer prints on a medium (e.g., paper) an image that visualizes the screen 35 displayed by the input screen information 105.

[0224] In the above embodiment, lesion 42 is given as an example of an "internal characteristic region" according to the present disclosure, but the present disclosure is not limited thereto, and the above-mentioned medical support processing can be performed even if a resection region, a bleeding region, a marking region, an organ, or a treatment tool (e.g., a hemostatic clip placed inside the body) is applied instead of lesion 42.

[0225] In the above embodiment, an example was given in which the frame 40 is input to the recognition unit 82A and the screen information 105 is output from the control unit 82B, but the present disclosure is not limited to this. For example, instruction data (so-called prompt) including the frame 40 may be input to a so-called generation AI, and the screen information 105 may be output from the generation AI. One example of the generation AI is GPT-4 (Internet search<https: / / openai.com / gpt-4> ) is used in ChatGPT.

[0226] Furthermore, instruction data including at least a portion of the information included in the screen information 105 (e.g., the frame 40 and the visual assist mark 106) may be used as input information for the generation AI. Examples of information output from the generation AI include information capable of identifying the positional relationship between the camera 52 and the lesion 42, information capable of identifying a change in the positional relationship between the camera 52 and the lesion 42, information indicating the operation content of the camera 52, information indicating the content of a medical procedure recommended to be performed during an endoscopic examination, and / or information indicating the content of a medical procedure recommended to be performed after an endoscopic examination. The information output from the generation AI may be stored in various storage areas (e.g., storage 76 and / or 86), displayed on the display device 18 as medical information 44, printed on a medium by a printer, or output as audio from a speaker.

[0227] In the above embodiment, the recognition process 96 using AI in the bounding box method is exemplified, but this is merely one example, and for example, recognition process using AI in the segmentation method may be performed instead of the recognition process 96 using AI in the bounding box method. Also, instead of the recognition process using AI, recognition process using a non-AI method (for example, a template matching method) may be performed, or recognition process using a combination of the non-AI method and the AI ​​method may be performed.

[0228] In the above embodiment, an example in which the medical support process is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processes included in the medical support process may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0229] 26 is a conceptual diagram showing an example of the configuration of an endoscope device 120. The endoscope device 120 is an example of the "endoscope device" according to the present disclosure. The endoscope device 120 differs from the endoscope device 10 described in the above embodiment in that it includes an external device 122.

[0230] External device 122 is communicatively coupled to computer 78 via a network 124 (eg, a WAN and / or a LAN, etc.).

[0231] An example of the external device 122 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 124. The external device 122 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 124. The external device 122 then executes processing according to the received processing execution instruction and transmits the processing result to the computer 78 via the network 124. In the computer 78, the processor 82 receives the processing result transmitted from the external device 122 via the network 124 and executes processing using the received processing result.

[0232] An example of the processing execution instruction is an instruction to cause the external device 122 to execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a processing to be executed by the external device 122) is the recognition processing 96. In this case, the external device 122 executes the recognition processing 96 in accordance with the processing execution instruction provided from the processor 82 via the network 124, and transmits the recognition result 98 to the computer 78 via the network 124. In the computer 78, the processor 82 receives the recognition result 98, and executes the same processing as in the above embodiment using the received recognition result 98.

[0233] A second example of at least a part of the medical support process (i.e., the process to be executed by the external device 122) is a process by the control unit 82B. In this case, the external device 122 executes the process by the control unit 82B according to a process execution instruction given from the processor 82 via the network 124, and transmits the process result (e.g., screen information 105, etc.) to the computer 78 via the network 124. In the computer 78, the processor 82 receives the process result, and executes the same process as in the above embodiment (e.g., display using the display device 18, etc.) using the received process result.

[0234] For example, the external device 122 is realized by cloud computing. Note that cloud computing is merely an example, and the external device 122 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 122. Also, the external device 122 may be a computing device with a communication function equipped with multiple types of AI functions.

[0235] In the above embodiment, an example in which the medical support program 90 is stored in the storage 86 has been described, but the present disclosure is not limited to this. For example, the medical support program 90 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10. The processor 82 executes medical support processing in accordance with the medical support program 90.

[0236] In addition, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscopic device 10 via a network, and the medical support program 90 may be downloaded and installed on the computer 78 upon request from the endoscopic device 10.

[0237] It is not necessary to store all of the medical support program 90 in a storage device such as another computer or server device connected to the endoscopic device 10, or to store all of the medical support program 90 in the storage 86; only a portion of the medical support program 90 may be stored.

[0238] The hardware resources for executing the medical support processing can be various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the medical support processing by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the medical support processing by using the memory.

[0239] The hardware resource for executing the medical support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support process may be a single processor.

[0240] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute medical support processing on a single IC chip, as typified by SoC. In this way, the medical support processing is realized using one or more of the above various processors as hardware resources.

[0241] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0242] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0243] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0244] 10,120 Endoscopic equipment 12. Doctor 14. Nurse 16 Endoscope 18 Display device 20 Light source device 22 Control device 24 Medical Support Equipment 26 Subject 28 Large intestine 30 light 32 Intestinal wall 34 Wagon 35 screens 36 1st display area 38 Second display area 39 Endoscopic video images 40 frames 42 Lesions 44 Medical Information 44A Text 46 Control section 48 Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 66,78 Computer 68,88 Bus 70,80 External I / F 72,82 processor 74,84 Memory 76,86 Storage 82A recognition part 82B Control section 84A First Storage Area 84B Second storage area 90 Medical Assistance Program 92 Recognition Model 96 Recognition Processing 98 Recognition results 98A Lesion presence / absence information 98B Geometric property information 98C Lesion location map 98D Lesion characteristics information 100 Edge 102 Feature position coordinates 104 Optical Flow 105 Screen Information 106,108 Visual Assist Mark 107 Code 109 Arrow 110 Display mode table 110A First display mode table 110B Second display mode table 110C Third display mode table 110D Fourth display mode table 110E Fifth display mode table 110F 6th display mode table 110G 7th display mode table 112 Magnetic field generating coil 114 Receiver 116 Screen generation information 117 Layout Information 118 Controller 122 External device 124 Network BB Bounding Box VC Yen VL Cross Line

Claims

1. A processor is provided. The processor, The camera captures images of the inside of the body, and the images are acquired. outputting screen generation information used for generating a screen on which the image and presence position information capable of identifying, outside the image, the presence position of an internal characteristic region recognized by executing an object recognition process using the image are displayed; The display position of the presence location information with respect to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic region. Medical support equipment.

2. The change is caused by the operation of the camera and / or the internal body movement. The medical support device according to claim 1 .

3. The display position is changed in accordance with the change to follow the operation and / or the body movement. The medical support device according to claim 2.

4. The display mode of the location information is changed according to the characteristics of the change. The medical support device according to claim 1 .

5. The characteristics include a rate of the change, an amount of the change, and / or a direction of the change. The medical support device according to claim 4.

6. the presence position information includes: within-angle-of-view position information that allows the presence position of the internal characteristic region to be specified outside the angle of view of the camera when the internal characteristic region is within the angle of view of the camera; and outside-angle-of-view position information that allows the presence position of the internal characteristic region to be specified outside the angle of view when the internal characteristic region is out of the angle of view; when the internal body feature region is within the field of view, the in-field of view position information is displayed on the screen; When the internal characteristic region is outside the angle of view, the outside angle of view position information is displayed on the screen. The medical support device according to claim 1 .

7. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, when the internal feature region is outside the angle of view, the outside angle of view position information is displayed on the screen; The display mode of the outside-angle-of-view position information is changed according to the characteristics of the change. The medical support device according to claim 1 .

8. The display mode includes whether or not to display, display intensity, display time, and / or speed at which the display intensity is changed. The medical support device according to claim 7.

9. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, when the internal feature region is outside the angle of view, the outside angle of view position information is displayed on the screen; the internal feature region is a lesion; The display manner of the outside angle of view position information is changed according to the malignancy of the lesion, the site where the lesion is present, the type of the lesion, the form of the lesion, the state of the boundary between the lesion and the periphery of the lesion, and / or the state of mucus adhesion to the lesion. The medical support device according to claim 1 .

10. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, When the internal feature region is out of the angle of view, the out-of-angle of view position information is displayed on the screen. The medical support device according to claim 1 .

11. The display of the outside-angle-of-view position information on the screen when the time within the angle of view during which the internal characteristic region is within the angle of view is less than a certain value is emphasized more than the display of the outside-angle-of-view position information on the screen when the time within the angle of view is equal to or greater than the certain value. The medical support device according to claim 10.

12. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, The out-of-angle-of-view position information is displayed on the screen on the condition that a predetermined time has elapsed since the internal feature region went out of the angle of view. The medical support device according to claim 1 .

13. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, When the internal characteristic region is out of the angle of view and the degree of the change exceeds a predetermined degree, the out-of-angle of view position information is displayed on the screen. The medical support device according to claim 1 .

14. the presence position information includes outside-angle-of-view position information that can specify the presence position outside the angle of view of the camera when the internal characteristic region is outside the angle of view of the camera, when the internal feature region is outside the angle of view, the outside angle of view position information is displayed on the screen; When the frequency of the internal feature region moving in and out of the angle of view within a unit time exceeds a predetermined frequency, the display of the outside angle of view position information on the screen is emphasized more than the display of the outside angle of view position information on the screen when the frequency is equal to or less than the predetermined frequency. The medical support device according to claim 1 .

15. the screen generation information includes the image and position indication information indicating a position of the presence position information within the screen, The position indication is updated according to the changes. The medical support device according to claim 1 .

16. The screen generation information includes the image, the location information, and the location instruction information. The medical support device according to claim 15.

17. The object recognition process includes a process of recognizing the internal feature region based on the image by using AI. The medical support device according to claim 1 .

18. The internal feature region is a lesion. The medical support device according to claim 1 .

19. The image is included in a plurality of frames obtained in time series by imaging the inside of the body with the camera, The processor, determining the change based on the plurality of frames; Changing the display position according to the identified change. The medical support device according to claim 1 .

20. The processor, Identifying the change based on a detection result by a sensor capable of detecting a behavior of the camera inside the body; Changing the display position according to the identified change. The medical support device according to claim 1 .

21. A processor is provided. The processor, The camera captures images of the inside of the body, and the images are acquired. outputting screen generation information used for generating a screen on which a medical image generated based on the image and presence position information capable of identifying, outside the image, the presence position of an internal characteristic region recognized by executing an object recognition process using the image are displayed; A display position of the presence location information with respect to the medical image on the screen is changed according to a change in a positional relationship between the camera and the internal characteristic region. Medical support equipment.

22. A medical support device according to any one of claims 1 to 21; The camera. Endoscopic device.

23. Obtaining an image obtained by imaging the inside of the body with a camera; and outputting screen generation information used to generate a screen on which the image and presence position information capable of identifying, outside the image, a presence position of an internal characteristic region recognized by executing an object recognition process using the image are displayed; The display position of the presence location information with respect to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic region. Medical assistance methods.

24. A program for causing a computer to execute a medical support process, The medical support process includes: Obtaining an image obtained by imaging the inside of the body with a camera; and outputting screen generation information used to generate a screen on which the image and presence position information capable of identifying, outside the image, a presence position of an internal characteristic region recognized by executing an object recognition process using the image are displayed; The display position of the presence location information with respect to the image on the screen is changed according to a change in the positional relationship between the camera and the internal characteristic region. program.

Images