Endoscopy support apparatus, method of operating endoscopy support apparatus, and endoscopy support program

Abstract

To provide an endoscopy support apparatus 1 capable of easily observing a target site of a stomach.SOLUTION: An endoscopy support apparatus 1 includes a memory 20 and a processor 10, the memory 20 stores a learned model learned by a learning data set in which a site name, a visual field direction, and an axial direction are annotated for each of a plurality of endoscope images, at least one target site name, and a positional relationship between a plurality of sites, and the processor 10 includes an AI processing unit 11 that infers the site name, the visual field direction, and the axial direction of the captured image by inputting the captured image to the learned model, and a navigation unit 12 that outputs a direction of the target site in the captured image on the basis of the positional relationship between the plurality of sites and the site name, the visual field direction, and the axial direction of the captured image inferred by the AI processing unit 11.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an endoscopic examination support device for observing a target region of the stomach, an operating method of an endoscopic examination support device for observing a target region of the stomach, and an endoscopic examination support program for observing a target region of the stomach. [Background technology]

[0002] In gastric examinations using an endoscope system, it is necessary to observe and photograph multiple examination sites to prevent overlooking lesions.

[0003] Japanese Patent Application Laid-Open Publication No. 2010-51399 discloses an endoscopic image recording device that automatically starts and stops recording endoscopic images. This endoscopic image recording device determines whether an image signal obtained by an endoscope contains a red color equal to or greater than a threshold, and automatically starts recording the image signal if it is determined that the image signal contains a red color equal to or greater than the threshold, and automatically stops recording the image signal if it is determined that the image signal does not contain a red color equal to or greater than the threshold.

[0004] International Publication No. 2021 / 144951 discloses an image processing device that uses AI calculations to infer whether an endoscopic image is of a specified target area and automatically records the image depending on the observation mode of the endoscopic image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-51399 [Patent Document 2] International Publication No. 2021 / 144951 Summary of the Invention [Problem to be solved by the invention]

[0006] It is not easy for the surgeon to comprehensively observe multiple examination sites within the stomach.

[0007] An embodiment of the present invention aims to provide an endoscopic examination support device capable of comprehensively observing the examination site of the stomach, an operating method of an endoscopic examination support device capable of comprehensively observing the examination site of the stomach, and an endoscopic examination support program capable of comprehensively observing the examination site of the stomach. [Means for solving the problem]

[0008] An image processing device of one embodiment of the present invention includes a memory and a processor. The memory stores a trained model trained using a training dataset in which the part name, field of view direction, and axial direction are annotated for each of a plurality of endoscopic images of the inside of the stomach, at least one target part name, and the positional relationship of a plurality of parts. The processor has an AI processing unit that infers the part name, field of view direction, and axial direction of the captured image by inputting the captured image into the trained model, and a navigation unit that outputs the direction of the target part in the captured image based on the positional relationship of the plurality of parts and the part name, field of view direction, and axial direction of the captured image inferred by the AI ​​processing unit.

[0009] In addition, a method of operating an image processing device according to one aspect of the present invention involves inputting the captured image into a trained model that has been trained using a training dataset in which the part name, field of view direction, and axial direction are annotated for each of a plurality of endoscopic images of the inside of the stomach, thereby inferring the part name, field of view direction, and axial direction of the captured image, and outputting the direction of a specified target part in the captured image based on the pre-stored positional relationship of a plurality of parts and the inferred part name, field of view direction, and axial direction of the captured image.

[0010] An image processing program according to one aspect of the present invention causes a computer to execute the following steps: inputting a captured image into a trained model that has been trained using a training data set in which a part name, a viewing direction, and an axial direction are annotated for each of a plurality of endoscopic images of the inside of the stomach, thereby inferring the part name, the viewing direction, and the axial direction of the captured image; and outputting the direction of a predetermined target part in the captured image based on a positional relationship of a plurality of parts that is stored in advance and the inferred part name, the viewing direction, and the axial direction of the captured image. [Effects of the Invention]

[0011] According to an embodiment of the present invention, it is possible to provide an endoscopic examination support device capable of comprehensively observing the examination site of the stomach, an operating method of an endoscopic examination support device capable of comprehensively observing the examination site of the stomach, and an endoscopic examination support program capable of comprehensively observing the examination site of the stomach. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an endoscope system including an endoscopic examination support device according to an embodiment. [Figure 2] FIG. 2 is a configuration diagram of an endoscope system including the endoscopic examination support device of the first embodiment. [Figure 3] FIG. 3 is a flowchart of the operation method of the endoscopic examination support device of the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining names of parts of the stomach and the viewing direction of the endoscope. [Figure 5] FIG. 5 is a diagram for explaining names of parts of the stomach. [Figure 6A] FIG. 6A is a diagram showing the positional relationship of parts of the stomach when the viewing direction is looking down. [Figure 6B] FIG. 6B is a diagram showing the positional relationship of the stomach region when the viewing direction is looking up (J-turn). [Figure 6C] FIG. 6C is a diagram showing the positional relationship of the stomach region when the viewing direction is looking up (U-turn). [Figure 7]FIG. 7 shows an example of a training data set. [Figure 8] FIG. 8 is an example of a navigation screen. [Figure 9] FIG. 9 is a configuration diagram of an endoscope system including an endoscopic examination support device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of the operation method of the endoscopic examination support device of the second embodiment. [Figure 11] FIG. 11 is a configuration diagram of an endoscope system including an endoscopic examination support device according to the third embodiment. [Figure 12] FIG. 12 is a flowchart of the operation method of the endoscopic examination support device according to the third embodiment. [Figure 13] FIG. 13 is a configuration diagram of an endoscope system including an endoscopic examination support device according to the fourth embodiment. [Figure 14] FIG. 14 is a flowchart of the operation method of the endoscopic examination support device according to the fourth embodiment. [Figure 15] FIG. 15 is a configuration diagram of an endoscope system including an endoscopic examination support device according to the fifth embodiment. BEST MODE FOR CARRYING OUT THE INVENTION

[0013] First Embodiment 1 and 2, an endoscopic examination support device 1 (hereinafter referred to as "support device 1") of this embodiment constitutes an upper gastrointestinal endoscopic system 2 together with an endoscope 9, a video processor 8, a monitor 7, a storage device 5, and a server 6. The storage device 5 and the server 6 may be shared with other medical devices, and are not essential components of the support device 1.

[0014] In the following description, the drawings based on each embodiment are schematic. The relationship between the thickness and width of each part, the thickness ratio of each part, and the relative angle are different from the actual ones. The drawings also include parts with different dimensional relationships and ratios. In addition, some components are not shown.

[0015] The endoscope 9 has a tip portion 9B equipped with an imaging unit 9A, a bending tube 9C that is bendable and allows for changing the direction of the tip portion 9B, a flexible tube 9D extending from the bending tube 9C, an operating unit 9E, and a universal cord 9F extending from the operating unit 9E. By operating the operating unit 9E, the surgeon can bend the bending tube 9C and change the direction of the tip portion 9B inserted into the stomach PS of the subject P, i.e., the field of view of the imaging unit 9A.

[0016] The universal cord 9F is connected to the video processor 8 via a connector. The video processor 8 controls the entire endoscope system 2, performs signal processing on the imaging signal output by the imaging unit 9A, and outputs a captured image. The monitor 7 displays the captured image output by the video processor 8. The monitor 7 is a display such as an LCD monitor. In this specification, the term "captured image" refers to an "endoscopic image" output from the endoscope 9 during an examination.

[0017] The support device 1 reads a program stored in a non-temporary storage device 5 (e.g., a magnetic disk, an optical disk) or a server 6 connected via a line such as the Internet, and supports endoscopic examination of the stomach PS of the subject P.

[0018] 2, the assistance device 1 includes a processor 10 and a memory 20. The processor 10, which is a CPU, includes an AI processing unit 11, a navigation unit 12, and a notification unit 13.

[0019] As will be described in detail later, the memory 20, which is, for example, a RAM, stores trained models used for estimation by the AI ​​processing unit 11, names of target parts, etc. Trained models created in advance may be transferred to the memory 20 from the storage medium 5 or the like.

[0020] The AI ​​processing unit 11 inputs the captured image into a trained model to infer the part name, field of view direction, and axial direction of the captured image. The navigation unit 12 outputs the direction of the target part in the captured image based on the part name, field of view direction, and axial direction of the captured image inferred by the AI ​​processing unit 11. The notification unit 13 outputs to the monitor 7 or the video processor 8, based on the output data of the navigation unit 12, the direction of the target part as data that can be displayed on the monitor 7.

[0021] All or part of the functions of processor 10 may be configured using logic circuits or analog circuits, and various processes may be implemented using electronic circuits such as FPGAs (Field Programmable Gate Arrays). Processor 10 may also be configured using multiple semiconductors. For example, AI processing unit 11 is preferably a semiconductor dedicated to AI processing.

[0022] <How the support device operates> The operation method of the assistance device 1 will be described below with reference to the flowchart shown in FIG.

[0023] <Step S10> Storing target body part name / body part positional relationship First, the names of the parts of the stomach PS are shown in Figures 4 and 5. The classifications and names of the parts are not limited to these. Note that Figure 4 shows the viewing direction of the endoscope, and Figure 5 shows the axial direction.

[0024] The viewing direction is either "looking up toward the cardia" or "looking down toward the pylorus." The "looking up" direction is further classified into a J-turn, where the tip turns toward the lesser curvature, and a U-turn, where the tip turns toward the greater curvature.

[0025] The axial direction is at least one of a first axial direction from the greater curvature to the lesser curvature and a second axial direction from the anterior wall to the posterior wall. In Fig. 7, the axial direction is represented by the coordinates of both ends of the first axis in the XY coordinate system of the captured image, but this is not limiting. It goes without saying that the first axial direction may be from the lesser curvature to the greater curvature, and the second axial direction may be from the posterior wall to the anterior wall.

[0026] The sequence of observations / photographs in endoscopic examinations varies depending on the surgeon. Below, we will explain the sequence as an example, starting from the cardia, observing multiple examination sites in an antegrade direction with the observation field looking down toward the pylorus, reaching the pylorus, then turning the observation field looking up toward the cardia to observe multiple examination sites, and returning to the cardia.

[0027] 1. Upper body (greater curvature, anterior wall, posterior wall) 2. Middle body (greater curvature, anterior wall, posterior wall) 3.Lower body (greater curvature, anterior wall, posterior wall) 4.Greater curvature of the gastric angle 5.Antrum (greater curvature, anterior wall, posterior wall), pylorus (J-turn maneuver) 6, vestibular curvature 7. Gastric angle (lesser curvature, anterior wall, posterior wall) 8.Lower body (lesser curvature, anterior wall, posterior wall) 9. Middle body (lesser curvature, anterior wall, posterior wall) 10. Upper body (lesser curvature, anterior wall, posterior wall) 11.Cardia (anterior wall, posterior wall, lesser curvature) (U-turn maneuver) 12. Cardia (greater curvature), vault

[0028] The relative positional relationships of the above-mentioned multiple parts are stored in memory 20.

[0029] Figures 6A to 6C show the relative positional relationships of multiple parts in the endoscopic field of view (image). In each figure, the smaller numbers indicate the upper side of the figure, and the larger numbers indicate the lower side of the figure.

[0030] It is also possible for the support device 1 to set all of the above multiple regions as target regions for navigation. However, for regions that are easy to observe and do not require navigation, navigation becomes cumbersome for the surgeon. For this reason, for example, the target regions can be set by the surgeon's settings.

[0031] Among the multiple examination regions, the posterior wall of the gastric angle and the lesser curvature of the cardia are particularly difficult to observe and are prone to becoming blind spots. For this reason, the following description will be given taking the posterior wall of the gastric angle and the lesser curvature of the cardia as examples of the regions to be navigated.

[0032] <Step S20> Capture image The tip 9B of the endoscope 9 is inserted into the stomach PS of the subject P, and an image of the inside of the stomach PS (hereinafter referred to as the "captured image") is acquired by the imaging unit 9A. The captured image is a still image automatically extracted from a video, or an image captured by the operator's operation.

[0033] <Step S30> AI processing A trained model is created in advance in the assistance device 1. The trained model is trained (for example, by deep learning) using a plurality of training data sets (training datasets) in which the part name, viewing direction, and axial direction are annotated for each of a plurality of endoscopic images (training images) of the inside of the stomach. The annotations are made by an experienced surgeon.

[0034] The annotation of the body parts is performed, for example, by image classification annotation, segmentation, bounding box, or keypoint annotation.

[0035] In the training data illustrated in Figure 7, the endoscopic image is annotated with the part name "lesser curvature of the vestibule," the viewing direction "downward," and the axial direction "first axis, (X1, 1) → (X2, 0)."

[0036] The created trained model is stored in the memory 20. The AI ​​processing unit 11 uses the trained model to estimate the part name, the field of view direction, and the axial direction for the captured image input from the video processor 8.

[0037] <Step S40> Target area? If the captured image estimated by the AI ​​processing unit 11 is an image of the target area for which navigation processing is to be performed (YES), the processor 10 performs processing from step S50. On the other hand, if the captured image is not an image of the target area (NO), the processor 10 performs navigation processing from step S80.

[0038] <Step S50> No recorded image? For example, the memory 20 stores the target region names that the AI ​​processing unit 11 has inferred so far in the current examination of the subject P (see S60). If the region name of the captured target image has already been recorded in the memory 20 (YES), the processor 10 repeats the processing from step S20. That is, a new captured image is processed. If the captured image has not been recorded (NO), the processing of step S60 is performed.

[0039] <Step S60> Record image and store part name The captured image is stored in the memory 20 or the storage device 5. The memory 20 also stores the name of the part of the stored captured image.

[0040] <Step S70> Done? The processor 10 repeats the process from step S20 until the observation / photography / image storage of all target sites is completed (YES).

[0041] The memory 20 may store not only the name of the target area but also the name of the examination area, and if the captured image is estimated to be an image of the target area or the examination area in step S40, the processing of steps S50 and S60 may be performed.

[0042] <Step S80> Target part direction output The navigation unit 12 calculates the direction of the target part in the captured image from the estimated part name, viewing direction, and axial direction of the captured image.

[0043] That is, the navigation unit 12 outputs the direction of the target part based on the positional relationship data of the multiple parts shown in Figures 6A to 6C. For example, as will be described later, the direction of the target part is expressed as an angle θ with respect to the center point of the captured image.

[0044] In the example shown in Figure 6B, when the viewing direction is (looking up, J-turn) and the lesser curvature of the upper body is being observed, the lesser curvature of "2: upper body" is the current position. The lesser curvature of the target region, "1: cardia," is upward relative to the lesser curvature of "2: upper body."

[0045] <Step S90> Display instruction mark The notification unit 13 outputs the direction of the target area output by the navigation unit 12 to the monitor 7 or the video processor 8. For example, as shown in Fig. 8, an arrow mark M indicating the direction of the target area is displayed around the captured image on the monitor 7 together with patient information and the like.

[0046] In FIG. 8, for the sake of explanation, the name of the part of the captured image, the field of view direction, the axial direction, and the direction of the target part (illustrated) are also displayed on the monitor 7, but these do not have to be displayed. Furthermore, the direction of the target part is shown using a clockwise angle θ with the center of the captured image as the origin and the upward direction as an angle of 0 degrees, but the direction display direction is not limited to this. For example, the direction of the target part may be four directions, "up, down, right, left," or eight directions, each of which may be obtained by adding a diagonal direction (for example, an upward-right diagonal direction) to the four directions. Furthermore, the direction of the target part may be notified to the surgeon by voice instead of by the arrow mark M.

[0047] When the surgeon operates the endoscope 9 while referring to the arrow mark indicating the direction of the target region to acquire a new captured image, the processing from step S20 is repeated.

[0048] As described above, the operating method of the endoscopic examination support device of the embodiment involves inputting the captured image into a trained model that has been trained using a training dataset in which the part name, field of view direction, and axial direction are annotated for each of a plurality of endoscopic images of the inside of the stomach, thereby inferring the part name, field of view direction, and axial direction of the captured image, and outputting the direction of a specified target part in the captured image based on the pre-stored positional relationship of a plurality of parts and the inferred part name, field of view direction, and axial direction of the captured image.

[0049] The endoscopic examination assistance program of the embodiment causes a computer to execute the following steps: inputting the captured image into a trained model that has been trained using a training dataset in which the part name, the viewing direction, and the axial direction are annotated for each of a plurality of endoscopic images of the inside of the stomach, thereby inferring the part name, the viewing direction, and the axial direction of the captured image; and outputting the direction of a specified target part in the captured image based on the positional relationship of a plurality of parts that has been stored in advance and the inferred part name, the viewing direction, and the axial direction of the captured image.

[0050] According to the embodiments of the present invention, it is possible to provide an endoscopic examination support device that can easily observe target areas in the stomach, an operating method of an endoscopic examination support device that can easily observe target areas in the stomach, and an endoscopic examination support program that can easily observe target areas in the stomach.

[0051] Second Embodiment The second embodiment described below is similar to the first embodiment and has the same effects as the first embodiment, so components with the same functions are given the same reference numerals and descriptions thereof will be omitted.

[0052] 9, the support device 1A of this embodiment has a start-up control unit 14. When observing / photographing a plurality of parts in sequence, if a part far away is set as the target part, navigation by the support device becomes complicated for the surgeon.

[0053] In the support device 1A, the memory 20 stores the names of adjacent parts adjacent to the target part. The start-up control unit 14 controls the AI ​​processing unit 11 to infer the direction of the target part only when the captured image is an image of an adjacent part. In other words, when observing / photographing multiple parts in sequence, the navigation process is performed only when the target part is approached.

[0054] The operation method of the assistance device 1A will be described with reference to the flowchart of Fig. 10. The flowchart of Fig. 10 is the same as the flowchart of Fig. 3 except for step S15 and step S45, so the description thereof will be omitted.

[0055] <Step S15> Storing target part name / storing adjacent part data When the target region name is set, the processor 10 stores in the memory 20 region data (region names, relative positions, etc.) adjacent to the target region.

[0056] For example, if the target site is the posterior wall of the gastric angle, the adjacent sites are the lesser curvature of the gastric angle, the lesser curvature of the antrum, and the posterior wall of the antrum. Also, if the target site is the lesser curvature of the cardia, the adjacent sites are the anterior wall of the cardia, the posterior wall of the cardia, and the lesser curvature of the upper body.

[0057] <Step S45> Adjacent area? The activation control unit 14 determines whether the part inferred by the AI ​​processing unit 11 is an adjacent part. If it is an adjacent part (YES), the notification unit 13 outputs the direction of the target part in step 80. If it is not an adjacent part (NO), the processor 10 repeats the process from step S20.

[0058] The support device 1A does not display navigation information, which is the inference result of the AI ​​processing unit 11, on the monitor 7 until the examination target reaches a region adjacent to the target region. Therefore, the support device 1A can provide navigation that is more comfortable for the operator than the support device 1.

[0059] <Third embodiment> The third embodiment described below is similar to the first and second embodiments and has the same effects as the first and second embodiments. Therefore, components with the same functions are given the same reference numerals and descriptions thereof will be omitted.

[0060] As shown in FIG. 11, in a support device 1B of this embodiment, an AI processing unit 11 includes a first AI processing unit 11A and a second AI processing unit 11B.

[0061] The trained models stored in memory 20 include a first trained model trained using a first training data set in which body part names and viewing directions are annotated, and a second trained model trained using a second training data set in which axial directions are annotated. AI processing unit 11 includes a first AI processing unit 11A that infers body part names and viewing directions by inputting a captured image into the first trained model, and a second AI processing unit 11B that infers axial directions by inputting a captured image into the second trained model.

[0062] The first trained model annotates the part name and the viewing direction, and therefore can create a model with higher accuracy in a shorter time than a trained model that annotates the part name, the viewing direction, and the axial direction. Furthermore, the first AI processing unit 11A can make inferences with higher accuracy in a shorter time than an AI processing unit that infers the part name, the viewing direction, and the axial direction. Similarly, the second trained model can create a model with higher accuracy in a shorter time than a trained model that annotates the part name, the viewing direction, and the axial direction. Furthermore, the second AI processing unit 11B can make inferences with higher accuracy in a shorter time than an AI processing unit that infers the part name, the viewing direction, and the axial direction.

[0063] The operation method of the support device 1B will be described with reference to the flowchart of Fig. 12. The flowchart of Fig. 12 is the same as the flowchart of Fig. 10 except for step S31 (S30) and step 46, so the description will be omitted.

[0064] <Step S31> First AI processing (first trained model) The first AI processing unit 11A estimates the part name and the viewing direction of the captured image input from the video processor 8 using the first trained model.

[0065] <Step S46> Second AI processing (second trained model) The second AI processing unit 11B estimates the part name and the viewing direction of the captured image using the second trained model.

[0066] The assistance device 1B is capable of performing inference in a shorter time and with higher accuracy than the assistance device 1A.

[0067] <Modification of the third embodiment> Although not shown, in the support device 1B, it is preferable that the second trained model has multiple models in which the axial direction is annotated for each part name, and the second AI processing unit 11B infers the axial direction by inputting the captured image, the part name inferred by the first AI processing unit 11A, and the field of view direction into the second trained model.

[0068] The modified support device, which performs inference using a model of the part name inferred by the first AI processing unit 11A from among multiple models in which axial directions are annotated for each part name, is able to perform inference in a shorter time and with higher accuracy than the support device 1B of the third embodiment, which performs inference using a model including all parts.

[0069] <Fourth embodiment> The fourth embodiment described below is similar to the first to third embodiments and has the same effects as the first to third embodiments. Therefore, components with the same functions are given the same reference numerals and descriptions thereof will be omitted.

[0070] As shown in FIG. 13, in a support device 1C of this embodiment, the AI ​​processing unit 11 includes a first AI processing unit 11A, a second AI processing unit 11B, and a third AI processing unit 11C.

[0071] The trained model includes a third trained model trained using a training dataset annotated with image quality for endoscopic images. The AI ​​processing unit 11 includes a third AI processing unit 11C that infers the image quality of the captured image by inputting the captured image to the third trained model. When the image quality of the captured image satisfies a predetermined quality, the first AI processing unit 11A and the second AI processing unit 11B make an inference.

[0072] For example, image quality is quantified by the ratio of the areas of halation areas, out-of-focus areas, blurred areas, mucus residue areas, etc. to the total area of ​​the captured image.

[0073] The operation method of the support device 1B will be described with reference to the flowchart of Fig. 14. The flowchart of Fig. 14 is the same as the flowchart of Fig. 12 except for steps S25 and S26, so the description thereof will be omitted.

[0074] <Step S25> Third AI processing (third trained model) The third AI processing unit 11C estimates the image quality of the captured image using the third trained model.

[0075] <Step S26> Appropriate image? If the image quality of the captured image satisfies the predetermined quality (YES), the navigation process starts from step S31. If the image quality does not satisfy the predetermined quality (No), the navigation process is not performed and the process from step S20 is repeated.

[0076] The assistance device 1B can efficiently proceed with the examination because the navigation process using an inappropriate captured image is not performed. Of course, the inappropriate captured image is not stored in the memory 20.

[0077] Step S25 (third AI processing) may be performed after step S50. That is, the captured image may be stored in the memory 20 only when the image quality of the captured image inferred by the first AI processing unit 11A satisfies a predetermined quality.

[0078] Fifth Embodiment The fifth embodiment is similar to the first to fourth embodiments and has the same effects as the first to fourth embodiments. Therefore, components with the same functions are given the same reference numerals and descriptions thereof will be omitted.

[0079] As shown in FIG. 15, in a support device 1D of this embodiment, the AI ​​processing unit 11 includes a fourth AI processing unit 11D, a fifth AI processing unit 11E, and a sixth AI processing unit 11F.

[0080] The trained models include a fourth trained model trained using a fourth training dataset in which endoscopic images of the inside of the stomach are annotated with part names, a fifth trained model trained using a fifth training dataset in which endoscopic images are annotated with the viewing direction, and a sixth trained model trained using a sixth training dataset in which endoscopic images are annotated with at least one of a first axis direction from the greater curvature to the lesser curvature and a second axis direction from the anterior wall to the posterior wall.

[0081] The AI ​​processing unit 11 infers the direction of the target part using the fourth trained model, the fifth trained model, and the sixth trained model.

[0082] The assistance device 1D uses a dedicated model for each estimation and performs AI processing for each, so the circuitry is smaller and more accurate inference can be made in a shorter time than assistance devices that perform AI processing using models that annotate multiple factors.

[0083] It goes without saying that the support device 1D may also perform navigation processing after first eliminating inappropriate images, as in the support device 1C.

[0084] <Additional Notes> AI processing and the like may be executed, for example, in the server 6. That is, the trained model is stored in the server 6, and the processor 10 transmits the input captured image to the server 6, where the AI ​​processing is performed. The processor 10 may then receive the judgment result and the estimation result from the server 6. In this case, the server 6 is considered to be part of the processor 10.

[0085] The programs for the endoscopic examination assistance devices 1, 1A-1D to execute the operations described above are stored in whole or in part in a computer-readable state in a portable medium such as a flexible disk or a CD-ROM, or in a non-transitory computer-readable storage medium such as a hard disk. That is, a program for causing a computer to execute the endoscopic examination assistance program of the embodiment is stored in the non-transitory computer-readable storage medium (external device).

[0086] The program may be distributed or provided in whole or in part via a communications network. A user may download the program via a communications network and install it on a computer, or install it from a recording medium onto a computer, thereby implementing the assistance device of the present invention.

[0087] The steps in the flowcharts in this specification may be executed in a different order, may be executed simultaneously, or may be executed in a different order, as long as this does not contradict the nature of the steps.

[0088] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0089] 1. 1A-1D Endoscopy support device 2. Endoscope system 5...Storage device 6. Server 7. Monitor 8. Video processor 9. Endoscopy 9A Imaging unit 9B...Tip 9C··Bent pipe 9D··Soft tube 9E···Operation unit 9F··Universal Code 10 processors 11. AI Processing Unit 12. Navigation section 13. Information Department 14. Start control section 20 Memory

Claims

1. a memory and a processor, The memory includes: a trained model trained using a training dataset in which the name of the region, the direction of view, and the axial direction are annotated for each of a plurality of endoscopic images of the stomach; storing at least one target body part name and a positional relationship of a plurality of body parts; The processor: an AI processing unit that infers the part name, the field of view direction, and the axial direction of the captured image by inputting the captured image into the trained model; an endoscopy support device comprising: a navigation unit that outputs the direction of the target area in the captured image based on the positional relationship of the plurality of areas, and the area name, the field of view direction, and the axial direction inferred by the AI ​​processing unit.

2. 2. The endoscopic examination support device according to claim 1, wherein the viewing direction is an upward direction toward the cardia or a downward direction toward the pylorus.

3. 3. The endoscopic examination support device according to claim 2, wherein the axial direction is at least one of a first axial direction from the greater curvature to the lesser curvature and a second axial direction from the front wall to the rear wall.

4. The endoscopic examination support device according to claim 1, wherein the processor includes a notification unit that outputs the direction of the target region so that a mark is displayed around the captured image displayed on a monitor.

5. 2. The endoscopic examination support device according to claim 1, wherein the target region includes at least one of the posterior wall of the angle of the stomach and the lesser curvature of the cardia.

6. The memory stores the part names that the AI ​​processing unit has inferred up to that point, The endoscopic examination support device according to claim 1, wherein the processor stores the captured image in the memory when the name of the target region is not stored in the memory.

7. the memory stores names of adjacent parts adjacent to the target part as the positional relationship of the plurality of parts; The endoscopic examination support device according to claim 6, wherein the processor has a startup control unit that controls the AI ​​processing unit to infer the direction of the target area only when the captured image is an image of the adjacent area.

8. The trained model includes a first trained model trained using a first training data set in which the body part name and the viewing direction are annotated, and a second trained model trained using a second training data set in which the axial direction is annotated; The endoscopic examination support device according to claim 3, characterized in that the AI ​​processing unit includes a first AI processing unit that infers the part name and the field of view direction by inputting the captured image into the first trained model, and a second AI processing unit that infers the axial direction by inputting the captured image into the second trained model.

9. The second trained model has a plurality of models in which the axis direction is annotated for each of the part names, The endoscopic examination support device according to claim 8, characterized in that the second AI processing unit infers the axial direction by inputting the captured image, the part name inferred by the first AI processing unit, and the field of view direction into the second trained model.

10. The trained model includes a third trained model trained using a training dataset annotated with image quality for the endoscopic image, The AI ​​processing unit includes a third AI processing unit that infers the image quality of the captured image by inputting the captured image to the third trained model, The endoscopic examination support device according to claim 8, wherein the captured image is stored when the image quality of the captured image satisfies a predetermined quality.

11. The trained model is a fourth trained model trained using a fourth training dataset in which the site names are annotated for the endoscopic images capturing the inside of the stomach, a fifth trained model trained using a fifth training dataset in which the viewing direction is annotated for the endoscopic image; a sixth trained model trained using a sixth training dataset in which at least one of a first axis direction from the greater curvature to the lesser curvature and a second axis direction from the anterior wall to the posterior wall is annotated for the endoscopic image; The AI ​​processing unit The endoscopic examination support device according to claim 1, characterized in that the direction of the target region is inferred using the fourth trained model, the fifth trained model, and the sixth trained model.

12. The captured image is input to a trained model trained using a training data set in which a part name, a viewing direction, and an axial direction are annotated for each of a plurality of endoscopic images of the stomach, and the captured image is inferred to include the part name, the viewing direction, and the axial direction; An operating method for an endoscopic examination support device, characterized by outputting the direction of a specified target part in the captured image based on the positional relationship of multiple parts stored in advance and the inferred part name, field of view direction, and axial direction of the captured image.

13. The captured image is input to a trained model trained using a training data set in which a part name, a viewing direction, and an axial direction are annotated for each of a plurality of endoscopic images of the stomach, and the captured image is inferred to include the part name, the viewing direction, and the axial direction; An endoscopic examination support program that causes a computer to execute the following: outputting the direction of a specified target area in the captured image based on the positional relationship of multiple areas that is stored in advance and the inferred area name, field of view direction, and axial direction of the captured image.

Images