Image processing device, display device, endoscope device, image processing method, image processing program, trained model, trained model generation method, and trained model generation program

Abstract

This image processing device comprises a processor. In accordance with a trained model obtained by machine learning based on positional relationships between a plurality of divided regions obtained by dividing an image obtained by imaging a tubular organ using a camera provided to an endoscope scope and a lumen-corresponding region included in the image, the processor acquires from the image a lumen direction that is a direction for inserting the endoscope scope, and outputs lumen direction information which pertains to information indicating the lumen direction.

Description

Image processing device, display device, endoscope device, image processing method, image processing program, trained model, trained model generation method, and trained model generation program

[0001] The technology disclosed herein relates to an image processing device, a display device, an endoscope device, an image processing method, an image processing program, a trained model, a trained model generation method, and a trained model generation program.

[0002] Japanese Patent No. 4077716 discloses an endoscope insertion direction detecting device, which comprises an image input means for inputting an endoscopic image from an endoscope inserted into a body cavity, a pixel extraction means for extracting pixels of a predetermined density value from the endoscopic image input by the image input means or for extracting pixels forming the endoscopic image whose density value has a predetermined gradient with respect to neighboring pixels, a region shape estimation means for determining the shape of a specific region formed by the pixels extracted by the pixel extraction means, and an insertion direction determination means for determining the insertion direction of the endoscope into the body cavity from the shape of the specific region determined by the region shape estimation means.

[0003] Japanese Patent No. 5687583 discloses a method for detecting an endoscope insertion direction, which includes a first step of inputting an endoscopic image, a first detection step of performing processing to detect the insertion direction of the endoscope based on the endoscopic image, based on one of the brightness gradient in the endoscopic image, the shape of halation in the endoscopic image, and the field of view movement of the endoscopic image, a determination step of determining whether the insertion direction of the endoscope was detected in the first detection step, and a second detection step of performing processing different from the first detection step to detect the insertion direction of the endoscope based on the endoscopic image, based on one of the brightness gradient in the endoscopic image, the shape of halation in the endoscopic image, and the field of view movement of the endoscopic image, which is different from the processing performed in the first detection step, if it is determined in the determination step that the insertion direction of the endoscope cannot be detected.

[0004] One embodiment of the technology disclosed herein provides an image processing device, a display device, an endoscopic device, an image processing method, an image processing program, a trained model, a trained model generation method, and a trained model generation program that realize the output of accurate lumen direction information.

[0005] A first aspect of the technology of the present disclosure is an image processing device that includes a processor, which acquires a lumen direction, which is the direction for inserting an endoscope, from an image in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided regions into which an image obtained by imaging a tubular organ with a camera provided on an endoscope is divided and a lumen-corresponding region included in the image, and outputs lumen direction information, which is information indicating the lumen direction.

[0006] A second aspect of the technique of the present disclosure is the image processing device according to the first aspect, in which the lumen corresponding region is a region of a predetermined range that includes a lumen region in the image.

[0007] A third aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which the lumen corresponding region is the end of the observation range of the camera in the direction in which the position of the lumen region is estimated from the fold region in the image.

[0008] A fourth aspect of the technology of the present disclosure is an image processing device according to any one of the first to third aspects, in which the direction of a divided area that overlaps with a lumen corresponding area among the multiple divided areas is the lumen direction.

[0009] A fifth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourth aspects, in which the trained model is a data structure configured to cause a processor to estimate the position of a lumen region based on the shape and / or orientation of a fold region in an image.

[0010] A sixth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fifth aspects, in which the lumen direction is the direction in which, among multiple divided regions, the divided region with the largest area overlapping with the lumen corresponding region in the image is located.

[0011] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to fifth aspects, in which the lumen direction is the direction in which the first divided area exists, which is the divided area among the multiple divided areas that has the largest area overlapping with the lumen corresponding area in the image, and the direction in which the second divided area exists, which is the divided area that has the next largest area overlapping with the lumen corresponding area after the first divided area.

[0012] An eighth aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which the divided area has a central area of ​​the image and multiple radial areas that radiate from the central area toward the outer edge of the image.

[0013] A ninth aspect according to the technique of the present disclosure is the image processing device according to the eighth aspect, in which there are eight radial regions arranged radially.

[0014] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which the divided area has a central area of ​​the image and multiple peripheral areas located closer to the outer edge of the image than the central area.

[0015] An eleventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which the divided area is obtained by dividing the image into areas in three or more directions starting from the center of the image toward the outer edge of the image.

[0016] A twelfth aspect of the technology of the present disclosure is an image processing device relating to any one of the first to seventh aspects, in which the divided area has a central area of ​​the image and multiple peripheral areas located on the outer edge side of the image than the central area, and the peripheral areas are obtained by dividing the area on the outer edge side of the image than the central area in three or more directions from the central area toward the outer edge of the image.

[0017] A thirteenth aspect of the technology of the present disclosure is a display device that displays information corresponding to lumen direction information output by a processor of an image processing device relating to any one of the first to twelfth aspects.

[0018] A fourteenth aspect of the technique of the present disclosure is an endoscope apparatus including the image processing device according to any one of the first to twelfth aspects and an endoscope.

[0019] A fifteenth aspect of the technology of the present disclosure is an image processing method that includes obtaining a lumen direction, which is the direction for inserting an endoscope, from an image in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided regions into which an image obtained by imaging a tubular organ with a camera provided on an endoscope is divided and a lumen-corresponding region included in the image, and outputting lumen direction information that is information indicating the lumen direction.

[0020] A sixteenth aspect of the technology of the present disclosure is an image processing program for causing a first computer to perform image processing, including obtaining a lumen direction, which is the direction for inserting an endoscope, from an image in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided regions into which an image obtained by imaging a tubular organ with a camera provided on an endoscope is divided and a lumen-corresponding region included in the image, and outputting lumen direction information, which is information indicating the lumen direction.

[0021] A seventeenth aspect of the technology of the present disclosure is a trained model obtained by machine learning based on the positional relationship between a plurality of divided regions into which an image obtained by imaging a tubular organ with a camera mounted on an endoscope is divided and a lumen-corresponding region included in the image.

[0022] An eighteenth aspect of the technology of the present disclosure is a trained model generation method that includes acquiring an image of a tubular organ captured by a camera mounted on an endoscope, and performing machine learning on the model based on the positional relationship between a plurality of divided regions into which the image is divided and a lumen-corresponding region included in the image.

[0023] A nineteenth aspect of the technology of the present disclosure is a trained model generation program for causing a second computer to execute a trained model generation process, which includes acquiring an image of a tubular organ captured by a camera mounted on an endoscope, and performing machine learning on the model based on the positional relationship between a plurality of divided regions into which the image is divided and a lumen-corresponding region included in the image.

[0024] 1 is a conceptual diagram showing an example of an aspect in which an endoscopic system is used. FIG. 1 is a conceptual diagram showing an example of the overall configuration of an endoscopic system. FIG. 2 is a block diagram showing an example of the hardware configuration of an endoscopic device. FIG. 3 is a block diagram showing an example of the configuration of an endoscopic device. FIG. 4 is a block diagram showing an example of the hardware configuration of an information processing device. FIG. 5 is a conceptual diagram showing an example of the processing content of a calculation unit of an information processing device. FIG. 6 is a conceptual diagram showing an example of the processing content of a calculation unit of an information processing device. FIG. 7 is a conceptual diagram showing an example of the processing content of a teacher data generation unit and a learning execution unit of an information processing device. FIG. 8 is a conceptual diagram showing an example of the processing content of a luminal direction estimation unit of a control device. FIG. 9 is a conceptual diagram showing an example of the processing content of a luminal direction estimation unit, an information generation unit, and a display control unit of a control device. FIG. 10 is a conceptual diagram showing an example of the processing content of a luminal direction estimation unit, an information generation unit, and a display control unit of a control device. 10A and 10B are conceptual diagrams showing an example of processing contents of a calculation unit according to a second modified example and a third modified example, respectively;

[0025] Below, examples of embodiments of an image processing device, a display device, an endoscope device, an image processing method, an image processing program, a trained model, a trained model generation method, and a trained model generation program relating to the technology of the present disclosure will be described with reference to the attached drawings.

[0026] First, the terms used in the following description will be explained.

[0027] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". CNN is an abbreviation for "Convolutional neural network". AI is an abbreviation for "Artificial Intelligence".

[0028] 1 , an endoscopic system 10 includes an endoscopic device 12. The endoscopic device 12 is used by a doctor 14 during an endoscopic examination. At least one assistant staff member 16 (e.g., a nurse) assists the doctor 14 in the endoscopic examination. Hereinafter, when it is not necessary to distinguish between the doctor 14 and the assistant staff member 16, they will also be referred to as "users" without being assigned reference numerals.

[0029] The endoscopic device 12 includes an endoscope 18 and is a device for performing medical examination on the inside of a subject 20 (e.g., a patient) via the endoscope 18. The endoscopic device 12 is an example of an "endoscopic device" according to the technology of the present disclosure.

[0030] The endoscope 18 captures images of the interior of the body of the subject 20 using a camera 38 (see FIG. 2 ), which will be described later, to obtain images showing the state inside the body. The endoscope 38 then outputs the images showing the state inside the body. The example shown in FIG. 1 illustrates a state in which the endoscope 18 is inserted into a body cavity from the anus of the subject 20. Note that in the example shown in FIG. 1 , the endoscope 18 is inserted into a body cavity from the anus of the subject 20, but this is merely an example, and the endoscope 18 may also be inserted into a body cavity from the mouth, nostril, perforation, or the like of the subject 20. The location at which the endoscope 18 is inserted is determined by the type of endoscope 18, the surgical procedure in which the endoscope 18 is used, and the like.

[0031] The display device 22 displays various information including images. Examples of the display device 22 include a liquid crystal display and an EL display. The display device 22 displays a plurality of screens arranged side by side. In the example shown in FIG. 1 , screens 24 and 26 are shown as examples of the plurality of screens. The display device 22 is an example of a "display device" according to the technology of the present disclosure.

[0032] An endoscopic image 28 is displayed on the screen 24. The endoscopic image 28 is an image acquired by capturing an image of an observation target region within the body cavity of the subject 20 using a camera 38 (see FIG. 2 ) provided on the endoscope 18. An example of the observation target region is the inner wall of the large intestine. The inner wall of the large intestine is merely one example, and the observation target region may be the inner or outer wall of another region, such as the small intestine, duodenum, or stomach.

[0033] The endoscopic image 28 displayed on the screen 24 is one frame included in a moving image that includes multiple frames. That is, the screen 24 displays multiple frames of the endoscopic image 28 at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second).

[0034] The screen 26 displays, for example, subject identification information 29. The subject identification information 29 is information related to the subject 20. The subject identification information 29 includes, for example, the name of the subject 20, the age of the subject 20, and an identification number by which the subject 20 can be identified.

[0035] As shown in FIG. 2 , the endoscope 18 includes an operating unit 32 and an insertion section 34. The operating unit 32 includes a rotary operating knob 32A, an air / water supply button 32B, and a suction button 32C. The insertion section 34 is tubular. The outer contour of the insertion section 34 is circular in cross section. The insertion section 34 is partially curved or rotated about its axis by operating the rotary operating knob 32A of the operating unit 32. As a result, the insertion section 34 is delivered deep into the body while bending according to the shape of the body (e.g., the shape of a tubular organ) or rotating about its axis according to the location within the body. Furthermore, operating the air / water supply button 32B delivers water or air into the body from the distal end 36, and operating the suction button 32C suctions water or air from the body.

[0036] The distal end 36 is provided with a camera 38, an illumination device 40, and a treatment tool opening 42. The camera 38 captures images of the inside of a tubular organ using an optical method. An example of the camera 38 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 38 is an example of a "camera" according to the technology of the present disclosure.

[0037] The illumination device 40 has an illumination window 40A and an illumination window 40B. The illumination device 40 emits light through the illumination window 40A and the illumination window 40B. Examples of the type of light emitted from the illumination device 40 include visible light (e.g., white light), invisible light (e.g., near-infrared light), and / or special light. Examples of the special light include light for BLI and / or light for LCI.

[0038] The treatment tool opening 42 is an opening for allowing a treatment tool to protrude from the distal end portion 36. The treatment tool opening 42 also functions as a suction port for sucking blood, internal waste, etc. The treatment tool is inserted into the insertion portion 34 from the treatment tool insertion port 45. The treatment tool passes through the insertion portion 34 and protrudes to the outside from the treatment tool opening 42. Examples of treatment tools include a puncture needle, a wire, a scalpel, grasping forceps, a guide sheath, and an ultrasonic probe.

[0039] The endoscope device 12 includes a control device 46 and a light source device 48. The endoscope 18 is connected to the control device 46 and the light source device 48 via a cable 50. The control device 46 is a device that controls the entire endoscope device 12. The light source device 48 is a device that emits light under the control of the control device 46 and supplies the light to the illumination device 40.

[0040] The control device 46 is provided with a plurality of hard keys 52. The plurality of hard keys 52 accept instructions from the user. The screen of the display device 22 is provided with a touch panel 54. The touch panel 54 is electrically connected to the control device 46 and accepts instructions from the user. The display device 22 is also electrically connected to the control device 46.

[0041] 3, the control device 46 includes a computer 56. The computer 56 is an example of an "image processing device" and a "first computer" according to the techniques of the present disclosure. The computer 56 includes a processor 58, a RAM 60, and an NVM 62, which are electrically connected to each other. The processor 58 is an example of a "processor" according to the techniques of the present disclosure.

[0042] The control device 46 includes a hard key 52 and an external I / F 64. The hard key 52, the processor 58, the RAM 60, the NVM 62, and the external I / F 64 are connected to a bus 65.

[0043] For example, the processor 58 has a CPU and a GPU, and controls the entire control device 46. The GPU operates under the control of the CPU and is responsible for executing various graphic-related processes. The processor 58 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0044] The RAM 60 is a memory that temporarily stores information and is used as a work memory by the processor 58. The NVM 62 is a non-volatile storage device that stores various programs, various parameters, and the like. An example of the NVM 62 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 62 may be another non-volatile storage device such as an HDD, or may be a combination of two or more types of non-volatile storage devices.

[0045] The hard keys 52 receive instructions from the user and output a signal indicating the received instruction to the processor 58. In this way, the instruction received by the hard keys 52 is recognized by the processor 58.

[0046] The external I / F 64 controls the exchange of various information between devices that exist outside the control device 46 (hereinafter also referred to as "external devices") and the processor 58. An example of the external I / F 64 is a USB interface.

[0047] The external I / F 64 is connected to the endoscope 18 as one of the external devices, and the external I / F 64 controls the exchange of various information between the endoscope 18 and the processor 58. The processor 58 controls the endoscope 18 via the external I / F 64. The processor 58 also acquires, via the external I / F 64, endoscopic images 28 (see FIG. 1 ) obtained by capturing images of the inside of a tubular organ with the camera 38.

[0048] The light source device 48 is connected to the external I / F 64 as one of the external devices, and the external I / F 64 controls the exchange of various information between the light source device 48 and the processor 58. The light source device 48 supplies light to the illumination device 40 under the control of the processor 58. The illumination device 40 irradiates the light supplied from the light source device 48.

[0049] A display device 22 is connected to the external I / F 64 as one of the external devices, and the processor 58 controls the display device 22 via the external I / F 76 to display various information on the display device 22.

[0050] The external I / F 64 is connected to the touch panel 54 as one of the external devices, and the processor 58 acquires instructions received by the touch panel 54 via the external I / F 64 .

[0051] An information processing device 66 is connected as one of the external devices to the external I / F 64. A server is an example of the information processing device 66. Note that the server is merely an example, and the information processing device 66 may also be a personal computer.

[0052] The external I / F 64 controls the exchange of various information between the information processing device 66 and the processor 58. The processor 58 requests the information processing device 66 to provide a service via the external I / F 64, and acquires a trained model 116 (see FIG. 4 ) from the information processing device 66 via the external I / F 64.

[0053] When observing the inside of a tubular organ (e.g., the large intestine) in the body using the camera 38 provided on the endoscope 18, the endoscope 18 is inserted along the lumen. In this case, it may be difficult for the user to determine the lumen direction, which is the direction in which the endoscope 18 is inserted. Furthermore, if the endoscope 18 is inserted in a direction different from the lumen direction, the endoscope 18 will hit the inner wall of the tubular organ, which may impose unnecessary strain on the subject 20 (e.g., a patient).

[0054] In view of these circumstances, in this embodiment, endoscopic image processing is performed by the processor 58 of the control device 46. As an example, as shown in FIG. 4 , an endoscopic image processing program 62A is stored in the NVM 62. The processor 58 reads the endoscopic image processing program 62A from the NVM 62 and executes the read endoscopic image processing program 62A on the RAM 60. The endoscopic image processing is realized by the processor 58 operating as a lumen direction estimation unit 58A, an information generation unit 58B, and a display control unit 58C in accordance with the endoscopic image processing program 62A executed on the RAM 60.

[0055] As an example, as shown in Figure 5, machine learning processing is performed by a processor 78 (see Figure 5) of an information processing device 66. The information processing device 66 is a device used for machine learning. The information processing device 66 is used by an annotator 76 (see Figure 6). The annotator 76 refers to an operator who adds annotations for machine learning to given data (i.e., an operator who performs labeling).

[0056] The information processing device 66 includes a computer 70, a reception device 72, a display 74, and an external I / F 76. The computer 70 is an example of a "second computer" according to the technology of the present disclosure.

[0057] The computer 70 includes a processor 78, an NVM 80, and a RAM 82. The processor 78, the NVM 80, and the RAM 82 are connected to a bus 84. The reception device 72, the display 74, and the external I / F 76 are also connected to the bus 84.

[0058] The processor 78 controls the entire information processing device 66. The processor 78, the NVM 80, and the RAM 82 are hardware resources similar to the processor 58, the NVM 62, and the RAM 60 described above.

[0059] The reception device 72 receives instructions from the annotator 76. The processor 78 operates in accordance with the instructions received by the reception device 72.

[0060] The external I / F 76 is a hardware resource similar to the above-described external I / F 64. The external I / F 76 is connected to the external I / F 64 of the endoscope device 12 and controls the exchange of various information between the endoscope device 12 and the processor 78.

[0061] A machine learning processing program 80A is stored in the NVM 80. The processor 78 reads the machine learning processing program 80A from the NVM 80 and executes the read machine learning processing program 80A on the RAM 82. The processor 78 performs machine learning processing in accordance with the machine learning processing program 80A executed on the RAM 82. The machine learning processing is realized by the processor 78 operating as a calculation unit 86, a teacher data generation unit 88, and a learning execution unit 90 in accordance with the machine learning processing program 80A. The machine learning processing program 80A is an example of a "trained model generation program" according to the technology of the present disclosure.

[0062] As shown in FIG. 6 as an example, the calculation unit 86 first displays the endoscopic image 28 on the display 74. Here, the endoscopic image 28 is, for example, an image acquired during a previous examination and / or treatment and pre-stored in the NVM 80, but this is merely an example. The endoscopic image 28 may be an image stored in an external device, such as an image server (not shown), and acquired via the external I / F 76 (see FIG. 5). With the endoscopic image 28 displayed on the display 74, the annotator 76 specifies a lumen-corresponding region 94 in the endoscopic image 28 to the computer 70 via the reception device 72 (e.g., keyboard 72A and / or mouse 72B). For example, the annotator 76 specifies a lumen region 28A in the endoscopic image 28 displayed on the display 74 using a pointer (not shown). Here, the lumen region 28A refers to an image region in the endoscopic image 28 that indicates a lumen.

[0063] The calculation unit 86 recognizes a lumen-corresponding region 94 specified by the annotator 76 via the reception device 72. Here, the lumen-corresponding region 94 is a region within a predetermined range (e.g., a range of a 64-pixel radius from the center of the lumen region 28A) that includes the lumen region 28A in the endoscopic image 28. The lumen-corresponding region 94 is an example of a "lumen-corresponding region" according to the technology of the present disclosure. Furthermore, the calculation unit 86 virtually divides the endoscopic image 28 to obtain multiple divided regions 96. The divided regions 96 are an example of a "divided region" according to the technology of the present disclosure. For example, the lumen-corresponding region 94 is a region that includes the lumen region 28A in the endoscopic image 28 and is large enough to be inscribed in a divided region 96, which will be described later.

[0064] In the example shown in FIG. 6 , the endoscopic image 28 is divided into a central region 96A and eight radial regions 96B. The central region 96A is, for example, a circular region centered at the center C of the endoscopic image 28. The radial regions 96B are regions that exist radially from the central region 96A toward the outer edge of the endoscopic image 28. Although eight radial regions 96B are shown here, this is merely an example. For example, the number of radial regions 96B may be seven or less, or nine or more. The central region 96A is an example of a "central region" according to the technology of the present disclosure, and the radial regions 96B are an example of a "radial region" according to the technology of the present disclosure.

[0065] The calculation unit 86 determines the direction of the divided region 96 that overlaps with the lumen-corresponding region 94 among the multiple divided regions 96 as the lumen direction. Specifically, the calculation unit 86 derives the divided region 96 that has the largest area of ​​overlap with the lumen-corresponding region 94 among the multiple divided regions 96. For example, the calculation unit 86 identifies the region where each of the multiple divided regions 96 overlaps with the lumen-corresponding region 94. The calculation unit 86 calculates the area of ​​the region where the divided region 96 overlaps with the lumen-corresponding region 94. Then, the calculation unit 86 identifies the divided region 96 where the region where the divided region 96 overlaps with the lumen-corresponding region 94 has the largest area.

[0066] The calculation unit 86 determines the direction of the divided region 96 that has the largest overlapping area with the lumen-corresponding region 94 as the lumen direction, and generates this as the correct answer data 92. In the example shown in Fig. 6, a second region 96B1 of the radial region 96B is shown as an example of the correct answer data 92. The second region 96B1 is a region that indicates the lumen direction (i.e., the direction for inserting the camera 38).

[0067] Here, an example in which the luminal region 28A is captured in the endoscopic image 28 has been described, but the technology of the present disclosure is not limited to this. For example, the luminal region 28A may not be captured in the endoscopic image 28. In this case, as shown in FIG. 7 as an example, the annotator 76 estimates the luminal region 28A by referring to the position and / or shape of a fold region 28B in the endoscopic image 28 displayed on the display 74. Here, the fold region 28B refers to an image region in the endoscopic image 28 that represents the folds within a tubular organ. Then, the end of the observation range in the endoscopic image 28 is designated as a lumen-corresponding region 94 using a pointer (not shown).

[0068] The calculation unit 86 derives, from among the multiple divided regions 96, the divided region 96 that has the largest area overlapping with the lumen-corresponding region 94. Then, the calculation unit 86 generates the divided region 96 that has the largest area overlapping with the lumen-corresponding region 94 as the correct answer data 92. In the example shown in Fig. 7, a seventh region 96B3 of the radial region 96B is shown as an example of the correct answer data 92. The seventh region 96B3 is a region that indicates the lumen direction.

[0069] 8 , the teacher data generation unit 88 acquires an endoscopic image 28 as an inference image from the calculation unit 86, and generates teacher data 95 by linking the acquired endoscopic image 28 with supervised answer data 92. The learning execution unit 90 acquires the teacher data 95 generated by the teacher data generation unit 88. Then, the learning execution unit 90 executes machine learning using the teacher data 95.

[0070] In the example shown in FIG. 8 , the learning execution unit 90 includes a CNN 110. The learning execution unit 90 inputs endoscopic images 28 included in the teacher data 95 to the CNN 110. Note that, while an example in which the endoscopic images 28 are input to the CNN 110 one by one has been described here, the technology of the present disclosure is not limited to this. Multiple frames (e.g., two to three frames) of the endoscopic images 28 may be input to the CNN 110 at once. When the endoscopic images 28 are input, the CNN 110 performs inference and outputs a CNN signal 110A indicating the inference result (e.g., an image area predicted as an image area indicating the direction in which a lumen exists among all image areas constituting the endoscopic image 28). The learning execution unit 90 calculates an error 112 between the CNN signal 110A and the correct answer data 92 included in the teacher data 95.

[0071] The learning execution unit 90 optimizes the CNN 110 by adjusting a plurality of optimization variables in the CNN 110 so as to minimize the error 112. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in the CNN 110.

[0072] The learning execution unit 90 repeatedly performs a learning process using multiple teacher data 95, including inputting endoscopic images 28 to the CNN 110, calculating the error 112, and adjusting multiple optimization variables in the CNN 110. That is, the learning execution unit 90 optimizes the CNN 110 by adjusting multiple optimization variables in the CNN 110 so as to minimize the error 112 for each of the multiple endoscopic images 28 included in the multiple teacher data 95. By optimizing the CNN 110 in this manner, a trained model 116 is generated. The trained model 116 is stored in a storage device by the learning execution unit 90. An example of the storage device is the NVM 62 of the endoscopic device 12, but this is merely an example. The storage device may also be the NVM 80 of the information processing device 66. The trained model 116 stored in a default storage device is used, for example, in the lumen direction estimation process in the endoscopic device 12. The trained model 116 is an example of a "trained model" according to the technology of the present disclosure.

[0073] As an example, as shown in FIG. 9 , the endoscopic device 12 performs a luminal direction estimation process using a trained model 116 generated by an information processing device 66. First, endoscopic images 28 are obtained by capturing images of the inside of a tubular organ in time series using the camera 38. The endoscopic images 28 are temporarily stored in the RAM 60. The luminal direction estimation unit 58A performs a luminal direction estimation process based on the endoscopic images 28. In this case, the luminal direction estimation unit 58A acquires the trained model 116 from the NVM 62. The luminal direction estimation unit 58A then inputs the endoscopic images 28 to the trained model 116. Upon receiving the endoscopic images 28, the trained model 116 outputs an estimation result 118 of the luminal direction in the endoscopic images 28. The estimation result 118 is, for example, the probability that a luminal direction exists for each divided region 96. The trained model 116 outputs a probability distribution p indicating nine probabilities corresponding to the nine divided regions 96 as the estimation result 118.

[0074] As an example, as shown in FIG. 10 , the lumen direction estimation unit 58A outputs an estimation result 118 to the information generation unit 58B. The information generation unit 58B generates lumen direction information 120 based on the estimation result 118. The lumen direction information 120 is information indicating the lumen direction. The lumen direction information 120 is also an example of "lumen direction information" according to the technology of the present disclosure. For example, the information generation unit 58B generates the lumen direction information 120 by setting the direction of the divided region 96 showing the highest probability value in the probability distribution p indicated by the estimation result 118 as the lumen direction. The information generation unit 58B outputs the lumen direction information 120 to the display control unit 58C.

[0075] The display control unit 58C acquires the endoscopic image 28 temporarily stored in the RAM 60. Furthermore, the display control unit 58C generates an image 122 in which the lumen direction indicated by the lumen direction information 120 is superimposed on the endoscopic image 28. The display control unit 58C causes the display device 22 to display the image 122. In the example shown in Fig. 10 , an arc 122A is displayed on the outer periphery of the observation range of the endoscopic image 28 within the image 122 as an indication of the lumen direction.

[0076] As an example, as shown in FIG. 11 , the image 122 displayed on the display device 22 is updated each time an endoscopic image 28 is acquired. Specifically, the luminal direction estimation unit 58A performs luminal direction estimation processing (see FIG. 10 ) each time an endoscopic image 28 is acquired from the camera 38. The luminal direction estimation unit 58A then outputs an estimation result 118 obtained by the luminal direction estimation processing to the information generation unit 58B. The information generation unit 58B generates luminal direction information 120 based on the estimation result 118. The display control unit 58C causes the display device 22 to update the image 122 based on the luminal direction information 120 and the endoscopic image 28 acquired from the camera 38. As a result, the display indicating the luminal direction in the image 122 changes according to the luminal direction in the endoscopic image 28. In the example shown in FIG. 11 , the luminal direction moves in the endoscopic image 28 from the left to the center and then to the right as viewed from the front side of the page. 11 shows an example in which the image 122 is updated in the order of arc 122A, X-shaped display 122B, and arc 122C as displays indicating the lumen direction. Note that, although an example in which arcs 122A and 122C and X-shaped display 122B are used as displays indicating the lumen direction has been described here, the technology of the present disclosure is not limited thereto. For example, a symbol such as an arrow or text such as "upper right" may be used as a display indicating the lumen direction. Instead of or in addition to displaying the lumen direction on the display device 22, the lumen direction may be notified by voice.

[0077] Next, the operation of the information processing device 66 will be described with reference to Fig. 12. Fig. 12 shows an example of the flow of machine learning processing performed by the processor 78. The flow of machine learning processing shown in Fig. 12 is an example of a "trained model generation method" according to the technology of the present disclosure.

[0078] 12, first, in step ST110, the calculation unit 86 displays the endoscopic image 28 on the display 74. After the processing of step ST110 is executed, the machine learning processing proceeds to step ST112.

[0079] In step ST112, the calculation unit 86 accepts the designation of the lumen-corresponding region 94 for the endoscopic image 28 displayed on the display 74 in step ST110, input by the annotator 76 via the reception device 72. After the processing of step ST112 is executed, the machine learning processing proceeds to step ST114.

[0080] In step ST114, the calculation unit 86 generates the correct answer data 92 based on the positional relationship between the lumen-corresponding region 94 received in step ST112 and the divided region 96. After the processing of step ST114 is executed, the machine learning processing proceeds to step ST116.

[0081] In step ST116, the teacher data generation unit 88 generates teacher data 95 by linking the supervised answer data 92 generated in step ST114 with the endoscopic image 28. After the processing of step ST116 is executed, the machine learning processing proceeds to step ST118.

[0082] In step ST118, the learning execution unit 90 acquires the endoscopic image 28 included in the training data 95 generated in step ST116. After the processing of step ST118 is executed, the machine learning processing proceeds to step ST120.

[0083] In step ST120, the learning execution unit 90 inputs the endoscopic image 28 acquired in step ST118 to the CNN 110. After the process of step ST120 is executed, the machine learning process proceeds to step ST122.

[0084] In step ST122, the learning execution unit 90 calculates the error 112 by comparing the CNN signal 110A obtained by inputting the endoscopic image 28 to the CNN 110 in step ST120 with the correct answer data 92 linked to the endoscopic image 28. After step ST122 is executed, the machine learning process proceeds to step ST124.

[0085] In step ST124, the learning execution unit 90 adjusts the optimization variables of the CNN 110 so as to minimize the error 112 calculated in step ST122. After step ST124 is executed, the machine learning process proceeds to step ST126.

[0086] In step ST126, the learning execution unit 90 determines whether a condition for terminating machine learning (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that the error 112 calculated in step ST124 is equal to or less than a threshold. If the termination condition has not been satisfied in step ST126, the determination is negative, and the machine learning process proceeds to step ST118. If the termination condition has been satisfied in step ST126, the determination is positive, and the machine learning process proceeds to step ST128.

[0087] In step ST128, the learning execution unit 90 outputs the trained model 116, which is the CNN 110 for which machine learning has been completed, to an external device (for example, the NVM 62 of the endoscope device 12). After step ST128 is executed, the machine learning process ends.

[0088] Next, the operation of the endoscopic device 12 will be described with reference to Fig. 13. Fig. 13 shows an example of the flow of endoscopic image processing performed by the processor 58. The flow of endoscopic image processing shown in Fig. 13 is an example of an "image processing method" according to the technology of the present disclosure.

[0089] In the endoscopic image processing shown in FIG. 13 , first, in step ST10, the luminal direction estimation unit 58A determines whether a luminal direction estimation start trigger is ON. Examples of the luminal direction estimation start trigger include whether a luminal direction estimation start instruction from a user (for example, operation of a button (not shown) on the endoscope 18) has been received. If the luminal direction estimation start trigger is not ON in step ST10, the determination is negative, and the endoscopic image processing proceeds to step ST10 again. If the luminal direction estimation start trigger is ON in step ST10, the determination is positive, and the endoscopic image processing proceeds to step ST12. Note that, although an example in which a determination is made in step ST10 as to whether the luminal direction estimation start trigger is ON has been described, the technology of the present disclosure is not limited thereto. The technology of the present disclosure also applies to an embodiment in which the determination in step ST10 is omitted and luminal direction estimation processing is always performed.

[0090] In step ST12, the lumen direction estimation unit 58A acquires the endoscopic image 28 from the RAM 60. After the processing of step ST12 is executed, the endoscopic image processing proceeds to step ST14.

[0091] In step ST14, the lumen direction estimation unit 58A starts estimating the lumen direction in the endoscopic image 28 using the trained model 116. After the processing of step ST14 is executed, the endoscopic image processing proceeds to step ST16.

[0092] In step ST16, the lumen direction estimation unit 58A determines whether or not estimation of the lumen direction has been completed. If estimation of the lumen direction has not been completed in step ST16, the determination is negative, and the endoscopic image processing proceeds to step ST16 again. If estimation of the lumen direction has been completed in step ST16, the determination is positive, and the endoscopic image processing proceeds to step ST18.

[0093] In step ST18, the information generator 58B generates lumen direction information 120 based on the estimation result 118 obtained in step ST16. After the processing of step ST18 is executed, the endoscopic image processing proceeds to step ST20.

[0094] In step ST20, the display control unit 58C outputs the lumen direction information 120 generated in step ST18 to the display 74. After the processing of step ST20 is executed, the endoscopic image processing proceeds to step ST22.

[0095] In step ST22, the display control unit 58C determines whether a condition for terminating the endoscopic image processing (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the endoscopic image processing has been received via the touch panel 54. If the termination condition has not been satisfied in step ST22, the determination is negative, and the endoscopic image processing proceeds to step ST12. If the termination condition has been satisfied in step ST22, the determination is positive, and the endoscopic image processing terminates.

[0096] In step ST10, the luminal direction estimation start trigger has been described as being, for example, whether or not a luminal direction estimation start instruction from a user (for example, operation of a button (not shown) on the endoscope 18) has been received. However, the technology of the present disclosure is not limited to this. The luminal direction estimation start trigger may also be whether or not it has been detected that the endoscope 18 has been inserted into a tubular organ. When it has been detected that the endoscope 18 has been inserted, the luminal direction estimation start trigger is turned ON. In this case, the processor 58 detects whether or not the endoscope 18 has been inserted into the tubular organ, for example, by performing image recognition processing using AI on the endoscopic image 28. Another luminal direction estimation start trigger may also be whether or not a specific site within the tubular organ has been recognized. When the specific site has been detected, the luminal direction estimation start trigger is turned ON. In this case, the processor 58 also detects whether or not the specific site has been detected, for example, by performing image recognition processing using AI on the endoscopic image 28.

[0097] Although the embodiment has been described in which the termination condition in step ST22 is that an instruction to terminate endoscopic image processing is received via the touch panel 54, the technology of the present disclosure is not limited to this. For example, the termination condition may be that the processor 58 detects that the endoscope 18 has been removed from the body. In this case, the processor 58 detects that the endoscope 18 has been removed from the body, for example, by performing image recognition processing using AI on the endoscopic image 28. Another termination condition may be that the processor 58 detects that the endoscope 18 has reached a specific location within a tubular organ (e.g., the ileocecal portion of the large intestine). In this case, the processor 58 detects that the endoscope 18 has reached a specific location within the tubular organ, for example, by performing image recognition processing using AI on the endoscopic image 28.

[0098] As described above, in the endoscopic device 12 according to this embodiment, the endoscopic image 28 captured by the camera 38 is input into the trained model 116 to acquire the lumen direction. The trained model 116 is obtained by machine learning processing based on the positional relationship between a plurality of segmented regions 96 obtained by segmenting an image showing a tubular organ (e.g., the large intestine) and the lumen-corresponding region 94 included in the endoscopic image 28. Furthermore, the processor 58 outputs lumen direction information 120, which is information indicating the lumen direction. Therefore, this configuration enables accurate output of the lumen direction information 120. The lumen direction information 120 is used, for example, to display the lumen direction to the user.

[0099] For example, compared to prediction of the lumen direction by image processing that applies empirical prediction of the lumen direction during a doctor's examination (e.g., prediction of the lumen direction from the arc shape of halation), this configuration makes it possible to predict the lumen direction even when using an image in a state where prediction accuracy is reduced based on empirical rules (e.g., an image without halation). Therefore, this configuration realizes the output of accurate lumen direction information 120.

[0100] Furthermore, in the endoscopic device 12 according to the present embodiment, a predetermined range including the lumen region 28A within the endoscopic image 28 is set as the lumen-corresponding region 94. The lumen direction is then estimated in accordance with a trained model 116 obtained by machine learning based on the positional relationship between the divided regions 96 and the lumen-corresponding region 94. By setting the predetermined range as the lumen-corresponding region 94, the existence of the lumen region 28A is more easily recognized in machine learning, improving the accuracy of the machine learning. Therefore, the accuracy of the estimation of the lumen direction using the trained model 116 is also improved. As a result, the processor 58 outputs highly accurate lumen direction information 120. Therefore, with this configuration, accurate lumen direction information 120 can be output.

[0101] For example, if only the lumen region 28A is used as the lumen-corresponding region 94, the lumen-corresponding region 94 will be small and dot-like in the image, and the machine learning will not accurately recognize the lumen-corresponding region 94, resulting in a decrease in the accuracy of the machine learning. On the other hand, in this configuration, the lumen-corresponding region 94 is set to a predetermined range, improving the accuracy of the machine learning. As a result, the processor 58 outputs highly accurate lumen direction information 120. Therefore, this configuration enables the output of accurate lumen direction information 120.

[0102] Furthermore, in the endoscopic device 12 according to this embodiment, the end of the observation range of the camera 38 in the direction in which the position of the lumen is estimated from the fold region 28B in the endoscopic image 28 is determined to be the lumen-corresponding region 94. The lumen direction is then estimated according to a trained model 116 obtained by machine learning based on the positional relationship between the segmented region 96 and the lumen-corresponding region 94. Because the end of the observation range of the camera 38 in the direction in which the position of the lumen is estimated from the fold region 28B is determined to be the lumen-corresponding region 94, machine learning can be performed even when the lumen region 28A is not included in the image. This increases the number of endoscopic images 28 to be used for learning, thereby improving the accuracy of machine learning. Therefore, the accuracy of the estimation of the lumen direction using the trained model 116 is also improved. As a result, the processor 58 outputs highly accurate lumen direction information 120. Therefore, this configuration enables accurate output of lumen direction information 120.

[0103] Furthermore, in the endoscopic device 12 according to the present embodiment, in the positional relationship between the lumen-corresponding region 94 and the divided region 96 in machine learning, the direction of the divided region 96 that overlaps with the lumen-corresponding region 94 is the lumen direction. The direction of the divided region 96 is determined in advance by dividing the endoscopic image 28. Therefore, with this configuration, the load in estimating the lumen direction is reduced compared to when the lumen direction is calculated each time depending on the position of the lumen-corresponding region 94.

[0104] Furthermore, in the endoscopic device 12 according to this embodiment, the trained model 116 is a data structure configured to cause the processor 58 to estimate the position of the lumen based on the shape and / or orientation of the fold region 28B. This allows the position of the lumen to be accurately estimated. Therefore, this configuration enables accurate output of lumen direction information 120.

[0105] For example, compared to prediction of the lumen direction by image processing that applies empirical prediction of the lumen direction during a doctor's examination (e.g., prediction of the lumen direction from the arc shape of halation), this configuration makes it possible to predict the lumen direction even when using an image in a state where prediction accuracy is reduced based on empirical rules (e.g., an image without halation). Therefore, this configuration realizes the output of accurate lumen direction information 120.

[0106] Furthermore, in the endoscopic device 12 according to this embodiment, the lumen direction is defined as the direction in which the divided region 96 with the largest overlapping area with the lumen-corresponding region 94 exists. A large overlapping area between the lumen-corresponding region 94 and the divided region 96 means that a lumen exists in the direction in which the divided region 96 exists. This makes it possible to uniquely determine the lumen direction in machine learning. Therefore, this configuration enables the output of accurate lumen direction information 120.

[0107] Furthermore, in the endoscopic device 12 according to this embodiment, the divided regions 96 include a central region 96A of the endoscopic image 28 and multiple radial regions 96B that radiate from the central region 96A toward the outer edge of the endoscopic image 28. The central region 96A frequently captures the lumen region 28A within the endoscopic image 28. Therefore, it is necessary to indicate the lumen direction even when a lumen is present in the central region 96A. Furthermore, by dividing the endoscopic image 28 radially, it becomes easier to indicate the direction in which the lumen exists. Dividing the endoscopic image 28 into the central region 96A and the radial regions 96B in this manner makes it easier to grasp which direction is the lumen direction. Therefore, this configuration makes it possible to indicate the lumen direction in a way that is easy for the user to understand.

[0108] Furthermore, in the endoscope device 12 according to this embodiment, eight radial regions 96B are radially arranged. The existence of eight radial regions 96B makes it easier to indicate the direction in which the lumen is located. Furthermore, the lumen direction is indicated to the user in divisions that are not too fine. Therefore, with this configuration, it is possible to indicate the lumen direction in a way that is easy for the user to understand.

[0109] Furthermore, in the endoscopic device 12 according to this embodiment, information corresponding to the lumen direction information 120 output by the processor 58 is displayed on the display device 22. Therefore, with this configuration, it becomes easier for the user to recognize the lumen direction.

[0110] Furthermore, the trained model 116 according to this embodiment is obtained by machine learning processing based on the positional relationship between a plurality of divided regions 96 obtained by dividing the endoscopic image 28 and a lumen-corresponding region 94 included in the endoscopic image 28. The trained model 116 is used to output lumen direction information 120 by the processor 58. Therefore, this configuration enables the output of accurate lumen direction information 120. The lumen direction information 120 is used, for example, to display the lumen direction to a doctor.

[0111] For example, compared to prediction of the lumen direction by endoscopic image processing that applies empirical prediction of the lumen direction during a doctor's examination (e.g., prediction of the lumen direction from the arc shape of halation), this configuration makes it possible to predict the lumen direction even when using an image in a state where prediction accuracy is reduced based on empirical rules (e.g., an image without halation). Therefore, this configuration realizes the output of accurate lumen direction information 120.

[0112] Second Embodiment In the above-described first embodiment, an example was described in which the calculation unit 86 generates, as the correct answer data 92, the direction of the divided region 96 having the largest area overlapping with the lumen-corresponding region 94. However, the technology of the present disclosure is not limited to this. In the present second embodiment, the calculation unit 86 generates, as the correct answer data 92, the direction of the divided region 96 having the largest area overlapping with the lumen-corresponding region 94 and the direction of the divided region 96 having the second largest area overlapping with the lumen-corresponding region.

[0113] 14 as an example, first, the calculation unit 86 displays the endoscopic image 28 on the display 74. With the endoscopic image 28 displayed on the display 74, the annotator 76 specifies a lumen-corresponding region 94 in the endoscopic image 28 to the computer 70 via the reception device 72 (e.g., keyboard 72A and / or mouse 72B).

[0114] The calculation unit 86 receives a designation of a lumen-corresponding region 94 in the endoscopic image 28 from the annotator 76 via the reception device 72. The calculation unit 86 virtually divides the endoscopic image 28 to obtain a plurality of divided regions 96. In the example shown in Fig. 14, the endoscopic image 28 is divided into a central region 96A and eight radial regions 96B.

[0115] The calculation unit 86 derives, from among the multiple divided regions 96, the divided region 96 having the largest area overlapping with the lumen-corresponding region 94 and the divided region 96 having the second largest area overlapping with the lumen-corresponding region 94. For example, the calculation unit 86 identifies the region where each of the multiple divided regions 96 overlaps with the lumen-corresponding region 94. The calculation unit 86 also calculates the area of ​​the region where the divided region 96 overlaps with the lumen-corresponding region 94. The calculation unit 86 then identifies the divided region 96 having the largest area where the divided region 96 overlaps with the lumen-corresponding region 94 and the divided region 96 having the second largest area. The divided region 96 having the largest area where the divided region 96 overlaps with the lumen-corresponding region 94 is an example of a "first divided region" according to the technology disclosed herein, and the divided region 96 having the second largest area is an example of a "second divided region" according to the technology disclosed herein.

[0116] In the example shown in Figure 14, the correct data 92 is an example in which the direction in which the second region 96B1 and the first region 96B2 of the radial region 96B exist is the lumen direction (i.e., the direction for inserting the camera 38).

[0117] Here, an example in which the luminal region 28A is captured in the endoscopic image 28 has been described, but the technology of the present disclosure is not limited to this. For example, the luminal region 28A may not be captured in the endoscopic image 28, as in FIG. 7 .

[0118] The teacher data generation unit 88 (see FIG. 8) acquires the endoscopic image 28 from the calculation unit 86 (see FIG. 8) and generates teacher data 95 (see FIG. 8) by linking the acquired endoscopic image 28 with the supervised answer data 92. The learning execution unit 90 acquires the teacher data 95 generated by the teacher data generation unit 88. The learning execution unit 90 then executes machine learning using the teacher data 95. The trained model 116A generated as a result of the machine learning is stored by the learning execution unit 90 in the NVM 62 of the endoscope device 12, which serves as a storage device.

[0119] 15 as an example, in the endoscopic device 12, a lumen direction estimation process is performed using a trained model 116A generated in the information processing device 66. The lumen direction estimation unit 58A performs the lumen direction estimation process based on the endoscopic image 28. The lumen direction estimation unit 58A acquires the trained model 116A from the NVM 62. The lumen direction estimation unit 58A then inputs the endoscopic image 28 to the trained model 116A. When the endoscopic image 28 is input, the trained model 116A outputs an estimation result 118A of the lumen direction in the endoscopic image 28. The estimation result 118A is, for example, a probability distribution p of whether or not a lumen direction exists for each divided region 96.

[0120] 16 as an example, the lumen direction estimation unit 58A outputs an estimation result 118 to the information generation unit 58B. The information generation unit 58B generates lumen direction information 120 based on the estimation result 118A. For example, the information generation unit 58B generates the lumen direction information 120 by setting the direction of the divided region 96 showing the highest probability distribution value and the direction of the divided region 96 showing the second highest probability distribution value in the probability distribution p shown by the estimation result 118A as the lumen direction. The information generation unit 58B outputs the lumen direction information 120 to the display control unit 58C.

[0121] The display control unit 58C generates an image 122 in which the lumen direction indicated by the lumen direction information 120 is superimposed on the endoscopic image 28. The display control unit 58C causes the display device 22 to display the image 122. In the example shown in Fig. 16 , arcs 122D and 122E are displayed on the outer periphery of the observation range of the endoscopic image 28 within the image 122 as an indication of the lumen direction.

[0122] As described above, in the endoscopic device 12 according to the present embodiment, the lumen direction is defined as the direction in which the divided region 96 with the largest overlapping area with the lumen-corresponding region 94 exists, and the direction in which the divided region 96 with the second largest overlapping area with the lumen-corresponding region 94 exists. A large overlapping area between the lumen-corresponding region 94 and the divided region 96 means that there is a high possibility that a lumen exists in the direction in which the divided region 96 exists. This makes it possible to determine a direction in which the lumen direction is likely to exist through machine learning. Therefore, with this configuration, it is possible to output lumen direction information 120 in which there is a high possibility that the lumen direction exists.

[0123] (First Modification) In the second embodiment, an example has been described in which the estimation result 118A output from the trained model 116A is used as is to generate the lumen direction information 120. However, the technology of the present disclosure is not limited to this. The estimation result 118A may be corrected to generate a correction result 124, which may be used to generate the lumen direction information 120.

[0124] 17 as an example, the lumen direction estimation unit 58A performs lumen direction estimation processing based on the endoscopic image 28. The lumen direction estimation unit 58A inputs the endoscopic image 28 to the trained model 116A. When the endoscopic image 28 is input, the trained model 116A outputs an estimation result 118A of the lumen direction in the endoscopic image 28.

[0125] The luminal direction estimation unit 58A performs an estimation result correction process on the estimation result 118A. The luminal direction estimation unit 58A extracts only the probability that a luminal direction exists from the probability distribution p of each divided region 96 of the estimation result 118A. Furthermore, the luminal direction estimation unit 58A performs weighting starting from the largest probability in the probability distribution p. Specifically, the luminal direction estimation unit 58A obtains a weighting coefficient 126 from the NVM 62 and multiplies the extracted probability by the weighting coefficient 126. For example, the weighting coefficient 126 is set such that the coefficient corresponding to the largest probability is 1 and the coefficient corresponding to the probability adjacent to the largest probability is 0.8. The weighting coefficient 126 is set as appropriate based on, for example, past estimation results 118A.

[0126] The weighting coefficients 126 may be set according to the probability distribution p. For example, if the probability of the central region 96A of the divided region 96 is the highest, the coefficient of the weighting coefficients 126 corresponding to the highest probability may be set to 1, and the coefficients other than the coefficient corresponding to the highest probability may be set to 0.

[0127] The luminal direction estimation unit 58A then acquires the threshold value 128 from the NVM 62 and sets the probability equal to or greater than the threshold value 128 as the modified result 124. The threshold value 128 is, for example, 0.5, but this is merely an example. For example, the threshold value 128 may be, for example, 0.4 or 0.6. The threshold value 128 is set appropriately based on, for example, the past estimation result 118A.

[0128] The lumen direction estimation unit 58A outputs the correction result 124 to the information generation unit 58B. The information generation unit 58B generates lumen direction information 120 based on the correction result 124. The information generation unit 58B outputs the lumen direction information 120 to the display control unit 58C.

[0129] As described above, in the endoscope device 12 according to the first modification, the estimation result 118A is corrected by the estimation result correction process. In the estimation result correction process, the estimation result 118A is corrected using the weighting coefficient 126 and the threshold value 128. This makes the lumen direction indicated by the estimation result 118A more accurate. Therefore, this configuration enables the output of accurate lumen direction information 120.

[0130] Although the first modification has been described with reference to an example in which the estimation result correction process is performed on the estimation result 118A, the technology of the present disclosure is not limited to this. A calculation equivalent to the estimation result correction process may be incorporated into the trained model 116A.

[0131] In the first and second embodiments, the divided region 96 includes the central region 96A and the radial regions 96B. However, the technology of the present disclosure is not limited to this. In the second modified example, the divided region 96 includes the central region 96A and a plurality of peripheral regions 96C that are located closer to the outer edge of the endoscopic image 28 than the central region 96A.

[0132] 18 , the calculation unit 86 receives a designation of a lumen-corresponding region 94 in the endoscopic image 28 from the annotator 76 via the reception device 72. The calculation unit 86 virtually divides the endoscopic image 28 to obtain a plurality of divided regions 96.

[0133] The divided region 96 has a central region 96A and a peripheral region 96C. The central region 96A is, for example, a circular region centered at the center C of the endoscopic image 28. The peripheral regions 96C are regions that exist on the outer edge side of the endoscopic image 28 relative to the central region 96A. In the example shown in FIG. 18 , three peripheral regions 96C exist on the outer edge side of the endoscopic image 28. Although three peripheral regions 96C are shown here, this is merely an example. The number of peripheral regions 96C may be two, or may be four or more. The peripheral region 96C is an example of a "peripheral region" according to the technology of the present disclosure.

[0134] The calculation unit 86 derives, from among the multiple divided regions 96, the divided region 96 that has the largest area overlapping with the lumen-corresponding region 94. For example, the calculation unit 86 identifies the region where each of the multiple divided regions 96 overlaps with the lumen-corresponding region 94. The calculation unit 86 calculates the area of ​​the region where the divided region 96 overlaps with the lumen-corresponding region 94. Then, the calculation unit 86 identifies the divided region 96 that has the largest area where the divided region 96 overlaps with the lumen-corresponding region 94.

[0135] The calculation unit 86 generates, as the correct answer data 92, the direction of the divided region 96 having the largest overlapping area with the lumen-corresponding region 94. In the example shown in Fig. 18, the correct answer data 92 is an example in which the direction in which the third region 96C1 of the peripheral region 96C exists is the lumen direction.

[0136] As described above, in this second modified example, the divided region 96 includes a central region 96A of the endoscopic image 28 and multiple peripheral regions 96C located closer to the outer edge of the endoscopic image 28 than the central region 96A. The lumen region 28A appears relatively frequently in the endoscopic image 28. Therefore, it is necessary to indicate the lumen direction even when a lumen is present in the central region 96A. Furthermore, by dividing the peripheral region 96C into multiple regions, it becomes easier to indicate the direction in which the lumen exists. By dividing the endoscopic image 28 into the central region 96A and multiple peripheral regions 96C in this way, it becomes easier to grasp which direction is the lumen direction. Therefore, this configuration makes it possible to indicate the lumen direction in a way that is easy for the user to understand.

[0137] In the second modified example, the divided regions 96 are obtained by dividing the peripheral region 96C of the endoscopic image 28 toward the outer edge of the endoscopic image 28 from the central region 96A in three or more directions from the central region 96A toward the outer edge of the endoscopic image 28. The lumen region 28A is relatively frequently captured in the central region 96A within the endoscopic image 28. Therefore, it is necessary to indicate the lumen direction even when a lumen is present in the central region 96A. By dividing the endoscopic image 28 in three or more directions toward the outer edge, it becomes easier to indicate the direction in which the lumen exists. By dividing the endoscopic image 28 into the central region 96A and three or more peripheral regions 96C in this manner, it becomes easier to grasp which direction is the lumen direction. Therefore, this configuration makes it possible to indicate the lumen direction in an easy-to-understand manner for the user.

[0138] In the first and second embodiments, the divided region 96 has been described as having a central region 96A and radial regions 96B, but the technology of the present disclosure is not limited to this. In this third modified example, the divided region 96 is obtained by dividing the endoscopic image 28 into regions in three or more directions starting from the center C of the endoscopic image 28 toward the outer edge of the endoscopic image 28.

[0139] 19 , the calculation unit 86 receives a designation of a lumen-corresponding region 94 in the endoscopic image 28 from the annotator 76 via the reception device 72. The calculation unit 86 virtually divides the endoscopic image 28 to obtain a plurality of divided regions 96.

[0140] The divided regions 96 are regions obtained by dividing the endoscopic image 28 in three directions from the center C of the endoscopic image 28 toward the outer edge of the endoscopic image 28. In the example shown in Fig. 19, three divided regions 96 exist on the outer edge side of the endoscopic image 28. Although three divided regions 96 are shown here, this is merely an example. The number of divided regions 96 may be two, four or more.

[0141] The calculation unit 86 derives, from among the multiple divided regions 96, the divided region 96 that has the largest area overlapping with the lumen-corresponding region 94. For example, the calculation unit 86 identifies the region where each of the multiple divided regions 96 overlaps with the lumen-corresponding region 94. The calculation unit 86 calculates the area of ​​the region where the divided region 96 overlaps with the lumen-corresponding region 94. Then, the calculation unit 86 identifies the divided region 96 that has the largest area where the divided region 96 overlaps with the lumen-corresponding region 94.

[0142] The calculation unit 86 generates the direction of the divided region 96 having the largest overlapping area with the lumen-corresponding region 94 as the correct answer data 92. In the example shown in Fig. 19, the correct answer data 92 is an example in which the direction in which the third region 96C1 of the peripheral region 96C exists is the lumen direction.

[0143] As described above, in the third modified example, the divided regions 96 are obtained by dividing the endoscopic image 28 in three or more directions starting from the center C of the endoscopic image 28 toward the outer edge. Dividing the endoscopic image 28 in three or more directions starting from the center C toward the outer edge makes it easier to indicate in which direction the lumen direction exists. Dividing the image into regions in three or more directions in this way makes it easier to understand which direction is the lumen direction. Therefore, with this configuration, it is possible to indicate the lumen direction in an easy-to-understand manner for the user.

[0144] In the above embodiments, an example in which endoscopic image processing is performed by the processor 58 of the endoscopic device 12 has been described. However, the technology of the present disclosure is not limited to this. For example, the device that performs endoscopic image processing may be provided externally to the endoscopic device 12. An example of a device that is provided externally to the endoscopic device 12 is a server. For example, the server may be implemented using cloud computing. While cloud computing is used here as an example, this is merely an example. For example, the server may be implemented using a mainframe or network computing such as fog computing, edge computing, or grid computing. Although a server is used here as an example of a device that is provided externally to the endoscopic device 12, this is merely an example. Instead of a server, at least one personal computer or the like may be used. Furthermore, endoscopic image processing may be performed in a distributed manner by multiple devices, including the endoscopic device 12 and devices provided externally to the endoscopic device 12.

[0145] Furthermore, in the above-described embodiments, an example in which the endoscopic image processing program 62A is stored in the NVM 62 has been described, but the technology of the present disclosure is not limited to this. For example, the endoscopic image processing program 62A may be stored in a portable storage medium such as an SSD or USB memory. The storage medium is a non-transitory computer-readable storage medium. The endoscopic image processing program 62A stored in the storage medium is installed in the computer 56 of the control device 46. The processor 58 performs endoscopic image processing in accordance with the endoscopic image processing program 62A.

[0146] Furthermore, in each of the above embodiments, an example in which the machine learning processing is performed by the processor 78 of the information processing device 66 has been described, but the technology of the present disclosure is not limited to this. For example, the machine learning processing may be performed in the endoscope device 12. Furthermore, the machine learning processing may be performed in a distributed manner by multiple devices including the endoscope device 12 and the information processing device 66.

[0147] Furthermore, in the above-described embodiments, an example has been described in which the lumen direction is displayed based on the estimation result 118 obtained by inputting the endoscopic image 28 into the trained model 116. However, the technology of the present disclosure is not limited to this. For example, the lumen direction may be displayed by using, in addition to the estimation result 118 for one endoscopic image 28, the estimation result 118 for another endoscopic image 28 (for example, the endoscopic image 28 several frames (for example, one to two frames) before the one endoscopic image 28 is obtained).

[0148] In each of the above embodiments, the computer 56 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 56. Furthermore, instead of the computer 56, a combination of a hardware configuration and a software configuration may be used.

[0149] The hardware resources for executing the various processes described in the above embodiments can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing endoscopic image processing by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute endoscopic image processing.

[0150] The hardware resource for executing endoscopic image processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a processor and an FPGA). Also, the hardware resource for executing endoscopic image processing may be a single processor.

[0151] As an example of a system configured with a single processor, first, one processor is configured by combining one or more processors with software, and this processor functions as a hardware resource that executes endoscopic image processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute endoscopic image processing, on a single IC chip, as typified by SoC. In this way, endoscopic image processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0152] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. Furthermore, the above-described endoscopic image processing is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0153] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0154] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0155] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0156] The disclosure of Japanese Patent Application No. 2022-115110, filed on July 19, 2023, is incorporated herein by reference in its entirety.

Claims

1. An image processing device comprising: a processor that acquires a lumen direction, which is the direction for inserting the endoscope, from an image obtained by imaging a tubular organ with a camera attached to the endoscope, in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided areas into which an image obtained by imaging a tubular organ with a camera attached to the endoscope is divided and a lumen-corresponding area included in the image; and outputs lumen direction information, which is information indicating the lumen direction.

2. The image processing device according to claim 1, wherein the lumen-corresponding region is a region of a predetermined range that includes a lumen region in the image.

3. The image processing device according to claim 1, wherein the lumen corresponding region is an end of the observation range of the camera in a direction in which the position of the lumen region is estimated from the fold region in the image.

4. The image processing device according to claim 1, wherein the direction of the divided region that overlaps with the lumen corresponding region among the plurality of divided regions is the lumen direction.

5. The image processing device according to claim 1, wherein the trained model is a data structure configured to cause the processor to estimate the position of a lumen region based on the shape and / or orientation of a fold region in the image.

6. The image processing device according to claim 1, wherein the lumen direction is the direction in which one of the plurality of divided regions has the largest area overlapping with the lumen corresponding region in the image.

7. The image processing device according to claim 1, wherein the lumen direction is the direction in which a first divided area exists, which is the divided area among the plurality of divided areas that has the largest area overlapping with the lumen corresponding area in the image, and the direction in which a second divided area exists, which is the divided area that has the next largest area overlapping with the lumen corresponding area after the first divided area.

8. The image processing device according to claim 1, wherein the divided regions have a central region of the image and a plurality of radial regions that radiate from the central region toward the outer edge of the image.

9. The image processing device according to claim 8, wherein there are eight radial regions arranged radially.

10. The image processing device according to claim 1, wherein the divided regions include a central region of the image and a plurality of peripheral regions located on the outer edge side of the image relative to the central region.

11. The image processing device according to claim 1, wherein the divided regions are obtained by dividing the image into regions in three or more directions starting from the center of the image toward the outer edge of the image.

12. The image processing device according to claim 1, wherein the divided regions have a central region of the image and a plurality of peripheral regions located on the outer edge side of the image than the central region, and the peripheral regions are obtained by dividing the outer edge side of the image than the central region in three or more directions from the central region toward the outer edge of the image.

13. A display device that displays information according to the lumen direction information output by the processor of the image processing device according to any one of claims 1 to 12.

14. An endoscope apparatus comprising: the image processing device according to any one of claims 1 to 12; and the endoscope.

15. An image processing method including: obtaining a lumen direction, which is the direction for inserting an endoscope, from an image obtained by imaging a tubular organ with a camera attached to an endoscope, in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided areas into which an image obtained by imaging a tubular organ with a camera attached to an endoscope is divided and a lumen-corresponding area included in the image; and outputting lumen direction information, which is information indicating the lumen direction.

16. An image processing program for causing a first computer to execute image processing, including: acquiring a lumen direction, which is the direction for inserting the endoscope, from the image in accordance with a trained model obtained by machine learning based on the positional relationship between a plurality of divided areas into which an image obtained by imaging a tubular organ with a camera provided on an endoscope and a lumen-corresponding area included in the image; and outputting lumen direction information, which is information indicating the lumen direction.

17. A trained model obtained by machine learning based on the positional relationship between multiple divided regions obtained by dividing an image obtained by capturing an image of a tubular organ using a camera attached to an endoscope and the lumen-corresponding region contained in the image.

18. A method for generating a trained model, comprising: acquiring an image of a tubular organ captured by a camera mounted on an endoscope; and performing machine learning on the model based on the positional relationship between a plurality of divided regions into which the image is divided and a lumen-corresponding region contained in the image.

19. A trained model generation program for causing a second computer to execute a trained model generation process, the process including: acquiring an image of a tubular organ captured by a camera mounted on an endoscope; and performing machine learning on a model based on the positional relationship between a plurality of divided regions into which the image is divided and a lumen-corresponding region included in the image.

Images