Image processing device and control method and program for image processing device

Abstract

Provided are an image processing device capable of assisting an endoscope in having an optimum observation visual field, and a control method and program for the image processing device. The image processing device is provided with a processor. The processor: acquires a medical image obtained by processing imaging data sequentially output from an image sensor of the endoscope; generates first assistance information on the basis of the medical image; and outputs the first assistance information. The first assistance information includes information pertaining to a relationship between a region of interest in the medical image and a visual field direction of the image sensor.

Description

Image processing device, control method and program for image processing device

[0001] The present disclosure relates to an image processing device, a control method for an image processing device, and a program.

[0002] Gastroesophageal reflux disease (GERD) is a pathological condition caused by the reflux of stomach contents, including a large amount of stomach acid, into the esophagus. Conventionally, an endoscope system using an endoscope has been proposed as a system for diagnosing gastroesophageal reflux disease (see, for example, Patent Document 1). In this technology, an insertion portion of the endoscope is inserted into the stomach, the endoscope is set to be able to image the gastric cardia, and air is blown into the stomach. The endoscope system then evaluates the function of the lower esophageal sphincter based on the pressure change inside the stomach detected during gas blowing into the stomach and the state of the gastric cardia observed from the image captured using the endoscope.

[0003] US Patent Application Publication No. 2020 / 0375485

[0004] In cases where GERD symptoms are confirmed, a physician must observe the condition in Phase 3 of the aforementioned Patent Document 1 to identify the cause and determine a treatment method. In this case, the physician must observe the esophageal peristalsis during clinically characteristic changes in organ activity, such as lower esophageal sphincter (LES) relaxation, scope holding sign (SHS) opening, peristalsis, belching, peristalsis, and LES contraction, while observing endoscopic images captured by the endoscope. At this time, the physician must bend the endoscope to adjust it so that the gastric cardia and the insertion portion of the endoscope itself fit within the observation field. Specifically, the physician must manipulate the endoscope to adjust it so that not only the gastric cardia but also the LES relaxation occurring during continuous pneumoperitoneum and the peristalsis of the esophageal body occurring after LES relaxation fit within the observation field.

[0005] However, the technique of adjusting the gastric cardia and the insertion part of the endoscope itself so that they fit within the field of view is difficult for anyone other than an experienced physician, and there has been a demand for technology that can assist the endoscope in finding the optimal field of view, regardless of the physician's skill level.

[0006] The present disclosure has been made in view of the above, and aims to provide an image processing device, a control method for an image processing device, and a program that can assist an endoscope in obtaining an optimal observation field.

[0007] In order to solve the above-mentioned problems and achieve the objectives, the image processing device of the present disclosure is an image processing device having a processor, wherein the processor acquires a medical image by processing imaging data sequentially output from an image sensor of an endoscope, generates first support information based on the medical image, and outputs the first support information, wherein the first support information includes information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

[0008] In addition, a control method for an image processing device according to the present disclosure is a control method for an image processing device equipped with a processor, in which the processor acquires a medical image by processing imaging data sequentially output from an image sensor of an endoscope, generates first support information based on the medical image, and outputs the first support information, the first support information including information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

[0009] In addition, the program according to the present disclosure is a program executed by an image processing device having a processor, which causes the processor to acquire a medical image by processing imaging data sequentially output from an image sensor of an endoscope, generate first support information based on the medical image, and output the first support information, wherein the first support information includes information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

[0010] According to the present disclosure, an effect is achieved in that an endoscope can be assisted to obtain an optimal observation field.

[0011] FIG. 1 is a diagram illustrating a configuration of an endoscopic system according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating a configuration of an endoscopic system according to the first embodiment of the present disclosure. FIG. 3 is a diagram illustrating the configuration and arrangement of a pressure sensor. FIG. 4 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to the first embodiment of the present disclosure. FIG. 8 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to the first embodiment of the present disclosure. FIG. 9 is a block diagram illustrating functional configurations of a storage unit and a control unit of a processing device according to the first embodiment of the present disclosure. FIG. 10 is a flowchart illustrating an overview of processing executed by the processing device 4 according to the first embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of first support information. FIG. 12 is a diagram illustrating a schematic relationship between a region of interest in an endoscopic image and the field of view of an imaging element under the conditions of FIG. 11. FIG. 13 is a diagram illustrating another example of the first support information. FIG. 14 is a diagram schematically illustrating the relationship between a region of interest in an endoscopic image and the field of view direction of an imaging element under the circumstances of FIG. 13 . FIG. 15 is a diagram illustrating the functional configuration of a storage unit and a control unit of a processing device according to a second embodiment of the present disclosure. FIG. 16 is a flowchart illustrating an outline of processing performed by the processing device according to the second embodiment of the present disclosure. FIG. 17 is a diagram illustrating an example of second support information. FIG. 18 is a diagram illustrating another example of the second support information. FIG. 19 is a diagram illustrating another example of the second support information. FIG. 20 is a diagram illustrating another example of the second support information. FIG. 21 is a diagram illustrating the functional configuration of an endoscopic system according to a third embodiment of the present disclosure. FIG. 22 is a flowchart illustrating an outline of processing performed by a processing device 4B according to the third embodiment of the present disclosure. FIG. 23 is a diagram illustrating the functional configuration of an endoscopic system according to a fourth embodiment of the present disclosure.

[0012] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0013] (Embodiment 1) [Configuration of Endoscopic System] Fig. 1 is a diagram illustrating the configuration of an endoscopic system according to embodiment 1 of the present disclosure. Fig. 2 is a diagram illustrating the functional configuration of the endoscopic system according to embodiment 1 of the present disclosure. The endoscopic system 1 shown in Figs. 1 and 2 is used in the medical field and is a system for diagnosing gastroesophageal reflux disease in a subject using an endoscope 2. As shown in Figs. 1 and 2, the endoscopic system 1 includes an endoscope 2, a light source device 3, a processing device 4, a display device 5, an air supply device 6, a gastric pressure measuring device 7, and a microphone 8.

[0014] In this embodiment, the endoscope 2 is a so-called flexible endoscope. A portion of the endoscope 2 is inserted into a living body, captures images of the living body, and outputs image signals generated by the image capture. As shown in FIG. 1 , the endoscope 2 includes an insertion section 21, an operation section 22, and a universal cord 23.

[0015] The insertion section 21 is a section that has at least a portion that is flexible and is inserted into a living body. As shown in Figures 1 and 2, the insertion section 21 includes a tip section 24, a freely bendable bending section 25 (Figure 1) composed of a plurality of bending pieces, a long, flexible flexible tube section 26 (Figure 1) that is connected to the base end side of the bending section 25, a first microphone 81, and a second microphone 82. An image sensor 244 (Figure 2) is built into the tip section 24. The insertion section 21 is inserted into a body cavity of a subject, and captures an image of a subject, such as biological tissue, that is located in a position that is not accessible by external light, using the image sensor 244.

[0016] Here, a pressure sensor 9 that detects pressure applied to the outer peripheral surface is provided on the outer peripheral surface of the insertion portion 21. The detailed configuration and arrangement of the pressure sensor 9 will be described later in the section "Configuration and arrangement of pressure sensor."

[0017] The operation unit 22 is connected to the base end portion of the insertion section 21. The operation unit 22 receives various operations for the endoscope 2. As shown in Fig. 1 , the operation unit 22 includes a bending knob 221 for bending the bending section 25 in the up-down and left-right directions, a treatment tool insertion section 222 that extends from the operation unit 22 to the tip of the insertion section 21 and inserts treatment tools such as biopsy forceps, an electric scalpel, and an examination probe into the body cavity of the subject, an air supply conduit 223 (Fig. 2) that extends from the operation unit 22 to the tip of the insertion section 21 and supplies air into the body cavity of the subject, and a plurality of switches 224 for operating peripheral devices such as the air supply device 6 and a water supply device (not shown).

[0018] The universal cord 23 incorporates at least a light guide 241 ( FIG. 2 ) and a cable assembly 245 ( FIG. 2 ) that bundles one or more signal lines. The light guide 241 is made of glass fiber or the like and serves as a light guide path for light emitted by the light source device 3. As shown in FIG. 1 , the universal cord 23 branches at the end opposite the end connected to the operation unit 22. The branched ends of the universal cord 23 are provided with a connector 231 that is detachable from the light source device 3 and a connector 232 that is detachable from the processing device 4. A portion of the light guide 241 extends from the end of the connector 231. The universal cord 23 transmits illumination light emitted from the light source device 3 to the distal end 24 via the connector 231 (light guide 241), the operation unit 22, and the flexible tube portion 26. The universal cord 23 also transmits image signals captured by an image sensor 244 provided in the distal end 24 to the processing device 4 via the connector 232. The cable assembly 245 includes a signal line for transmitting an image signal, a signal line for transmitting a drive signal for driving the image sensor 244, and a signal line for transmitting and receiving information including unique information related to the endoscope 2 (image sensor 244). Note that, in this embodiment, the signal lines are described as transmitting electrical signals, but they may also be used to transmit optical signals, or may be used to transmit signals between the endoscope 2 and the processing device 4 by wireless communication.

[0019] The output end side of the light guide 241 is inserted into the tip portion 24. As shown in Fig. 2, the tip portion 24 includes an illumination lens 242, an optical system 243 for collecting light, and an image sensor 244 that is provided at the imaging position of the optical system 243 and receives the light collected by the optical system 243, photoelectrically converts the light into an electrical signal, and performs predetermined signal processing.

[0020] The optical system 243 is configured using one or more lenses, and forms an observation image on the light receiving surface of the image sensor 244. The optical system 243 may have an optical zoom function that changes the angle of view and a focus function that changes the focus.

[0021] The image sensor 244 photoelectrically converts light from the optical system 243 to generate an electrical signal (image signal). The image sensor 244 is configured with a plurality of pixels arranged in a matrix, each of which has a photodiode that accumulates an electric charge according to the amount of light and a capacitor that converts the electric charge transferred from the photodiode into a voltage level. The image sensor 244 photoelectrically converts light incident on each pixel via the optical system 243 to generate an electric signal, sequentially reads out the electric signals generated by pixels arbitrarily designated as readout targets among the plurality of pixels, and outputs the electric signals as an image signal. The image sensor 244 is realized, for example, using a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0022] For ease of explanation, the image signal generated by the image sensor 244 capturing an image will be referred to as a captured image below.

[0023] Here, the endoscope 2 has a memory (not shown) that stores execution programs and control programs for the image sensor 244 to perform various operations, as well as data including identification information of the endoscope 2. The identification information includes the endoscope 2's unique information (ID), model year, specification information, transmission method, etc. The memory may also temporarily store captured images generated by the image sensor 244.

[0024] As shown in FIG. 2 , the light source device 3 includes a light source unit 31 , an illumination control unit 32 , and a light source driver 33 .

[0025] The light source unit 31 emits light under the control of the illumination control unit 32. The light source unit 31 emits light having a wavelength band of visible light (white light (illumination light)). The light source unit 31 is realized using any light source such as an LED (Light Emitting Diode) light source, a laser light source, a xenon lamp, or a halogen lamp. The light source unit 31 may also include one or more lenses. The light generated by the light source unit 31 passes through the light guide 241 and the illumination lens 242 and is emitted from the tip of the tip unit 24 toward the subject.

[0026] The light emitted from the light source unit 31 is not limited to white light, but may be narrowband light having light in a specific wavelength band, or may be excitation light that excites substances contained in the object of observation.

[0027] The light source driver 33 supplies current to the light source unit 31 under the control of the illumination control unit 32, thereby causing the light source unit 31 to emit light.

[0028] The processing device 4 includes an image processing unit 41 , a synchronization signal generating unit 42 , an input unit 43 , a control unit 44 , and a storage unit 45 .

[0029] Under the control of the control unit 44, the image processing unit 41 performs predetermined image processing on the captured image received from the endoscope 2 to generate an endoscopic image.

[0030] Examples of image processing performed by the image processing unit 41 include optical black subtraction processing (clamping processing), white balance adjustment processing, demosaic processing, color correction matrix processing, gamma correction processing, YC processing that converts RGB signals into luminance color difference signals (Y, Cb / Cr signals), digital gain adjustment that multiplies by digital gain, noise removal, and filter processing that emphasizes structure.

[0031] The image processing unit 41 described above is configured using dedicated processors such as memory and dedicated processors having hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processing) or an FPGA (Field Programmable Gate Array), or various arithmetic circuits that execute specific functions, such as an ASIC (Application Specific Integrated Circuit).

[0032] The synchronization signal generation unit 42 generates a clock signal (synchronization signal) that serves as a reference for the operation of the processing device 4, and outputs the generated synchronization signal to the light source device 3, the image processing unit 41, the control unit 44, and the endoscope 2. Here, the synchronization signal generated by the synchronization signal generation unit 42 includes a horizontal synchronization signal and a vertical synchronization signal. Therefore, the light source device 3, the image processing unit 41, the control unit 44, and the endoscope 2 operate in synchronization with each other using the generated synchronization signal.

[0033] The input unit 43 is realized using a keyboard, a mouse, a switch, and a touch panel, and accepts various operations for instructing the operation of the endoscope system 1. The input unit 43 may include a switch provided on the operation unit 22 or a portable terminal such as an external tablet computer.

[0034] The control unit 44 is configured using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuits that execute specific functions such as an ASIC. In the first embodiment, the control unit 44 functions as the processor of the present disclosure.

[0035] The control unit 44 also acquires endoscopic images, which are medical images obtained by processing the imaging data sequentially output from the imaging element 244 of the endoscope 2, generates first support information based on the endoscopic images, and outputs the first support information. Details of the functions and processing of the control unit 44 will be described later. In the first embodiment, the control unit 44 functions as the image processing device of the present disclosure.

[0036] The storage unit 45 stores various programs executed by the control unit 44 and data including various parameters necessary for the processing of the control unit 44. Specifically, the storage unit 45 has a program storage unit 451 that stores various programs. The various programs can be recorded on computer-readable recording media such as a hard disk, flash memory, CD-ROM, DVD-ROM, or flexible disk and widely distributed. The various programs can also be obtained by downloading them via a communication network. The communication network referred to here is realized, for example, by an existing public line network, a LAN (Local Area Network), a WAN (Wide Area Network), or the like, and can be wired or wireless.

[0037] The storage unit 45 having the above configuration is realized using a ROM (Read Only Memory) in which various programs etc. are pre-installed, and a RAM, hard disk, SSD (Solid State Drive) etc. that store calculation parameters and data etc. for each process.

[0038] In this embodiment, the light source device 3 and the processing device 4 are provided in separate housings, but this is not limiting, and they may be provided integrally in the same housing.

[0039] The display device 5 displays the display image received from the processing device 4 (image processing unit 41) via the video cable. The display device 5 is configured using a monitor such as a liquid crystal or organic EL (Electro Luminescence) monitor.

[0040] The gas supply device 6 adjusts the pressure of gas supplied from a gas supply source (not shown, for example, a carbon dioxide gas cylinder) to a predetermined pressure and discharges the gas from the tip of the insertion section 21 into the space where the tip is located through the gas supply conduit 223. As shown in Fig. 1, the gas supply device 6 includes a gas supply unit 61, a flow rate measurement unit 62, and a control unit 63.

[0041] Although not specifically shown, the gas supply unit 61 includes a primary pressure reducer, a secondary pressure reducer, and a flow control valve. These primary pressure reducer, secondary pressure reducer, and flow control valve are connected in this order by an air supply conduit made of silicone, fluororesin, or the like. Gas supplied from a gas supply source (not shown) passes through the primary pressure reducer, secondary pressure reducer, and flow control valve in this order via the air supply conduit. After being adjusted to a predetermined pressure and flow rate, the gas is discharged from the air supply tube TU ( FIG. 1 ) via the flow rate measuring unit 62. The control unit 63 controls the flow control valve provided in the gas supply unit 61 to adjust the flow rate of gas supplied to the endoscope 2 to a predetermined value. The flow control valve is, for example, a type of electromagnetically driven valve, and is configured as a regulating valve using an electromagnetic coil in the drive unit. The opening degree of the valve unit is controlled by controlling the position of the plunger depending on the magnitude of the current flowing through the electromagnetic coil, thereby adjusting the flow rate of gas flowing through the air supply conduit to a predetermined value.

[0042] The gastric pressure measuring device 7 corresponds to an internal pressure sensor according to the present invention. This gastric pressure measuring device 7 detects the pressure inside the space where the tip is located via a pressure measurement probe 71 inserted to the tip of the insertion section 21 through the treatment tool insertion section 222. A signal (hereinafter referred to as gastric pressure information) related to the pressure (hereinafter referred to as gastric pressure) detected by the gastric pressure measuring device 7 is output to the processing device 4.

[0043] The microphone 8 is placed on the throat of the subject and collects belching sounds (burping sounds) emitted from the esophagus of the subject. The signal related to the belching sounds collected by the microphone 8 (hereinafter referred to as belching sound information) is output to the processing device 4.

[0044] [Configuration and Arrangement of Pressure Sensor] Next, the configuration and arrangement of the pressure sensor 9 will be described. FIG.

[0045] The pressure sensor 9 shown in Fig. 3 is configured by, for example, a resistance-type pressure sensor or a capacitance-type pressure sensor that detects pressure using a known method. As shown in Fig. 3, the pressure sensor 9 according to the first embodiment is a circular pressure sensor that is provided around the entire circumference in the rotational direction around a central axis along the axial direction of the insertion portion 21. Note that the pressure sensor 9 is not limited to a circular pressure sensor, and a point-type pressure sensor that is provided only around a portion of the entire circumference in the rotational direction may also be used.

[0046] 3, a total of 36 pressure sensors 9 are arranged at 1 cm intervals along the axial direction of the insertion section 21 on the outer peripheral surface of the insertion section 21. As a result, when the insertion section 21 is inserted into the stomach, the pressure sensors 9 are arranged at positions that detect contraction pressure and relaxation pressure at various locations inside the esophagus from the upper esophageal sphincter to the lower esophageal sphincter.

[0047] The number of pressure sensors 9 is not limited to 36, but may be, for example, 6 or more, 12 or more, or even more.

[0048] Here, the pressure sensors 9 may be positioned a predetermined distance from one or more of the nearest other pressure sensors 9. Optionally, the spacing between each pressure sensor 9 may be substantially the same. The spacing may be 3 cm or less, for example 2 cm or less, for example 1 cm, or even less than 1 cm.

[0049] The signal relating to the pressure detected by the pressure sensor 9 described above (hereinafter referred to as esophageal pressure information) is output to the processing device 4.

[0050] [Method for Diagnosing Gastroesophageal Reflux Disease] Next, a method for diagnosing gastroesophageal reflux disease will be described. FIGS. 4 to 8 are diagrams illustrating a method for diagnosing gastroesophageal reflux disease according to embodiment 1. Specifically, FIG. 4 is a flowchart illustrating the method for diagnosing gastroesophageal reflux disease. FIG. 5 is a cross-sectional view showing the vicinity of the gastric cardia, illustrating steps S1 to S3. For ease of explanation, FIG. 5 omits the pressure sensor 9 provided on the outer peripheral surface of the insertion section 21. FIGS. 6 and 7 are diagrams illustrating the structures of the stomach and esophagus. FIG. 6 is a cross-sectional view showing the structures of the stomach and esophagus. FIG. 7 is a view of the stomach and esophagus as viewed from the outside. FIG. 8 is a diagram illustrating images captured in Phases 1 to 3, which correspond to dynamic changes that occur when air is supplied to the stomach (during pneumoperitoneum).

[0051] First, a user such as a doctor inserts the insertion portion 21 into the subject through a natural opening such as the mouth or nose, and introduces the insertion portion 21 into the stomach through the esophagus, as shown in FIG. 5 (step S1).

[0052] After step S1, a user such as a doctor operates (bends) the bending knob 221 to set the field of view to include the gastric cardia (step S2). In step S2, in response to the operation of the bending knob 221, the insertion section 21 is set into a J-shape with the tip pointing toward the gastric cardia, as shown in Fig. 5. In this state, the pressure sensor 9 is disposed at a position to detect the contraction pressure and relaxation pressure at various locations inside the esophagus from the upper esophageal sphincter to the lower esophageal sphincter.

[0053] After step S2, a user such as a doctor operates switch 224 to start supplying air from the air supply device 6 to the stomach through the air supply conduit 223 (step S3). In Fig. 5, the state in which intragastric pressure increases due to air supply is represented by a hollow arrow.

[0054] When gas is supplied to the stomach (pneumoperitoneum), if the subject is healthy, the dynamic changes shown below occur in Phases 1 to 3. Before explaining Phases 1 to 3, the structure of the stomach and esophagus will be described with reference to Figures 6 and 7.

[0055] The intramural anti-reflux barrier complex (IM-ARB complex), which is part of the anti-reflux mechanism at the gastroesophageal junction, is composed of three main components: the collar sling muscle fibers (Figure 7), the clasp muscle fibers (Figure 7), and the lower esophageal sphincter (Figures 6 and 7).

[0056] The Collar Sling Muscle Fibers are oblique muscles located along the greater curvature of the stomach (Figure 6) and are arranged in a sling-like fashion to surround the upper part of the stomach (Figure 7). These Collar Sling Muscle Fibers constrict the gastric cardia, preventing reflux of gastric contents into the esophagus. From inside the stomach, the Collar Sling Muscle Fibers can be seen as a gastroesophageal flap valve (Figure 6). The gastroesophageal flap valve (GEFV) is a protrusion within the gastroesophageal junction formed by the acute angle between the esophagus and the gastric cardia, and is a type of mucosal flap valve (MFV). The MFV located at the gastroesophageal junction is called the gastroesophageal flap valve. The MFV is a flap-shaped portion of the mucosa. The gastroesophageal flap valve changes shape depending on the contraction and relaxation of muscles such as the Collar Sling Muscle Fibers and Clasp Muscle Fibers.

[0057] Clasp muscle fibers are located on the lesser curvature side of the stomach (Figure 6) and consist of a circular muscle layer. These clasp muscle fibers tighten the gastric cardia to prevent the reflux of stomach contents into the esophagus.

[0058] The lower esophageal sphincter (LES) is a ring of muscle located at the junction of the esophagus and stomach that normally contracts to close the esophagus and prevent stomach contents from refluxing.

[0059] Phase 1 begins with the amount of gas sent into the stomach being 0. During Phase 1, the gastroesophageal flap valve and the longitudinal folds of the lesser curvature are observed, as shown in Figures 8(a) and 8(b). The longitudinal folds of the lesser curvature are mucosal folds that extend vertically along the inner wall of the stomach. Their shape changes with the expansion and contraction and relaxation of muscles such as clasp muscle fibers, and they gradually stretch and flatten as intragastric pressure increases. Note that the esophageal mucosa is not observed during Phase 1. During Phase 1, as the amount of gas sent into the stomach increases, the longitudinal folds of the lesser curvature are stretched, the gastroesophageal flap valve gradually flattens, and the crura open, as shown in Figure 8(b). However, the esophageal mucosa is not observed.

[0060] In other words, in Phase 1, it is possible to evaluate the valve function (anti-reflux mechanism) of the stomach structure formed by the gastroesophageal flap valve and the longitudinal folds of the lesser curvature.

[0061] Phase 2 occurs after Phase 1. In Phase 2, as shown in Figure 8(c), the esophageal mucosa is observed beyond the squamocolumnar junction (SCJ) in Figure 6). The SCJ is the intersection of the esophageal squamous epithelium at the gastroesophageal junction (GEJ) and the gastric columnar epithelium, marking the boundary between different epithelial cells in the digestive tract. The GEJ is located near the border between the esophagus and the stomach and is composed of various anatomical components that form a barrier to prevent reflux of gastric contents. The GEJ also includes collar sling muscle fibers, clasp muscle fibers, the lower esophageal sphincter, the gastroesophageal flap valve, and the SCJ. If the subject is healthy, the scope holding sign (SHS) is observed. The SHS refers to the phenomenon in which the insertion tube 21 is held in place by contraction of the lower esophageal sphincter when intragastric pressure increases. The state shown in FIG. 8(c) is the SHS.

[0062] That is, in Phase 2, it becomes possible to evaluate the valve function (anti-reflux mechanism) of the lower esophageal sphincter.

[0063] Phase 3 occurs after Phase 2. In Phase 3, intragastric pressure exceeds the contractile force of the lower esophageal sphincter, causing the lower esophageal sphincter to relax. In Figure 8(d), the arrow indicates that gas leaks into the esophagus (producing a belching sound) due to the relaxation of the lower esophageal sphincter. If the subject is healthy, peristaltic waves will subsequently descend from the upper esophagus, and SHS will be observed again.

[0064] That is, in Phase 3, it becomes possible to evaluate the acid clearance function by esophageal peristalsis.

[0065] After step S3, a user such as a doctor determines whether the state corresponds to the above-mentioned Phase 1 based on the diagnostic support information displayed on the display device 5, and evaluates the valve function (state of the gastric cardia) based on the structure of the stomach side (step S4).

[0066] After step S4, a user such as a doctor determines whether or not the state corresponds to the above-mentioned Phase 2 based on the diagnostic support information displayed on the display device 5, and evaluates the valve function of the lower esophageal sphincter (the state of the lower esophageal sphincter) (step S5).

[0067] After step S5, a user such as a doctor determines whether or not the patient is in a state corresponding to the above-mentioned Phase 3 based on the diagnostic support information displayed on the display device 5, and evaluates the acid clearance function due to esophageal peristalsis (the state of peristaltic movement in the esophagus) (step S6).

[0068] After step S6, a user such as a doctor diagnoses gastroesophageal reflux disease based on the evaluation results of steps S4 to S6 (step S7).

[0069] [Functional Configuration of Storage Unit and Control Unit] Next, a description will be given of the functional configuration of the storage unit 45 and the control unit 44. FIG.

[0070] [Functional Configuration of Storage Unit] First, the functional configuration of the storage unit 45 will be described. The storage unit 45 includes a program storage unit 451 that stores various programs executed by the processing device 4, and a first trained model storage unit 452 that stores trained models. The first trained model stored in the first trained model storage unit 452 is, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). In this case, the first trained model is a model that learns using training data that associates multiple endoscopic images, the gastric cardia contained in each of the multiple endoscopic images, and the field of view direction of the image sensor 244, receives the endoscopic images as input parameters, and outputs, as output parameters, information indicating whether the relationship between the gastric cardia, which is the region of interest, and the field of view direction L1 (the optical axis direction of the optical system 243) of the image sensor 244 is appropriate.

[0071] [Functional Configuration of Control Unit] Next, a description will be given of the functional configuration of the control unit 44. The control unit 44 has an acquisition unit 441, a generation unit 442, and a display control unit 443.

[0072] The acquisition unit 441 sequentially acquires endoscopic images obtained by the image processing unit 41 sequentially processing the image data sequentially captured by the endoscope 2 .

[0073] The generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first learned model stored in the first learned model memory unit 452, and generates first support information based on the output result output by the first learned model.

[0074] The display control unit 443 superimposes the first support information generated by the generation unit 442 on the endoscopic image and outputs the superimposed image to the display device 5 .

[0075] [Processing by Processing Device] Next, the processing executed by the processing device 4 will be described. Fig. 10 is a flowchart showing an outline of the processing executed by the processing device 4. Note that Fig. 10 describes the processing executed by the processing device 4 in a state where the tip portion 24 of the endoscope 2 is set in a J-shape facing the gastric cardia at the timing when gas supply into the stomach via the gas supply conduit 223 from the gas supply device 6 shown in Fig. 5 described above is started.

[0076] As shown in FIG. 10, first, the acquisition unit 441 starts sequential acquisition of endoscopic images obtained by the image processing unit 41 sequentially processing the image data sequentially captured by the endoscope 2 (step S101).

[0077] Next, the generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first learned model stored in the first learned model memory unit 452, and generates first support information based on the output result output by the first learned model (step S102).

[0078] Thereafter, the display control unit 443 superimposes the first support information generated by the generation unit 442 on the endoscopic image and outputs the superimposed image to the display device 5 (step S103).

[0079] Fig. 11 is a diagram showing an example of first support information. Fig. 12 is a diagram schematically explaining the relationship between the region of interest in an endoscopic image and the direction of view of the image sensor 244 under the circumstances of Fig. 11. Fig. 13 is a diagram showing another example of first support information. Fig. 14 is a diagram schematically explaining the relationship between the region of interest in an endoscopic image and the direction of view of the image sensor 244 under the circumstances of Fig. 13.

[0080] 11 , the display control unit 443 superimposes the first support information A1 generated by the generation unit 442 on the endoscopic image P1 and outputs the superimposed information to the display device 5. The first support information A1 includes information on whether or not the relationship between the gastric cardia W1, which is the region of interest, and the field of view of the image sensor 244 (the optical axis direction of the optical system 243) is appropriate. Here, the appropriate state refers to a state in which the gastric cardia and a portion of the insertion section 21 of the endoscope 2 are included in the field of view of the image sensor 244 within the endoscopic image, and which includes at least one of a state in which a portion of the esophagus can be observed when the cardia opens and a state in which the movement of the esophagus can be observed when peristaltic movement of the esophagus occurs.

[0081] 11 , the first trained model outputs an estimation result estimating whether the relationship between the cardia W1 of the stomach, which is the region of interest, and the field of view L1 (the optical axis direction of the optical system 243) of the image sensor 244 is appropriate, in response to the input of the endoscopic image P1 by the generation unit 442. Specifically, the first trained model outputs an estimation result (output result) estimating whether the field of view L1 of the image sensor 244 and the long axis direction L2 of the esophagus W2 connected to the cardia W1 are arranged in a substantially straight line, in response to the input of the endoscopic image P1 by the generation unit 442. For example, in the case shown in FIG. 11 , the first trained model outputs an estimation result estimating that the relationship between the cardia W1 of the stomach, which is the region of interest, and the field of view L1 of the image sensor 244 is not appropriate because the field of view L1 of the image sensor 244 and the long axis direction L2 of the esophagus W2 connected to the cardia W1 of the stomach, which is the region of interest, are not arranged in a substantially straight line. As a result, as shown in Figure 10, the generation unit 442 generates a message "NG posture" as first support information indicating that the observation posture of the endoscope 2 is inappropriate, based on the estimation result that the first trained model estimates that the gastric cardia W1, which is the region of interest, and the field of view direction L1 of the image sensor 244 are not in an appropriate state.

[0082] 13, the display control unit 443 superimposes the first support information A2 generated by the generation unit 442 on the endoscopic image P2 and outputs the superimposed information to the display device 5. The first support information A2 includes information on whether or not the relationship between the gastric cardia W1, which is the region of interest, and the field of view direction L1 of the imaging element 244 (the optical axis direction of the optical system 243) is appropriate.

[0083] 13 , the first trained model outputs an estimation result estimating whether the relationship between the cardia W1 of the stomach, which is the region of interest, and the field of view L1 (the optical axis direction of the optical system 243) of the image sensor 244 is appropriate, in response to the input of the endoscopic image P2 by the generation unit 442. Specifically, the first trained model outputs an estimation result (output result) estimating whether the field of view L1 of the image sensor 244 and the long axis direction L2 of the esophagus W2 connected to the cardia W1 are arranged in a substantially straight line, in response to the input of the endoscopic image P2 by the generation unit 442. For example, in the case shown in FIG. 14 , the first trained model outputs an estimation result estimating that the relationship between the cardia W1 of the stomach, which is the region of interest, and the field of view L1 of the image sensor 244 is appropriate, because the field of view L1 of the image sensor 244 and the long axis direction L2 of the esophagus W2 connected to the cardia W1 of the stomach, which is the region of interest, are arranged in a substantially straight line. 13, the generation unit 442 generates, as the first support information, a message "GOOD" indicating that the observation posture of the endoscope 2 is appropriate, based on the estimation result that the first trained model estimates that the gastric cardia W1, which is the region of interest, and the field of view direction L1 of the image sensor 244 are appropriate. Furthermore, as shown in Fig. 13, the endoscopic image P2 is in a state in which the gastric cardia W1 and a part of the insertion section 21 of the endoscope 2 are included in the field of view of the image sensor 244, and includes at least one of a state in which a part of the esophagus W2 can be observed when the cardia W1 opens and a state in which the movement of the esophagus W2 can be observed when peristaltic movement of the esophagus W2 occurs.

[0084] Returning to Fig. 10, the description of step S104 and subsequent steps will be continued. In step S104, the control unit 44 determines whether an instruction signal to end observation has been input from the input unit 43. If the control unit 44 determines that an instruction signal to end observation has been input (step S104: Yes), the processing device 4 ends this processing. On the other hand, if the control unit 44 determines that an instruction signal to end observation has not been input (step S104: No), the processing device 4 returns to step S102.

[0085] According to the above-described embodiment 1, the generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first learned model stored in the first learned model memory unit 452, generates first support information based on the output result output by the first learned model, and the display control unit 443 superimposes the first support information generated by the generation unit 442 on the endoscopic image and outputs it to the display device 5, thereby allowing the doctor to move the endoscope 2 to the optimal observation field and supporting the endoscope 2 to the optimal observation field.

[0086] In embodiment 1, the generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first trained model, and the first trained model outputs and generates a message (text) that superimposes the first support information on the endoscopic image based on the output result.However, for example, instead of a message, audio may be used.

[0087] (Embodiment 2) Next, embodiment 2 will be described. The endoscopic system 1 according to embodiment 2 differs from the above-described embodiment 1 in the configurations of the control unit 44 and storage unit 45 of the processing device 4, and also in the processing executed by the treatment device. Specifically, in embodiment 2, navigation information is generated to support the current posture of the insertion unit of the endoscope and the relative positional relationship between the tip of the endoscope and the cardia of the stomach, based on the first support information and operation information of the endoscope. Below, the configurations of the control unit and storage unit according to embodiment 2 will be described, and then the processing executed by the processing device according to embodiment 2 will be described. Note that the same components as those in the endoscopic system 1 according to embodiment 1 will be assigned the same reference numerals, and detailed description thereof will be omitted.

[0088] [Functional Configuration of Storage Unit and Control Unit] Fig. 15 is a diagram showing the functional configuration of the storage unit and control unit of a processing device according to embodiment 2. The processing device 4A shown in Fig. 15 includes a storage unit 45A and a control unit 44A instead of the storage unit 45 and control unit 44 of embodiment 1 described above. Note that Fig. 15 shows only the main parts of the processing device 4A that have been changed from the processing device 4 according to embodiment 1 described above.

[0089] [Functional Configuration of Storage Unit] First, a functional configuration of the storage unit 45 A will be described. The storage unit 45 A further includes a second trained model storage unit 453 in addition to the configuration of the storage unit 45 according to the first embodiment.

[0090] The second trained model stored in the second trained model storage unit 453 is expected to be, for example, a CNN, an RNN, etc. In this case, the second trained model is a model that learns using training data that associates a plurality of endoscopic images, a plurality of pieces of support information generated by the generation unit 442, the cardia of the stomach included in each of the plurality of endoscopic images, the field of view of the image sensor 244, and the attitude of the insertion unit 21 of the endoscope 2 and the relative positional relationship between the tip of the endoscope and the cardia of the stomach based on the cardia of the stomach included in each of the plurality of endoscopic images and the field of view of the image sensor 244, inputs first line of sight information as an input parameter, and outputs information regarding the attitude of the insertion unit 21 of the endoscope 2 and the relative positional relationship between the tip of the endoscope and the cardia of the stomach as an output parameter.

[0091] [Functional Configuration of Control Unit] Next, a description will be given of the functional configuration of the control unit 44 A. The control unit 44 A further includes a navigation generation unit 444 in addition to the functional configuration of the control unit 44 according to the first embodiment described above.

[0092] The navigation generation unit 444 inputs the first support information generated by the generation unit 442, the operation information detected by the operation sensor 27, and the endoscopic image into the second learned model stored in the second learned model memory unit 453, and generates second support information based on the output results output by the second learned model.

[0093] [Processing by Processing Device] Next, the processing executed by the processing device 4A will be described. Fig. 16 is a flowchart showing an outline of the processing executed by the processing device 4A. Note that Fig. 16 describes the processing executed by the processing device 4 when the distal end portion 24 of the endoscope 2 is set in a J-shape facing the gastric cardia at the timing when gas supply into the stomach via the gas supply conduit 223 from the gas supply device 6 shown in Fig. 5 described above is started.

[0094] As shown in FIG. 16, first, the acquisition unit 441 starts sequentially acquiring endoscopic images obtained by processing the imaging data sequentially captured by the endoscope 2 by the image processing unit 41 and operation information detected by the operation sensor 27 (step S201).

[0095] Next, the generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first learned model stored in the first learned model memory unit 452, and generates first support information based on the output result output by the first learned model (step S202).

[0096] Then, the navigation generation unit 444 inputs the first support information generated by the generation unit 442, the operation data detected by the operation sensor 27, and the endoscopic image into the second learned model stored in the second learned model memory unit 453, and generates second support information based on the output results output by the second learned model (step S203).

[0097] Next, the display control unit 443 superimposes the second support information generated by the navigation generation unit 444 on the endoscopic image and outputs it to the display device 5 (step S204).

[0098] FIG. 17 is a diagram showing an example of second support information. As shown in FIG. 17 , the display control unit 443 superimposes second support information A10 generated by the navigation generation unit 444 on an endoscopic image P10 and outputs the superimposed information to the display device 5. The second support information A10 is navigation information including a message for guiding the observation field of the endoscope 2 to an appropriate field of view so that the relationship between the gastric cardia W1, which is the region of interest, and the field of view direction of the imaging element 244 (the optical axis direction of the optical system 243) is appropriate. This message includes information regarding the operation of the endoscope. Specifically, FIG. 17 includes the message "Return the angle of the endoscope 2 to its original position." The message includes information regarding the forward / backward operation of the insertion section 21 of the endoscope 2 in the insertion direction and information regarding the bending operation of the endoscope 211. For example, the types of messages include "Slightly advance insertion," "Set the UP angle to the maximum," "Set the UP angle back by 10 degrees," "Set the R angle by 10 degrees," and "The optimal position has been reached," depending on the generation by the navigation generation unit 442. This allows the doctor to operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information A10, thereby achieving an optimal observation field.

[0099] FIG. 18 is a diagram showing another example of the second support information. As shown in FIG. 18 , the display control unit 443 superimposes second support information A11 generated by the navigation generation unit 444 on an endoscopic image P11 and outputs the superimposed information to the display device 5. The second support information A11 is navigation information of a transition diagram for returning the angle of the endoscope 2 to a state in which the relationship between the gastric cardia W1, which is the region of interest, and the field of view of the image sensor 244 (the optical axis direction of the optical system 243) is appropriate. This transition diagram includes a shape for returning the current shape of the endoscope 2 to a state in which the relationship between the gastric cardia W1, which is the region of interest, and the field of view of the image sensor 244 (the optical axis direction of the optical system 243) is appropriate, by bending the operation unit 22 using the endoscope 2. This allows the doctor to operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information A11 to obtain an optimal observation field.

[0100] FIG. 19 is a diagram showing another example of second support information. As shown in FIG. 19 , the display control unit 443 superimposes second support information A12 generated by the navigation generation unit 444 on an endoscopic image P12 and outputs the superimposed information to the display device 5. The second support information A12 is navigation information using an arrow to navigate the angle of the endoscope 2 to a state where the relationship between the gastric cardia W1, which is the region of interest, and the field of view of the image sensor 244 (the optical axis direction of the optical system 243) is appropriate. This navigation information includes information for navigating the endoscope 2 in the normal direction by performing an advance / retreat operation from the current angle position of the endoscope 2 to a position where the relationship between the gastric cardia W1, which is the region of interest, and the field of view of the image sensor 244 (the optical axis direction of the optical system 243) is appropriate. This allows the doctor to operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information A12 to obtain an optimal observation field.

[0101] 20 is a diagram showing another example of the second support information. As shown in FIG. 20 , the display control unit 443 may output the second support information A13 generated by the navigation generation unit 444 to a display device (not shown) different from the display device 5. The second support information A13 is navigation information that uses arrows to navigate the angle of the endoscope 2 so that the relationship between the gastric cardia W1, which is the region of interest, and the field of view direction of the imaging element 244 (the optical axis direction of the optical system 243) is appropriate. This navigation information includes an operation method of the endoscope 2 indicated by the arrows. This allows the doctor to operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information A12 to achieve an optimal observation field.

[0102] Returning to FIG. 16 , the description of step S205 and subsequent steps will be continued. In step S205, the control unit 44A determines whether an instruction signal to end observation has been input from the input unit 43. If the control unit 44A determines that an instruction signal to end observation has been input (step S205: Yes), the processing device 4A ends this processing. On the other hand, if the control unit 44A determines that an instruction signal to end observation has not been input (step S205: No), the processing device 4A returns to step S202.

[0103] According to the second embodiment described above, the navigation generation unit 444 inputs the first support information generated by the generation unit 442, the operation data detected by the operation sensor 27, and the endoscopic image into the second learned model stored in the second learned model memory unit 453, generates second support information based on the output results output by the second learned model, and the display control unit 443 superimposes the second support information generated by the navigation generation unit 444 on the endoscopic image and outputs it to the display device 5.Therefore, the doctor can operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information to achieve an optimal observation field.

[0104] (Embodiment 3) Next, embodiment 3 will be described. The endoscopic system according to embodiment 3 has a different configuration from the endoscopic system 1 according to embodiment 1 described above. Specifically, the endoscopic system according to embodiment 3 further includes an UPD device (Endoscope Position Detecting Unit) that detects the shape of the endoscope inserted inside the subject, and a magnetic coil is disposed in the insertion portion of the endoscope. Below, the configuration of the endoscopic system according to embodiment 3 will be described, and then the processing executed by the processing device according to embodiment 3 will be described. Note that the same components as those in the endoscopic systems according to embodiments 1 and 2 described above will be assigned the same reference numerals, and detailed description thereof will be omitted.

[0105] [Configuration of Endoscope System] Fig. 21 is a diagram showing the functional configuration of an endoscope system according to embodiment 3. The endoscope system 1B shown in Fig. 21 includes an endoscope 2B and a processing device 4B instead of the endoscope 2 and the processing device 4 according to embodiment 1. Furthermore, the endoscope system 1B further includes a UPD device 10.

[0106] The endoscope 2B further includes a magnetic coil 28 in addition to the configuration of the endoscope 2 according to the above-described embodiment 1. A plurality of magnetic coils 28 are provided in the insertion section 21 of the endoscope 2B, and generate a magnetic field.

[0107] The processing device 4B includes a storage unit 45A and a control unit 44A according to the second embodiment, instead of the storage unit 45 and the control unit 44 of the processing device 4 according to the first embodiment described above.

[0108] The UPD device 10 receives magnetic fields from each of the multiple magnetic coils 28 provided in the endoscope 2B, and displays a three-dimensional model of the endoscope 2B that shows the current shape of the endoscope 2B, in which the positions of each of the received multiple magnetic coils 28 are connected by smooth curves in a three-dimensional model. The UPD device 10 also outputs shape data (shape information) of the three-dimensional model of the endoscope 2B to the processing device 4B.

[0109] [Processing by Processing Device] Next, the processing executed by the processing device 4A will be described. Fig. 22 is a flowchart showing an outline of the processing executed by the processing device 4B. Note that Fig. 22 describes the processing executed by the processing device 4B when the distal end portion 24 of the endoscope 2B is set in a J-shape facing the gastric cardia at the timing when gas supply into the stomach via the gas supply conduit 223 from the gas supply device 6 shown in Fig. 5 described above is started.

[0110] As shown in FIG. 22, first, the acquisition unit 441 starts sequentially acquiring endoscopic images obtained by the image processing unit 41 sequentially processing the image data sequentially captured by the endoscope 2B and shape data of the endoscope 2B from the UPD device 10 (step S301).

[0111] Next, the generation unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the first learned model stored in the first learned model memory unit 452, and generates first support information based on the output result output by the first learned model (step S302).

[0112] Thereafter, the navigation generation unit 444 inputs the first support information generated by the generation unit 442, the shape data detected by the UPD device 10, and the endoscopic image into the second trained model stored in the second trained model storage unit 453, and generates second support information based on the output result output by the second trained model (step S303). Specifically, the navigation generation unit 444 generates the second support information described in the above-mentioned second embodiment.

[0113] Steps S304 and S305 are similar to the processes in steps S204 and S205 in FIG. 16 described above, and therefore detailed description thereof will be omitted.

[0114] According to the third embodiment described above, similar to the second embodiment described above, the doctor can operate the endoscope 2 and adjust the angle of the endoscope 2 in accordance with the second support information to obtain an optimal observation field.

[0115] (Embodiment 4) Next, embodiment 3 will be described. The endoscopic system according to embodiment 4 has a different configuration from the endoscopic system 1 according to embodiment 1 described above. Specifically, the endoscopic system according to embodiment 4 further includes a shape sensor that detects the shape of the endoscope when inserted into the subject. The configuration of the endoscopic system according to embodiment 4 will be described below. Note that the same components as those in the endoscopic systems according to embodiments 1 to 3 described above are assigned the same reference numerals, and detailed description thereof will be omitted.

[0116] [Configuration of Endoscope System] Fig. 23 is a diagram showing the functional configuration of an endoscope system according to embodiment 4. An endoscope system 1C shown in Fig. 21 includes an endoscope 2C and a processing device 4C instead of the endoscope 2 and the processing device 4 according to embodiment 1.

[0117] The endoscope 2C further includes a shape sensor 29 in addition to the configuration of the endoscope 2 according to the above-described embodiment 1. The shape sensor 29 is provided along the insertion section 21 of the endoscope 2C, detects the shape of the endoscope 2C, and outputs the detection result to the processing device 4C.

[0118] The processing device 4B includes a storage unit 45A and a control unit 44A according to the second embodiment, instead of the storage unit 45 and the control unit 44 of the processing device 4 according to the first embodiment described above.

[0119] In the endoscope system 1C configured in this manner, the processing device 4C executes the same processing as that in the third embodiment described above based on the shape data detected by the shape sensor 29.

[0120] According to the fourth embodiment described above, similarly to the second and third embodiments described above, the doctor can operate the endoscope 2 in accordance with the second support information and adjust the angle of the endoscope 2 to obtain an optimal observation field.

[0121] (Other Embodiments) Various inventions can be formed by appropriately combining multiple components disclosed in the endoscope system according to the above-described embodiment of the present disclosure. For example, some components may be omitted from all of the components described in the medical support system according to the above-described embodiment of the present disclosure. Furthermore, the components described in the medical support system according to the above-described embodiment of the present disclosure may be appropriately combined.

[0122] In one embodiment of the present disclosure described above, in order to diagnose gastroesophageal reflux disease, at least two pieces of information, namely, an endoscopic image and esophageal pressure information, are required, and it is not necessary to use all of the information, namely, the endoscopic image, the esophageal pressure information, the intragastric pressure information, and the eructation information.

[0123] Although the above-described embodiment of the present disclosure has been described as a technology capable of diagnosing gastroesophageal reflux disease, this is not intended to be limiting. For example, the technology can be used to evaluate the relaxation dysfunction of the lower esophageal sphincter, which can be useful for diagnosing esophageal achalasia. It can also be used to evaluate the function of the major sphincters present in the human digestive tract. Specifically, it can evaluate the external anal sphincter, i.e., physiological anorectal function associated with aging.

[0124] In addition, in one embodiment, the processing performed by the control unit 44 may be performed by the image processing unit 41, or external devices or servers connectable to the processing device 4 and the image processing unit 41 may be provided with the functions of the control unit 44, i.e., the functions of the acquisition unit 441, the generation unit 442, the display control unit 443, and the navigation generation unit 444.

[0125] Furthermore, in the endoscope system according to an embodiment of the present disclosure, the above-described "unit" can be read as "means," "circuit," etc. For example, a control unit can be read as control means or a control circuit.

[0126] In addition, a program to be executed by an endoscopic system according to one embodiment of the present disclosure is provided as file data in an installable format or an executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0127] Furthermore, the program executed by the medical support system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0128] In the description of the flowcharts in this specification, expressions such as "first," "then," and "continue" are used to clearly indicate the order of processing between steps, but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed within a consistent range. Furthermore, programs are not limited to those consisting of simple branching processing, and branching can be achieved by comprehensively determining more judgment items.

[0129] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention.

[0130] DESCRIPTION OF SYMBOLS 1, 1B, 1C Endoscope system 2 Endoscope 3 Light source device 4, 4A, 4B, 4C Processing device 5 Display device 6 Air supply device 7 Gastric pressure measuring device 8 Microphone 9 Pressure sensor 10 UPD device 21 Insertion section 22 Operation section 23 Universal cord 24 Tip section 25 Bending section 26 Flexible tube section 27 Operation sensor 28 Magnetic coil 29 Shape sensor 31 Light source section 32 Lighting control section 33 Light source driver 41 Image processing section 42 Synchronization signal generation section 43 Input section 44, 44A, 44B, 44C Control section 45, 45A Memory section 61 Air supply section 62 Flow rate measurement section 63 Control section 71 Pressure measurement probe 221 Bending knob 222 Treatment tool insertion section 223 Air supply conduit 224 Switch 231, 232 Connector 241 Light guide 242 Illumination lens 243 Optical system 244 Imaging element 245 Collective cable 441 Acquisition unit 442 Generation unit 443 Display control unit 444 Navigation generation unit 451 Program storage unit 452 First trained model storage unit 453 Second trained model storage unit TU Air supply tube

Claims

1. An image processing device having a processor, wherein the processor acquires a medical image by processing imaging data sequentially output from an image sensor of an endoscope, generates first support information based on the medical image, and outputs the first support information, the first support information including information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

2. An image processing device according to claim 1, wherein the processor inputs the medical image into a first trained model and generates the first support information based on the output result output by the first trained model.

3. An image processing device according to claim 2, further comprising a memory unit, wherein the memory unit stores the first trained model.

4. An image processing device according to claim 1, wherein the first support information includes information indicating whether the relationship between the region of interest and the field of view of the image sensor is appropriate.

5. An image processing device according to claim 1, wherein the processor inputs the medical image and operation information for the insertion portion of the endoscope into a first trained model, and outputs the first support information based on the output result output by the first trained model.

6. An image processing device according to claim 1, wherein the processor inputs the medical image and shape information of the insertion portion of the endoscope into a first trained model, and generates the first support information based on the output results output by the first trained model.

7. An image processing device according to claim 1, wherein the region of interest is a cardia of the stomach, and the first support information includes information relating to the relationship between the cardia and the field of view direction of the image sensor.

8. An image processing device according to claim 7, wherein the first support information includes information relating to the relationship between the longitudinal direction of the esophagus connected to the cardia and the field of view of the image sensor.

9. An image processing device according to claim 8, wherein the first support information includes information indicating whether or not the relationship between the cardia and the field of view of the image sensor is appropriate.

10. An image processing device according to claim 9, wherein the appropriate state is a state in which a part of the esophagus can be observed when the cardia is opened.

11. An image processing device according to claim 10, wherein the appropriate state is a state in which the movement of the esophagus can be observed when peristaltic movement of the esophagus occurs.

12. An image processing device according to claim 9, wherein the processor determines whether the longitudinal direction of the esophagus connected to the cardia and the field of view of the image sensor are arranged in a substantially straight line.

13. An image processing device according to claim 7, wherein the medical image is an image in which the cardia of the stomach and a part of the insertion portion of the endoscope are included in the field of view.

14. An image processing device according to claim 1, wherein the processor generates second support information based on the first support information.

15. An image processing device according to claim 14, wherein the second support information is navigation information for guiding the observation field of the endoscope to an appropriate field of view.

16. An image processing device according to claim 14, wherein the second support information is navigation information including information relating to the operation of the endoscope.

17. An image processing device according to claim 14, wherein the second support information is navigation information including information relating to the forward and backward operation of the insertion portion of the endoscope in the insertion direction.

18. An image processing device according to claim 14, wherein the second support information is navigation information including information relating to the bending operation of the endoscope.

19. An image processing device according to claim 14, wherein the processor inputs the first support information to a second trained model and causes the second trained model to output the second support information.

20. A control method for an image processing device having a processor, wherein the processor acquires a medical image by processing imaging data sequentially output from an image sensor of an endoscope, generates first support information based on the medical image, and outputs the first support information, the first support information including information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

21. A program executed by an image processing device having a processor, the program causing the processor to acquire medical images by processing imaging data sequentially output from an image sensor of an endoscope, generate first support information based on the medical images, and output the first support information, the first support information including information regarding the relationship between a region of interest in the medical image and the field of view direction of the image sensor.

Images