Image processing device, method for controlling image processing device, and program

Abstract

Provided are: an image processing device that can extract desired data from a large amount of data outputted from a sensor; a method for controlling the image processing device; and a program. The image processing device comprises a processor. The processor sequentially acquires endoscopic images obtained by processing imaging data sequentially outputted from an image sensor of an endoscope, detects a region of interest on the basis of the endoscopic images, determines a first action and a second action for the region of interest, and, on the basis of the determination results, subjects first data outputted from a first sensor to a data section extraction process for extracting data.

Description

Image processing device, control method and program for image processing device

[0001] The present disclosure relates to an image processing device, a control method for an image processing device, and a program.

[0002] Gastroesophageal reflux disease (GERD) is a pathological condition caused by the reflux of stomach contents, including a large amount of stomach acid, into the esophagus. Conventionally, an endoscope system using an endoscope has been proposed as a system for diagnosing gastroesophageal reflux disease (see, for example, Patent Document 1). In this technology, an insertion portion of the endoscope is inserted into the stomach, the endoscope is set to be able to image the gastric cardia, and air is blown into the stomach. The endoscope system then evaluates the function of the lower esophageal sphincter based on the pressure change inside the stomach detected during gas blowing into the stomach and the state of the gastric cardia observed from the image captured using the endoscope.

[0003] US Patent Application Publication No. 2020 / 0375485

[0004] In cases where GERD symptoms are confirmed, a physician must observe the condition in Phase 3 of the above-mentioned Patent Document 1 to identify the cause and determine a treatment method. In this case, the physician must observe the peristaltic movement of the esophagus in clinically characteristic changes in organ activity, such as lower esophageal sphincter (LES) relaxation, scope holding sign (SHS) opening, peristalsis, belching, peristalsis, and LES contraction, while observing endoscopic images captured by the endoscope. In this case, the amount of data collected from various sensors becomes enormous depending on the number of sensors and the sensing time, which causes a problem in that it takes a long time for the physician to select the data for evaluation.

[0005] The present disclosure has been made in consideration of the above, and aims to provide an image processing device, a control method for an image processing device, and a program that can extract desired data from a huge amount of data output from a sensor.

[0006] In order to solve the above-mentioned problems and achieve the objective, the image processing device of the present disclosure is an image processing device equipped with a processor, which sequentially acquires endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope, detects a region of interest based on the endoscopic images, determines a first operation and a second operation in the region of interest, and performs a data section extraction process to extract data from the first data output from a first sensor based on the determination result.

[0007] In addition, a control method for an image processing device according to the present disclosure is a control method for an image processing device equipped with a processor, in which the processor sequentially acquires endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope, detects a region of interest based on the endoscopic images, determines a first operation and a second operation in the region of interest, and performs a data section extraction process to extract data from the first data output from a first sensor based on the determination result.

[0008] In addition, the program according to the present disclosure is a program executed by an image processing device having a processor, which causes the processor to sequentially acquire endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope, detect a region of interest based on the endoscopic images, determine a first operation and a second operation in the region of interest, and, based on the result of the determination, perform a data section extraction process to extract data from the first data output from a first sensor.

[0009] According to the present disclosure, it is possible to extract desired data from a huge amount of data output from a sensor.

[0010] FIG. 1 is a diagram illustrating a configuration of an endoscopic system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a configuration of an endoscopic system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating the configuration and arrangement of a pressure sensor. FIG. 4 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating a method for diagnosing gastroesophageal reflux disease according to an embodiment of the present disclosure. FIG. 9 is a block diagram illustrating functional configurations of a storage unit and a control unit according to an embodiment of the present disclosure. FIG. 10 is a flowchart illustrating an outline of processing executed by a processing device according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an outline of extracted content extracted by a processing unit according to an embodiment of the present disclosure.

[0011] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0012] [Configuration of Endoscope System] Fig. 1 is a diagram showing the configuration of an endoscopic system according to an embodiment of the present disclosure. Fig. 2 is a diagram showing the functional configuration of an endoscopic system according to an embodiment of the present disclosure. The endoscopic system 1 shown in Figs. 1 and 2 is used in the medical field and is a system for diagnosing gastroesophageal reflux disease in a subject using an endoscope 2. As shown in Figs. 1 and 2, the endoscopic system 1 includes an endoscope 2, a light source device 3, a processing device 4, a display device 5, an air supply device 6, a gastric pressure measuring device 7, and a microphone 8.

[0013] In this embodiment, the endoscope 2 is a so-called flexible endoscope. A portion of the endoscope 2 is inserted into a living body, captures images of the living body, and outputs image signals generated by the image capture. As shown in FIG. 1 , the endoscope 2 includes an insertion section 21, an operation section 22, and a universal cord 23.

[0014] The insertion section 21 is a section that has at least a portion that is flexible and is inserted into a living body. As shown in Figures 1 and 2, the insertion section 21 includes a tip section 24, a freely bendable bending section 25 (Figure 1) composed of a plurality of bending pieces, a long, flexible flexible tube section 26 (Figure 1) that is connected to the base end side of the bending section 25, a first microphone 81, and a second microphone 82. An image sensor 244 (Figure 2) is built into the tip section 24. The insertion section 21 is inserted into a body cavity of a subject, and captures an image of a subject, such as biological tissue, that is located in a position that is not accessible by external light, using the image sensor 244.

[0015] Here, a pressure sensor 9 that detects pressure applied to the outer peripheral surface is provided on the outer peripheral surface of the insertion portion 21. The detailed configuration and arrangement of the pressure sensor 9 will be described later in the section "Configuration and arrangement of pressure sensor."

[0016] The detailed configuration and arrangement of the first microphone 81 and the second microphone 82 will be described later in the section "Configuration and arrangement of the first microphone 81 and the second microphone 82."

[0017] The operation unit 22 is connected to the base end portion of the insertion section 21. The operation unit 22 receives various operations for the endoscope 2. As shown in Fig. 1 , the operation unit 22 includes a bending knob 221 for bending the bending section 25 in the up-down and left-right directions, a treatment tool insertion section 222 that extends from the operation unit 22 to the tip of the insertion section 21 and inserts treatment tools such as biopsy forceps, an electric scalpel, and an examination probe into the body cavity of the subject, an air supply conduit 223 (Fig. 2) that extends from the operation unit 22 to the tip of the insertion section 21 and supplies air into the body cavity of the subject, and a plurality of switches 224 for operating peripheral devices such as the air supply device 6 and a water supply device (not shown).

[0018] The universal cord 23 incorporates at least a light guide 241 ( FIG. 2 ) and a cable assembly 245 ( FIG. 2 ) that bundles one or more signal lines. The light guide 241 is made of glass fiber or the like and serves as a light guide path for light emitted by the light source device 3. As shown in FIG. 1 , the universal cord 23 branches at the end opposite the end connected to the operation unit 22. The branched ends of the universal cord 23 are provided with a connector 231 that is detachable from the light source device 3 and a connector 232 that is detachable from the processing device 4. A portion of the light guide 241 extends from the end of the connector 231. The universal cord 23 transmits illumination light emitted from the light source device 3 to the distal end 24 via the connector 231 (light guide 241), the operation unit 22, and the flexible tube portion 26. The universal cord 23 also transmits image signals captured by an image sensor 244 provided in the distal end 24 to the processing device 4 via the connector 232. The cable assembly 245 includes a signal line for transmitting an image signal, a signal line for transmitting a drive signal for driving the image sensor 244, and a signal line for transmitting and receiving information including unique information related to the endoscope 2 (image sensor 244). Note that, in this embodiment, the signal lines are described as transmitting electrical signals, but they may also be used to transmit optical signals, or may be used to transmit signals between the endoscope 2 and the processing device 4 by wireless communication.

[0019] The output end side of the light guide 241 is inserted into the tip portion 24. As shown in Fig. 2, the tip portion 24 includes an illumination lens 242, an optical system 243 for collecting light, and an image sensor 244 that is provided at the imaging position of the optical system 243 and receives the light collected by the optical system 243, photoelectrically converts the light into an electrical signal, and performs predetermined signal processing.

[0020] The optical system 243 is configured using one or more lenses, and forms an observation image on the light receiving surface of the image sensor 244. The optical system 243 may have an optical zoom function that changes the angle of view and a focus function that changes the focus.

[0021] The image sensor 244 photoelectrically converts light from the optical system 243 to generate an electrical signal (image signal). The image sensor 244 is configured with a plurality of pixels arranged in a matrix, each of which has a photodiode that accumulates an electric charge according to the amount of light and a capacitor that converts the electric charge transferred from the photodiode into a voltage level. The image sensor 244 photoelectrically converts light incident on each pixel via the optical system 243 to generate an electric signal, sequentially reads out the electric signals generated by pixels arbitrarily designated as readout targets among the plurality of pixels, and outputs the electric signals as an image signal. The image sensor 244 is realized, for example, using a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0022] For ease of explanation, the image signal generated by the image sensor 244 capturing an image will be referred to as a captured image below.

[0023] Here, the endoscope 2 has a memory (not shown) that stores execution programs and control programs for the image sensor 244 to perform various operations, as well as data including identification information of the endoscope 2. The identification information includes the endoscope 2's unique information (ID), model year, specification information, transmission method, etc. The memory may also temporarily store captured images generated by the image sensor 244.

[0024] As shown in FIG. 2 , the light source device 3 includes a light source unit 31 , an illumination control unit 32 , and a light source driver 33 .

[0025] The light source unit 31 emits light under the control of the illumination control unit 32. The light source unit 31 emits light having a wavelength band of visible light (white light (illumination light)). The light source unit 31 is realized using any light source such as an LED (Light Emitting Diode) light source, a laser light source, a xenon lamp, or a halogen lamp. The light source unit 31 may also include one or more lenses. The light generated by the light source unit 31 passes through the light guide 241 and the illumination lens 242 and is emitted from the tip of the tip unit 24 toward the subject.

[0026] The light emitted from the light source unit 31 is not limited to white light, but may be narrowband light having light in a specific wavelength band, or excitation light that excites substances contained in the object of observation.

[0027] The light source driver 33 supplies current to the light source unit 31 under the control of the illumination control unit 32, thereby causing the light source unit 31 to emit light.

[0028] The processing device 4 includes an image processing unit 41 , a synchronization signal generating unit 42 , an input unit 43 , a control unit 44 , and a storage unit 45 .

[0029] Under the control of the control unit 44, the image processing unit 41 performs predetermined image processing on the captured image received from the endoscope 2 to generate an endoscopic image.

[0030] Examples of image processing performed by the image processing unit 41 include optical black subtraction processing (clamping processing), white balance adjustment processing, demosaic processing, color correction matrix processing, gamma correction processing, YC processing that converts RGB signals into luminance color difference signals (Y, Cb / Cr signals), digital gain adjustment that multiplies by digital gain, noise removal, and filter processing that emphasizes structure.

[0031] The image processing unit 41 described above is configured using dedicated processors such as memory and dedicated processors having hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processing) or an FPGA (Field Programmable Gate Array), or various arithmetic circuits that execute specific functions, such as an ASIC (Application Specific Integrated Circuit).

[0032] The synchronization signal generation unit 42 generates a clock signal (synchronization signal) that serves as a reference for the operation of the processing device 4, and outputs the generated synchronization signal to the light source device 3, the air supply device 6, the image processing unit 41, the control unit 44, and the endoscope 2. Here, the synchronization signal generated by the synchronization signal generation unit 42 includes a horizontal synchronization signal and a vertical synchronization signal. Therefore, the light source device 3, the air supply device 6, the image processing unit 41, the control unit 44, and the endoscope 2 operate in synchronization with each other based on the generated synchronization signal.

[0033] The input unit 43 is realized using a keyboard, a mouse, a switch, and a touch panel, and accepts various operations for instructing the operation of the endoscope system 1. The input unit 43 may include a switch provided on the operation unit 22 or a portable terminal such as an external tablet computer.

[0034] The control unit 44 is configured using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuits that execute specific functions such as an ASIC. In the first embodiment, the control unit 44 functions as the processor of the present disclosure.

[0035] The control unit 44 also sequentially acquires endoscopic images obtained by processing the image data sequentially output from the imaging element 244 of the endoscope 2 by the image processing unit 41, detects a region of interest based on the endoscopic image, determines a first motion and a second motion in the region of interest, and performs a data section extraction process to extract data from the first data output from the first sensor based on the determination result. Details of the process performed by the control unit 44 will be described later.

[0036] The storage unit 45 stores various programs executed by the control unit 44 and data including various parameters necessary for the processing of the control unit 44. Specifically, the storage unit 45 has a program storage unit 451 that stores various programs. The various programs can be recorded on computer-readable recording media such as a hard disk, flash memory, CD-ROM, DVD-ROM, or flexible disk and widely distributed. The various programs can also be obtained by downloading them via a communication network. The communication network referred to here is realized, for example, by an existing public line network, a LAN (Local Area Network), a WAN (Wide Area Network), or the like, and can be wired or wireless.

[0037] The storage unit 45 having the above configuration is realized using a ROM (Read Only Memory) in which various programs etc. are pre-installed, and a RAM, hard disk, SSD (Solid State Drive) etc. that store calculation parameters and data etc. for each process.

[0038] In this embodiment, the light source device 3 and the processing device 4 are provided in separate housings, but this is not limiting, and they may be provided integrally in the same housing.

[0039] The display device 5 displays the display image received from the processing device 4 (image processing unit 41) via the video cable. The display device 5 is configured using a monitor such as a liquid crystal or organic EL (Electro Luminescence) monitor.

[0040] The gas supply device 6 adjusts the pressure of gas supplied from a gas supply source (not shown, for example, a carbon dioxide gas cylinder) to a predetermined pressure and discharges the gas from the tip of the insertion section 21 into the space where the tip is located through the gas supply conduit 223. As shown in Fig. 1, the gas supply device 6 includes a gas supply unit 61, a flow rate measurement unit 62, and a control unit 63.

[0041] Although not specifically shown, the gas supply unit 61 includes a primary pressure reducer, a secondary pressure reducer, and a flow control valve. These primary pressure reducer, secondary pressure reducer, and flow control valve are connected in this order by an air supply conduit made of silicone, fluororesin, or the like. Gas supplied from a gas supply source (not shown) passes through the primary pressure reducer, secondary pressure reducer, and flow control valve in this order via the air supply conduit. After being adjusted to a predetermined pressure and flow rate, the gas is discharged from the air supply tube TU ( FIG. 1 ) via the flow rate measuring unit 62. The control unit 63 controls the flow control valve provided in the gas supply unit 61 to adjust the flow rate of gas supplied to the endoscope 2 to a predetermined value. The flow control valve is, for example, a type of electromagnetically driven valve, and is configured as a regulating valve using an electromagnetic coil in the drive unit. The opening degree of the valve unit is controlled by controlling the position of the plunger depending on the magnitude of the current flowing through the electromagnetic coil, thereby adjusting the flow rate of gas flowing through the air supply conduit to a predetermined value.

[0042] The gastric pressure measuring device 7 corresponds to an internal pressure sensor according to the present invention. This gastric pressure measuring device 7 detects the pressure (internal pressure data) inside the space where the tip is located, via a pressure measurement probe 71 inserted through the treatment tool insertion section 222 to the tip of the insertion section 21. A signal (hereinafter referred to as "gastric pressure information") related to the pressure detected by the gastric pressure measuring device 7 (hereinafter referred to as "gastric pressure data") is output to the processing device 4.

[0043] The microphone 8 is placed on the throat of the subject and collects belching sounds (burping sounds) emitted from the esophagus of the subject. The signal related to the belching sounds collected by the microphone 8 (hereinafter referred to as "belching sound information") is output to the processing device 4.

[0044] [Configuration and Arrangement of Pressure Sensor] Next, the configuration and arrangement of the pressure sensor 9 will be described. FIG.

[0045] The pressure sensor 9 shown in Fig. 3 is configured by, for example, a resistance-type pressure sensor or a capacitance-type pressure sensor that detects pressure data using a known method. As shown in Fig. 3, the pressure sensor 9 according to this embodiment is a circular pressure sensor that is provided around the entire circumference in the rotational direction around a central axis along the axial direction of the insertion portion 21. Note that the pressure sensor 9 is not limited to a circular pressure sensor, and a point-type pressure sensor that is provided only around a portion of the entire circumference in the rotational direction may also be used.

[0046] 3, a total of 36 pressure sensors 9 are arranged at 1 cm intervals along the axial direction of the insertion section 21 on the outer peripheral surface of the insertion section 21. As a result, when the insertion section 21 is inserted into the stomach, the pressure sensors 9 are arranged at positions that detect contraction pressure and relaxation pressure at various locations inside the esophagus from the upper esophageal sphincter to the lower esophageal sphincter.

[0047] The number of pressure sensors 9 is not limited to 36, but may be, for example, 6 or more, 12 or more, or even more.

[0048] Here, the pressure sensors 9 may be positioned a predetermined distance from one or more of the nearest other pressure sensors 9. Optionally, the spacing between each pressure sensor 9 may be substantially the same. The spacing may be 3 cm or less, for example 2 cm or less, for example 1 cm, or even less than 1 cm.

[0049] The signal relating to the pressure detected by the pressure sensor 9 described above (hereinafter referred to as esophageal pressure information) is output to the processing device 4.

[0050] [Method for Diagnosing Gastroesophageal Reflux Disease] Next, a method for diagnosing gastroesophageal reflux disease will be described. FIGS. 4 to 8 are diagrams illustrating a method for diagnosing gastroesophageal reflux disease according to embodiment 1. Specifically, FIG. 4 is a flowchart illustrating the method for diagnosing gastroesophageal reflux disease. FIG. 5 is a cross-sectional view showing the vicinity of the gastric cardia, illustrating steps S1 to S3. For ease of explanation, FIG. 5 omits the pressure sensor 9 provided on the outer peripheral surface of the insertion section 21. FIGS. 6 and 7 are diagrams illustrating the structures of the stomach and esophagus. FIG. 6 is a cross-sectional view showing the structures of the stomach and esophagus. FIG. 7 is a view of the stomach and esophagus as viewed from the outside. FIG. 8 is a diagram illustrating images captured in Phases 1 to 3, which correspond to dynamic changes that occur when air is supplied to the stomach (during pneumoperitoneum).

[0051] First, a user such as a doctor inserts the insertion portion 21 into the subject through a natural opening such as the mouth or nose, and introduces the insertion portion 21 into the stomach through the esophagus, as shown in FIG. 5 (step S1).

[0052] After step S1, a user such as a doctor operates (bends) the bending knob 221 to set the field of view to include the gastric cardia (step S2). In step S2, in response to the operation of the bending knob 221, the insertion section 21 is set into a J-shape with the tip pointing toward the gastric cardia, as shown in Fig. 5. In this state, the pressure sensor 9 is disposed at a position to detect the contraction pressure and relaxation pressure at various locations inside the esophagus from the upper esophageal sphincter to the lower esophageal sphincter.

[0053] After step S2, a user such as a doctor operates switch 224 to start supplying air from the air supply device 6 to the stomach through the air supply conduit 223 (step S3). In Fig. 5, the state in which intragastric pressure increases due to air supply is represented by a hollow arrow.

[0054] When gas is supplied to the stomach (pneumoperitoneum), if the subject is healthy, the dynamic changes shown below occur in Phases 1 to 3. Before explaining Phases 1 to 3, the structure of the stomach and esophagus will be described with reference to Figures 6 and 7.

[0055] The Intramural Anti-Reflux Barrier Complex (IM-ARB complex), which is part of the anti-reflux mechanism at the gastroesophageal junction, is composed of three main components: Collar Sling Muscle fibers (Figure 7), Claps Muscle fibers (Figure 7), and the lower esophageal sphincter (Figures 6 and 7).

[0056] The Collar Sling Muscle Fibers are oblique muscles located along the greater curvature of the stomach (Figure 6) and are arranged in a sling-like fashion to surround the upper part of the stomach (Figure 7). These Collar Sling Muscle Fibers tighten the gastric cardia to prevent reflux of gastric contents into the esophagus. From inside the stomach, the Collar Sling Muscle Fibers can be seen as a gastroesophageal flap valve (Figure 6). The gastroesophageal flap valve (GEFV) is a protrusion within the gastroesophageal junction formed by the acute angle between the esophagus and the gastric cardia and is a type of mucosal flap valve (MFV). The MFV located at the gastroesophageal junction is called the gastroesophageal flap valve. The MFV is a flap-shaped portion of the mucosa. The gastroesophageal flap valve changes shape depending on the contraction and relaxation of muscles such as the Collar Sling Muscle Fibers and Claps Muscle Fibers.

[0057] The claps muscle fibers are located on the lesser curvature side of the stomach (Figure 6) and consist of a circular muscle layer. These claps muscle fibers tighten the gastric cardia to prevent the reflux of stomach contents into the esophagus.

[0058] The lower esophageal sphincter (LES) is a ring of muscle located at the junction of the esophagus and stomach that normally contracts to close the esophagus and prevent stomach contents from refluxing.

[0059] Phase 1 begins with the amount of gas sent into the stomach being 0. During Phase 1, the gastroesophageal flap valve and the longitudinal folds of the lesser curvature are observed, as shown in Figures 8(a) and 8(b). The longitudinal folds of the lesser curvature are mucosal folds that extend vertically along the inner wall of the stomach. Their shape changes with the expansion and contraction and relaxation of muscles such as claps and muscle fibers, and they gradually stretch and flatten as intragastric pressure increases. Note that the esophageal mucosa is not observed during Phase 1. During Phase 1, as the amount of gas sent into the stomach increases, the longitudinal folds of the lesser curvature are stretched, the gastroesophageal flap valve gradually flattens, and the crura open, as shown in Figure 8(b). However, the esophageal mucosa is not observed.

[0060] In other words, in Phase 1, it is possible to evaluate the valve function (anti-reflux mechanism) of the stomach structure formed by the gastroesophageal flap valve and the longitudinal folds of the lesser curvature.

[0061] Phase 2 occurs after Phase 1. In Phase 2, as shown in Figure 8(c), the esophageal mucosa is observed beyond the squamocolumnar junction (SCJ) in Figure 6). The SCJ is the intersection of the esophageal squamous epithelium at the gastroesophageal junction (GEJ) and the gastric columnar epithelium, marking the boundary between different epithelial cells in the digestive tract. The GEJ is located near the border between the esophagus and the stomach and is composed of various anatomical components that form a barrier to prevent reflux of gastric contents. The GEJ also includes collar sling muscle fibers, clap muscle fibers, the lower esophageal sphincter, the gastroesophageal flap valve, and the SCJ. If the subject is healthy, the scope holding sign (SHS) is observed. The SHS refers to the phenomenon in which the insertion tube 21 is held in place by contraction of the lower esophageal sphincter when intragastric pressure increases. The state shown in FIG. 8(c) is the SHS.

[0062] That is, in Phase 2, it becomes possible to evaluate the valve function (anti-reflux mechanism) of the lower esophageal sphincter.

[0063] Phase 3 occurs after Phase 2. In Phase 3, intragastric pressure exceeds the contractile force of the lower esophageal sphincter, causing the lower esophageal sphincter to relax. In Figure 8(d), the arrow indicates that gas leaks into the esophagus (producing a belching sound) due to the relaxation of the lower esophageal sphincter. If the subject is healthy, peristaltic waves will subsequently descend from the upper esophagus, and SHS will be observed again.

[0064] That is, in Phase 3, it becomes possible to evaluate the acid clearance function by esophageal peristalsis.

[0065] After step S3, a user such as a doctor determines whether the state corresponds to the above-mentioned Phase 1 based on the diagnostic support information displayed on the display device 5, and evaluates the valve function (state of the gastric cardia) based on the structure of the stomach side (step S4).

[0066] After step S4, a user such as a doctor determines whether or not the state corresponds to the above-mentioned Phase 2 based on the diagnostic support information displayed on the display device 5, and evaluates the valve function of the lower esophageal sphincter (the state of the lower esophageal sphincter) (step S5).

[0067] After step S5, a user such as a doctor determines whether or not the patient is in a state corresponding to the above-mentioned Phase 3 based on the diagnostic support information displayed on the display device 5, and evaluates the acid clearance function due to esophageal peristalsis (the state of peristaltic movement in the esophagus) (step S6).

[0068] After step S6, a user such as a doctor diagnoses gastroesophageal reflux disease based on the evaluation results of steps S4 to S6 (step S7).

[0069] [Functional Configuration of Storage Unit and Control Unit] Next, a description will be given of the functional configuration of the storage unit 45 and the control unit 44. FIG.

[0070] [Functional Configuration of Storage Unit] First, the functional configuration of the storage unit 45 will be described. The storage unit 45 includes a program storage unit 451 that stores various programs executed by the processing device 4, and a trained model storage unit 452 that stores trained models. The trained model stored in the trained model storage unit 452 is, for example, a convolutional neural network (CNN). In this case, the trained model is trained using training data that associates multiple endoscopic images with the lower esophageal sphincter, stomach, cardia, and esophagus contained in each of the multiple endoscopic images, receives the endoscopic images as input parameters, and detects (estimates) and outputs, as output parameters, the position of the lower esophageal sphincter as a region of interest in the endoscopic images, for example, the coordinates (position information) of the lower esophageal sphincter in the endoscopic image or the presence or absence of the lower esophageal sphincter.

[0071] [Functional Configuration of Control Unit] Next, a description will be given of the functional configuration of the control unit 44. The control unit 44 has an acquisition unit 441, a detection unit 442, a creation unit 443, a determination unit 444, a decision unit 445, and a processing unit 446.

[0072] The acquisition unit 441 sequentially acquires endoscopic images obtained by processing the image data sequentially captured by the endoscope 2 by the image processing unit 41 .

[0073] The detection unit 442 detects a region of interest in an endoscopic image using a trained model stored in the trained model storage unit 452. Here, the region of interest is the lower esophageal sphincter. Although the detection unit 442 has been described as detecting the lower esophageal sphincter as the region of interest, in reality, it is the "mucosal portion of the esophagus that is moved" by the movement of the lower esophageal sphincter. However, for convenience of explanation, the region of interest will hereinafter be referred to as the lower esophageal sphincter.

[0074] The creation unit 443 tracks the region of interest detected by the detection unit 442, synchronizes with the synchronization signal generated by the synchronization signal generation unit 42, and creates tracking data of the lower esophageal sphincter of the region of interest that associates a series of detection results related to the lower esophageal sphincter with time data.

[0075] The determination unit 444 determines a first movement of the lower esophageal sphincter based on the tracking data created by the creation unit 443. Furthermore, the determination unit 444 determines a second movement of the lower esophageal sphincter based on the tracking data created by the creation unit 443. Here, the first movement is an opening movement of the lower esophageal sphincter, and the second movement is a closing movement of the lower esophageal sphincter.

[0076] Based on the judgment result of the judgment unit 444, the decision unit 445 sets a data interval from which data is extracted for the first data output from each of the multiple pressure sensors 9, which are first sensors that are synchronized in time with the tracking data created by the creation unit 443, and the second data output from the gastric pressure measuring device 7 (internal pressure sensor), which is the second sensor.

[0077] The processing unit 446 performs a data interval setting and extraction process to extract the data interval set by the determination unit 445 from the first data output from each of the multiple pressure sensors 9, which are first sensors that are synchronized in time with the tracking data created by the creation unit 443, and the second data output from the gastric pressure measuring device 7 (internal pressure sensor), which is the second sensor.

[0078] [Processing by Processing Device] Next, the processing executed by the processing device 4 will be described. Fig. 10 is a flowchart showing an outline of the processing executed by the processing device 4. Note that Fig. 10 describes the processing executed by the processing device 4 in a state where the tip portion 24 of the endoscope 2 is set in a J-shape facing the gastric cardia at the timing when the acquisition unit 441 acquires the gas supply status from the gas supply device 6 shown in Fig. 5 described above and starts supplying gas from the gas supply device 6 to the inside of the stomach through the gas supply conduit 223.

[0079] As shown in FIG. 10, first, the acquisition unit 441 starts sequential acquisition of endoscopic images obtained by processing the image data sequentially captured by the endoscope 2 by the image processing unit 41 (step S101).

[0080] Next, the detection unit 442 detects a region of interest in the endoscopic image using the trained model stored in the trained model storage unit 452 (step S102). Specifically, the detection unit 442 inputs the endoscopic image acquired by the acquisition unit 441 into the trained model stored in the trained model storage unit 452, and detects a region of interest using output parameters output from the trained model. More specifically, the detection unit 442 uses the trained model to detect the lower esophageal sphincter in the endoscopic image based on position information or presence or absence of the lower esophageal sphincter in the endoscopic image output from the trained model.

[0081] Thereafter, the creation unit 443 tracks the region of interest detected by the detection unit 442 (step S104), and creates tracking data of the lower esophageal sphincter in which a series of detection results related to the lower esophageal sphincter are associated with time data in synchronization with the synchronization signal generated by the synchronization signal generation unit 42 (step S105). Specifically, the creation unit 443 tracks the region of interest between endoscopic images sequentially acquired by the acquisition unit 441 by the detection unit 442, and creates tracking data of a group of endoscopic images including the lower esophageal sphincter in which a series of detection results related to the lower esophageal sphincter are associated with time data.

[0082] Next, the determination unit 444 determines the first movement of the lower esophageal sphincter based on the tracking data created by the creation unit 443 (step S106). Specifically, the determination unit 444 determines the opening movement of the lower esophageal sphincter as the first movement using well-known pattern matching or the like for the lower esophageal sphincter included in the tracking data, and determines that the first movement has occurred if the opening movement of the lower esophageal sphincter has occurred. For example, the determination unit 444 performs pattern matching on the lower esophageal sphincter included in the tracking data with a predetermined shape of the opening movement of the lower esophageal sphincter, and determines that the first movement has occurred if the degree of match is equal to or greater than a threshold. If the determination unit 444 determines that the first movement of the lower esophageal sphincter has occurred (step S106: Yes), the processing device 4 determines that the first movement has occurred (step S107). After step S107, the processing device 4 proceeds to step S108. On the other hand, if the determination unit 444 determines that there is no first movement of the lower esophageal sphincter (step S106: No), the determination unit 444 continues this determination until there is a first movement from the tracking data.

[0083] Thereafter, the determination unit 444 determines the second movement of the lower esophageal sphincter based on the tracking data created by the creation unit 443 (step S108). Specifically, the determination unit 444 determines the closing movement of the lower esophageal sphincter as the second movement using well-known pattern matching or the like for the lower esophageal sphincter included in the tracking data, and determines that the second movement has occurred if the closing movement of the lower esophageal sphincter has occurred. For example, the determination unit 444 performs pattern matching on the lower esophageal sphincter included in the tracking data with a predetermined shape of the closing movement of the lower esophageal sphincter, and determines that the second movement has occurred if the degree of match is equal to or greater than a threshold. If the determination unit 444 determines that the second movement of the lower esophageal sphincter has occurred (step S108: Yes), the processing device 4 determines that the second movement has occurred (step S109). After step S109, the processing device 4 proceeds to step S110. On the other hand, if the determination unit 444 determines that there is no second movement of the lower esophageal sphincter (step S108: No), the determination unit 444 continues this determination until there is a second movement from the tracking data.

[0084] Next, based on the determination result of the determination unit 444, the determination unit 445 sets a data interval for extracting data for the first data output from each of the multiple pressure sensors 9 serving as a first sensor that are temporally synchronized with the tracking data created by the creation unit 443, and the second data output from the gastric pressure measuring device 7 (internal pressure sensor) serving as a second sensor (step S110). Specifically, the determination unit 445 sets the start timing and end timing of peristaltic movement based on the determination by the determination unit 444 of the first and second movements of the lower esophageal sphincter, i.e., opening and closing. More specifically, the determination unit 445 sets the opening movement, i.e., the first movement, as the start timing of peristaltic movement, and the closing movement, i.e., the second movement, as the end timing of peristaltic movement. That is, the determination unit 445 sets a data interval for extracting data related to peristaltic movement.

[0085] Then, the processing unit 446 performs a data interval setting and extraction process to extract the data interval set by the determination unit 445 for the first data output from each of the multiple pressure sensors 9, which are the first sensors that are synchronized in time with the tracking data created by the creation unit 443, and the second data output from the gastric pressure measuring device 7 (internal pressure sensor), which is the second sensor (step S111).

[0086] FIG. 10 is a diagram illustrating an overview of the extracted content extracted by the processing unit 446. As shown in FIG. 10 , the processing unit 446 extracts the data intervals set by the determination unit 445 from the sensor data (first data) output from each of the multiple pressure sensors 9 (first sensors) that are temporally synchronized with the tracking data created by the creation unit 443, and the sensor data (second data) output from the gastric pressure measuring device 7 (internal pressure sensor) that is the second sensor. Specifically, the processing unit 446 extracts peristaltic movement data D1 related to the peristaltic movement period. This allows desired data to be extracted from the vast amount of data output from each sensor. In this case, the processing unit 446 generates diagnostic support information based on at least one of the extracted first data and second data and outputs it to the display device 5, an external printer, or the like. This diagnostic support information is information indicating whether peristalsis is functioning normally based on the peristaltic movement data D1. Of course, the processing unit 446 may generate the waveform of at least one of the extracted first data and second data as diagnostic assistance information and output it to the display device 5 or an external printer or the like.

[0087] Next, the determination unit 444 determines whether an instruction signal to end the observation of the subject has been input via the input unit 43 (step S112). If the determination unit 444 determines that an instruction signal to end the observation of the subject has been input (step S112: Yes), the processing device 4 ends this process. On the other hand, if the determination unit 444 determines that an instruction signal to end the observation of the subject has not been input (step S112: No), the processing device 4 returns to the above-mentioned step S102.

[0088] According to the embodiment described above, the control unit 44 sequentially acquires endoscopic images obtained by processing the image data sequentially output from the imaging element 244 of the endoscope 2 by the image processing unit 41, detects a region of interest based on the endoscopic image, determines the first and second movements in the region of interest, and performs a data section extraction process to extract data from the first data output from the first sensor based on the determination result, thereby making it possible to extract desired data from the vast amount of data output from each sensor.

[0089] Furthermore, according to one embodiment, the control unit 44 extracts peristaltic movement data D1 relating to the peristaltic movement period associated with peristaltic movement, so that data associated with peristaltic movement can be extracted from the vast amount of data output from various sensors.

[0090] In one embodiment, the control unit 44 extracts the data interval set by the determination unit 445 from the sensor data (first data) output from each of the multiple pressure sensors 9, which are the first sensors, and the sensor data (second data) output from the gastric pressure measuring device 7 (internal pressure sensor), which is the second sensor. However, a data interval extraction process may be performed to extract data from a signal related to a belching sound from the microphone 8.

[0091] In addition, in one embodiment, the processing performed by the control unit 44 may be performed by the image processing unit 41, or an external device or server connectable to other processing devices 4 may be provided with the functions of the control unit 44, i.e., an acquisition unit 441, a detection unit 442, a creation unit 443, a judgment unit 444, a decision unit 445 and a processing unit 446.

[0092] (Other Embodiments) Various inventions can be formed by appropriately combining multiple components disclosed in the endoscope system according to the above-described embodiment of the present disclosure. For example, some components may be omitted from all of the components described in the medical support system according to the above-described embodiment of the present disclosure. Furthermore, the components described in the medical support system according to the above-described embodiment of the present disclosure may be appropriately combined.

[0093] Furthermore, in one embodiment of the present disclosure, in order to diagnose gastroesophageal reflux disease, at least two pieces of information, namely, an endoscopic image and esophageal pressure information, are required, and it is not necessary to use all of the information, namely, the endoscopic image, the esophageal pressure information, the intragastric pressure information, and the eructation information.

[0094] Furthermore, although one embodiment of the present disclosure has been described as a technology capable of diagnosing gastroesophageal reflux disease, this is not limited to this. For example, it can also evaluate the relaxation dysfunction of the lower esophageal sphincter, which can be useful for diagnosing esophageal achalasia. It can also evaluate the function of the major sphincters present in the human digestive tract. Specifically, it can evaluate the external anal sphincter, i.e., physiological anorectal function associated with aging.

[0095] Furthermore, in one embodiment of the present disclosure, an example has been shown in which the control unit 44 performs a data interval extraction process to extract data based on the first and second movements of the lower esophageal sphincter, but a similar method can also be used to perform a data interval extraction process to extract data from the start of insufflation into the stomach to the first movement.

[0096] Furthermore, in the endoscope system according to an embodiment of the present disclosure, the above-described "unit" can be read as "means," "circuit," etc. For example, a control unit can be read as control means or a control circuit.

[0097] In addition, a program to be executed by an endoscopic system according to one embodiment of the present disclosure is provided as file data in an installable format or an executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0098] In addition, the program executed by the medical support system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0099] In the description of the flowcharts in this specification, expressions such as "first," "then," and "continue" are used to clearly indicate the order of processing between steps, but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed within a consistent range. Furthermore, programs are not limited to those consisting of simple branching processing, and branching can be achieved by comprehensively determining more judgment items.

[0100] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention.

[0101] REFERENCE SIGNS LIST 1 Endoscope system 2 Endoscope 3 Light source device 4 Processing device 5 Display device 6 Air supply device 7 Gastric pressure measuring device 8 Microphone 9 Pressure sensor 21 Insertion section 22 Operation section 23 Universal cord 24 Tip section 25 Bending section 26 Flexible tube section 31 Light source section 32 Lighting control section 33 Light source driver 41 Image processing section 42 Synchronization signal generation section 43 Input section 44 Control section 45 Memory section 61 Air supply section 62 Flow rate measurement section 63 Control section 71 Pressure measurement probe 221 Bending knob 222 Treatment tool insertion section 223 Air supply conduit 224 Switch 231, 232 Connector 241 Light guide 242 Illumination lens 243 Optical system 244 Image pickup element 245 Collective cable 441 Acquisition section 442 Detection unit 443 Creation unit 444 Judgment unit 445 Decision unit 446 Processing unit 451 Program storage unit 452 Learned model storage unit TU Air supply tube

Claims

1. An image processing device having a processor, wherein the processor sequentially acquires endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope, detects a region of interest based on the endoscopic images, determines a first motion and a second motion in the region of interest, and performs data section extraction processing to extract data from the first data output from the first sensor based on the determination result.

2. An image processing device according to claim 1, wherein the imaging data is imaging data of the area around the gastric cardia.

3. An image processing device according to claim 2, wherein the processor detects the lower esophageal sphincter in the endoscopic image as the region of interest.

4. An image processing device according to claim 3, wherein the processor detects the region of interest by inputting the endoscopic image into a trained model that has been trained using training data that combines multiple images of a subject and the region of interest.

5. An image processing device according to claim 3, wherein the processor tracks the movement of the region of interest.

6. An image processing device according to claim 5, wherein the processor creates tracking data of the lower esophageal sphincter by tracking the region of interest, the tracking data associating a series of detection results relating to the lower esophageal sphincter with time data.

7. An image processing device according to claim 6, wherein the processor determines the opening and closing of the lower esophageal sphincter based on the tracking data.

8. An image processing device according to claim 7, wherein the first movement of the region of interest is an opening movement of the lower esophageal sphincter, and the second movement of the region of interest is a closing movement of the lower esophageal sphincter.

9. An image processing device according to claim 8, wherein the processor determines the start and end timings of peristaltic movement based on the determination of whether the lower esophageal sphincter is open or closed.

10. An image processing device according to claim 9, wherein the processor determines the opening of the lower esophageal sphincter as the start timing of peristaltic movement, and the closing of the lower esophageal sphincter as the end timing of peristaltic movement.

11. An image processing device according to claim 6, wherein the first data is synchronized in time with the tracking data.

12. An image processing device according to claim 11, wherein a data interval setting process is performed to extract data related to peristaltic movement from the first data that is synchronized in time with the tracking data.

13. An image processing device according to claim 1, wherein the first sensor is a pressure sensor, and the first data is pressure data output by the pressure sensor.

14. An image processing device according to claim 13, wherein the first sensor is provided on the outer periphery of the insertion portion of the endoscope.

15. An image processing device according to claim 14, wherein the first sensor observes changes in pressure due to relaxation and contraction of at least the lower esophageal sphincter.

16. An image processing device according to claim 13, wherein the first sensor includes a plurality of pressure sensors.

17. An image processing device according to claim 16, wherein the first sensors are arranged at intervals of 3 cm or less.

18. An image processing device according to claim 16, wherein the first sensors are arranged at intervals of 1 cm.

19. An image processing device according to claim 16, wherein the first sensor observes changes in pressure due to relaxation and contraction at various points in the esophagus from the upper esophageal sphincter to the lower esophageal sphincter.

20. The image processing device according to claim 1, further comprising a second sensor that outputs second data.

21. An image processing device according to claim 20, wherein the second sensor is an internal pressure sensor, and the second data is internal pressure data output by the internal pressure sensor.

22. An image processing device according to claim 21, wherein the second sensor observes the internal pressure of the space in which the tip of the endoscope is located.

23. An image processing device according to claim 22, wherein the second data is time-synchronized with tracking data of the lower esophageal sphincter, and the processor performs a data interval extraction process to extract data related to peristaltic movement from the second data output from the second sensor based on a determination result of a first movement and a second movement of the region of interest.

24. An image processing device according to claim 23, wherein the processor outputs at least one of the first data and the second data that have been subjected to the data section extraction processing.

25. An image processing device according to claim 24, wherein the processor generates and outputs diagnostic assistance information based on at least one of the first data and the second data that have been subjected to the data section extraction processing.

26. An image processing device according to claim 25, wherein the diagnosis support information is information on whether or not peristalsis is functioning normally.

27. An image processing device according to claim 26, wherein the first sensor is a pressure sensor, and the first data is pressure data.

28. An image processing device according to claim 27, wherein the processor tracks movement of the region of interest.

29. An image processing device according to claim 28, wherein, by tracking the region of interest, tracking data of the lower esophageal sphincter is created in which a series of detection results relating to the lower esophageal sphincter are associated with time data.

30. An image processing device according to claim 29, wherein said first data is time-synchronized with said tracking data.

31. An image processing device according to claim 30, wherein the processor performs a data interval setting process to extract data related to peristaltic movement from the first data that is synchronized in time with the tracking data.

32. An image processing device according to claim 31, wherein the processor acquires an air supply status from an air supply device, and synchronizes the first data and the tracking data in time from at least the timing when the air supply device starts to supply air.

33. An image processing device according to claim 32, wherein the processor processes the first data by distinguishing between data associated with peristaltic movement and data not associated with peristaltic movement based on the determination result of the first movement and the second movement of the region of interest.

34. A control method for an image processing device having a processor, wherein the processor sequentially acquires endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope, detects a region of interest based on the endoscopic images, determines first and second movements in the region of interest, and performs data section extraction processing to extract data from the first data output from the first sensor based on the determination result of the first and second movements in the region of interest.

35. A program executed by an image processing device having a processor, causing the processor to: sequentially acquire endoscopic images obtained by processing imaging data sequentially output from an image sensor of an endoscope; detect a region of interest based on the endoscopic images; determine first and second movements in the region of interest; and, based on the determination result of the first and second movements in the region of interest, perform data section extraction processing to extract data from the first data output from the first sensor.

Images