Endoscopic examination assisting device, endoscopic examination assisting method, and computer-readable recording medium

Abstract

The present invention provides an image selection assisting device that can easily grasp the site where treatment was performed during an endoscopic examination. In the endoscopic examination assisting device, a plurality of captured image data obtained by multiple shots with the endoscope while the endoscope is inserted into the subject and treatment information indicating the treatment performed using the endoscope in this state are stored in a memory. Then, in the endoscopic examination assisting device, a large intestine expansion image (91) of the subject's body is generated by synthesizing the plurality of captured image data based on the stored plurality of captured image data and treatment information, wherein the large intestine expansion image (91) is attached with a biopsy site mark (91a) indicating the location in the subject where the treatment was performed, and the generated large intestine expansion image (91) is displayed on a display device.

Description

Technical Field

[0001] The present invention relates to an endoscope examination assisting device, an endoscope examination assisting method, and an endoscope examination assisting program for assisting an endoscope-based examination. Background Art

[0002] Conventionally, there are known systems for displaying images acquired through an endoscope in association with the time of capture during an endoscopic inspection. For example, Patent Document 1 describes a system for generating an inspection transition screen in which events extracted from changes in inspection data status values are arranged in time series.

[0003] Furthermore, Patent Document 2 describes a configuration for generating an expanded image that virtually represents a state in which the lumen organ is cut and expanded in its extending direction, based on a plurality of endoscopic images obtained by imaging the inner surface of the lumen organ.

[0004] Patent Document 3 describes a structure in which a source coil is provided on a biopsy forceps as a treatment instrument, and a biopsy position is recorded by triggering an on-state of a biopsy operation signal, thereby automatically recording the position where a biopsy was performed in a desired examination area in a body cavity.

[0005] Previous technical literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-86685

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 2016-43033

[0009] Patent Document 3: Japanese Patent Application Laid-Open No. 2006-280592 Summary of the Invention

[0010] Technical issues to be solved by the invention

[0011] However, the conventional techniques described above cannot easily identify the site where a biopsy or other procedure was performed during endoscopic examination. In particular, the inner surface of the large intestine is difficult to determine the observation site using only an image, making it difficult to identify the site within the large intestine where a biopsy or other procedure was performed.

[0012] For example, as in Patent Document 1, in a configuration that displays an examination transition screen in which extracted events are arranged in time series, it is possible to grasp the time when a procedure such as a biopsy was performed, but it is difficult to grasp the site where the procedure such as the biopsy was performed.

[0013] Furthermore, as in Patent Document 2, in a configuration that generates an expanded image of a lumen organ, the state of the inner surface of the lumen organ can be grasped, but the site where a procedure such as a biopsy was performed cannot be grasped.

[0014] Furthermore, Patent Document 3 does not disclose any means for solving the above-mentioned problems.

[0015] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an endoscope support device, an endoscope support method, and an endoscope support program that can easily grasp a site where treatment has been performed during endoscope inspection.

[0016] Means for solving technical problems

[0017] The endoscopic examination auxiliary device of the present invention has a processor and a memory, the memory stores a plurality of captured image data obtained by multiple shots of the endoscope when the endoscope is inserted into the subject and treatment information indicating treatment performed using the endoscope in the above state, the processor generates an expanded image of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein the expanded image is attached with information indicating the position inside the subject where the treatment was performed, and the generated expanded image is displayed on a display device.

[0018] The endoscopic examination assisting method of the present invention includes the following processing: storing a plurality of captured image data obtained by multiple shots of the endoscope in a state where the endoscope is inserted into the subject and treatment information indicating treatment performed using the endoscope in the above state, generating an expanded image of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the above treatment information, wherein the expanded image is attached with information indicating the position within the subject where the treatment was performed, and displaying the generated expanded image on a display device.

[0019] The endoscopic examination auxiliary program of the present invention is used to enable a computer to perform the following processing: storing a plurality of captured image data obtained by multiple shots of the endoscope in a state where the endoscope is inserted into the subject and treatment information indicating treatment performed using the endoscope in the above state, generating an expanded image of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the above treatment information, wherein the expanded image is attached with information indicating the position within the subject where the treatment was performed, and displaying the generated expanded image on a display device.

[0020] Effects of the Invention

[0021] According to the present invention, it is possible to provide an endoscope support device, an endoscope support method, and an endoscope support program that can easily grasp a site where treatment has been performed during endoscope inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This diagram shows an example of the overall structure of the hospital internal system.

[0023] Figure 2 Yes Figure 1 A diagram showing the schematic configuration of the endoscope department system in the system shown.

[0024] Figure 3 Yes Figure 2 FIG. 2 is a block diagram showing the internal structure of the client PC 22 in the endoscopy department system 20.

[0025] Figure 4 This is a diagram showing an example of a basic screen of an application on the client PC 22 .

[0026] Figure 5 It is a diagram showing an example of the endoscope device 24.

[0027] Figure 6 Yes Figure 5 A schematic diagram of the internal structure of the endoscope device 24 is shown.

[0028] Figure 7 It means by Figure 6 FIG. 1 is a diagram showing an example of a spectrum of light generated by the light source device 5 shown.

[0029] Figure 8 This is a flowchart showing an example of a still image generation process by the image processing device 23 .

[0030] Figure 9 This figure shows an example of an endoscope examination and a large intestine expansion image. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] (Example of Implementation)

[0033] Figure 1 This diagram shows an example of the overall structure of the hospital internal system. Figure 1The illustrated system includes a hospital information system (HIS) 10 , an endoscopy department system 20 , a pathology department system 30 , a medical image storage system 40 , and other department systems 50 , which are connected to a hospital internal LAN (Local Area Network) 60 and can cooperate with each other.

[0034] The HIS 10 is an integrated system including a medical affairs accounting system, a medical reservation system, a medical information system, etc., and has an electronic medical record database, etc. The electronic medical record database stores electronic medical records recording medical information of patients.

[0035] When information regarding a request (instruction) for an examination is issued from another medical department to the endoscopy department (hereinafter referred to as examination request information), the information is transmitted to the endoscopy department system 20 via the HIS 10 .

[0036] Examination request information includes, for example, patient information, instruction key information (such as "instruction number," "date and time of issue"), request source information (such as "requesting department," "requesting doctor," and "request date"), instruction information (such as "requested disease," "examination purpose," "examination type," "examination items," "examination site," and "comments"), and examination reservation information (such as "examination date and time"). Patient information is patient-related information and includes patient-specific information such as "patient ID," "patient name," "date of birth," "age," "gender," and "inpatient / outpatient status."

[0037] The endoscope department system 20 is a system for managing the endoscope department.

[0038] The pathology department system 30 is a system for managing the pathology department.

[0039] Medical image storage system 40 electronically stores, searches, and analyzes images from medical diagnostic imaging devices such as endoscopes, CT (Computed Tomography), and MRI (Magnetic Resonance Imaging). Medical image storage system 40 can be, for example, a PACS (Picture Archiving and Communication System) or another system capable of storing medical images.

[0040] The other department system 50 is a system for managing other departments.

[0041] Figure 2Yes Figure 1 The endoscope department includes a reception area 20A, a pre-treatment room 20B, a plurality of endoscope examination rooms (hereinafter referred to as examination rooms) 20C, a cleaning room 20D, and a conference room 20E.

[0042] The reception desk 20A is where the examination is accepted. The pre-treatment room 20B is used for consultations and pre-treatments before endoscopic examinations. The examination room 20C is used for endoscopic examinations. The cleaning room 20D is used to clean the scopes used in endoscopic examinations.

[0043] Figure 2 The illustrated endoscopy department system 20 includes an endoscopy department server 21, multiple client PCs (Personal Computers) 22, an image processing device 23, an endoscope device 24, a cleaning management device 25, and a reprocessor 26. The endoscopy department server 21, multiple client PCs 22, the image processing device 23, and the cleaning management device 25 are connected to an internal department LAN 27. The internal department LAN 27 is connected to a hospital internal LAN 60. As an example, the endoscopy inspection support device of the present invention can be applied to the image processing device 23.

[0044] The endoscope device 24 includes an insertion portion (scope) having an imaging element at its distal end, and inputs continuous captured image data obtained by continuous imaging of the imaging element into the image processing device 23. Figure 5 In the embodiment of the present invention, a configuration example of the endoscope device 24 will be described later.

[0045] The image processing device 23 is connected to the endoscope device 24 in the inspection room 20C where the image processing device 23 is installed. Image data obtained by continuous imaging by the endoscope device 24 is continuously input to the image processing device 23. The image processing device 23 generates a still image based on image data that satisfies specific conditions among the image data continuously input from the endoscope device 24.

[0046] Specifically, the image processing device 23 uses AI (Artificial Intelligence) to analyze captured image data input from the endoscope 24. The image processing device 23 then generates a still image based on the analyzed captured image data from the captured image data continuously input from the endoscope 24.

[0047] For example, when the image processing device 23 detects that the tip of the scope of the endoscope device 24 has reached a singular point (such as the anus or ileocecal region) through image recognition based on captured image data input from the endoscope device 24, it extracts a plurality of captured image data from that moment on. The image processing device 23 then generates a still image for each of the extracted plurality of captured image data.

[0048] Furthermore, when the image processing device 23 detects the performance of a procedure using the endoscope of the endoscope device 24 through image recognition based on captured image data input from the endoscope device 24, it extracts the captured image data at that moment. This procedure may be, for example, a biopsy using a biopsy forceps attached to the endoscope of the endoscope device 24. When a biopsy is performed using a biopsy forceps, the biopsy forceps is reflected in the captured image data. Therefore, the image processing device 23 can detect a biopsy using a biopsy forceps by detecting the biopsy forceps reflected in the image through image recognition based on the captured image data.

[0049] Furthermore, the image processing device 23 adds, to the generated still image, shooting time information indicating the internal time of the image processing device 23 when the still image was obtained.

[0050] When the endoscope inspection assisting device of the present invention is applied to the image processing device 23, the image processing device 23 stores a plurality of captured image data obtained through multiple captures by the endoscope device 24 and treatment information indicating treatments performed using the endoscope device 24. Furthermore, the image processing device 23 generates an expanded image of the interior of the subject obtained by synthesizing the plurality of captured image data based on the stored plurality of captured image data and treatment information, wherein the expanded image is supplemented with information indicating the location within the subject where the treatment was performed. The image processing device 23 then displays the generated expanded image on another display device (e.g., the display unit 22b of the client PC 22). The processing performed by the image processing device 23 will be described later.

[0051] The image processing device 23 sends the generated still image, shooting time and expanded image to the endoscope department server 21 via the department internal LAN 27. The still image and shooting time information sent by the image processing device 23 are, for example, controlled by the endoscope department server 21. Figure 1 The medical image archiving system 40 shown stores the images.

[0052] The cleaning machine 26 and the cleaning management device 25 are installed in the cleaning room 20D. The cleaning machine 26 is a device that cleans the scope and the like used in endoscopic examination.

[0053] The cleaning management device 25 is connected to the reprocessor 26 and is a computer for storing information such as the cleaning history by the reprocessor 26 in the endoscope department server 21 .

[0054] The endoscope department server 21 is a computer that centrally controls the client PC 22, the image processing device 23, and the cleaning management device 25. The endoscope department server 21 has a built-in database DB, which stores various information (inspection request information, inspection result information, etc.).

[0055] A predetermined application program is installed in the client PC 22 , and the program enables reference and editing of data recorded in the database DB, storage of data in the database, and the like.

[0056] Figure 3 Yes Figure 2 FIG. 2 is a block diagram showing the internal structure of the client PC 22 in the endoscope department system 20. Figure 3 As shown, the client PC 22 is composed of an input unit 22a, a display unit 22b, a recording unit 22c, a transceiver unit 22d, and a control unit 22e.

[0057] The input unit 22 a is an input mechanism for performing various inputs, and is composed of an input device such as a keyboard and a touch panel, or a pointing device such as a mouse and a trackball.

[0058] The display unit 22 b is a display for performing various displays such as images and reports, and is composed of an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube).

[0059] The recording unit 22c is composed of a hard disk or the like for recording various data.

[0060] The transceiver unit 22 d is composed of a transceiver interface circuit and the like, and executes processing for transmitting and receiving various instructions, various requests, and various data via the internal department LAN 27 .

[0061] The control unit 22e includes various processors that execute programs to perform processing, RAM (Random Access Memory), and ROM (Read Only Memory).

[0062] Various processors include general-purpose processors such as CPUs (Central Processing Units) that execute programs and perform various processes, FPGAs (Field Programmable Gate Arrays) and other processors whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs) or ASICs (Application Specific Integrated Circuits), which are processors with circuit structures specifically designed to perform specific processes, such as dedicated circuits.

[0063] More specifically, the various processors are configured as circuits formed by combining circuit elements such as semiconductor elements.

[0064] The control unit 22e may be composed of one of various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0065] The control unit 22e controls each unit of the client PC 22 according to various requests sent from the outside via the above-mentioned program and the department internal LAN 27, and instruction information input from the input unit 22a.

[0066] The internal structure of the client PC 22 has been described, but the image processing device 23 is composed of, for example, Figure 3 The image processing device 23 communicates with the endoscope device 24 or the endoscope department server 21 via the transceiver 22d. The image processing device 23 analyzes the captured image data via the control unit 22e.

[0067] Figure 4 This is a diagram showing an example of a basic screen of an application on the client PC 22. When a user starts an application on the client PC 22 and logs in, the control unit 22e obtains basic screen data from the database DB and uses Figure 4 The basic screen shown is displayed on the display unit 22b. In addition, the client PC 22 is, for example, the client PC 22 of the inspection room 20C, but is not limited thereto. Figure 2 The other client PC 22 is shown.

[0068] This basic screen is composed of an area A displaying a list of inspection request information (partially extracted information), an area B displaying various operation buttons, and an area C displaying a launcher valid for the inspection request information selected in the list in area A.

[0069] In the list of area A, processing items such as "consultation", "acceptance", "inspection", and "accounting" are set for each inspection request information. When the processing shown in each processing item for each inspection request information is completed, an "○" mark is displayed in each processing item.

[0070] The data used to display this mark is stored in the database DB by the endoscopy department server 21 upon completion of each process. For example, upon completion of an examination based on examination request information, the endoscopy department server 21 associates information indicating the completion of the examination with the examination request information and stores it in the database DB. This displays a "○" mark in the "Examination" process item. Information indicating the completion of each process can be manually input or automatically notified by the client PC 22 or the endoscope device 24.

[0071] This application has an "image display" button C1 as an operation button. The "image display" button C1 is displayed on Figure 4 The basic screen is shown in area C. The "image display" button C1 is a button for displaying a captured image obtained by an inspection based on the inspection request information selected from the list in area A.

[0072] The control unit 22e of the client PC 22 periodically acquires data for displaying the basic screen from the database DB, and displays the data on the display unit 22b.

[0073] Figure 5 FIG is a diagram showing an example of an endoscope device 24. Figure 5 As shown, the endoscope device 24 includes an endoscope 1 , and a control device 4 and a light source device 5 connected to the endoscope 1 .

[0074] The control device 4 is connected to a display device 7 that displays images captured by the endoscope 1 inside the subject, and an input unit 6 that is an interface for inputting various information to the control device 4. The control device 4 controls the endoscope 1, the light source device 5, and the display device 7.

[0075] The display device 7 has a display surface with two-dimensionally arranged display pixels, and displays an image based on the image data by drawing pixel data constituting the image data on each display pixel of the display surface. The display device 7 constitutes a display unit that switches the displayed image according to instructions from the control device 4.

[0076] The endoscope 1 includes: an insertion portion 110, which is a tubular component extending in one direction and is inserted into the subject; an operating portion 11, which is arranged at the base end of the insertion portion 110 and is provided with operating components for performing observation mode switching operations, video recording operations, forceps operations, air and water supply operations, and suction operations; an angle button 12, which is provided adjacent to the operating portion 11; and a universal cord 13, which includes connector portions 13A and 13B for freely connecting the endoscope 1 to the control device 4 and the light source device 5, respectively.

[0077] In addition, although Figure 5 Although omitted in the figure, various channels such as a forceps hole for inserting a biopsy forceps for collecting biological tissues such as cells or polyps, a channel for supplying air and water, and a channel for suction are provided inside the operation portion 11 and the insertion portion 110.

[0078] The insertion portion 110 is composed of a flexible flexible portion 10A, a curved portion 10B provided at the distal end of the flexible portion 10A, and a rigid distal end portion 10C provided at the distal end of the curved portion 10B.

[0079] The bending portion 10B is configured to be bendable by rotating the angle knob 12. The bending portion 10B can be bent in any direction and at any angle depending on the part of the subject to which the endoscope 1 is applied, and the distal end portion 10C can be directed in a desired direction.

[0080] Figure 6 Yes Figure 5 A schematic diagram of the internal structure of the endoscope device 24 is shown. Figure 7 It means by Figure 6 FIG. 1 is a diagram showing an example of a spectrum of light generated by the light source device 5 shown.

[0081] The light source device 5 can switch between emitting normal light and special light as illumination light. Normal light is light with a light spectrum suitable for human recognition, such as white light. Special light is light with a light spectrum different from normal light and suitable for computer analysis, such as IEE (Image-Enhanced Endoscopy).

[0082] Specifically, the light source device 5 includes a light source processor 151 , a light source unit 152 , and an optical path coupling unit 154 . The light source processor 151 is connected to the system control unit 44 of the control device 4 , and controls the light source unit 152 based on instructions from the system control unit 44 .

[0083] The light source unit 152 includes, for example, a plurality of semiconductor light sources, which are individually lit or extinguished. When lit, the light output of each semiconductor light source is controlled to emit illumination light that illuminates the object being observed. In this embodiment, the light source unit 152 includes four color LEDs: a V-LED (Violet Light Emitting Diode) 152a, a B-LED (Blue Light Emitting Diode) 152b, a G-LED (Green Light Emitting Diode) 152c, and an R-LED (Red Light Emitting Diode) 152d.

[0084] The light source processor 151 can independently control the V-LED 152a, B-LED 152b, G-LED 152c and R-LED 152d to emit purple light V, blue light B, green light G or red light R with different light amounts. Figure 7 As shown, V-LED 152a generates violet light V with a central wavelength of 405±10 nm and a wavelength range of 380-420 nm. B-LED 152b generates blue light B with a central wavelength of 450±10 nm and a wavelength range of 420-500 nm. G-LED 152c generates green light G with a wavelength range of 480-600 nm. R-LED 152d generates red light R with a central wavelength of 620-630 nm and a wavelength range of 600-650 nm.

[0085] When emitting normal light, the light source processor 151 controls the LEDs 152a to 52d to emit white light with a light intensity ratio of Vc:Bc:Gc:Rc among the violet light V, blue light B, green light G, and red light R. Furthermore, Vc, Bc, Gc, and Rc>0.

[0086] In addition, when irradiating special light, the light source processor 151 controls each LED 152a~52d so that the light intensity ratio between the purple light V, blue light B, green light G and red light R, which are short-wavelength narrow-band light, becomes Vs:Bs:Gs:Rs special light.

[0087] The light intensity ratio Vs:Bs:Gs:Rs differs from the light intensity ratio Vc:Bc:Gc:Rc used when irradiating normal light and is determined appropriately depending on the purpose of observation. For example, to emphasize superficial blood vessels, Vs is preferably greater than the other Bs, Gs, and Rs. To emphasize mid- to deep-layer blood vessels, Gs is preferably greater than the other Vs, Gs, and Rs.

[0088] The optical path coupling unit 154 couples the light emitted from the V-LED 152a, B-LED 152b, G-LED 152c, and R-LED 152d, and emits the coupled light as illumination light. The illumination light emitted from the optical path coupling unit 154 of the light source unit 152 enters a light guide 153 (described later) built into the universal cord 13, passes through the illumination lens 150 provided at the distal end portion 10C of the insertion portion 110, and is illuminated onto the subject.

[0089] The distal end portion 10C of the endoscope 1 is provided with an imaging optical system including an objective lens 121 and a lens group 122, an imaging element 123 for capturing an image of a subject through the imaging optical system, a memory 125 such as a RAM, a communication interface (I / F) 126, an imaging drive unit 127, and a light guide 153 for guiding illumination light emitted from a light source unit 152 to an illumination lens 150. The imaging element 123 constitutes the imaging unit of the present invention.

[0090] The light guide 153 extends from the distal end portion 10C to the connector 13A of the universal cord 13 . When the connector 13A of the universal cord 13 is connected to the light source device 5 , illumination light emitted from the light source 152 of the light source device 5 can enter the light guide 153 .

[0091] The imaging element 123 may be a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. In this embodiment, the imaging element 123 is a CMOS using a rolling shutter.

[0092] The imaging element 123 has a light-receiving surface with a plurality of pixels arranged two-dimensionally. The optical image formed on the light-receiving surface by the imaging optical system is converted into an electrical signal (imaging signal) at each pixel. The imaging element 123 then converts the converted imaging signal from an analog signal into a digital signal with a predetermined number of bits and outputs the converted imaging signal to the memory 125. For example, the imaging element 123 can use an imaging element equipped with a primary color filter or a complementary color filter.

[0093] The imaging element 123 may be disposed at the distal end portion 10C with its light receiving surface perpendicular to the optical axis Ax of the objective lens 121 or parallel to the optical axis Ax of the objective lens 121 .

[0094] The imaging optical system provided in the endoscope 1 is composed of optical components such as lenses and prisms (including the lens group 122) located on the optical path of light from the subject between the imaging element 123 and the objective lens 121, and the objective lens 121. The imaging optical system may be composed of only the objective lens 121.

[0095] The memory 125 temporarily records the digital imaging signal output from the imaging element 123 .

[0096] The communication I / F 126 is connected to the communication interface (I / F) 41 of the control device 4 . The communication I / F 126 transmits the imaging signal recorded in the memory 125 to the control device 4 via the signal line in the universal cord 13 .

[0097] The imaging drive unit 127 is connected to the system control unit 44 of the control device 4 via the communication I / F 126 . The imaging drive unit 127 drives the imaging element 123 and the memory 125 in accordance with a command received from the system control unit 44 via the communication I / F 126 .

[0098] The control device 4 includes a communication I / F 41 connected to the communication I / F 126 of the endoscope 1 via a universal cord 13 , a signal processing unit 42 , a display controller 43 , a system control unit 44 , and a recording medium 45 .

[0099] The communication I / F 41 receives the imaging signal transmitted from the communication I / F 126 of the endoscope 1 and transfers the image pickup signal to the signal processing unit 42 .

[0100] The signal processing unit 42 includes a built-in memory for temporarily recording the image signals received from the communication I / F 41. The signal processing unit 42 processes the image signals (such as demosaicing and gamma correction) recorded in the memory to generate image information in a format that allows for recognition. The image information generated by the signal processing unit 42 is recorded on a recording medium 45 such as a hard disk or flash memory.

[0101] The display controller 43 displays the captured image based on the captured image information generated by the signal processing unit 42 on the display device 7. The coordinates of each pixel data constituting the captured image information generated by the signal processing unit 42 are associated with the coordinates of any display pixel constituting the display surface of the display device 7 and are managed.

[0102] The system control unit 44 controls the various components of the control device 4 and sends instructions to the imaging driver 127 of the endoscope 1 and the light source processor 151 of the light source device 5, thereby centrally controlling the entire endoscope device 24. For example, the system control unit 44 controls the imaging element 123 via the imaging driver 127. Furthermore, the system control unit 44 controls the light source unit 152 via the light source processor 151.

[0103] The system control unit 44 or the signal processing unit 42 includes various processors, RAM, and ROM that execute programs to perform processing.

[0104] Various processors include general-purpose processors such as CPUs and FPGAs that execute programs and perform various processes; processors whose circuit structures can be changed after manufacturing, such as programmable logic devices or ASICs, which are processors with circuit structures specially designed to perform specific processes, such as dedicated circuits.

[0105] More specifically, the various processors are configured as circuits formed by combining circuit elements such as semiconductor elements.

[0106] The system control unit 44 or the signal processing unit 42 may be composed of one of various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0107] Figure 8 This is a flowchart showing an example of a still image generation process by the image processing device 23. The image processing device 23 to which the endoscope inspection support device of the present invention is applied executes, for example, Figure 8 Here, a case of a large intestine examination will be described, in which an endoscope 1 (scope) is inserted from the anus into a subject's body to the ileocecal region, and then the endoscope 1 is slowly withdrawn while performing observation.

[0108] First, the image processing device 23 begins analyzing the captured image data continuously input from the endoscope device 24 using AI (step S81). Specifically, this analysis involves image recognition based on the captured image data. For example, the image processing device 23 uses this image recognition to detect when the tip of the endoscope 1 reaches a singular point, or to detect the presence of a biopsy forceps reflected in the image to detect the implementation of a biopsy. This singular point can be, for example, a specific location within the large intestine, such as the ascending colon, transverse colon, descending colon, sigmoid colon, or rectum.

[0109] Next, the image processing device 23 determines whether the distal end of the endoscope 1 has been detected reaching the ileocecal region based on the analysis results started in step S81 (step S82 ), and waits until reaching the ileocecal region is detected (step S82 : No loop).

[0110] In step S82 , when reaching the ileocecal region is detected (step S82 : YES), the image processing device 23 determines whether reaching the singular point of the distal end of the endoscope 1 is detected based on the result of the analysis started in step S81 (step S83 ).

[0111] If arrival at the singular point is detected in step S83 (step S83: YES), the image processing device 23 generates a still image based on the captured image data input from the endoscope device 24 at that time and capture time information indicating the time when the captured image data was captured (step S84), and the process returns to step S83. The generation of the still image and capture time information in step S84 can be performed at multiple times after the arrival at the singular point is detected.

[0112] In step S83 , when reaching a singular point is not detected (step S83 : No), the image processing device 23 determines whether the implementation of a biopsy is detected based on the result of the analysis started in step S81 (step S85 ).

[0113] If, in step S85, the implementation of a biopsy is detected (step S85: YES), the image processing device 23 generates a still image based on the captured image data input from the endoscope device 24 at that time and capture time information indicating the time when the captured image data was captured (step S86), and then returns to step S83. The generation of the still image and the capture time information in step S86 can be performed at multiple times after the implementation of a biopsy is detected.

[0114] In step S85, if the implementation of the biopsy is not detected (step S85: No), the image processing device 23 determines whether the distal end of the endoscope 1 has been detected reaching the anus based on the results of the analysis started in step S81 (step S87). If the anus is not detected (step S87: No), the image processing device 23 returns to step S83.

[0115] In step S87, if it is detected that the anus has been reached (step S87: "Yes"), the image processing device 23 generates a large intestine expansion image based on the multiple captured image data input from the endoscope device 24 (step S88). This large intestine expansion image virtually represents the state of the large intestine of the subject being cut and expanded along the extension direction, and can be generated by synthesizing multiple still images based on the images taken inside the large intestine. Figure 9 In the following, a specific example of a large intestine expanded image is described.

[0116] For example, the image processing device 23 may generate a large intestine expanded image based on the still images generated in steps S84 and S86. Alternatively, the image processing device 23 may generate a large intestine expanded image based on at least a portion of the captured image data input from the endoscope device 24 between the time the distal end of the endoscope 1 reaches the ileocecal region and the time the distal end of the endoscope 1 reaches the anus. That is, in addition to the still images generated in steps S84 and S86, the image processing device 23 may also generate a still image for generating a large intestine expanded image based on captured image data acquired from the endoscope device 24.

[0117] Next, the image processing device 23 adds biopsy position information indicating the position where the biopsy was performed to the expanded image of the large intestine generated in step S88 (step S89). The biopsy position information is, for example, an image indicating the position where the biopsy was performed in the expanded image of the large intestine. Figure 9 In the following, a specific example of biopsy position information is described.

[0118] Next, the image processing device 23 transmits the expanded large intestine image with biopsy location information added in step S89, the still images generated in steps S84 and S86, and the captured time information to the endoscopy department server 21 (step S90), thus concluding the series of processing steps. This allows the expanded large intestine image generated by the image processing device 23 to be displayed on a display device of another device (e.g., the display unit 22b of the client PC 22).

[0119] That is, the information sent to the endoscope department server 21 in step S90 is Figure 1 The medical image storage system 40 shown in FIG. 1 stores the medical image and displays the medical image on, for example, the client PC 22. Figure 9 In the following, a specific example of display based on the client PC 22 is described.

[0120] Figure 9 This figure shows an example of an endoscopy and a large intestine expansion image. Figure 9 In FIG, the horizontal axis T1 represents the time series from the start to the end of the endoscope examination.

[0121] As shown on the horizontal axis T1, Figure 9 In the example shown, the distal end of the endoscope 1 starts to be inserted into the subject at time t1 (inspection starts). Then, at time t2 after time t1, the distal end of the endoscope 1 reaches the ileocecal region of the subject (reaches the ileocecal region).

[0122] After time t2, observation is performed while the distal end of the endoscope 1 is pulled out from the subject. Then, at time t3 after time t2 and at time t4 after time t3, biopsy is performed using the biopsy forceps of the endoscope 1.

[0123] At time t3, the image processing device 23 detects the implementation of a biopsy by recognizing the reflection of the biopsy forceps based on the captured image data from the endoscope device 24. The image processing device 23 then automatically acquires a still image at time t3 based on the detection result.

[0124] Similarly, at time t4, the image processing device 23 detects the implementation of a biopsy by recognizing the reflection of the biopsy forceps based on the captured image data from the endoscope device 24. The image processing device 23 then automatically acquires a still image at time t4 based on the detection result.

[0125] Then, at time t5 after time t4, the distal end of the endoscope 1 reaches the anus of the subject, and the endoscope 1 is pulled out from the subject (examination is completed).

[0126] Furthermore, in addition to detecting the implementation of a biopsy, the image processing device 23 also generates a still image when it detects that the distal end of the endoscope 1 has reached a singular point in the large intestine based on captured image data from the endoscope device 24 .

[0127] Furthermore, the image processing device 23 adds, to the generated still image, shooting time information indicating the internal time of the image processing device 23 when the still image was obtained.

[0128] The image processing device 23 generates a developed image of the large intestine based on the captured image data obtained from the endoscope device 24 , and adds biopsy position information to the generated developed image of the large intestine.

[0129] As described above, the still images, imaging time information, and large intestine expanded images generated by the image processing device 23 are stored in the medical image storage system 40 .

[0130] Then, for example, in the client PC 22, Figure 9 The endoscope accepts the display of the image (e.g., pressing Figure 4 When the "Image Display" button C1 is pressed, the client PC 22 displays the image on the display unit 22b based on the still image, the shooting time information and the large intestine expanded image stored in the medical image storage system 40. Figure 9 Image 90 is shown.

[0131] The image 90 includes a large intestine expanded image 91 , a thumbnail 92 , a thumbnail 93 , a biopsy mark 93 a , a biopsy site mark 91 a , and a biopsy number notification unit 94 .

[0132] The large intestine expanded image 91 is, for example, Figure 8The expanded large intestine image 91 is generated by the image processing device 23 in step S88. For example, the image processing device 23 generates the expanded large intestine image 91 by combining the generated still images so as to overlap each other's overlapping portions.

[0133] For example, the expanded image 91 of the large intestine is an expanded image obtained by synthesizing still images based on image data captured at times t2 to t5. Figure 9 In the figure, the left end of the expanded large intestine image 91 is the portion captured at time t2, and the right end is the portion captured at time t5. Specifically, the expanded large intestine image 91 includes the ascending colon, transverse colon, descending colon, sigmoid colon, and rectum in order from the left end.

[0134] For example, the biopsy site mark 91a is Figure 8 The figure shows an example of biopsy location information added to the expanded large intestine image by the image processing device 23 in step S89. For example, based on the capturing time information of each still image used to generate the expanded large intestine image 91, the image processing device 23 superimposes a biopsy site marker 91a on the portion of the expanded large intestine image 91 captured at the time the biopsy was detected. As a result, the biopsy site marker 91a is added to the location in the expanded large intestine image 91 where the biopsy was performed.

[0135] Thumbnail 92 is, for example, Figure 8 The thumbnail 93 is a thumbnail (reduced image) of a still image generated by the image processing device 23 together with the shooting time information in step S84. Figure 8 The thumbnail of the still image is generated by the image processing device 23 together with the shooting time information in step S86.

[0136] For each of thumbnails 92 and 93, for example, image processing device 23 identifies the portion of expanded large intestine image 91 captured at the time the still image corresponding to the target thumbnail was captured, based on the capturing time information of each still image used to generate expanded large intestine image 91. The image processing device 23 then places the target still image above the identified portion. Thus, image 90 is generated, including thumbnails 92 and 93 associated with expanded large intestine image 91 based on the capturing time.

[0137] When the client PC 22 receives a selection operation (e.g., clicking on a thumbnail 92 or 93) to select one of the displayed thumbnails 92 or 93, the display unit 22b displays the still image corresponding to the selected thumbnail 92 or 93. The still image corresponding to the thumbnail may be displayed superimposed on the image 90, displayed instead of the image 90, or displayed in parallel with the image 90.

[0138] Furthermore, a biopsy mark 93a indicating the still image when the biopsy was performed may be added to the still image 93 obtained by detecting the biopsy among the thumbnails 92 and 93. This makes it easy to understand that the thumbnail 93 among the thumbnails 92 and 93 corresponds to the still image when the biopsy was performed.

[0139] Furthermore, the client PC 22 may display the number of biopsies performed (for example, the number of thumbnails 93) on the biopsy number notification section 94. Figure 9 In the example shown, since the execution of the biopsy is detected at time t3 and time t4, the client PC 22 displays 2 (Biopsy: 2 times) as the number of performed biopsies on the biopsy count notification unit 94 .

[0140] In this way, according to the image processing device 23 of the endoscopic inspection auxiliary device to which the present invention is applied, it is possible to display on the client PC 22 an expanded image inside the subject obtained by synthesizing the above-mentioned multiple captured image data obtained by multiple shootings of the endoscope 1 when the endoscope 1 is inserted into the subject and treatment information indicating the treatment performed using the endoscope 1 in this state, which is attached with information indicating the position inside the subject where the above-mentioned treatment was performed.

[0141] Therefore, the user observing the expanded image can easily grasp the state of the inner surface of the subject into which the endoscope 1 is inserted. Furthermore, the expanded image is supplemented with information indicating the position within the subject where the above-mentioned treatment was performed, thereby making it easy to grasp the location where treatments such as biopsy were performed during the endoscopic examination.

[0142] (Variation)

[0143] While the configuration described above includes storing various information generated by the image processing device 23 in the medical image storage system 40, and displaying the image 90 and the like based on the information stored in the medical image storage system 40, the present invention is not limited to this configuration. For example, a configuration may also be employed in which the information generated by the image processing device 23 is stored in the endoscopy department server 21, and the client PC 22 displays the image 90 and the like based on the information stored in the endoscopy department server 21.

[0144] Furthermore, the configuration in which the expanded image is displayed on the display unit 22b of the client PC 22 has been described, but the present invention is not limited to this configuration. For example, the expanded image may be displayed on the display device 7 of the endoscope device 24 or another display device (not shown).

[0145] Furthermore, although biopsy using a biopsy forceps of the endoscope 1 has been described as a specific example of treatment using the endoscope 1, treatment using the endoscope 1 is not limited thereto. For example, treatment using the endoscope 1 may also include removal of polyps or the like.

[0146] Furthermore, while the expanded image of the large intestine has been described as an example of an expanded image, the expanded image is not limited thereto. For example, when examining the stomach of a subject using the endoscope 24, a stomach expanded image is displayed that virtually shows the stomach of the subject being incised and expanded along its extension direction.

[0147] While the configuration of the endoscope inspection supporting device of the present invention has been described as being applied to the image processing device 23, the endoscope inspection supporting device of the present invention can also be applied to devices other than the image processing device 23. For example, the endoscope inspection supporting device of the present invention can also be applied to the endoscope device 24, the endoscope department server 21, the medical image storage system 40, and the like.

[0148] As described above, the present specification discloses the following matters. (1)

[0150] An endoscope inspection assisting device having a processor and a memory,

[0151] The memory stores a plurality of captured image data obtained by a plurality of imaging operations performed by the endoscope while the endoscope is inserted into a subject, and treatment information indicating treatment performed using the endoscope in the state.

[0152] The processor generates, based on the plurality of captured image data and the treatment information, an expanded image of the interior of the subject obtained by synthesizing the plurality of captured image data, wherein the expanded image is supplemented with information indicating a position within the subject where the treatment was performed.

[0153] The generated expanded image is displayed on a display device. (2)

[0155] The endoscope inspection assisting device according to (1), wherein

[0156] The memory stores still images obtained by photographing with the endoscope.

[0157] The processor generates the expanded image associated with the still image. (3)

[0159] The endoscope inspection assisting device according to (2), wherein:

[0160] The still image includes a still image obtained based on a detection result of the captured image data according to treatment using the endoscope. (4)

[0162] The endoscope inspection assisting device according to (2) or (3), wherein:

[0163] The still image includes a still image obtained based on a detection result based on the captured image data that the distal end portion of the endoscope has reached a specific site within the subject. (5)

[0165] The endoscope inspection assisting device according to any one of (1) to (4), wherein:

[0166] The plurality of captured image data are obtained after it is detected that the distal end portion of the endoscope has reached the end of the inspection region within the subject. (6)

[0168] The endoscope inspection assisting device according to any one of (1) to (5), wherein:

[0169] The memory stores the plurality of captured image data and the treatment information in association with time information.

[0170] The processor generates the expanded image to which information indicating a position within the subject where the treatment was performed is added based on the time information. (7)

[0172] The endoscope inspection assisting device according to any one of (1) to (5), wherein:

[0173] The processor generates the expanded image to which information indicating a position within the subject where the treatment was performed is added, based on a result of detecting the performance of the treatment using the endoscope based on the captured image data. (8)

[0175] An endoscopic examination assisting method, comprising the following steps:

[0176] storing a plurality of captured image data obtained by a plurality of imaging operations with the endoscope inserted into a subject and treatment information indicating treatment performed using the endoscope in the aforementioned state,

[0177] generating an expanded image of the interior of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein the expanded image is supplemented with information indicating a position within the subject where the treatment was performed;

[0178] The generated expanded image is displayed on a display device. (9)

[0180] An endoscopy assisting program for causing a computer to execute the following processing:

[0181] storing a plurality of captured image data obtained by a plurality of imaging operations with the endoscope inserted into a subject and treatment information indicating treatment performed using the endoscope in the aforementioned state,

[0182] generating an expanded image of the interior of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein the expanded image is supplemented with information indicating a position within the subject where the treatment was performed;

[0183] The generated expanded image is displayed on a display device.

[0184] While various embodiments have been described above with reference to the accompanying drawings, the present invention is not limited to these embodiments. It goes without saying that those skilled in the art will readily appreciate that various variations and modifications are possible within the scope of the claims, and that these variations and modifications are understood to fall within the technical scope of the present invention. Furthermore, the various components of the above embodiments may be arbitrarily combined without departing from the spirit of the invention.

[0185] In addition, this application is based on the Japanese patent application (Japanese patent application 2020-032313) for which it applied on February 27, 2020, and the content is used as a reference in this application.

[0186] Explanation of symbols

[0187] 1-Endoscope, 4-Control device, 5-Light source device, 6, 22a-Input unit, 7-Display device, 10-HIS, 10A-Flexible unit, 10B-Bending unit, 10C-Front end, 11-Operation unit, 12-Angle button, 13-Universal cord, 13A-Connector unit, 13B-Connector unit, 20-Endoscope department system, 20A-Reception desk, 20B-Pre-treatment room, 20C-Examination room, 20D-Cleaning room, 20E-Consultation room, 21-Endoscope department server, 22-Client PC, 22b-Display unit, 22c-Recording unit, 22d-Transmitter / receiver unit, 22e-Control unit, 23-Image processing device, 24-Endoscope device, 25-Cleaning management device, 26-Cleaning machine, 27-Internal department LAN, 30-Pathology department system, 40-Medical image storage system, 41, 126 - Communication I / F, 42 - Signal processing unit, 43 - Display controller, 44 - System control unit, 45 - Recording medium, 50 - Other department systems, 60 - Hospital internal LAN, 90 - Image, 91 - Expanded large intestine image, 91a - Biopsy site mark, 92, 93 - Thumbnails, 93a - Biopsy mark, 94 - Biopsy number notification unit, 110 - Insertion unit, 121 - Objective lens, 122 - Lens group, 123 - Image sensor, 125 - Memory, 127 - Image driver, 150 - Illumination lens, 151 - Light source processor, 152 - Light source unit, 152a - V-LED, 152b - B-LED, 152c - G-LED, 152d - R-LED, 153 - Light guide, 154 - Optical path coupling unit, t1 to t5 - Time, C1 - Button.

Claims

1. An endoscope inspection assisting device comprising a processor and a memory, The memory stores a plurality of captured image data obtained by a plurality of imaging operations with the endoscope inserted into a subject, and treatment information related to treatment performed on the subject using the endoscope in the state, the treatment information including information indicating a position where the treatment was performed. The processor generates an expanded image of the inside of the subject obtained by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein: The expanded image is added with information indicating the position in the subject where the treatment was performed. The generated expanded image is displayed on a display device.

2. The endoscope inspection assisting device according to claim 1, wherein: The memory stores still images obtained by photographing with the endoscope. The processor generates the expanded image associated with the still image.

3. The endoscope inspection assisting device according to claim 2, wherein: The still image includes a still image obtained based on a detection result of the captured image data according to a treatment using the endoscope.

4. The endoscope inspection assisting device according to claim 2, wherein: The still image includes a still image obtained based on a detection result based on the captured image data that the distal end portion of the endoscope has reached a specific site within the subject.

5. The endoscope inspection assisting device according to claim 3, wherein: The still image includes a still image obtained based on a detection result based on the captured image data that the distal end portion of the endoscope has reached a specific site within the subject.

6. The endoscope inspection assisting device according to any one of claims 1 to 5, wherein: The plurality of captured image data are obtained after it is detected that the distal end portion of the endoscope has reached the end of the inspection region within the subject.

7. The endoscope inspection assisting device according to any one of claims 1 to 5, wherein: The memory stores the plurality of captured image data and the treatment information in association with time information. The processor generates the expanded image to which information indicating the position within the subject where the treatment was performed is added based on the time information.

8. The endoscope inspection assisting device according to claim 6, wherein: The memory stores the plurality of captured image data and the treatment information in association with time information. The processor generates the expanded image to which information indicating the position within the subject where the treatment was performed is added based on the time information.

9. The endoscope inspection assisting device according to any one of claims 1 to 5, wherein: The processor generates the expanded image to which information indicating a position within the subject where the treatment was performed is added, based on a result of detecting the performance of the treatment using the endoscope based on the captured image data.

10. The endoscope inspection assisting device according to claim 6, wherein: The processor generates the expanded image to which information indicating a position within the subject where the treatment was performed is added, based on a result of detecting the performance of the treatment using the endoscope based on the captured image data.

11. An endoscopic examination assisting method, comprising the following steps: storing a plurality of captured image data obtained by a plurality of imaging operations with the endoscope inserted into a subject, and treatment information related to treatment performed on the subject using the endoscope in the state, the treatment information including information indicating a position where the treatment was performed, An expanded image of the interior of the subject is generated by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein: The expanded image is added with information indicating the position in the subject where the treatment was performed. The generated expanded image is displayed on a display device.

12. A computer-readable recording medium storing an endoscopy assisting program for causing a computer to execute the following processing: storing a plurality of captured image data obtained by a plurality of imaging operations with the endoscope inserted into a subject, and treatment information related to treatment performed on the subject using the endoscope in the state, the treatment information including information indicating a position where the treatment was performed, An expanded image of the interior of the subject is generated by synthesizing the plurality of captured image data based on the plurality of captured image data and the treatment information, wherein: The expanded image is added with information indicating the position in the subject where the treatment was performed. The generated expanded image is displayed on a display device.

Images