Program product, information processing method, and endoscope system

Abstract

The program of this invention enables a computer to perform the following processing: acquiring endoscopic images of a subject by taking pictures of the subject through multiple imaging units arranged on the end face and circumferential surface of the cylindrical front end portion of the endoscope insertion section; generating a composite image based on the acquired multiple endoscopic images, which is synthesized in the full circumferential direction relative to the front end portion, including the anterior and posterior fields of view; acquiring the insertion distance and rotation angle of the endoscope inserted into the subject's body when the endoscopic images were taken; and outputting the composite image in association with the acquired insertion distance and rotation angle of the endoscope. Furthermore, it is capable of displaying an endoscopic unfolded diagram and comparing it with a virtual endoscopic unfolded diagram from an X-ray CT scan, outputting comparison information.

Description

Technical Field

[0001] This technology relates to a program, an information processing method, and an endoscope system.

[0002] This application claims priority based on Japanese Application No. 2020-167752, filed on October 2, 2020, and incorporates all disclosures set forth in that Japanese application. Background Technology

[0003] An endoscope is a medical device that allows observation and treatment of desired areas by inserting it into a subject's body cavity. It includes: an imaging unit, the front end of which is embedded in an insertion tube within the body cavity; and an illumination device that illuminates the imaging field of view of the imaging unit. Patent Document 1 discloses an endoscope equipped with an illumination device that provides illumination over a wide angle of 180° or more and enables wide-angle observation.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-16021 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, the endoscopic device described in Reference 1 has the following problems: it does not consider generating an image based on images captured by multiple imaging units. Furthermore, it lacks a method for displaying images captured by multiple imaging units using multiple display devices. Additionally, the endoscopic device lacks a display method such as displaying a virtual endoscopic image unfolding diagram like that used in X-ray CT scans, which is also a problem.

[0009] On one hand, an object of the present invention is to provide a program or the like capable of generating images based on images captured by multiple imaging units. On the other hand, an object of the present invention is to display the image using the same unfolded image display method as that used in X-ray CT virtual endoscopy, so as to correspond to the virtual endoscopy image of X-ray CT.

[0010] Technical solutions for solving the problem

[0011] In one embodiment of this disclosure, a program causes a computer to perform the following processing: acquiring endoscopic images of a subject by taking pictures of the subject through a plurality of imaging units disposed on the end face and circumferential surface of the cylindrical front end portion of the endoscope insertion portion; generating a composite image based on the acquired plurality of endoscopic images, which is synthesized in the full circumferential direction relative to the front end portion, including anterior and posterior fields of view; acquiring the insertion distance and rotation angle of the endoscope inserted into the subject's body when the endoscopic images were taken; and outputting the composite image in association with the acquired insertion distance and rotation angle of the endoscope.

[0012] The information processing method in one embodiment of this disclosure causes a computer to perform the following processing: acquiring endoscopic images of a subject by taking pictures of the subject through a plurality of imaging units disposed on the end face and circumferential surface of the cylindrical front end portion of the endoscope insertion portion; generating a composite image based on the plurality of acquired endoscopic images, which is synthesized in the full circumferential direction relative to the front end portion, including the anterior and posterior fields of view; acquiring the insertion distance and rotation angle of the endoscope inserted into the subject's body when the endoscopic images were taken; and outputting the composite image in association with the acquired insertion distance and rotation angle of the endoscope.

[0013] In one embodiment of this disclosure, a program causes a computer to perform the following processing: acquiring endoscopic images of a subject by means of multiple imaging units disposed on the front end of an endoscope insertion section; generating a composite image based on the acquired multiple endoscopic images, which is synthesized in a circumferential direction relative to the front end, including anterior and posterior fields of view; and outputting the generated composite image. The processing of generating the composite image includes the following processing: acquiring a reference image of a predetermined reference object by means of the multiple imaging units; deriving image synthesis parameters based on the acquired reference image; and generating the composite image using the derived image synthesis parameters. This program enables a computer to perform processing including the following: generating an endoscopic unfolded image corresponding to a unfolded display of a virtual endoscopic image from an X-ray CT scan, and comparing the unfolded display from the X-ray CT scan with the endoscopic unfolded image.

[0014] An information processing method according to one embodiment of this disclosure enables a computer to perform the following processing: acquiring endoscopic images of a subject by means of multiple imaging units disposed on the front end of an endoscope insertion section; generating a composite image based on the acquired multiple endoscopic images, which is synthesized in a circumferential direction relative to the front end, including anterior and posterior fields of view; and outputting the generated composite image. The processing of generating the composite image includes the following processing: acquiring a reference image of a predetermined reference object by means of the multiple imaging units; deriving image synthesis parameters based on the acquired reference image; and generating the composite image using the derived image synthesis parameters. This information processing method enables a computer to perform processing including the following processing: generating an endoscopic unfolded image corresponding to an unfolded display of a virtual endoscopic image from an X-ray CT scan, and comparing the unfolded display from the X-ray CT scan with the unfolded endoscopic image.

[0015] An endoscopic system according to one embodiment of this disclosure includes an endoscope and a control unit for processing endoscopic images captured by the endoscope. The endoscope includes: an insertion part for insertion into a subject's body, and a plurality of imaging parts disposed at a cylindrical front end located at the front end of the insertion part; the plurality of imaging parts capture multiple endoscopic images, including anterior and posterior fields of view, in a circumferential direction relative to the front end; the control unit acquires the endoscopic images obtained by capturing images of the subject through the plurality of imaging parts, synthesizes the plurality of endoscopic images captured at multiple times, generates a composite image including a posterior field of view relative to the front end, and outputs the generated composite image. This endoscopic system can output the following information: it can generate an endoscopic unfolded image corresponding to the unfolded image display of a virtual endoscopic image from an X-ray CT scan, and compare the unfolded image display from the X-ray CT scan with the endoscopic unfolded image display.

[0016] Invention Effects

[0017] According to this disclosure, a program or similar tool can be provided that can generate an image based on images captured by multiple imaging units. Attached Figure Description

[0018] Figure 1 This is a schematic diagram showing the outline of the endoscope system involved in Embodiment 1 (Endoscope unfolded diagram).

[0019] Figure 2 This is a block diagram illustrating a structural example of an endoscope device included in an endoscope system.

[0020] Figure 3 This is a block diagram illustrating a structural example of an information processing device included in an endoscope system.

[0021] Figure 4 It is a perspective view schematically showing the front end (circumferential surface) of the insertion part.

[0022] Figure 5 It is a schematic front view showing the front end (end face) of the insertion part.

[0023] Figure 6 This is an explanatory diagram schematically showing the measuring section.

[0024] Figure 7 This is a functional block diagram illustrating the functional units included in the control section of an information processing device.

[0025] Figure 8 This is an explanatory diagram showing the endoscope insertion distance (S-coordinate value).

[0026] Figure 9 This is an explanatory diagram schematically showing how a photograph is taken through a photographing part located at the front end of the insertion part.

[0027] Figure 10 This is an explanatory diagram of the endoscope deployment.

[0028] Figure 11 This is a flowchart illustrating an example of the processing steps performed by the control unit.

[0029] Figure 12 This is a schematic front view of the front end (end face) of the insertion part involved in Embodiment 2 (the imaging part located around the channel).

[0030] Figure 13 It is a schematic side view showing the front end (circumferential surface) of the insertion part.

[0031] Figure 14 This is a schematic side view of the front end (circumferential surface) of the insertion part involved in Embodiment 3 (wireless communication unit / waveguide).

[0032] Figure 15 This is a functional block diagram of the functional units included in the control unit of the information processing device according to Embodiment 4 (stereoscopic vision image).

[0033] Figure 16 This is an explanatory diagram schematically showing how a photograph is taken through a photographing part located at the front end of the insertion part.

[0034] Figure 17 This is an explanatory diagram about the relationship between synthetic images and stereoscopic images in the full circumference direction.

[0035] Figure 18 This is a flowchart illustrating an example of the processing steps performed by the control unit.

[0036] Figure 19 This is a functional block diagram of the functional units included in the control unit of the information processing apparatus according to Embodiment 5 (image synthesis parameters).

[0037] Figure 20 This is an illustrative diagram about using surface projection, which projects endoscopic images onto a continuous curved surface, for geometric distortion correction.

[0038] Figure 21 This is an illustration of how to reproject a composite image onto a curved surface (ellipsoidal display, cylindrical display) for display.

[0039] Figure 22 This is a flowchart illustrating an example of the processing steps performed by the control unit. Detailed Implementation

[0040] (Implementation Method 1)

[0041] The invention will be described in detail below with reference to the accompanying drawings illustrating its embodiments. Figure 1 This is a schematic diagram showing an outline of the diagnostic support system involved in Embodiment 1 (endoscopic unfolded view). The endoscopic system S includes: an endoscope device 10, and an information processing device 6 communicatively connected to the endoscope device 10.

[0042] The endoscope device 10 transmits the images (captured images) captured by the imaging unit 446 of the endoscope 40 to the endoscope processor 20. The endoscope processor 20 performs various image processing operations, such as gamma correction, white balance correction, and shading correction, to generate an endoscope image that is easy for the operator to view. The endoscope device 10 outputs (sends) the generated endoscope image to the information processing device 6. After acquiring the endoscope images sent from the endoscope device 10, the information processing device 6 performs various information processing operations based on these endoscope images and outputs information supporting the diagnosis.

[0043] The endoscope device 10 includes an endoscope processor 20, an endoscope 40, and a display device 50. The display device 50 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0044] The display device 50 is located on the upper shelf of the wheeled storage rack 16. The endoscope processor 20 is stored in the middle shelf of the storage rack 16. The storage rack 16 is positioned near the endoscope examination bed (not shown). The storage rack 16 has a drawer-type shelf for holding the keyboard 15 connected to the endoscope processor 20.

[0045] The endoscope processor 20 is generally rectangular in shape and has a touch panel 25 on one side. A reading unit 28 is arranged below the touch panel 25. The reading unit 28 is an interface for reading and writing portable recording media, such as a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive.

[0046] The endoscope 40 has an insertion section 44, an operating section 43, a universal flexible cable 49, and an observer connector 48. The operating section 43 is equipped with a control button 431. The insertion section 44 is elongated, with one end connected to the operating section 43 via a bend-stop section 45. From the operating section 43 side, the insertion section 44 has a flexible section 441, a bending section 442, and a front end 443 in sequence. The bending section 442 bends in response to the operation of the bending knob 433. Physical detection devices such as a 3-axis accelerometer, a gyroscope sensor, a geomagnetic sensor, a magnetic coil sensor, or an endoscope insertion shape observation device (Colonavi) are installed in the insertion section 44, and the detection results of these physical detection devices can be obtained when the endoscope 40 is inserted into the body of the subject.

[0047] The general-purpose flexible cable 49 is elongated, with one end connected to the operating part 43 and the second end connected to the observer connector 48. The general-purpose flexible cable 49 is flexible. The observer connector 48 is approximately cuboid in shape. The observer connector 48 has air and water supply ports 36 for connecting air and water supply pipes (see reference). Figure 2 ).

[0048] Figure 2 This is a block diagram illustrating a structural example of the endoscope apparatus included in an endoscope system. The control unit 21 is an operational control device for executing the program (program product) of this embodiment. The control unit 21 uses one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), or multi-core CPUs, etc. The control unit 21 is connected to the various hardware components constituting the endoscope processor 20 via a bus.

[0049] The main storage device 22 is, for example, an SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main storage device 22 temporarily stores information required for processing by the control unit 21, as well as the program (program product) currently being executed in the control unit 21. The auxiliary storage device 23 is, for example, an SRAM, flash memory, or hard disk, and is a storage device with a larger capacity than the main storage device 22. For example, acquired images and generated endoscopic images can be stored as intermediate data in the auxiliary storage device 23.

[0050] The communication unit 24 is a communication module or communication interface for communicating with the information processing device 6 via a network through wired or wireless means, such as a narrowband wireless communication module like Wi-Fi or Bluetooth, or a broadband wireless communication module like 4G or LTE. The touch panel 25 includes a display unit such as a liquid crystal display panel and an input unit superimposed on the display unit.

[0051] Display device I / F26 is an interface for connecting the endoscope processor 20 and the display device 50. Input device I / F27 is an interface for connecting the endoscope processor 20 and input devices such as the keyboard 15.

[0052] The light source 33 is, for example, a high-brightness white light source such as a white LED or a xenon lamp. The light source 33 is connected to the bus via a driver (not shown). The lighting, extinguishing, and brightness changes of the light source 33 are controlled by the control unit 21. The illumination light emitted from the light source 33 is incident on the optical connector 312. The optical connector 312 engages with the observer connector 48 and provides illumination light to the endoscope 40.

[0053] Pump 34 generates pressure for supplying air and water to the endoscope 40. Pump 34 is connected to a bus via a driver (not shown). The on / off state of pump 34 and pressure changes are controlled by control unit 21. Pump 34 is connected to air and water supply ports 36 provided on the observer connector 48 via water supply tank 35.

[0054] The following describes the functional overview of the endoscope 40 connected to the endoscope processor 20. Fiber optic cables, cable bundles, air supply tubes, and water supply tubes are inserted inside the observer connector 48, universal flexible cable 49, operation section 43, and insertion section 44. Illumination light emitted from the light source 33 is emitted through the illumination window provided on the front end portion 443 via the optical connector 312 and the fiber optic cable. The area illuminated by the illumination light is captured by the imaging section provided on the front end portion 443. The captured images are transmitted from the imaging section to the endoscope processor 20 via the cable bundle and electrical connector 311.

[0055] The control unit 21 of the endoscope processor 20 executes the program stored in the main storage device 22, thereby performing the function of the image processing unit 211. The image processing unit 211 performs various image processing on the image (captured image) output from the endoscope 40, such as gamma correction, white balance correction, and shadow correction, and outputs it as an endoscope image.

[0056] Figure 3 This is a block diagram illustrating an example structure of the information processing device 6 included in an endoscope system. The information processing device 6 includes a control unit 62, a communication unit 61, a storage unit 63, and an input / output (I / O) unit 64. The information processing device 6 can be, for example, a server device or a personal computer. The server device includes not only a single server device but also a cloud server device or a virtual server device composed of multiple computers. The information processing device 6 can also be configured as a cloud server located on an external network accessible from the endoscope processor 20.

[0057] The control unit 62 includes one or more arithmetic processing devices with timing functions, such as CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units). By reading and executing the program P (program product) stored in the storage unit 63, it can perform various information processing and control processing involving the information processing device 6. Alternatively, the control unit 62 may be constructed from a quantum computer chip, and the information processing device 6 may be a quantum computer.

[0058] Storage unit 63 includes volatile storage areas such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory, as well as non-volatile storage areas such as EEPROM or hard disk. Programs (program products) and data to be referenced during processing are pre-stored in storage unit 63. The programs (program products) stored in storage unit 63 can also be programs (program products) read from recording medium 630 that can be read from information processing device 6. Alternatively, programs (program products) can be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in storage unit 63. Storage unit 63 stores actual files (instance files of neural networks (NN)) constituting the learning models (lesion learning model 631, region of interest learning model 632) described later. These actual files can also be part of the program (program product).

[0059] The communication unit 61 is a communication module or communication interface for communicating with the endoscope device 10 via wired or wireless means, such as a narrowband wireless communication module like Wi-Fi (registered trademark) or Bluetooth (registered trademark) or a broadband wireless communication module like 4G or LTE.

[0060] The I / O input / output I / F64 conforms to communication standards such as USB or DSUB and is a communication interface for serial communication with external devices connected to the I / O input / output I / F64. The I / O input / output I / F64 is connected to a display unit 7 such as a monitor and an input unit 8 such as a keyboard. The control unit 62 outputs the information processing results based on the execution commands or events input from the input unit 8 to the display unit 7.

[0061] Figure 4 It is a perspective view schematically showing the front end (circumferential surface 445) of the insertion part. Figure 5 This is a schematic front view showing the front end (end face 444) of the insertion part. Multiple imaging parts 446 and an imaging light source 447 are provided at the front end 443 of the endoscope 40.

[0062] The plurality of imaging units 446 include four imaging units 446 (shown as four in the figure) arranged circumferentially at equal intervals on the circumferential surface 445 of the cylinder in the front end portion 443, and one imaging unit 446 (shown as one in the figure) provided on the end face 444 of the cylinder in the front end portion 443. The imaging unit 446 is, for example, composed of an image sensor such as a CMOS sensor and a substrate on which the image sensor is mounted. The substrate on which the image sensor is mounted (the imaging unit 446) is connected to the endoscope processor 20, for example, via a communication line and an electrical connector 311.

[0063] The imaging portion 446 on the end face 444 of the cylindrical portion 443 takes pictures forward relative to the insertion direction of the endoscope 40. Multiple imaging portions 446 on the peripheral surface 445 of the cylindrical portion 443 take pictures laterally relative to the insertion direction of the endoscope 40. That is, the multiple imaging portions 446 on the peripheral surface 445 of the cylindrical portion 443 can each take pictures of a part of the body further posterior to the part of the body photographed by the imaging portion 446 on the end face 444 of the cylindrical portion 443, relative to the insertion direction of the endoscope 40. In this embodiment, there are four imaging portions 446 on the peripheral surface of the cylindrical front portion 443, and each imaging portion 446 is arranged at equal intervals of 90-degree circumferential angles, but this is not a limitation. For example, there may be three imaging portions 446 on the peripheral surface of the cylindrical front portion 443, arranged at equal intervals of 120-degree circumferential angles.

[0064] The multiple imaging light sources 447 include four imaging light sources 447 (shown as four in the figure) arranged circumferentially at equal intervals on the circumferential surface 445 of the cylinder in the front end portion 443, and two imaging light sources 447 (shown as two in the figure) provided on the end face 444 of the cylinder in the front end portion 443. The imaging light sources 447 can be light sources that emit illumination light from a light source 33 guided by an optical fiber bundle through an illumination window provided in the front end portion 443, or they can be light sources such as white LEDs provided in the front end portion 443. If white LEDs are provided in the front end portion 443, the light source 33 included in the endoscope device may not be necessary. The multiple imaging light sources 447 are respectively provided corresponding to the imaging portions 446 provided on the end face 444 of the cylinder in the front end portion 443 and the multiple imaging portions 446 provided on the circumferential surface 445 of the cylinder in the front end portion 443, thus providing sufficient light intensity when these multiple imaging portions 446 are used to image internal parts.

[0065] In this way, by setting multiple imaging units 446 on the circumferential surface 445 and end face 444 of the cylinder in the front end 443, multiple endoscopic images can be acquired from multiple different viewpoints from a single (same) viewpoint position.

[0066] Figure 6 This is an explanatory diagram schematically showing the measuring part 9 (mouthpiece, accessory). The measuring part 9 is configured, for example, as a mouthpiece installed in the mouth or an accessory installed in the anus, and is an annular member with a hole formed for inserting the endoscope 40 into the insertion part 44.

[0067] A detection section 91 is provided circumferentially within the annular measuring section 9. The detection section 91 is composed of an optical sensor or a magnetic sensor. A subject 449 corresponding to the detection section 91 is provided in the insertion section 44 of the endoscope 40.

[0068] The object to be detected 449 is an elongated body arranged along the axial direction of the insertion part 44 of the endoscope 40. It is composed of optical reflective members or magnetic bodies, etc., and is detected by the detection part 91, which is composed of optical sensors or magnetic sensors, etc.

[0069] When the insertion part 44 of the endoscope 40 is inserted into the body through the measuring part 9, the measuring part 9 detects the object 449 disposed in the insertion part 44 and calculates the insertion distance and rotation angle of the insertion part 44, and outputs the calculated insertion distance and rotation angle to the processor of the endoscope 40. That is, the measuring part 9 performs the functions of an insertion distance measuring part and a rotation angle measuring part.

[0070] When the insertion part 44 of the endoscope 40 is inserted into the body, if it is an upper endoscope for imaging, it is inserted into the body through the mouthpiece (measuring part 9) installed in the oral cavity; if it is a lower endoscope for imaging, it is inserted into the body through the accessory (measuring part 9) installed in the anus. Therefore, in any case, the insertion distance and rotation angle of the insertion part 44 of the endoscope 40 can be accurately determined.

[0071] In this embodiment, the rotation angle of the insertion part 44 is measured using a measuring part 9 consisting of a mouthpiece or an accessory, but it is not limited to this. Alternatively, a gyroscope or accelerometer may be provided on the operating part 43 of the endoscope 40 to measure the rotation of the operating part 43, and the rotation angle of the insertion part 44 may be derived based on the measured rotation.

[0072] Figure 7 This is a functional block diagram illustrating the functional units included in the control unit 62 of the information processing device 6. The control unit 21 of the endoscope processor 20 (endoscope device 10) performs the function of the image processing unit 211 by executing the program stored in the main storage device 22. The control unit 62 of the information processing device 6 performs the functions of the acquisition unit 621, the composite image generation unit 622, the unfolded image generation unit 623, the lesion region determination unit 624 (lesion learning model 631), and the output unit 625 by executing the program stored in the storage unit 63.

[0073] The image processing unit 211 of the endoscope processor 20 performs various image processing operations, such as gamma correction, white balance correction, and shadow correction, on the images (multiple captured images) output from the endoscope 40 (captured by multiple imaging units 446), and outputs them as multiple endoscopic images. The image processing unit 211 outputs (sends) the generated endoscopic images and the examination date and time based on the capture time of the endoscopic images to the information processing device 6. The image processing unit 211 can also output the subject ID input from the keyboard 15 to the information processing device 6.

[0074] In order to measure the surrounding environment of the endoscope 40, the endoscope processor 20 (image processing unit 211) outputs information about the insertion distance (S-coordinate) and rotation angle of the endoscope 40 from sensors such as the measuring unit 9 (mouthpiece, accessory) provided corresponding to the insertion part 44 (flexible tube) of the endoscope 40 to the information processing device 6. The image processing unit 211 may also overlay the information about the insertion distance and rotation angle of the endoscope 40 obtained from the measuring unit 9 onto the endoscope image and display it on the display device 50.

[0075] The acquisition unit 621 acquires multiple endoscopic images, S-coordinates (insertion distance), and rotation angles output by the endoscopic processor 20. Figure 8This is an explanatory diagram showing the insertion distance (S-coordinate value) of the endoscope 40. As shown, when a digestive organ or the like is represented in 3D by an image captured by the endoscope 40, a space is formed inside the inner wall of the digestive organ or the like, and this space becomes the insertion path for inserting the endoscope 40. The insertion distance of the endoscope 40 (insertion part 44), i.e., the S-coordinate, is located inside this insertion path (inside the inner wall of the digestive organ or the like), and is a position where the path length of the insertion path is approximately equal to this insertion distance. Therefore, based on the S-coordinate, the coordinates of the tip 443 of the endoscope 40, located inside the inner wall of the digestive organ or the like, can be derived.

[0076] In the rotation angle of the endoscope 40 (insertion part 44), with the Z-axis defined as parallel to the subject's body axis, the Z-axis represents the vertical direction, the Y-axis represents the front-back direction, and the Z-axis represents the left-right direction. These axes are used as the rotation axes to define the rotation angle. When taking endoscopic images using the upper endoscope, the rotation angle is the angle with the vertical direction (Z-axis) parallel to the subject's body axis as the rotation axis. When taking endoscopic images using the lower endoscope, the rotation angle is the angle with the front-back direction parallel to the length of the subject's body cavity as the rotation axis.

[0077] The acquisition unit 621 outputs the acquired multiple endoscopic images, S-coordinates (insertion distance), and rotation angles to the composite image generation unit 622. When outputting this information to the composite image generation unit 622, the acquisition unit 621 can establish and output the S-coordinates (insertion distance) and rotation angles separately with the multiple endoscopic images. These multiple endoscopic images are endoscopic images captured at the same shooting time by the imaging unit 446 provided on the front end portion 443 (end face 444, peripheral face 445). The acquisition unit 621 can establish and output these multiple endoscopic images separately with the shooting time (time information) to the composite image generation unit 622. The acquisition unit 621 can also, for example, acquire a virtual endoscopic image of the subject ID and a unfolded view of the virtual endoscopic image from a communicatively connected CT device (X-ray CT) based on the subject ID output from the endoscopic processor 20. The virtual endoscopic image is generated (reconstructed) based on three-dimensional medical images from X-ray CT, MRI, or cone-beam X-ray CT, and displays the internal condition of an organ (body cavity) through a virtual endoscope. Based on the virtual endoscopic image, an unfolded diagram is generated by setting the vertical axis to the rotation angle and the horizontal axis to the insertion distance. This unfolded diagram (the unfolded diagram of the virtual endoscopic image) includes regions corresponding to multiple endoscopic images acquired by the acquisition unit 621. The acquisition unit 621 outputs the unfolded diagram of the virtual endoscopic image acquired from X-ray CT to the output unit 625.

[0078] The composite image generation unit 622, based on the acquired multiple endoscopic images, includes the front and rear sides in the field of view in the direction of travel of the insertion unit 44, and generates a full-circumference (360-degree FOV image) composite image relative to the front end 443. Figure 9 This is an explanatory diagram schematically illustrating the imaging process performed by the imaging unit 446 located at the front end portion 443 of the insertion portion 44. As shown in the accompanying drawings of this embodiment, each imaging unit 446 located on the front end portion 443 (end face 444, peripheral face 445) includes the front, side, and rear views in the direction of travel of the insertion portion 44. Therefore, a 360-degree FOV image can be effectively synthesized based on multiple acquired endoscopic images. In particular, the imaging unit 446 located on the peripheral face 445 can capture images of the portion including the front (upper) side of the tumor relative to the front (above) view of the tumor, thereby improving observation efficiency and enhancing the accuracy of diagnostic support information through AI and other means based on the endoscopic images captured in this way.

[0079] When synthesizing multiple endoscopic images, the image synthesis unit 622 can also perform geometric correction on each endoscopic image. Geometric correction can be, for example, spherical correction using surface projection (projection transformation), i.e., projecting the acquired endoscopic image onto a continuous curved surface. Alternatively, geometric correction can be performed based on image processing algorithms such as affine transformation, pseudo-affine transformation, quadratic isogonal transformation, or two-dimensional projective transformation. Alternatively, geometric correction can be based on the specifications or optical characteristics of the lenses included in the imaging unit 446, performing optical simulation on a large number of images, and using artificial intelligence to perform machine learning on the results to generate a dictionary corresponding to the position of the captured image. This dictionary is then used to restore and correct the image without aberrations. By performing geometric correction, the extraction accuracy of repetitive regions (repeated areas) in these multiple endoscopic images can be improved, resulting in an image that is easy to synthesize (a coordinate system image that is easy to stitch together).

[0080] The composite image generation unit 622 generates a composite image by overlapping overlapping regions (regions where the same object is observed) in multiple geometrically corrected endoscopic images to achieve alignment. The composite image generation unit 622 can correlate the generated composite image with the insertion distance and rotation angle of the endoscope 40 when the multiple original endoscopic images of the composite image were captured, and further with relevant time information from the time the multiple original endoscopic images of the composite image were captured (the capturing time). The composite image generation unit 622 outputs the generated composite image to the unfolded image generation unit 623.

[0081] The unfolded image generation unit 623 generates an endoscope unfolded image with the vertical axis being the rotation angle (second axis) and the horizontal axis being the insertion distance (first axis) based on the acquired composite image and the insertion distance and rotation angle associated with the composite image. Figure 10This is an explanatory diagram regarding endoscopic unfolding diagrams. As shown in the accompanying drawings of this embodiment, by converting a composite image into an endoscopic unfolding diagram, the visibility of the observed area can be improved. The unfolding diagram generation unit 623 can align multiple endoscopic unfolding diagrams unfolded from these composite images based on rotation angles when acquiring multiple composite images at different times, and display them side-by-side in the order of the acquisition time (order of insertion distance). The unfolding diagram generation unit 623 outputs the generated endoscopic unfolding diagrams to the lesion area determination unit 624 and the output unit 625. Furthermore, in the endoscopic unfolding diagram, the vertical axis is the rotation angle and the horizontal axis is the insertion distance, but it is also possible to represent (unfold) the image by setting the horizontal axis to the rotation angle and the vertical axis to the insertion distance.

[0082] The lesion region determination unit 624 includes a lesion learning model 631 that outputs lesion regions such as tumors when an endoscopic image is input. The lesion region determination unit 624 inputs the acquired endoscopic unfolded image into the lesion learning model 631 to determine whether the endoscopic unfolded image contains lesions such as tumors. Since the endoscopic unfolded image is a composite image of multiple endoscopic images, by inputting the endoscopic unfolded image into the lesion learning model 631, information on whether the endoscopic unfolded image contains lesions such as tumors can be obtained. If a tumor is present, the lesion region determination unit 624 determines the region of the tumor (tumor region) in the endoscopic unfolded image based on the insertion distance and rotation angle of the endoscope 40, and outputs the relevant information of the tumor region to the output unit 625.

[0083] The lesion learning model 631 is a neural network that learns using training data, and it is assumed to be used as a program module, which is part of artificial intelligence software. The lesion learning model 631 is used in an information processing device 6, which, as described above, includes a control unit 62 (CPU, etc.) and a storage unit 63. Thus, the information processing device 6, having computational processing capabilities, performs related operations, thereby constituting a neural network system. Specifically, the control unit 62 of the information processing device 6 operates as follows: based on the instructions of the lesion learning model 631 stored in the storage unit 63, it performs calculations to extract feature quantities from the endoscopic image input to the input layer, and outputs diagnostic support information, including the presence or absence of lesions, from the output layer.

[0084] The input layer has multiple neurons that receive pixel values ​​from the endoscopic image and pass the input pixel values ​​and distance information to the intermediate layer. The intermediate layer has multiple neurons that extract image features from the endoscopic image and pass the extracted image features to the output layer. The output layer has one or more neurons that output information related to the presence or absence of lesions and symptom staging, and outputs this information based on the image features output from the intermediate layer. For example, when the lesion learning model 631 is a CNN (Convolutional Neural Network), the intermediate layer has an alternating structure of convolutional layers for convolving the pixel values ​​of each pixel input from the input layer and pooling layers for mapping (compressing) the pixel values ​​convolved in the convolutional layers, compressing the pixel information of the endoscopic image while ultimately extracting the feature values ​​of the endoscopic image. The output layer has one or more neurons that output information related to the presence or absence of lesions in the body parts contained in the endoscopic image, and outputs this information based on the image features output from the intermediate layer. The output information, such as the presence or absence of lesions, is used by doctors operating the endoscope 40 as diagnostic support information.

[0085] In this embodiment, the data input to the lesion learning model 631 or the data to be synthesized, which is the data processed as an endoscopic image, are described, but this is not a limitation. The data processed can also be the image captured by the imaging unit 446 of the endoscope 40 (the original image). That is, the lesion learning model 631 can also output relevant information such as the presence or absence of lesions by inputting the captured image. In this embodiment, the lesion learning model 631 is described as a neural network (NN) such as a CNN, but the lesion learning model 631 is not limited to an NN. It can also be a lesion learning model 631 constructed using other learning algorithms such as SVM (Support Vector Machine), Bayesian network, or regression tree. Alternatively, any object detection algorithm such as RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), or YOLO (YouOnly Look Once) can be used instead of a CNN.

[0086] The output unit 625, based on relevant information about the tumor region (insertion distance and rotation angle) obtained from the lesion region determination unit 624 and the endoscopic unfolding diagram obtained from the unfolding diagram generation unit 623, overlays a red frame or similar element surrounding the determined tumor region on the endoscopic unfolding diagram and outputs the highlighted endoscopic unfolding diagram to the display unit 7. The display unit 7 displays the highlighted endoscopic unfolding diagram on the display screen included in this unit based on the data output from the output unit 625. The output unit 625 can also generate data that displays the endoscopic unfolding diagram obtained from the unfolding diagram generation unit 623 and the unfolding diagram of the virtual endoscopic image obtained from X-ray CT via the acquisition unit side-by-side, and output this data to the display unit 7. Furthermore, the output unit 625 can also generate data that displays the composite image (360-degree FOV image) and the virtual endoscopic image side-by-side, and output this data to the display unit 7. As described above, the unfolded diagram of the virtual endoscopic image acquired from X-ray CT includes regions corresponding to the original data of multiple endoscopic images acquired from the unfolded diagram generation unit 623, all of which include the same observation area. Therefore, by comparing and displaying the unfolded diagrams of endoscopic images (endoscopic unfolded diagrams) for the same observation site and the unfolded diagrams of the virtual endoscopic images acquired from X-ray CT side by side, effective diagnostic support information can be provided to physicians and others.

[0087] The display format for comparing and displaying the unfolded images of the endoscope and the virtual endoscope is not limited to displaying these unfolded images side by side; it can also be a format where these unfolded images are displayed overlapping. For example, when displaying the unfolded image of the endoscope overlaid on the unfolded image of the virtual endoscope, the unfolded image of the endoscope can be converted to semi-transparent, and alignment can be performed using the insertion distance (S-coordinate) and rotation angle, thereby overlapping these unfolded images. Similarly, a composite image (360-degree FOV image) and a virtual endoscope image can also be displayed overlappingly. In addition, when overlapping these unfolded images, each unfolded image (endoscope unfolded image, virtual endoscope image unfolded image) whose lesion region is determined by the lesion region determination unit 624 can also be displayed overlappingly. For the synthetic image (360-degree FOV image) and the virtual endoscope image, the lesion area can also be determined by an AI (learning model) composed of the same CNN as the lesion area determination unit 624, and the synthetic image (360-degree FOV image) and the virtual endoscope image, which have been determined by annotations such as red boxes, are overlaid and displayed.

[0088] In this embodiment, the functional units in a series of processes are divided into functional units provided by the control unit 21 of the endoscope processor 20 and functional units provided by the control unit 62 of the information processing device 6, and described accordingly. However, the division of these functional units is merely an example and is not limited thereto. The control unit 21 of the endoscope processor 20 can function as all the functional units executed by the control unit 62 of the information processing device 6. That is, the endoscope processor 20 may substantially include the information processing device 6. Alternatively, the control unit 21 of the endoscope processor 20 may only output the image captured by the imaging unit 446, and the control unit 62 of the information processing device 6 may function as all the functional units that perform subsequent processing. Alternatively, the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing device 6 may cooperate as functional units in a series of processes by, for example, performing inter-process communication.

[0089] Figure 11 This is a flowchart illustrating an example of the processing steps performed by the control unit 62. For example, the information processing device 6 begins processing according to the content input from the input unit 8 connected to this device.

[0090] The control unit 62 of the information processing device 6 acquires endoscopic images captured by multiple imaging units 446 (S101). Each imaging unit 446 provided on the front end portion 443 (end face 444, peripheral face 445) includes the front, side and rear views in the direction of travel of the insertion portion 44. The control unit 62 of the information processing device 6 acquires multiple endoscopic images obtained by capturing images of the observation parts located in the front, side and rear views in the direction of travel of the insertion portion 44.

[0091] The control unit 62 of the information processing device 6 acquires the insertion distance and rotation angle of the endoscope 40 (S102). The control unit 62 of the information processing device 6 acquires the insertion distance and rotation angle of the endoscope 40 output from the measuring unit 9, which is configured as, for example, a mouthpiece or an accessory. Regarding the rotation angle, when taking endoscopic images using the upper endoscope, the rotation axis is the vertical direction parallel to the subject's body axis, while when taking endoscopic images using the lower endoscope, the rotation axis is the front-back direction perpendicular to the subject's body axis.

[0092] The control unit 62 of the information processing device 6 performs geometric correction on multiple endoscopic images (S103). By performing geometric correction on each of the acquired multiple endoscopic images, the control unit 62 improves the extraction accuracy of repetitive regions (repeated areas) in these multiple endoscopic images, correcting them into images that are easy to synthesize (coordinate system images that are easy to stitch together). Geometric correction can be, for example, spherical correction using surface projection (projection transformation), i.e., projecting the acquired endoscopic image onto a continuous curved surface. Alternatively, geometric correction can be performed based on image processing algorithms such as affine transformation, pseudo-affine transformation, quadratic isogonal transformation, or two-dimensional projective transformation. Alternatively, geometric correction can also be based on the specifications or optical characteristics of the lenses included in the imaging unit 446, performing optical simulation on a large number of objects, and using artificial intelligence (AI) to perform machine learning on the results, thereby generating a dictionary (lookup table) corresponding to the position of the captured image, and using this dictionary to restore and correct without aberrations.

[0093] The control unit 62 of the information processing device 6 generates a composite image based on the corrected multiple endoscopic images (S104). The control unit 62 of the information processing device 6 generates the composite image by overlapping overlapping regions (regions that capture the same object of observation) in the geometrically corrected multiple endoscopic images to achieve alignment. Multiple imaging units 446 are provided on the end face 444 and the peripheral face 445 of the front end portion 443 of the insertion portion 44. The imaging units 446 provided on the peripheral face 445 are arranged at equal intervals along the circumferential direction. The imaging units 446 on the end face 444 capture images forward in the direction of travel of the insertion portion 44, and the imaging units 446 on the peripheral face 445 capture images to the side and rear relative to the direction of travel of the insertion portion 44. Therefore, by synthesizing the multiple endoscopic images captured by these multiple imaging units 446, the front and rear can be included in the field of view in the direction of travel of the insertion portion 44, generating a composite image (360-degree FOV image) with a full circumference (360 degrees) relative to the front end portion 443.

[0094] The control unit 62 of the information processing device 6 establishes a correlation between the generated composite image and the insertion distance and rotation angle of the endoscope 40 when the composite image was captured. Furthermore, the control unit 62 of the information processing device 6 can also establish a correlation between the generated composite image and relevant time information (capture time) when the composite image was captured.

[0095] The control unit 62 of the information processing device 6 generates an endoscope unfolding diagram based on the synthesized image (S105). The control unit 62 of the information processing device 6 generates an endoscope unfolding diagram with the rotation angle as the vertical axis and the insertion distance as the horizontal axis, based on the synthesized image and the insertion distance and rotation angle associated with it. The control unit 62 of the information processing device 6 can align the endoscope unfolding diagrams generated from each of the multiple synthesized images based on the rotation angle, and display them side-by-side in order of insertion distance (the order in which the images were captured).

[0096] The control unit 62 of the information processing device 6 determines the lesion region based on the endoscopic unfolding diagram (S106). For example, the control unit 62 of the information processing device 6 uses a lesion learning model 631 stored in the storage unit 63 of the device and inputs the endoscopic unfolding diagram into the lesion learning model 631 to determine whether the endoscopic unfolding diagram contains lesions such as tumors. If it contains a tumor, it determines the region of the tumor (tumor region) in the endoscopic unfolding diagram based on the insertion distance and rotation angle of the endoscope 40.

[0097] The control unit 62 of the information processing device 6 outputs an endoscopic unfolding diagram highlighting the lesion area (S107). In cases where the lesion includes a tumor, the control unit 62 overlays a red frame or similar element surrounding the identified tumor area onto the endoscopic unfolding diagram and outputs the highlighted endoscopic unfolding diagram to the display unit 7. Based on the output data, the display unit 7 displays the highlighted endoscopic unfolding diagram on a display screen included in this unit.

[0098] In this embodiment, a series of processes are performed by the control unit 62 of the information processing device 6, but it is not limited to this. This series of processes may also be performed by the control unit 21 of the endoscope processor 20. Alternatively, this series of processes may be performed collaboratively by the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing device 6 through, for example, inter-process communication.

[0099] According to this embodiment, a plurality of imaging units 446 are provided on the end face 444 and the peripheral surface 445 of the cylindrical front end portion 443 of the insertion portion 44 of the endoscope 40. Therefore, multiple endoscopic images, including the anterior and posterior fields of view relative to the front end portion 443, can be acquired from the plurality of imaging units 446. Based on the acquired multiple endoscopic images, a circumferential composite image including the anterior and posterior fields of view relative to the front end portion 443 can be effectively generated. The composite image is formed by combining multiple endoscopic images captured by the plurality of imaging units 446 arranged at equal intervals at predetermined circumferential angles on the end face 444 and the peripheral surface 445 of the front end portion 443. Therefore, in the circumferential direction of the cylindrical front end portion 443, there is a 360-degree FOV (Field of View) image that includes the entire circumference.

[0100] Endoscopic images are captured at multiple times by multiple imaging units 446. At this time, by associating the insertion distance and rotation angle of the endoscope 40 during image capture with the synthesized image, the position (coordinates) of the generated synthesized image within the body can be determined based on this insertion distance and rotation angle. That is, by assigning (associating) absolute coordinates to each captured endoscopic image, these images can be compared (endoscopic image comparison) in conjunction with the rotation direction. Therefore, it is possible to effectively perform coordinate reproducibility and temporal comparison (time direction comparison) for endoscopic images (360-degree FOV images) captured at multiple times.

[0101] According to this embodiment, the endoscope 40 includes, for example, an endoscope 40 for upper gastrointestinal endoscopy such as gastroscopy and an endoscope 40 for lower gastrointestinal endoscopy such as colonoscopy. The insertion distance for capturing endoscopic images using the upper endoscope is output from the measuring unit 9 (mouthpiece) located in the subject's mouth, and the insertion distance for capturing endoscopic images using the lower endoscope is output from the measuring unit 9 (accessory) located in the subject's anus. The rotation angle for capturing endoscopic images using the upper endoscope is an angle with the vertical direction parallel to the subject's body axis as the rotation axis, and the rotation angle for capturing endoscopic images using the lower endoscope is an angle with the front-back direction parallel to the length of the subject's body cavity as the rotation axis. These measuring units 9, for example, are mouthpieces installed in the subject's mouth or accessories installed in the subject's anus, and function as insertion distance measuring units and rotation angle measuring units for measuring insertion distance and rotation angle. Therefore, whether using upper or lower endoscopy, the insertion distance and rotation angle can be appropriately detected (measured).

[0102] According to this embodiment, multiple composite images can be generated based on endoscopic images captured at multiple times. The generated composite images are captured at different times, and the insertion distance and rotation angle of the endoscope 40 vary depending on the time. Based on these multiple composite images, an endoscopic unfolding diagram is generated with the insertion distance of the endoscope 40 as the horizontal axis and the rotation angle as the vertical axis, thereby effectively generating an endoscopic unfolding diagram with excellent visibility. Although the times at which the multiple composite images are captured (capture times) are different, and the insertion distance and rotation angle of the endoscope 40 are also different at those capture times, there are overlapping regions in these multiple composite images, i.e., regions where the same observation area is captured repeatedly, depending on the viewing angle of the imaging unit 446. Therefore, by extracting these overlapping regions and overlapping them, an endoscopic unfolding diagram with reduced gaps and repetitions can be effectively generated. By using endoscopic unfolding images, it is possible to visually identify all shadows of folds and wrinkles in the synthesized 360-degree endoscopic images (360-degree FOV images), such as when observing a tumor from above, thereby providing effective diagnostic support information for doctors and other endoscopic operators.

[0103] According to this embodiment, after inputting an endoscopic image, an endoscopic unfolding diagram is input to the lesion learning model 631 used to output lesion areas such as tumors, thereby determining whether the endoscopic unfolding diagram contains lesions such as tumors. If a tumor is contained, the region of the tumor (tumor region) in the endoscopic unfolding diagram is determined based on the insertion distance and rotation angle of the endoscope 40. For example, the red frame surrounding the determined tumor region is overlaid for emphasis, and the endoscopic unfolding diagram is output. Therefore, effective diagnostic support information can be provided to doctors and other operators of the endoscope 40.

[0104] (Implementation Method 2)

[0105] The endoscope 40 of Embodiment 2 differs from that of Embodiment 1 in that a plurality of imaging portions 446 are provided on the end face 444 of the front end portion 443. Figure 12 This is a schematic front view of the front end portion 443 (end face 444) of the insertion portion 44 involved in Embodiment 2 (the shooting portion 446 located around the channel 448). Figure 13 This is a schematic side view of the front end portion 443 (circumferential surface 445) of the insertion portion 44.

[0106] Similar to the endoscope 40 of Embodiment 1, the endoscope 40 of Embodiment 2 has an end face 444 of a cylindrical front end portion 443 and a plurality of imaging portions 446 provided on the peripheral surface 445. The end face 444 of the front end portion 443 of the endoscope 40 of Embodiment 2 has a channel 448 (working channel) at its center.

[0107] The working channel 448 is a hole for inserting treatment tools such as forceps. It extends from the end face 444 of the front end portion 443 toward the operating portion 43 and passes through the entire insertion portion 44. Centered on the working channel 448 located in the center of the end face 444, a plurality of imaging portions 446 are provided on the end face 444. For example, there are four imaging portions 446 provided on the end face 444, which are arranged at equal intervals with the same central angle (90 degrees in this embodiment).

[0108] A plurality of imaging portions 446 provided on the peripheral surface 445 of the front end portion 443 are inclined such that the optical axis angle of the imaging portions 446 is greater than 90 degrees relative to the insertion direction of the insertion portion 44 of the endoscope 40. The insertion direction of the insertion portion 44 of the endoscope 40 is a direction parallel to the axial direction of the insertion portion 44, which corresponds to the direction of travel (forward) when the insertion portion 44 is inserted into the body. As shown in the accompanying drawings of this embodiment, by tilting the imaging portions 446 of the peripheral surface 445 rearward, the angle (α) between the insertion direction (axial direction) of the insertion portion 44 and the optical axis of the imaging portions 446 of the peripheral surface 445 is greater than 90 degrees (α°>90°). The optical axis angle of the imaging portions 446 relative to the insertion direction of the insertion portion 44 of the endoscope 40 is less than 180 degrees, that is, the angle range is (180°>α°>90°).

[0109] According to this embodiment, around the channel 448 located at the center of the end face 444 of the front end portion 443, a plurality of imaging portions 446 for capturing images of the forward field of view relative to the front end portion 443 are arranged, for example, at equal intervals, thereby allowing the front end portion 443, i.e., the insertion portion 44 of the endoscope 40, to be made smaller. With respect to the insertion direction when the insertion portion 44 of the endoscope 40 is inserted into the body, the optical axis angle of the imaging portions 446 disposed on the peripheral surface 445 of the front end portion 443 is greater than 90 degrees. Therefore, with respect to the insertion direction of the insertion portion 44, the optical axis of the imaging portions 446 on the peripheral surface 445 faces rearward. Thus, a wider rearward field of view can be captured using the imaging portions 446 on the peripheral surface 445.

[0110] (Implementation Method 3)

[0111] The endoscope 40 in Embodiment 3 differs from that in Embodiment 1 in that it uses wireless communication to transmit the captured images output from the imaging unit 446. Figure 14 This is a schematic side view of the front end portion 443 (circumferential surface 445) of the insertion portion 44 involved in Embodiment 3 (wireless communication unit 4481 / waveguide).

[0112] A wireless communication unit 4481 is provided at the front end 443 of the endoscope 40. This unit is connected to the imaging unit 446 via a wire or the like, and wirelessly transmits the images output by the imaging unit 446. The wireless communication unit 4481 is, for example, disposed on the outer peripheral surface of the channel 448, and wirelessly transmits the images output by the imaging unit 446 to the communication unit 24 of the processor for the endoscope 40. In this case, the communication unit 24 of the processor for the endoscope 40 functions as a wireless communication receiver.

[0113] The wireless communication unit 4481 can use the channel 448 as a waveguide to transmit radio waves into the space formed by the inner wall of the channel 448 and send the captured image output by the imaging unit 446 to the wireless communication receiving unit provided on the operation unit 43.

[0114] According to this embodiment, endoscopic images can be output from the imaging unit 446 in the endoscope 40 to the control unit 62 in the processor of the endoscope 40 via wireless communication between the endoscope 40 and the processor for the endoscope 40. Therefore, there is no need to provide wiring for transmitting the images captured by the imaging unit 446 inside the insertion part 44 of the endoscope 40, allowing the insertion part 44 of the endoscope 40 to be made smaller. Furthermore, by using the channel 448 (working channel) formed by the hole that runs through the entire insertion part 44 as a waveguide, wireless communication waves can be transmitted very well.

[0115] (Implementation Method 4)

[0116] The difference between the endoscope system S in Embodiment 4 and Embodiment 1 is that it generates stereoscopic vision images based on synthetic images. Figure 15 This is a functional block diagram illustrating the functional units included in the control unit 62 of the information processing apparatus 6 according to Embodiment 4 (stereoscopic image). The control unit 21 of the endoscope processor 20 (endoscope device 10) performs the function of the image processing unit 211 by executing the program stored in the main storage device 22. Similar to Embodiment 1, the control unit 62 of the information processing apparatus 6 performs the functions of the acquisition unit 621, the composite image generation unit 622, and the output unit 625 by executing the program stored in the storage unit 63, and further performs the functions of the stereoscopic image generation unit 626 and the region of interest determination unit 627 (region of interest learning model 632).

[0117] Similar to Embodiment 1, the acquisition unit 621 acquires multiple endoscopic images, S-coordinates (insertion distance), and rotation angles output by the endoscopic processor 20. Also similar to Embodiment 1, the composite image generation unit 622, based on the acquired multiple endoscopic images, includes the area in front and behind the insertion unit 44 within the field of view, generates a composite image (360-degree FOV image) covering the entire circumference of the front end portion 443, and outputs the generated composite image to the stereoscopic image generation unit 626.

[0118] The stereoscopic image generation unit 626 generates a stereoscopic image composed of multiple endoscopes 40 arranged to capture images of the same observation area within the subject from different visual field directions, based on the acquired synthetic image and the insertion distance and rotation angle associated with the synthetic image. The stereoscopic image generation unit 626 segments the multiple synthetic images captured at different times according to a predetermined visual field angle corresponding to each visual field direction, aligns the segmented synthetic images based on the rotation angle of the endoscopes 40, and stitches them together according to the insertion distance of the endoscopes 40, thereby generating (reconstructing) a stereoscopic image.

[0119] Figure 16 This is an explanatory diagram schematically showing the imaging portion 446 provided at the front end 443 of the insertion portion 44 for taking pictures. As shown in the accompanying drawings of this embodiment, by inserting the insertion portion 44 of the endoscope 40 into the body at a predetermined speed (insertion speed), endoscopic images of the same observation site, such as tumors, can be taken at different field of view at multiple consecutive imaging moments.

[0120] In the illustration of this embodiment, the imaging unit 446 on the peripheral surface 445 captures images of the tumor (observation site) from the front at imaging time t[x], from the left side (front view) at imaging time t[x-1] before t[x], and from the right side (rear view) at imaging time t[x+1] after t[x]. In this way, a composite image (360-degree FOV image) can be generated based on the images obtained from capturing the same observation site (tumor) at multiple consecutive imaging times.

[0121] Figure 17 This is an explanatory diagram regarding the relationship between the composite image and the stereoscopic image in the full circumference direction. The composite image (360-degree FOV image) is synthesized from endoscopic images captured by multiple imaging units 446 located on the end face 444 and circumferential surface 445 of the front end portion 443 forming the cylinder. Therefore, as shown in the accompanying drawings of this embodiment, it can be represented using a curved surface coordinate system (cylindrical curved surface coordinate system) formed on the inner wall side of the cylinder.

[0122] The stereoscopic image generation unit 626 segments (contour segmentation) and synthesizes images according to a predetermined field of view angle in the cylindrical curved surface coordinate system, and extracts the segmented synthesized image containing the observation site (tumor) from any segmented synthesized image (segmented synthesized image). By performing such segmentation and extraction processing on synthesized images from multiple consecutive shooting times, segmented synthesized images containing the observation site (tumor) can be determined. The stereoscopic image generation unit 626 aligns these determined segmented synthesized images based on the rotation angle of the front end 443, and further arranges and stitches (reconstructs) them according to the order of shooting times, thereby generating a stereoscopic image. By generating this stereoscopic image, the same observation site (tumor) can be observed from different angles.

[0123] The stereoscopic image generation unit 626 can also extract segmented composite images with the same field of view from any segmented composite image (segmented composite image) from composite images captured at multiple consecutive shooting times, and stitch (reconstruct) these segmented composite images with the same field of view to generate a stereoscopic image. For example, using the shooting unit 446 on the peripheral surface 445 as the base point, and setting the same field of view angle to 90 degrees relative to the insertion direction (axial direction) of the insertion unit 44, segmented composite images obtained from shooting the observation area from the front (above) can be stitched together to generate a stereoscopic image. The stereoscopic image generation unit 626 outputs the generated stereoscopic image to the region of interest determination unit 627 and the output unit 625.

[0124] The lesion region determination unit 624 includes a region of interest (ROI) learning model 632. When an endoscopic image is input, it outputs diagnostic support information including the presence or absence of a tumor, lesion candidates, treatment tools, and regions of interest (ROIs) such as markers. The ROI determination unit 627 inputs the acquired stereoscopic image into the ROI learning model 632, determines whether the endoscopic unfolded image contains a ROI, and outputs diagnostic support information, including whether the stereoscopic image contains a tumor or other ROIs, to the output unit 625. Furthermore, the ROI determination unit 627 compares and displays the comparison information with the unfolded image of a virtual endoscope from an X-ray CT scan. Since the stereoscopic image is a composite image of multiple endoscopic images, by inputting the stereoscopic image into the ROI learning model 632, diagnostic support information, including whether the stereoscopic image contains a tumor or other ROIs, can be obtained. Similar to the lesion learning model 631 in Embodiment 1, the ROI learning model 632 is also composed of a neural network, such as a CNN, that learns using training data.

[0125] The output unit 625 outputs, for example, a stereoscopic image superimposed with diagnostic support information to the display unit 7 based on diagnostic support information related to the region of interest determined by the region of interest determination unit 627 and a stereoscopic image obtained by the stereoscopic image generation unit 626. The display unit 7 displays the stereoscopic image superimposed with diagnostic support information on a display screen included in this unit based on the data output from the output unit 625.

[0126] Figure 18 This is a flowchart illustrating an example of the processing steps performed by the control unit 62. For example, the information processing device 6 begins processing according to the content input from the input unit 8 connected to this device.

[0127] The control unit 62 of the information processing device 6 acquires endoscopic images captured by the multiple imaging units 446 (S401). The control unit 62 of the information processing device 6 acquires the insertion distance and rotation angle of the endoscope 40 (S402). The control unit 62 of the information processing device 6 performs geometric correction on the multiple endoscopic images (S403). Based on the corrected multiple endoscopic images, the control unit 62 of the information processing device 6 generates a composite image (S404). Similar to the processing S101 to S104 in Embodiment 1, the control unit 62 of the information processing device 6 performs the processing S401 to S404.

[0128] The control unit 62 of the information processing device 6 segments the composite image associated with the insertion distance and rotation angle according to the field of view direction (S405). As shown in the accompanying drawings of this embodiment, the composite image (360-degree FOV image) can be represented by a cylindrical curved surface coordinate system formed on the inner wall side of the cylinder. The control unit 62 of the information processing device 6 segments (contour segmentation) the composite image in this cylindrical curved surface coordinate system according to a predetermined field of view angle, and extracts the segmented composite image containing the observation site (tumor) from any of the segmented composite images (segmented composite images). Alternatively, the control unit 62 of the information processing device 6 can extract the segmented composite image with the same field of view angle from any of the segmented composite images (segmented composite images).

[0129] The control unit 62 of the information processing device 6 generates a stereoscopic image by stitching together multiple segmented composite images (S406). By performing such segmentation and extraction processing on composite images from multiple consecutive shooting times, a segmented composite image containing the observation site (tumor) can be determined. The control unit 62 of the information processing device 6 aligns these determined multiple segmented composite images based on the rotation angle of the front end 443, and further arranges and stitches (reconstructs) them according to the order of the shooting times, thereby generating a stereoscopic image.

[0130] The control unit 62 of the information processing device 6 determines the region of interest (ROI) based on the stereoscopic vision image (S407). For example, the control unit 62 of the information processing device 6 uses the ROI learning model 632 stored in the storage unit 63 of the device to input the stereoscopic vision image into the ROI learning model 632, thereby determining whether the stereoscopic vision image contains ROIs such as tumors, lesion candidates, treatment tools, and markers, and outputs information related to the ROI (including diagnostic support information such as whether or not there is an ROI).

[0131] The control unit 62 of the information processing device 6 outputs a stereoscopic image along with information related to the region of interest (S408). Based on the information related to the region of interest (diagnostic support information) and the stereoscopic image, the control unit 62 of the information processing device 6 outputs, for example, a stereoscopic image overlaid with diagnostic support information to the display unit 7. Based on the output data, the display unit 7 displays, for example, the stereoscopic image overlaid with diagnostic support information on a display screen included in this unit.

[0132] According to this embodiment, multiple composite images can be generated based on endoscopic images captured at multiple times. The generated composite images are captured at different times, and the insertion distance and rotation angle of the endoscope 40 vary depending on the time. In contrast, the composite images segmented according to each visual field direction are aligned based on the rotation angle of the endoscope 40, thereby absorbing the differences in the rotation angle of the endoscope 40 at the time the composite images were captured, and the segmented composite images (segmented composite images) are stitched together separately. Furthermore, the segmented composite images (segmented composite images) are stitched together separately according to the insertion distance (S-coordinate) of the endoscope 40, thereby matching and stitching these segmented composite images according to the insertion direction of the endoscope 40 based on the order of the capture times. Based on such multiple composite images, a stereoscopic visual image composed of multiple endoscopes 40s arranged to capture the same observation site within the subject from different visual field directions is generated, thereby facilitating stereoscopic confirmation (confirmation of height direction), and effectively generating stereoscopic visual images with excellent visibility.

[0133] According to this embodiment, when an endoscopic image is input, the region of interest (ROI) learning model 632 outputs diagnostic support information including the presence or absence of tumors, lesion candidates, treatment tools, and regions of interest (ROIs) such as markers. By inputting a stereoscopic image into the ROI learning model 632, diagnostic support information, including whether the stereoscopic image contains a region of interest such as a tumor, can be obtained. According to this embodiment, an endoscopic unfolding diagram or a stereoscopic image can be displayed, and it can be compared and displayed with a virtual endoscopic unfolding diagram from an X-ray CT scan, with comparison information output.

[0134] (Implementation Method 5)

[0135] Unlike Embodiment 1, the endoscope system S in Embodiment 5 uses synthesis parameters (image synthesis parameters) derived from a reference image to generate a synthesized image. Figure 19 This is a functional block diagram illustrating the functional units included in the control unit 62 of the information processing apparatus 6 according to Embodiment 5 (image synthesis parameters). The control unit 21 of the endoscope processor 20 (endoscope device 10) performs the function of the image processing unit 211 by executing the program stored in the main storage device 22. Similar to Embodiment 1, the control unit 62 of the information processing apparatus 6 performs the functions of the acquisition unit 621, the image synthesis generation unit 622, and the output unit 625 by executing the program stored in the storage unit 63, and further performs the functions of the synthesis parameter derivation unit 628 and the geometric distortion correction unit 629.

[0136] Similar to Embodiment 1, the acquisition unit 621 acquires multiple endoscopic images, S-coordinates (insertion distance), and rotation angles output by the endoscope processor 20. The acquisition unit 621 further acquires multiple reference images output by the endoscope processor 20.

[0137] When the endoscope 40's operating unit 43 is inserted into the body, before outputting the endoscopic image obtained by photographing the observed area inside the body, it passes through a mouthpiece or accessory that also functions as the measuring unit 9. These mouthpieces and accessories include a reference object composed of a regular pattern such as a grid. Each imaging unit 446 located on the front end portion 443 (end face 444, peripheral face 445) photographs this reference object through the mouthpiece, etc. The photographed image (reference image) obtained from the reference object is output from the endoscope 40's processor to the acquisition unit 621 (the control unit 62 of the information processing device 6), where the acquisition unit 621 acquires the reference image. The acquisition unit 621 then outputs the acquired reference image to the synthesis parameter export unit 628.

[0138] The synthesis parameter derivation unit 628 derives synthesis parameters based on the acquired reference images. The storage unit 63 of the information processing device 6 stores reference object information, such as the shape, size, and color, of reference objects composed of regular, prescribed patterns. The synthesis parameter derivation unit 628 compares the reference objects contained in each acquired reference image with the reference object information stored in the storage unit 63, deriving synthesis parameters in a manner that matches the reference objects contained in the synthesized image with the reference object information in the image coordinate system (the coordinate system of the synthesized image) when synthesizing these reference images. The synthesis parameter derivation unit 628 outputs the derived synthesis parameters to the synthesized image generation unit 622.

[0139] The acquisition unit 621 outputs the acquired multiple endoscopic images to the geometric distortion correction unit 629. The geometric distortion correction unit 629 performs geometric correction on each of the acquired endoscopic images, thereby improving the extraction accuracy of repetitive regions (repeated areas) in these multiple endoscopic images and correcting them into images that are easy to synthesize (coordinate system images that are easy to stitch together). Geometric correction can be, for example, spherical correction using a surface projection (projection transformation) that projects the acquired endoscopic images onto a continuous curved surface, or geometric correction based on image processing algorithms such as affine transformation, pseudo-affine transformation, quadratic isogonal transformation, and two-dimensional projective transformation. Alternatively, geometric correction can also be based on the specifications or optical characteristics of the lenses included in the imaging unit 446, performing optical simulation on a large number of objects, and using artificial intelligence to perform machine learning on the results to generate a dictionary corresponding to the position of the captured image. This dictionary is then used to restore and correct the image without aberrations. The geometric distortion correction unit 629 outputs the geometrically distorted multiple endoscopic images to the composite image generation unit 622.

[0140] The composite image generation unit 622 generates a composite image based on multiple corrected endoscopic images acquired from the geometric distortion correction unit 629 and composite parameters acquired from the composite parameter derivation unit 628. By using the composite parameters, the generation accuracy of the composite image can be improved. These composite parameters are derived based on measured values ​​using a reference object, and therefore, appropriate composite processing can be performed according to the usage environment of the imaging unit 446 provided on the front end 443 of the endoscope 40.

[0141] Figure 20 This is an explanatory diagram regarding geometric distortion correction using a surface projection method that projects endoscopic images onto a continuous curved surface. The composite image generation unit 622 generates a composite image by overlapping repeated regions (regions where the same object is observed) in multiple geometrically corrected endoscopic images to achieve alignment. When extracting repeated regions from multiple endoscopic images, the composite image generation unit 622 can also extract these repeated regions by performing pattern matching between regions divided into predetermined pixel units within the endoscopic images. Geometric correction is particularly effective for spherical correction of the peripheral portion of the endoscopic image. This peripheral portion, which is a region that is repeated with other endoscopic images, becomes an overlapping region during synthesis; therefore, geometric correction can correct (transform) it into an endoscopic image that is easily overlapped.

[0142] By synthesizing multiple endoscopic images captured by multiple imaging units 446 located on the insertion section 44 (end face 444 and peripheral face 445), a composite image (360-degree FOV image) with a full circumference (360 degrees) relative to the front end portion 443 can be generated, encompassing both the front and rear views in the direction of travel of the insertion section 44. The composite image generation unit 622 establishes a correlation between the generated composite image and the insertion distance and rotation angle of the endoscope 40 when capturing the multiple original endoscopic images of the composite image. The composite image generation unit 622 can further establish a correlation between the generated composite image and relevant time information (capture time) when capturing the multiple original endoscopic images of the composite image. As in Embodiment 1, the composite image generation unit 622 outputs the generated composite image to the output unit 625. As in Embodiment 1, the output unit 625 outputs the acquired composite image to the display unit 7.

[0143] Figure 21 This is an explanatory diagram regarding the reprojection of a composite image onto a curved surface (ellipsoidal display, cylindrical display) for display. When outputting the composite image to the display unit 7, the output unit 625 can also output it to the display unit 7 in a display form (ellipsoidal display, cylindrical display) after projection transformation, as shown in the accompanying drawings of this embodiment, in a manner that serves as the field of view of a coordinate ellipsoid or cylindrical surface. In this case, the display unit 7 includes multiple display screens with multiple windows in an independent display device or a single display device, and the composite image output in an ellipsoidal or cylindrical display form can be displayed separately on these multiple display screens. Among these multiple display screens, the central portion (front image) of the composite image can be displayed on the central display screen, and the left and right sides of the composite image can be displayed on the left and right display screens.

[0144] Figure 22 This is a flowchart illustrating an example of the processing steps performed by the control unit 62. For example, the information processing device 6 begins processing according to the content input from the input unit 8 connected to this device.

[0145] The control unit 62 of the information processing device 6 acquires reference images captured by multiple imaging units 446 (S501). When the operating unit 43 of the endoscope 40 is inserted into the body, if it is an upper endoscope, it is inserted into the body through a mouthpiece installed in the oral cavity; if it is a lower endoscope, it is inserted into the body through an accessory installed in the anus. These mouthpieces and accessories are provided with reference objects composed of regularly prescribed patterns, and the imaging units 446 capture images of these reference objects as they pass through the mouthpieces, etc. The control unit 62 of the information processing device 6 acquires images (reference images) of the reference objects captured by the imaging units 446.

[0146] The control unit 62 of the information processing device 6 derives synthesis parameters based on the acquired reference images (S502). The storage unit 63 of the information processing device 6 stores reference object information, such as the shape, size, and color, of a reference object composed of a regular, prescribed pattern. The control unit 62 of the information processing device 6 compares the reference objects contained in each acquired reference image with the reference object information stored in the storage unit 63, and derives synthesis parameters by matching the reference objects contained in the reference images with the reference object information in the image coordinate system (the coordinate system of the synthesized image) when synthesizing these reference images.

[0147] The control unit 62 of the information processing device 6 acquires endoscopic images captured by the multiple imaging units 446 (S503). The control unit 62 of the information processing device 6 acquires the insertion distance and rotation angle of the endoscope 40 (S504). The control unit 62 of the information processing device 6 performs geometric correction on the multiple endoscopic images (S505). Similar to the processes S101 to S103 in Embodiment 1, the control unit 62 of the information processing device 6 performs the processes S503 to S505.

[0148] The control unit 62 of the information processing device 6 generates a composite image using synthesis parameters based on the corrected multiple endoscopic images (S506). The control unit 62 of the information processing device 6 performs the same processing as S104 of Embodiment 1, and at the same time, generates a composite image using synthesis parameters derived from the reference image.

[0149] The control unit 62 of the information processing device 6 outputs the generated composite image (S507). When outputting the generated composite image, the control unit 62 of the information processing device 6 can also output it to the display unit 7 in a display form (ellipsoidal display, cylindrical display) after projection transformation into a coordinate ellipsoidal or cylindrical surface view. In this case, the display unit 7 includes multiple display screens of multiple windows in an independent display device or a single display device, and the composite image output in an ellipsoidal or cylindrical display form can be displayed separately on these multiple display screens. Among these multiple display screens, the central part (front image) of the composite image can be displayed on the central display screen, and the left and right sides of the composite image can be displayed on the left and right display screens.

[0150] According to this embodiment, when generating a composite image, a reference image containing the reference object is obtained by capturing a predetermined reference object with multiple imaging units 446. Image compositing parameters are derived based on each reference image captured by the multiple imaging units 446, and the composite image is generated using these image compositing parameters, thereby improving the accuracy of the compositing process. Since the reference image contains a regular, prescribed pattern, the image compositing parameters can be efficiently derived based on this pattern.

[0151] According to this embodiment, the geometric distortion correction included in the process of generating the synthesized image adopts the surface projection method, which projects the acquired endoscopic image onto a continuous curved surface. Therefore, especially when performing spherical correction on the periphery of the endoscopic image and synthesizing multiple endoscopic images, it is possible to effectively overlap (align the overlapping regions) the repeated regions in these endoscopic images.

[0152] It should be understood that the embodiments disclosed herein are illustrative in all respects and not restrictive. The technical features described in the various embodiments can be combined with each other, and all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within the scope of this invention.

[0153] Symbol Explanation

[0154] S Endoscopic System

[0155] 10 Endoscopic devices

[0156] 15 Keyboards

[0157] 16 containment racks

[0158] 20 Endoscope Processors

[0159] 21 Control Department

[0160] 211 Image Processing Department

[0161] 22 Main storage device

[0162] 23. Auxiliary storage device

[0163] 24 Ministry of Communications

[0164] 25 Touch Panel

[0165] 26 Display Device I / F

[0166] 27 Input Device I / F

[0167] 28 Reading Department

[0168] 31 Connectors for endoscopes

[0169] 311 Electrical Connector

[0170] 312 Optical Connector

[0171] 33 Light Source

[0172] 34 pumps

[0173] 35 Water Supply Tank

[0174] 36 Gas and water supply outlets

[0175] 40 Endoscope

[0176] 43 Operations Department

[0177] 431 Control Buttons

[0178] 433 Bend Knob

[0179] 44 Insertion section

[0180] 441 Flexible section

[0181] 442 Bending section

[0182] 443 Front end

[0183] 444 end face

[0184] 445 Weeks

[0185] 446 Filming Department

[0186] 447 Light source for shooting

[0187] 448 channels

[0188] 4481 Wireless Communications Department

[0189] 449 subjects tested

[0190] 45. Bending stop

[0191] 48 Connectors for Observers

[0192] 49 General-purpose flexible cord

[0193] 50 display devices

[0194] 6. Information processing device

[0195] 61 Ministry of Communications

[0196] 62 Control Department

[0197] 621 Acquisition Department

[0198] 622 Composite Image Generation Unit

[0199] 623 Unfolded Diagram Generation Department

[0200] 624 Lesion Area Determination Section

[0201] 625 Output Section

[0202] 626 Stereoscopic Image Generation Unit

[0203] 627 Region of Interest Determination Department

[0204] 628 Synthesis Parameter Export Section

[0205] 629 Geometric Distortion Correction Department

[0206] 63 Storage Department

[0207] 630 Recording Media

[0208] 631 Pathogenesis Learning Model

[0209] 632 Region of Interest Learning Model

[0210] 64 Input / Output I / F

[0211] 7 Display Section

[0212] 8 Input Section

[0213] 9. Measurement Department

[0214] 91. Testing Department.

Claims

1. A program product comprising a program that causes a computer to perform the following processes: Endoscopic images are obtained by taking pictures of the subject through multiple imaging points arranged on the end face and circumference of the cylindrical front end of the endoscope insertion part; Based on the multiple endoscopic images captured, a composite image is generated by combining the images in the full circumference direction relative to the front end, including the anterior and posterior fields of view. Acquire the insertion distance and rotation angle of the endoscope inserted into the subject's body during the acquisition of the endoscopic images; and The synthesized image is output in association with the acquired insertion distance and rotation angle of the endoscope. The process of generating the synthesized image includes the following steps: A reference image is obtained by capturing images of a predetermined reference object using multiple imaging units. Based on the acquired reference image, image synthesis parameters are derived, and The synthesized image is generated using the exported image synthesis parameters.

2. The program product according to claim 1, wherein, The optical axis of the imaging section disposed on the peripheral surface of the front end is at an angle greater than 90 degrees relative to the insertion direction of the insertion section of the endoscope.

3. The program product according to claim 1 or 2, wherein, When taking endoscopic images of the subject using a superior endoscope, the insertion distance of the endoscope is the insertion distance output from a measuring unit located in the subject's oral cavity. When taking endoscopic images of the subject using a lower endoscope, the insertion distance of the endoscope is the insertion distance output from the measuring section located at the anus of the subject.

4. The program product according to claim 1 or 2, wherein, When taking endoscopic images of the subject using the upper endoscope, the rotation angle of the endoscope inserted into the subject's body is an angle with the vertical direction parallel to the subject's body axis as the axis of rotation. When taking endoscopic images of the subject using a lower endoscope, the rotation angle of the endoscope inserted into the subject's body is an angle with the front-back direction parallel to the length of the subject's body cavity as the rotation axis.

5. The program product according to claim 1 or 2, wherein, Multiple synthetic images are generated based on the endoscopic images captured at multiple times. Based on multiple synthesized images, an endoscope unfolding diagram is generated with the insertion distance of the endoscope as the first axis and the rotation angle of the endoscope as the second axis. Output the generated endoscope unfolding diagram.

6. The program product according to claim 5, wherein, The process of generating the endoscope unfolding diagram from multiple synthetic images includes the following processes: Extract repeated regions from multiple synthesized images. The overlapping regions are used to generate the endoscope unfolding diagram.

7. The program product according to claim 5, wherein, The tumor region identified as a tumor in the endoscopic unfolding diagram is determined based on the insertion distance and rotation angle of the endoscope. The identified tumor region is highlighted and the endoscopic unfolded image is output, or it is compared and displayed with a virtual endoscopic image unfolded from an X-ray CT scan of the corresponding region.

8. The program product according to claim 1 or 2, wherein, Multiple synthetic images are generated based on the endoscopic images captured at multiple times. Based on the multiple synthesized images, a stereoscopic visual image is generated by arranging multiple endoscopic images obtained from taking pictures of the same observation site in the subject from different visual directions. Output the generated stereoscopic image.

9. The program product according to claim 8, wherein, The process of synthesizing multiple endoscopic images to generate the stereoscopic visual image includes the following steps: The multiple synthesized images are segmented according to each view direction, and... The segmented synthetic images are aligned based on the rotation angle of the endoscope, and then stitched together according to the insertion distance of the endoscope.

10. The program product according to claim 8, wherein, Extract the region of interest contained in the stereoscopic vision image. Based on the extracted region of interest, diagnostic support information is output.

11. An information processing method that causes a computer to perform the following processing: Endoscopic images are obtained by taking pictures of the subject through multiple imaging points arranged on the end face and circumference of the cylindrical front end of the endoscope insertion part; Based on the multiple endoscopic images captured, a composite image is generated by combining the images in the full circumference direction relative to the front end, including the anterior and posterior fields of view. Acquire the insertion distance and rotation angle of the endoscope inserted into the subject's body during the acquisition of the endoscopic images; and The synthesized image is output in association with the acquired insertion distance and rotation angle of the endoscope. The process of generating the synthesized image includes the following steps: A reference image is obtained by capturing images of a predetermined reference object using multiple imaging units. Based on the acquired reference image, image synthesis parameters are derived, and The synthesized image is generated using the exported image synthesis parameters.

12. A program product comprising a program that causes a computer to perform the following processes: Acquire endoscopic images of the subject by taking pictures of the subject through multiple imaging units arranged on the front end of the endoscope insertion part; Based on the multiple endoscopic images captured, a composite image is generated by combining the images in the full circumference relative to the anterior end, including both the anterior and posterior fields of view; and Output the generated composite image; The process of generating the synthesized image includes the following steps: A reference image is obtained by capturing images of a predetermined reference object using multiple imaging units. Based on the acquired reference image, image synthesis parameters are derived, and The synthesized image is generated using the exported image synthesis parameters.

13. The program product according to claim 12, wherein, The reference image contains a regular, prescribed pattern.

14. The program product according to claim 12 or 13, wherein, The process of generating the synthesized image includes the following steps: Geometric distortion correction is performed on the acquired endoscopic images, and Multiple endoscopic images corrected for geometric distortion are then synthesized.

15. The program product according to claim 14, wherein, The geometric distortion correction is performed using a surface projection method that projects the acquired endoscopic images onto a continuous curved surface.

16. An information processing method that causes a computer to perform the following processing: Acquire endoscopic images of the subject by taking pictures of the subject through multiple imaging units arranged on the front end of the endoscope insertion part; Based on the multiple endoscopic images captured, a composite image is generated by combining the images in the full circumference relative to the anterior end, including both the anterior and posterior fields of view; and Output the generated composite image; The process of generating the synthesized image includes the following steps: A reference image is obtained by capturing images of a predetermined reference object using multiple imaging units. Based on the acquired reference image, image synthesis parameters are derived, and The synthesized image is generated using the exported image synthesis parameters.

17. An endoscope system comprising an endoscope and a control unit for processing endoscopic images captured by the endoscope. The endoscope has the following features: The insertion point inserted into the subject's body, and Multiple shooting sections are provided at the cylindrical front end located at the front end of the insertion section; The multiple imaging units capture multiple endoscopic images, including the frontal and rear views, in a full circumference direction relative to the front end. The control unit Acquire endoscopic images of the subject by means of multiple imaging devices. Multiple endoscopic images captured at multiple moments are synthesized to generate a composite image that includes a posterior field of view relative to the anterior end portion. Output the generated composite image. The process of generating the synthesized image includes the following steps: A reference image is obtained by capturing images of a predetermined reference object using multiple imaging units. Based on the acquired reference image, image synthesis parameters are derived, and The synthesized image is generated using the exported image synthesis parameters.

18. The endoscope system according to claim 17, wherein, A channel is provided in the center of the end face of the front end. The camera is positioned around the channel on the end face.

19. The endoscopic system according to claim 17 or 18, wherein, The endoscope includes a wireless communication unit. The control unit acquires the endoscopic images output from the imaging unit via the wireless communication unit.

20. The endoscopic system according to claim 17 or 18, wherein, The processing performed by the control unit to generate the synthetic image includes geometric distortion correction using a surface projection method that projects the acquired endoscopic image onto a continuous curved surface. The processing performed by the control unit to output the synthesized image includes the process of outputting the synthesized image, which has been corrected for geometric distortion, to a display unit that reprojects it onto a curved or flat surface for display.

21. The endoscopic system according to claim 20, wherein, The processing performed by the control unit to output the composite image includes: outputting the composite image, which includes the front view and the rear view, in a full circumference direction relative to the front end, to the display unit, which includes multiple display screens.

22. The endoscopic system according to claim 21, wherein, The processing performed by the control unit to output the composite image includes the process of displaying the central portion of the composite image on the central display screen among the plurality of display screens.

Images