Endoscopic systems, lumen scanning methods, endoscopes and their operation methods

Abstract

The endoscope system (1) includes: an insertion section (2b); a subject light acquisition section (20) disposed in the insertion section (2b) for acquiring light from the subject, i.e., subject light; an imaging section for taking pictures based on the subject light, thereby acquiring an image within the field of view; a torsion mechanism (18) for rotating the subject light acquisition section (20); an advance / retraction mechanism (17) for moving the insertion section (2b) in an insertion direction or an withdrawal direction; and a control section (33) for controlling the torsion mechanism (18) and the advance / retraction mechanism (17) to scan the inner wall of the lumen through the field of view of the imaging section.

Description

Technical Field

[0001] This invention relates to endoscopic systems, lumen scanning methods, endoscopes, and methods for operating endoscopes. Background Technology

[0002] Previously, endoscopes were widely used in the medical and industrial fields. For example, in the medical field, doctors insert the endoscope into the body of the patient and observe the images displayed on a screen to examine the body, thus enabling endoscopic examinations.

[0003] For example, in colonoscopy-based examinations where the physician operates manually, uneven scanning of the colonic lumen can sometimes occur. This can result in the potential for missed lesions. Patent Document 1 discloses a method in which the actuator is controlled during insertion of a medical device, thereby automatically guiding the movement of the proximal end.

[0004] Furthermore, methods for constructing three-dimensional models of the intestine based on dynamic images captured by the endoscope have been studied with the aim of understanding unobserved areas during colonoscopy. For example, Non-Patent Literature 1 discloses a method for generating a mapping map of the colonic surface using a cylindrical model.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2017-164555

[0008] Non-patent literature

[0009] Non-patent document 1: Mohammad Ali Armin et al, "Automated visibility map of the internal colon surface from colonoscopy video", International Journal of Computer Assisted Radiology and Surgery, 2016, Volume 11, Issue number 9, p.1599-1610 Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] Patent Document 1 describes a method for automating the movement of a medical device toward a target area, but it does not disclose control for scanning the lumen. That is, Patent Document 1 cannot prevent omissions when observing the lumen structure. Furthermore, Non-Patent Document 1 does not disclose a method for controlling the movement of the insertion part 2b.

[0012] According to several aspects of the present invention, it is possible to provide an endoscope system that can suppress omissions in lumen structures and a lumen scanning method based on the endoscope system.

[0013] Methods for solving problems

[0014] One aspect of the present invention relates to an endoscope system comprising: an insertion section inserted into a lumen; a subject light acquisition section disposed in the insertion section for acquiring light from a subject, i.e., subject light; an imaging section for capturing an image based on the subject light, thereby acquiring an image within a field of view; a torsion mechanism for rotating the subject light acquisition section; an advance / retraction mechanism for moving the insertion section in an insertion direction or an withdrawal direction; and a control section for controlling the torsion mechanism and the advance / retraction mechanism, thereby controlling the movement of the field of view of the imaging section, the control section controlling the torsion mechanism and the advance / retraction mechanism to scan the inner wall of the lumen through the field of view.

[0015] Other aspects of the present invention relate to a method for scanning a lumen based on an endoscope system, comprising: inserting an insertion portion of an endoscope system into a lumen, the endoscope system having the insertion portion, a subject light acquisition portion disposed in the insertion portion and acquiring reflected light from a subject, i.e., subject light, and an imaging portion acquiring an image within a field of view based on the subject light; and performing a torsional motion to rotate the subject light acquisition portion and a forward / backward motion to move the insertion portion in an insertion direction or a withdrawal direction, so as to scan the inner wall of the lumen through the field of view. Attached Figure Description

[0016] Figure 1 This is a structural example of an endoscope system.

[0017] Figure 2 This is an example of the structure of an endoscope.

[0018] Figure 3 This is an example of the structure of each part of an endoscope system.

[0019] Figure 4 This is a structural example of an insertion part.

[0020] Figure 5 This diagram illustrates how the field of view of the camera is rotated by a torsion mechanism.

[0021] Figure 6 These are other structural examples of the insertion part.

[0022] Figure 7 These are other structural examples of the insertion part.

[0023] Figure 8 (A) Figure 8 (D) is another structural example of the insertion part.

[0024] Figure 9 (A) Figure 9 (B) is a diagram illustrating scanning based on the field of view of the camera.

[0025] Figure 10 This diagram illustrates scanning based on the camera's field of view.

[0026] Figure 11 This diagram illustrates scanning based on the camera's field of view.

[0027] Figure 12 Other structural examples of image processing devices.

[0028] Figure 13 This is a flowchart illustrating whether analysis, judgment, and processing are possible.

[0029] Figure 14 These are examples of photographic images where there are hidden parts due to folds, etc.

[0030] Figure 15 These are other structural examples of endoscope systems.

[0031] Figure 16 This is a structural example of a device for detecting the structure of a lumen.

[0032] Figure 17 This is a flowchart illustrating the acquisition and processing of information about the lumen structure.

[0033] Figure 18 This is an example of information about the structure of a lumen.

[0034] Figure 19 This is a flowchart illustrating the process of obtaining and processing information about the structure of a tube using bundle adjustment.

[0035] Figure 20 This is a schematic diagram illustrating the relationship between multiple feature points and the position and orientation of the front end.

[0036] Figure 21 Other structural examples of image processing devices.

[0037] Figure 22 This is an example of whether the correlation between analytical information and lumen structure information can be analyzed.

[0038] Figure 23 (A) Figure 23 (B) is an example of the positional relationship between the front end and the unanalyzable part. Detailed Implementation

[0039] The following describes this embodiment. Furthermore, the embodiments described below do not unduly limit the scope of the claims. Also, not all structures described in this embodiment are essential structural elements of the present invention.

[0040] 1. System Structure Example

[0041] In endoscopic examinations, minimizing the omission of areas of interest is crucial. Furthermore, an area of ​​interest refers to a region that, for the user, has a relatively higher priority for observation than other areas. When the user is a physician performing diagnosis or treatment, the area of ​​interest corresponds, for example, to the region where the lesion is captured, as described above. However, if the physician wants to observe vesicles or debris, the area of ​​interest can also be the region where those vesicles or debris are captured. In other words, the object of the user's attention varies depending on the purpose of the observation; however, during the observation, the region that has a relatively higher priority for observation than other areas becomes the area of ​​interest.

[0042] The following description uses an example where the endoscopic system is used to observe a living organism, and the object of observation is the large intestine. That is, in this embodiment, the lumen described later is narrowly defined as the intestine. However, the method of this embodiment can also target lumens other than the intestine. For example, the digestive tract other than the large intestine can be targeted, as well as lumen structures in other parts of the living organism. Furthermore, the endoscopic system can also be an industrial endoscope used to observe lumen-like components. Additionally, an example where the area of ​​interest is a lesion will be described below; however, as mentioned above, the area of ​​interest can be extended to areas other than lesions.

[0043] To minimize the risk of missing lesions, it is crucial to comprehensively image the entire surface of luminal structures such as the intestines. However, it has historically been difficult to accurately determine the location and movement of the endoscope within the luminal structure, and consequently, the extent of the luminal structure being imaged under specific imaging conditions. Furthermore, when physicians manually operate the endoscope, they must simultaneously perform procedures and conduct diagnoses based on the imagery. Consequently, in situations where the focus is solely on diagnosis, proper procedures may not be performed, resulting in uneven scanning of the large intestine. Additionally, when the focus is solely on procedures, it is difficult to fully review the imagery, potentially causing lesions to be missed even if they are captured.

[0044] Therefore, in the method of this embodiment, in the structure that controls the movement of the field of view of the camera unit by controlling the torsion mechanism 18 and the advance and retreat mechanism 17 of the endoscope system 1, control is performed such as scanning the inner wall of the lumen through the field of view.

[0045] Here, the field of view of the camera unit refers to a given space determined by the optical axis direction and field of view angle of the camera unit. For example, the field of view of the camera unit is a pyramidal or conical space with the position corresponding to the imaging element 15 as the vertex and the optical axis of the camera unit passing through the center of the vertex and the base. By aligning the optical axis of the camera unit towards the direction of the lesion or a direction close to it, the lesion can be captured within the field of view. For example, in the case where the subject light acquisition unit 20 acquires subject light and the camera unit receives the subject light to output a captured image, as described later, the relative positional relationship between the subject light acquisition unit 20 and the subject changes, thereby causing a change in the subject captured within the field of view.

[0046] Furthermore, in this embodiment, scanning refers to the action of sequentially photographing a predetermined range of the inner surface of the lumen by moving the aforementioned field of view according to a prescribed rule. Ideally, this predetermined range is the entire inner wall of the lumen.

[0047] According to the method of this embodiment, the endoscopic system is controlled to comprehensively image the inner wall of the lumen. Therefore, the workload of the physician on the endoscope can be reduced, and missed views due to the omission of a portion of the lumen structure being imaged can be suppressed. However, the method of this embodiment is a method that controls the possibility of missed views, but does not guarantee complete suppression of missed views. Furthermore, in this embodiment, a possible missed view may be determined if there is an area in the lumen structure that has never entered the field of view of the imaging unit. Alternatively, as explained later in the second embodiment, a possible missed view may be determined if there is an unanalyzable portion, which is a portion that was not imaged in an analytical state.

[0048] 2. First Implementation Method

[0049] 2.1 System Structure Example

[0050] Figure 1 This is a structural diagram of the endoscope system 1 according to this embodiment. The endoscope system 1 includes: an endoscope 2, an image processing device 3, a light source device 4, and a monitor 6 as a display device. A physician can use the endoscope system 1 to perform an endoscopic examination of the colon of a patient Pa lying supine on a bed 8. However, the endoscope system 1 is not limited to... Figure 1 The structure can be modified in various ways, such as omitting some structural elements or adding other structural elements. For example, as described later, the endoscope system 1 may also include a lumen structure detection device 5, a magnetic field generating device 7, etc.

[0051] In addition, Figure 1The illustration shows an example where the image processing device 3 is located near the endoscope 2, but it is not limited to this. For example, some or all of the functions of the image processing device 3 can also be implemented by a server system or the like that can be connected via a network. In other words, the image processing device 3 can also be implemented via cloud computing. The network here can be a private network such as an intranet, or a public communication network such as the Internet. Furthermore, the network can be wired or wireless.

[0052] Figure 2 This is a perspective view of endoscope 2. Endoscope 2 includes: an operating part 2a, a flexible insertion part 2b, and a universal cable 2c containing signal lines, etc. Endoscope 2 is a tubular insertion device into a body cavity through which the tubular insertion part 2b is inserted. A connector is provided at the front end of the universal cable 2c, through which endoscope 2 can be detachably connected to the light source device 4 and the image processing device 3. Here, endoscope 2 is an endoscope that can be inserted into the large intestine. Furthermore, although not shown, a light guide 22 is inserted through the universal cable 2c, through which the endoscope 2 allows illumination light from the light source device 4 to pass and exit from the front end of the insertion part 2b.

[0053] like Figure 2 As shown, the insertion part 2b has a front end portion 11, a bendable curved portion 12, and a flexible tube portion 13 extending from its front end towards its base end. The insertion part 2b is inserted into the lumen of the patient Pa, who is the subject. The base end portion of the front end portion 11 is connected to the front end portion of the curved portion 12, and the base end portion of the curved portion 12 is connected to the front end portion of the flexible tube portion 13. The front end portion 11 of the insertion part 2b is the front end portion of the endoscope 2 and is a rigid, frontal portion.

[0054] The bending section 12 can be bent in a desired direction according to the operation of the bending operation component 14 provided on the operation section 2a. The bending operation component 14 includes, for example, a left-right bending operation knob 14a and an up-down bending operation knob 14b. When the bending section 12 is bent to change the position and orientation of the front end 11 and capture the observation area in the subject's body within the field of view, illumination light is shone on the observation area. The bending section 12 has multiple bending blocks connected along the length axis of the insertion section 2b. Thus, while pushing the insertion section 2b into or pulling it out of the large intestine, the doctor can bend the bending section 12 in various directions, thereby enabling observation of the inside of the patient Pa's large intestine.

[0055] The left-right bending operation knob 14a and the up-down bending operation knob 14b pull and loosen the operation line that passes through and is inserted into the insertion part 2b, so that the bending part 12 bends. The bending operation component 14 also has a fixing knob 14c to fix the position of the bent part 12 after bending. In addition, in the operation part 2a, in addition to the bending operation component 14, various operation buttons such as a release button and an air / water supply button are also provided.

[0056] The flexible tube section 13 is flexible and bends according to external force. The flexible tube section 13 is a tubular component extending from the operating section 2a.

[0057] Furthermore, an imaging element 15, serving as an imaging device, is provided at the front end 11 of the insertion part 2b. The imaging element 15 captures images of the observation area within the large intestine illuminated by the light source device 4. Specifically, the imaging element 15 is configured as an imaging unit located at the front end 11 of the insertion part 2b, used to capture images of the subject. The imaging signal obtained by the imaging element 15 is provided to the image processing device 3 via the signal line within the universal cable 2c. Additionally, the location of the imaging element 15 is not limited to the front end 11 of the insertion part 2b. For example, by guiding the light from the subject, the imaging element 15 can be positioned further towards the base than the front end 11.

[0058] The image processing device 3 is a video processor that performs prescribed image processing on the received camera signal to generate a camera image. The image signal of the generated camera image is output from the image processing device 3 to the monitor 6, and the real-time camera image is displayed on the monitor 6. The doctor performing the examination can insert the front end 11 of the insertion part 2b into the anus of the patient Pa to observe the large intestine of the patient Pa.

[0059] The light source device 4 is a light source device capable of emitting normal light for normal light observation mode. Furthermore, when the endoscope system 1 has both a normal light observation mode and a special light observation mode, the light source device 4 selectively emits normal light for the normal light observation mode and special light for the special light observation mode. The light source device 4 emits either the normal light or the special light as illumination light depending on the state of the switching switch provided in the image processing device 3 for switching observation modes.

[0060] Figure 3 This is a schematic diagram showing the structure of the various parts of the endoscope system 1, which includes the image processing unit 3. The image processing unit 3 performs image processing and overall system control. The image processing unit 3 includes: an image acquisition unit 31, an image processing unit 32, a control unit 33, a storage unit 34, and a focus control unit 35. The insertion unit 2b includes: a subject light acquisition unit 20, an imaging element 15, an illumination lens 21, and a light guide 22. Specifically, the subject light acquisition unit 20 is an objective lens optical system that includes one or more lenses. For example, the subject light acquisition unit 20 includes a focusing lens 20a driven by an actuator 20b.

[0061] The light guide 22 guides the illumination light from the light source device 4 to the front end of the insertion part 2b. The illumination lens 21 irradiates the subject with the illumination light guided by the light guide 22. The subject light acquisition unit 20 acquires the reflected light, i.e., the subject light, reflected from the subject. The subject light acquisition unit 20 may also include a focusing lens 20a, and the position of the focused object can be changed according to the position of the focusing lens 20a. The actuator 20b drives the focusing lens 20a according to the instruction from the focusing control unit 35. Here, the focused object position refers to the position of the object when the system consisting of the lens system, the image plane, and the object is in a focused state. For example, when the image plane is set as the surface of the imaging element, the focused object position refers to the position of the subject in the captured image where the focus is ideally aligned when the subject image is captured using the imaging element via the above-described lens system.

[0062] The imaging element 15, which serves as the camera unit, can be a monochrome sensor or an element with a color filter. The color filter can be a widely known Bayer filter, a complementary color filter, or other filters. A complementary color filter is a filter that includes cyan, magenta, and yellow.

[0063] The image processing apparatus 3 of this embodiment is configured with the following hardware. The hardware can include at least one of circuitry for processing digital signals and circuitry for processing analog signals. For example, the hardware can be composed of one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an IC, an FPGA (field-programmable gate array), etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0064] Furthermore, the image processing apparatus 3 can also be implemented using the processor described below. The image processing apparatus 3 of this embodiment includes a memory for storing information and a processor for performing operations based on the information stored in the memory. The information includes, for example, programs and various types of data. The processor includes hardware. The processor can be various processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and DSP (Digital Signal Processor). The memory can be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), or a register, or a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disc drive. For example, the memory stores computer-readable commands, and the processor executes these commands, thereby implementing the functions of each part of the image processing apparatus 3 as processing. Specifically, each part of the image processing apparatus 3 includes… Figure 3 The various parts of the control unit 33 shown. The commands here can be commands that constitute a set of commands for a program, or commands that instruct the processor's hardware circuitry to perform actions.

[0065] Furthermore, each part of the image processing apparatus 3 in this embodiment can also be implemented as a module of a program running on a processor. For example, the control unit 33 is a control module that controls each part of the endoscope system 1. Specifically, the control unit 33 can also be a module that controls the use of... Figures 4-7 The control module controls the torsion mechanism 18 and the forward / backward mechanism 17, which will be explained later.

[0066] Furthermore, the program that implements the processing performed by each part of the image processing apparatus 3 in this embodiment can be stored, for example, in a computer-readable medium, i.e., an information storage device. The information storage device can be implemented, for example, by an optical disc, memory card, HDD, or semiconductor memory. The semiconductor memory is, for example, a ROM. The control unit 33 of the image processing apparatus 3 performs various processing operations according to the program stored in the information storage device. That is, the information storage device stores programs for enabling the computer to function as each part of the image processing apparatus 3. A computer is a device having an input device, a processing unit, a storage unit, and an output unit. The program is a program for causing the computer to execute the processing of each part of the image processing apparatus 3.

[0067] The image acquisition unit 31 acquires the captured images sequentially output from the camera unit and outputs them sequentially to the image processing unit 32 and the focus control unit 35. The image processing unit 32 performs various image processing operations on the captured images, such as white balance processing, de-mosaic (simultaneous) processing, noise reduction processing, color conversion processing, grayscale conversion processing, and contour enhancement processing, and outputs them sequentially to the monitor 6. The control unit 33 inputs and outputs various control signals.

[0068] The focus control unit 35 controls the focusing lens 20a based on the captured image. The focus control unit 35 controls the focusing lens 20a, for example, based on a known contrast-based autofocus (AF). However, in the endoscope system 1 of this embodiment, AF is not a necessary component, and the focus control unit 35 can be omitted.

[0069] The light source device 4 includes a light source control unit 41 and a light source 42. The light source control unit 41 controls the light quantity of the light source 42 according to the target light quantity of the light source sequentially output from the control unit 33. The light source 42 emits illumination light. The light source 42 can be a xenon light source, an LED (light emitting diode), or a laser light source. In addition, the light source 42 can also be other light sources, and the light emission method is not limited.

[0070] 2.2 Detailed structural example of the insertion part

[0071] Figure 4 This is a schematic diagram showing an example of the structure of the insertion part 2b. For example... Figure 4 As shown, the length direction of the insertion portion 2b is set as the reference axis AX1. Furthermore, the reference axis AX1, in a narrow sense, is the length direction of the flexible tube portion 13. In the unbent state, the length direction of the bent portion 12 is approximately aligned with the reference axis AX1. In other words, by performing a bending operation, the length direction of the bent portion 12 changes to a direction different from the reference axis AX1.

[0072] like Figure 4As shown, the advance / retract mechanism 17 includes, for example, an advance / retract roller 17a that moves the insertion portion 2b in a direction corresponding to the reference axis AX1, and a drive unit 19 that drives the advance / retract roller 17a. The corresponding direction can be the same direction or approximately the same direction. Approximately the same direction is a direction in which the angle with the reference axis AX1 is less than a predetermined threshold. The advance / retract roller 17a can rotate about AX2 in the direction shown by D1 or D2. A portion of the advance / retract roller 17a contacts the insertion portion 2b. Therefore, the insertion portion 2b moves in a proximal direction by rotating the advance / retract roller 17a in the direction of D1. Here, the proximal direction is the direction toward the base end side of the insertion portion 2b, which corresponds to the anal side during insertion. Furthermore, the insertion portion 2b moves in an inward direction by rotating the advance / retract roller 17a in the direction of D2. Here, the inward direction is the direction in which the insertion portion 2b is pushed toward the front end side, which corresponds to the cecum side during insertion.

[0073] Furthermore, the torsion mechanism 18 includes, for example, a rotating roller 18a that rotates the insertion portion 2b about the reference axis AX1, and a drive unit 19 that drives the rotating roller. Alternatively, the drive unit that drives the advance roller 17a and the drive unit that drives the rotating roller 18a may be provided separately. The rotating roller 18a is capable of rotating in a direction indicated by D3 or D4 with AX3 as the rotation axis. A portion of the rotating roller 18a contacts the insertion portion 2b. Therefore, the insertion portion 2b rotates in the opposite direction to the rotating roller 18a due to the rotation of the rotating roller 18a.

[0074] In addition, such as Figure 4 As shown, the bending portion 12 is capable of bending relative to the reference axis AX1. Specifically, the bending portion 12 can adjust the bending angle relative to the reference axis AX1 by operating at least one of the up-down bending operation knob 14b and the left-right bending operation knob 14a. Figure 4 The θ shown represents the bending angle.

[0075] Figure 5 This diagram illustrates the movement of the front end 11 when the rotating roller 18a is rotated while the bent portion 12 is bent relative to the reference axis AX1. Figure 5 F1 and F2 represent the insertion portion 2b in the bent state of the bent portion 12. Furthermore, Figure 5 F3 and F4 represent the field of view of the camera unit when the insertion part 2b is in the states of F1 and F2, respectively.

[0076] Figure 5 The F2 shown corresponds to the state of F1, in which the insertion part 2b is rotated about the reference axis AX1 by the rotating roller 18a. The state afterward. As the insertion section 2b rotates based on the rotating roller 18a, the field of view of the camera section also rotates around the reference axis AX1.

[0077] For example, when the curved section 12 is bent at an arbitrary bending angle θ relative to the reference axis AX1, rotating the rotating roller 18a 360° allows for comprehensive circumferential imaging of the cavity. Furthermore, by driving the advance / retract roller 17a, the field of view of the imaging unit can be moved along the length of the cavity. That is, by controlling the torsion mechanism 18 and the advance / retract mechanism 17, field of view control, such as scanning the inner wall of the cavity, is possible. Regarding specific scanning methods, using… Figure 9 (A) Figure 11 The explanation will follow.

[0078] Figure 6 This is a schematic diagram showing other structures of the insertion part 2b. Figure 4 The diagram shows a structure in which the insertion portion 2b is rotated as a whole around a reference axis AX1 using a rotating roller 18a. In contrast, in... Figure 6 In the structure shown, the torsion mechanism 18 has a rotation mechanism 18b, which rotates only a portion of the insertion part 2b near the front end 11 about the reference axis AX1. Figure 6 In the example shown, the front end portion 11 and the bent portion 12 in the insertion portion 2b can be rotated about the reference axis AX1 via the rotation mechanism 18b, while the flexible tube portion 13 does not rotate about the reference axis AX1. However, various modifications can be made regarding which part of the insertion portion 2b can be rotated.

[0079] like Figures 4-6 As shown, the torsion mechanism 18 is, for example, a mechanism that rotates at least a portion of the insertion portion 2b about the reference axis AX1 while the bent portion 12 is bent relative to the reference axis AX1. The insertion portion 2b may rotate as a whole or only a portion of the front end side may rotate.

[0080] Furthermore, the torsion mechanism 18 is only required to rotate the field of view of the camera unit, and is not limited to rotating the insertion part 2b around the reference axis AX1. As described above, the bending part 12 has an up-down bending operation knob 14b and a left-right bending operation knob 14a, and can adjust the bending direction to four directions (up, down, left, and right) based on the bending angle θ relative to the reference axis AX1. For example, when the upward bending is set as the reference position, the right or left bending corresponds to a rotation of ±90 degrees around the reference axis AX1. In addition, the downward bending corresponds to a rotation of ±180 degrees around the reference axis AX1. Furthermore, by adjusting the degree of bending in the up-down direction and the degree of bending in the left-right direction, rotation between 0° and ±90°, and between ±90° and ±180°, can also be achieved.

[0081] That is, the torsion mechanism 18 is a mechanism that drives the bending portion 12, and the torsion mechanism 18 can also rotate the subject light-acquiring portion 20 about the reference axis AX1 by changing the shape of the bending portion 12. For example, the torsion mechanism 18 includes a motor (not shown) for rotating the left-right bending operation knob 14a and the up-down bending operation knob 14b. The control unit 33 controls the motor to automatically operate the left-right bending operation knob 14a and the up-down bending operation knob 14b.

[0082] Figure 7 This is a schematic diagram showing other structures of the curved portion 12. For example... Figure 7 As shown, the bending section 12 may also include two or more bending sections that can be individually controlled. Figure 7 In the example, the bending section 12 includes a first bending section 12a, a second bending section 12b, and a third bending section 12c. The first bending section 12a to the third bending section 12c can, for example, be operated in the up-down direction and the left-right direction, respectively. Thus, by dividing the bending section 12 into multiple bending sections, the position and orientation of the subject light-acquiring section 20 relative to the cavity can be controlled with high precision. For example, even when the torsion mechanism 18 does not have a rotating roller 18a or a rotation mechanism 18b, dividing the bending section 12 into multiple bending sections makes it easy to finely control the interval of rotation, etc.

[0083] In addition, it is also possible to Figure 7 The structure shown is similar to Figure 4 The structure shown allows the entire insertion part 2b to rotate. Figure 6 The structure shown rotates the front end portion of the insertion part 2b, and other structures that rotate part or all of the insertion part 2b are combined.

[0084] Figure 8 (A) Figure 8 (D) is a schematic diagram showing other structures of the insertion part 2b. For example... Figure 8 (A) Figure 8 As shown in (D), the subject light acquisition unit 20 can also receive subject light from the side of the insertion unit 2b.

[0085] exist Figure 8 (A) Figure 8 In the structure shown in (B), part or all of the insertion part 2b can be connected with... Figure 4 or Figure 6 The same torsion mechanism 18 rotates. By rotating the insertion part 2b, the subject light acquisition part 20 provided on the side rotates about the reference axis AX1, and therefore, the field of view of the camera part rotates about the reference axis AX1. That is, the torsion mechanism 18 can be implemented with a structure that does not cause the bending part 12 to bend.

[0086] In addition, such as Figure 8 As shown in (C), the insertion part 2b can also be a non-rotating structure. Figure 8 In (C), the insertion part 2b may also have a transparent part 18c, inside which the subject light acquisition part 20 can rotate about the axis of the insertion part 2b. In this way, by rotating the subject light acquisition part 20 inside the insertion part 2b, the field of view of the camera unit can also be rotated about the reference axis AX1.

[0087] In addition, such as Figure 8 As shown in (D), the subject light acquisition unit 20 can also be a structure capable of changing the light-receiving direction of the subject light. For example, the torsion mechanism 18 may include a drive unit (not shown) that drives the objective lens optical system, which serves as the subject light acquisition unit 20. The control unit 33 drives the objective lens optical system, thereby changing the field of view of the imaging unit. For example, when the light-receiving direction of the subject light can be changed to the up-down direction and the left-right direction, the field of view of the imaging unit can be rotated around the reference axis AX1 by changing the light-receiving direction sequentially. Alternatively, the field of view of the imaging unit can be rotated around the reference axis AX1 by fixing the subject light acquisition direction at a predetermined angle relative to the reference axis AX1 and rotating part or all of the insertion unit 2b around the reference axis AX1 in this state.

[0088] In addition, in use Figure 8 (A) Figure 8 In the case of the structure shown in (D), the optical axis of the imaging unit becomes an axis in a direction different from the insertion / removal direction of the insertion unit 2b, i.e., along the reference axis AX1. Therefore, the imaging image becomes an image suitable for diagnosing the inner wall of the lumen, but it becomes an image unsuitable for insertion / removal. For example, even if an imaging image obtained by taking a picture in the lateral direction is displayed, it is not easy for the physician to insert the device in the inward direction based on the imaging image.

[0089] Therefore, the endoscope 2 can also include structures for lateral observation and structures for frontal observation. For example, the endoscope 2 includes a second subject light acquisition unit and a second imaging unit for frontal observation, in addition to a subject light acquisition unit 20 for lateral observation and an imaging unit. In this way, observations for insertion / removal and observations for diagnosis can be performed appropriately.

[0090] Alternatively, the endoscope 2 may include a subject light acquisition unit 20 for side observation and a second subject light acquisition unit for frontal observation, sharing a common imaging unit. In this case, the imaging unit can selectively receive either the subject light from the subject light acquisition unit 20 for side observation or the subject light from the second subject light acquisition unit for frontal observation. For example, subject light selection can be achieved by controlling a light-shielding member that is pluggable in an optical path that is the path from reflected light from the subject to the imaging unit via either the subject light acquisition unit 20 or the second subject light acquisition unit. This allows switching between using one imaging unit for pluggable observation and for diagnostic observation. Furthermore, when using... Figure 8 In the case of the structure shown in (D), the direction of light received by the subject can also be switched between the front direction for insertion and the side direction for observation.

[0091] 2.3 Examples of scanning

[0092] Next, a scanning example, specifically a control example based on the field of view of the torsion mechanism 18 and the advance / retract mechanism 17, will be described. Furthermore, as described above, the structure of the insertion part 2b can be adapted to various configurations; in any configuration, the field of view of the camera unit can be moved along the length of the cavity, and the field of view of the camera unit can be rotated circumferentially within the cavity. Therefore, the specific structure of the insertion part 2b will be omitted below, and the description will focus on the field of view of the camera unit. Moreover, for the sake of simplicity, the cavity will be described as a cylinder below.

[0093] Figure 9 (A) is a diagram illustrating the movement of the field of view during a spiral scan. For example, when the area on the intestinal surface included in the field of view of the imaging unit is defined as the imaging range, Figure 9 (A) is a graph illustrating the time-series changes of the reference point of the imaging range. The reference point of the imaging range is, for example, the center of the imaging range, corresponding to the intersection of the optical axis of the imaging unit and the surface of the intestine. Figure 9 (B) is a diagram illustrating the positional relationship between the camera range when the reference point of the camera range is located at C1 and the camera range when the reference point of the camera range is located at C2. Figure 9 In (B), C3 represents the camera range corresponding to C1, and C4 represents the camera range corresponding to C2.

[0094] The imaging element 15 acquires reflected light from the subject corresponding to the imaging range via the subject light acquisition unit 20, thereby generating an image corresponding to the imaging range. Hereinafter, an example will be described where the vertical direction of the image is along the reference axis AX1, and the horizontal direction is the rotational direction around the reference axis AX1. However, the relationship between the vertical and horizontal directions of the image and the direction of the reference axis AX1 is arbitrary. Furthermore, for ease of explanation, an example where the imaging range is rectangular will be described; however, the actual shape of the imaging range varies depending on the imaging angle and the surface shape of the intestine.

[0095] exist Figure 9 In the example shown in (A), the control unit 33 controls the torsion mechanism 18, thereby rotating the field of view of the camera unit around the reference axis AX1, and controls the advance / retreat mechanism 17, thereby moving the field of view of the camera unit in the direction along the reference axis AX1. Here, an example is given where the rotational speed around the reference axis AX1 and the movement speed in the direction along the reference axis AX1 are constant, but the speeds can also be variable.

[0096] At this time, the control unit 33 controls the torsion mechanism 18 so that the field of view of the camera in a given frame overlaps with the field of view of the camera in the next frame in the circumferential direction of the cavity. For example, when the camera interval is set to t (seconds), the rotation angle per unit time is set to vθ (degrees / second), and the horizontal field of view of the camera is set to θ1 (degrees), the control unit 33 sets vθ to satisfy the following formula (1). In this way, two consecutive images in the time sequence have overlapping areas. That is, it is possible to suppress the generation of uncaptured areas in the circumferential direction of the cavity.

[0097] t×vθ≦θ1…(1)

[0098] Furthermore, the control unit 33 sets the movement speed of the field of view based on the rotation speed of the field of view based on the torsion mechanism 18. For example, the forward / backward mechanism 17 is controlled so that the camera image in a given frame overlaps with the camera image after the field of view has rotated one revolution around the reference axis AX1 from that point in the direction along the reference axis AX1. Figure 9 In example (B), the control unit 33 makes the lower region of the camera range shown in C3 repeat the upper region of the camera range after one rotation, i.e., C4. For example, when the rotation period of the field of view of the camera unit based on the torsion mechanism 18 is set to T (seconds), the movement speed of the field of view of the camera unit based on the advance / retreat mechanism 17 is set to vz (cm / second), and the longitudinal length of the camera range is set to H (cm), the control unit 33 sets vz to satisfy the following formula (2). Here, T = 360 / vθ. Furthermore, H is determined by the vertical field of view angle θ2 of the camera unit and the distance L from the camera unit to the subject. In addition, as Figure 9 As shown in (A), assuming the cavity is cylindrical and the reference axis AX1 coincides with the axis passing through the center of the cylinder, the distance L is calculated based on the radius r of the cylinder. Figure 8 (A) Figure 8 In the case of a structure where the bend 12 is not bent, as in (D), the distance L can be approximated by the radius r. Furthermore, when using the bend 12, the distance L can be estimated based on known information such as the length and amount of bending of the bend 12. This also helps to suppress the formation of areas that become unphotographed gaps along the length of the lumen.

[0099] T×vz≦H…(2)

[0100] For example, the storage unit 34 of the image processing device 3 stores vθ satisfying equation (1) and vz satisfying equation (2) as control parameters. The control unit 33 controls the torsion mechanism 18 and the advance / retreat mechanism 17 based on vθ and vz read from the storage unit 34. Thus, the control of the torsion mechanism 18 and the advance / retreat mechanism 17, which scans the inner wall of the cavity through the field of view of the camera unit, is realized.

[0101] Figure 10 These are other diagrams illustrating the movement of the camera's field of view. For example... Figure 10 As shown, the control unit 33 can also alternately control the torsion mechanism 18 that rotates the field of view of the camera unit around the reference axis AX1 and the forward / backward mechanism 17 that moves the field of view of the camera unit in the direction along the reference axis AX1. If it is Figure 10 In the example, the control unit 33 first controls the torsion mechanism 18 to rotate the field of view around the reference axis AX1 by one revolution (D1), and then controls the forward and backward mechanism 17 to move the field of view in the direction along the reference axis AX1 (D2). Thereafter, the rotation of the field of view around the reference axis AX1 and the movement in the direction of the reference axis AX1 are repeated alternately.

[0102] and Figure 9 Similarly, the control unit 33 controls the torsion mechanism 18 based on vθ satisfying equation (1) above. However, since the torsion mechanism 18 exclusively controls the advance / retreat mechanism 17, the constraints related to control are small. Specifically, since the movement speed of the field of view of the advance / retreat mechanism 17 is arbitrary, the control unit 33 can control the amount of movement in one move. For example, the control unit 33 controls the advance / retreat mechanism 17 so that the amount of movement in one move shown in D2 is less than or equal to the longitudinal length of the imaging range, i.e., H. In this way, it is possible to suppress the generation of un-photographed areas in the length direction of the lumen.

[0103] In addition, Figure 9 (A) Figure 10The scanning process is described in the case where the field of view of the camera can rotate continuously in the same direction. However, depending on the structure of the torsion mechanism 18, the case where the number of rotations in the same direction is limited is also considered. For example, in cases such as... Figure 6 When a rotating mechanism 18b is provided at the front end of the insertion part 2b as shown, the portion including the rotating mechanism 18b becomes susceptible to cleaning, etc. Therefore, a mechanical seal resistant to cleaning is required to achieve multiple rotations. In contrast, sealing is easily achieved by limiting the number of rotations in the same direction. Thus, the control unit 33 can also perform control that does not include continuous rotation in the same direction.

[0104] Figure 11 These are other diagrams illustrating the movement of the camera's field of view. (And...) Figure 10 Similarly, in the example shown, the control unit 33 alternately controls the torsion mechanism 18 that rotates the field of view of the camera unit about the reference axis AX1 and the forward / backward mechanism 17 that moves the field of view of the camera unit in the direction along the reference axis AX1. However, the control unit 33 controls the torsion mechanism 18 so that the rotation of the field of view of the camera unit in a predetermined direction and the rotation in the opposite direction are performed alternately.

[0105] in the case of Figure 11 For example, the control unit 33 performs the following control: after rotating the field of view of the camera unit one full rotation in a predetermined direction (E1), it moves the camera unit by a predetermined amount in the direction along the reference axis AX1 (E2), and rotates the field of view of the camera unit one full rotation in the opposite direction to E1 (E3). Thereafter, the rotation direction is switched every time thereafter. Furthermore, an example is shown here where the consecutive number of rotations in the predetermined direction is one full rotation, but the number of rotations could also be two or more full rotations.

[0106] In this case, the control unit 33 also controls the torsion mechanism 18 and the advance / retreat mechanism 17 to suppress the generation of uncaptured areas in both the circumferential and longitudinal directions of the cavity. For example, the control unit 33 controls the torsion mechanism 18 according to vθ satisfying the above equation (1), and controls the advance / retreat mechanism 17 so that the amount of movement shown in E2 is less than or equal to the longitudinal length H of the imaging range.

[0107] As described above, the endoscope system 1 of this embodiment includes an insertion section 2b inserted into a lumen, a subject light acquisition section 20, an imaging section, a torsion mechanism 18, an advance / retract mechanism 17, and a control section 33. The subject light acquisition section 20 is provided in the insertion section 2b and acquires light from the subject, i.e., subject light. The imaging section captures images based on the subject light, thereby obtaining an image within the field of view. The torsion mechanism 18 rotates the subject light acquisition section 20. For example, when the axis of the insertion section 2b is set as the reference axis AX1, the torsion mechanism 18 rotates the subject light acquisition section 20 about the reference axis AX1. The advance / retract mechanism 17 moves the insertion section 2b in an insertion direction or an withdrawal direction. The insertion direction is the direction towards the inside of the lumen; in the case where the lumen is the intestine, the insertion direction is from the anus towards the vicinity of the cecum. The withdrawal direction is the opposite direction to the insertion direction. For example, the advance / retract mechanism 17 moves the insertion section 2b in a direction corresponding to the reference axis AX1. The control unit 33 controls the torsion mechanism 18 and the advance / retract mechanism 17, thereby controlling the movement of the camera's field of view. Furthermore, the control unit 33 controls the torsion mechanism 18 and the advance / retract mechanism 17 to scan the inner wall of the cavity through the field of view.

[0108] Here, the direction corresponding to the reference axis AX1 refers to the same or approximately the same direction as the reference axis AX1, and the direction corresponding to the reference axis AX1 indicates the direction in which the angle between the reference axis AX1 and the reference axis AX1 is below a given angle threshold.

[0109] According to the method of this embodiment, in an endoscope system 1 that automatically controls the insertion part 2b via a torsion mechanism 18 and an advance / retract mechanism 17, imaging can be performed to scan the inner wall of the lumen. Since the generation of unimaged portions during observation using the endoscope system 1 is suppressed, omissions of areas of interest can be prevented.

[0110] Furthermore, the control unit 33 performs scanning by combining the movement of the insertion unit 2b along the reference axis AX1 using the advance and retraction mechanism 17 and the periodic rotation of the subject light acquisition unit 20 in the circumferential direction of the cavity using the torsion mechanism 18.

[0111] The periodic rotation here can also be Figure 9 (A) refers to continuous, repeated rotations in the same direction. Furthermore, periodic rotations can also be intermittent, repeated rotations. Intermittent repetition, for example, refers to... Figure 10 After a rotation of a given amount in a given direction, followed by a period of movement along the reference axis AX1, a rotation of the same amount in the same direction is performed again. Furthermore, periodic rotation can also be performed as follows: Figure 11 That includes both rotation in a given direction and rotation in the opposite direction.

[0112] In this way, by performing periodic control, the generation of uncaptured portions can be suppressed. That is, if control can be achieved in one cycle, the same control can be repeated thereafter, thus reducing the amount of information required for control and facilitating control automation. The information required for control includes, for example, the control parameters such as vθ and vz mentioned above, and a control program for the control unit 33 to execute the control of the advance / retreat mechanism 17 and the torsion mechanism 18 based on these control parameters.

[0113] Furthermore, the image captured by the camera unit at a given time interval is designated as the first image. After the first image is captured, the control unit 33 performs control, and after this control, designates the image captured by the camera unit as the second image. This control is a combination of control that rotates the subject light acquisition unit 20 approximately one revolution around the reference axis AX1 and control that moves the insertion unit 2b in the direction along the reference axis AX1 using the advance / retract mechanism 17. In this case, the control unit 33 controls the torsion mechanism 18 and the advance / retract mechanism 17 so that the field of view when the first image is captured and the field of view when the second image is captured have overlapping portions. In other words, the control unit 33 controls the torsion mechanism 18 and the advance / retract mechanism 17 so that the first image and the second image have overlapping portions.

[0114] When considering capturing the entire inner surface of the tube without omission, it is important that the captured images repeat each other. For example, as shown in equation (1) above, it is necessary to control the rotation speed based on the torsion mechanism 18 so that the captured image in a given frame repeats with the captured image in the next frame. However, when observing the inner wall of the tube while rotating the field of view of the camera, it is also necessary to consider the repetition between a given rotation and the next rotation. Specifically, as shown in equation (2) above, it is necessary to control the moving speed or moving amount based on the advance / retreat mechanism 17.

[0115] The first camera image is, for example, an image obtained by capturing the camera range shown in C3, and the second camera image is, for example, an image obtained by capturing the camera range shown in C4. The first and second camera images have overlapping areas, thereby suppressing the gap between the camera range captured in a given camera image and the camera range captured in different camera images along the length of the lumen.

[0116] Furthermore, depending on the frame rate of the shot and the setting based on the rotation speed of the torsion mechanism 18, the first and second camera images are not necessarily obtained by shooting subject light from the same direction. That is, the rotation angle of approximately one full rotation mentioned above can be 360°, but is not limited to this, and can also be other angles below a given threshold that differ from 360°.

[0117] Furthermore, if we consider Figure 11 For example, the second camera image is not limited to an image captured with the field of view rotated approximately one full revolution relative to the first camera image. For instance, the camera unit is rotated by an angle of approximately [missing information - likely a rotation angle] from the reference axis AX1. The image obtained by capturing the subject light in the direction of the object is designated as the first image. Then, after capturing the first image, the control unit 33 performs the following control: after the control unit 33 performs this control, the image obtained by the camera unit from the object light rotating around the reference axis AX1 at an angle of... The image obtained by capturing the subject light in the direction of the subject is set as the second image. The control is a combination of the control that rotates the subject light acquisition unit 20 around the reference axis AX1 and the control that moves the insertion unit 2b in the direction along the reference axis AX1 using the advance and retreat mechanism 17. Indicates and The difference between them is an angle below a specified threshold. That is, the specific movement of the camera's field of view from the time the first image is captured until the second image is captured is arbitrary. For example, the camera's field of view can rotate one full circle in a given direction, or the rotation direction can be switched midway. In this case, control is also performed to repeat the first and second images, thereby suppressing the generation of uncaptured portions along the length of the lumen, wherein the first image is an image obtained by capturing a given image in a direction perpendicular to the reference axis AX1, and the second image is an image obtained by capturing an image in approximately the same direction after the first image.

[0118] In addition, such as Figure 9 As shown in (A), the control unit 33 can also perform rotation based on the torsion mechanism 18 and movement based on the advance / retreat mechanism 17, thereby controlling the subject light-acquiring unit 20 to move in a spiral shape. In a narrow sense, the rotation based on the torsion mechanism 18 and the movement based on the advance / retreat mechanism 17 are performed simultaneously. Furthermore, as... Figure 10 , Figure 11 As shown, the control unit 33 can also perform control that alternates between rotation based on the torsion mechanism 18 and movement based on the forward and backward mechanism 17.

[0119] By simultaneously performing rotation based on the torsion mechanism 18 and movement based on the advance / retract mechanism 17, the observation of the lumen can be accelerated. This reduces the burden on physicians and patients. By alternately performing rotation based on the torsion mechanism 18 and movement based on the advance / retract mechanism 17, it is easier to control the repetition of the camera's field of view. Therefore, it is possible to further suppress the generation of areas that are not captured.

[0120] Furthermore, the torsion mechanism 18 may also have a mechanism capable of rotating the front end of the insertion part 2b about the reference axis AX1. This mechanism specifically corresponds to... Figure 6 The rotating mechanism 18b shown.

[0121] In this way, it is not necessary to rotate the entire insertion part 2b, so the operating part 2a does not move along with the rotation of the insertion part 2b, making observation easier. On the other hand, in situations such as Figure 4 When the insertion part 2b is rotated as a whole as shown, it has the advantage that the sealing of the rotating mechanism does not need to be considered.

[0122] Furthermore, in this embodiment, the lumen can also be observed by inserting the insertion part 2b in the inward direction and then pulling it out in the forward direction. Insertion in the inward direction is usually an insertion from the anus to the cecum, or an insertion up to the deepest point that can be inserted before reaching the anus to the cecum. Then, the control unit 33 controls the torsion mechanism 18 and the forward and backward mechanism 17 at least when the insertion part 2b is pulled out in the forward direction.

[0123] In tubular structures such as the large intestine, the insertion of the insertion part 2b requires navigating various bends. During insertion, methods such as degassing the lumen and then pressing the front end 11 against the lumen wall are used to facilitate insertion, making it difficult to observe the lumen structure. Therefore, after insertion to the innermost part, the inner wall of the lumen is observed while air is being supplied to it.

[0124] The method of this embodiment relates to the control of the insertion portion 2b for suppressing the omission of the lumen structure, which is preferably controlled during withdrawal.

[0125] However, the control unit 33 can also control the torsion mechanism 18 and the advance / retraction mechanism 17 when the insertion part 2b is inserted into the lumen. As mentioned above, operations such as passing over the bend are required during insertion, and various methods such as the shaft-holding shortening method are known. By controlling the torsion mechanism 18 and the advance / retraction mechanism 17, the control unit 33 can appropriately assist the insertion of the insertion part 2b. For example, appropriate insertion can be performed regardless of the physician's skill level, thus reducing the burden on the patient.

[0126] Furthermore, the method of this embodiment can be applied to the following lumen operation method. The lumen operation method includes: inserting the insertion part 2b of the endoscope system 1 into the lumen; and performing a torsional motion that rotates the subject light acquisition part 20 and a forward / backward motion that moves the imaging part in the insertion or withdrawal direction of the insertion part 2b, so as to scan the inner wall of the lumen through the field of view of the imaging part. For example, the torsional motion is an action that rotates the subject light acquisition part 20 around a reference axis AX1. Furthermore, the forward / backward motion is an action that moves the insertion part 2b in the direction along the reference axis AX1. As described above, the endoscope system 1 here has an insertion part 2b, a subject light acquisition part 20 disposed in the insertion part 2b and acquiring reflected light, i.e., subject light, from the subject, and an imaging part that captures an image within the field of view based on the subject light.

[0127] In addition, the cavity operation method can also be used to perform scanning by combining the forward and backward motion with the torsional motion that causes the subject light-acquiring part 20 to rotate periodically in the circumferential direction of the cavity.

[0128] 2.4 Specific Controls During Scanning

[0129] By controlling the torsion mechanism 18 and the forward / backward mechanism 17 as described above, it is possible to comprehensively photograph the inner surface of the lumen structure. However, in cases where the lumen is prone to deformation, even with the above-described controls, it is difficult to achieve proper photographing depending on the lumen's condition. For example, if the lumen is an intestine, the intestine will expand and contract, and therefore the state of the folds will change according to the gas pressure inside the intestine. Insufficient air delivery can easily lead to problems with the use of... Figure 14 The hidden parts explained later are therefore easy to miss.

[0130] Therefore, when scanning the inner wall of the cavity by controlling the torsion mechanism 18 and the advance / retreat mechanism 17, the control unit 33 controls the state of the cavity. This prevents the front end 11 of the insertion part 2b, and specifically the subject light-acquiring part 20, from contacting the cavity wall, or other parts from being blocked by a portion of the cavity. In other words, it prevents the formation of hidden parts that are not fully captured in the image.

[0131] More specifically, the control unit 33 performs control to maintain the state of the lumen by inflating gas into the lumen and deflating gas out of the lumen. This maintains the gas pressure within the lumen, thus ensuring a suitable state for observation. Furthermore, as mentioned above, controlling the intestinal bulge through inflating is important in intestinal observation. However, excessive inflating increases the patient's burden. By performing both inflating and deflating, an appropriate state can be maintained.

[0132] Furthermore, the control unit 33 can also control the adjustment of the subject's position to maintain the state of the lumen. When observing a living part such as the intestine, the state of the intestine changes according to the subject's position. Such positions include, for example, right lateral decubitus, left lateral decubitus, and supine positions. By controlling the position, the intestine can be easily observed.

[0133] Furthermore, "maintaining the state of the lumen" here refers to keeping the lumen in a state suitable for observation. That is, when the body position suitable for observation is constant, controlling the state of the lumen is equivalent to controlling the body position. However, when the preferred body position differs for each part of the intestine, the state of the lumen can be maintained in a state suitable for observation by adopting a body position corresponding to that part. In this case, controlling the state of the lumen is equivalent to controlling the change of body position according to the body position.

[0134] Alternatively, the control for adjusting the subject's position can also be a control that provides information indicating a change in position. Or, the control for adjusting the subject's position can also be a control of the bed 8 on which the subject lies. For example, the bed 8 may include a drive unit for changing the tilt angle, and the control unit 33 outputs a drive signal to this drive unit, thereby adjusting the subject's position.

[0135] In addition, regarding the control of gas supply and degassing, as well as the control of adjusting the position of the subject, both of these can be performed, or either one can be performed.

[0136] 2.5 Rescan

[0137] Furthermore, in this embodiment, feedback on the scan can also be provided based on the imaging results. Specifically, the control unit 33 can also perform a rescan if it determines that there are portions in the scanned portion of the lumen that have not entered the field of view of the camera unit. Here, specifically, "not entering the field of view" means that the portion has not entered the field of view of the camera unit even once during the period from the start of the scan to the timer of the processing object.

[0138] For example, control unit 33 in Figure 9 If the two camera ranges shown in (B) C3 and C4 do not overlap, it is determined that there is a portion that is not within the field of view of the camera unit. More specifically, the control unit 33 performs a comparison process between the first camera image and the second camera image to determine whether there is a duplicate portion. For example, the control unit 33 performs template matching using a portion of the first camera image as a template image. Alternatively, as will be explained later in the third embodiment, if lumen structure information can be obtained, the control unit 33 can also determine that there is a portion that is not within the field of view of the camera unit even if the lumen structure information is missing.

[0139] For example, after inserting the insertion part 2b inward by a predetermined amount, a rescan can be performed using the same scanning conditions as before. These scanning conditions include at least the control conditions for the torsion mechanism 18 and the control conditions for the advance / retract mechanism 17. For example, the scanning conditions include the rotational speed of the torsion mechanism 18 and the moving speed or amount of movement of the advance / retract mechanism 17. Alternatively, as described later in the second embodiment, the scanning conditions can be changed during rescanning. For example, the control unit 33 suppresses the moving speed or amount of movement of the front end 11 based on the advance / retract mechanism 17 in the direction along the reference axis AX1.

[0140] 3. Second Implementation Method

[0141] Figure 12 This is a diagram illustrating an example of the structure of the image processing apparatus 3 according to this embodiment. Figure 12 As shown, the image processing device 3 can also be used Figure 3 Based on the structure described above, the image processing apparatus 3 also includes an analysis unit 36 ​​and a decision unit 37 for whether or not analysis is possible. However, the image processing apparatus 3 is not limited to... Figure 12 The structure allows for various modifications, such as omitting either the analysis unit 36 ​​or the analysis-determination unit 37, or adding other structural elements. Each part will be described in detail below. Furthermore, at least one of the analysis unit 36 ​​and the analysis-determination unit 37 may be located in a device different from the image processing apparatus 3.

[0142] 3.1 Analysis Department

[0143] The analysis unit 36 ​​performs analysis based on the camera image. For example, if the area of ​​interest is a lesion, the analysis can be detection processing to detect the lesion from the camera image, or classification processing to classify the lesion according to its malignancy. By including the analysis unit 36, not only the scanning of the insertion unit 2b can be assisted by the user in performing analysis related to the area of ​​interest.

[0144] The analysis unit 36 ​​performs image processing on the captured image, thereby performing detection or classification processing. Image processing here includes, for example, determining feature quantities from the image and whether those feature quantities meet specified conditions. Feature quantities can be image brightness, lightness, hue, chroma, the result of edge extraction processing, or the result of matching processing using a given template image. The template image is, for example, an image capturing the region of interest. Determining whether the conditions are met involves, for example, comparing the value of the feature quantity with a given threshold.

[0145] Alternatively, the analysis unit 36 ​​can also perform analysis using the learned model. For example, the learning device performs processing to generate a learned model based on learning data obtained by assigning forward resolution data to the learning image. The learning image is an image obtained by photographing the inside of a lumen, specifically an in vivo image obtained by photographing the intestine. Forward resolution data is information that determines the region of interest contained in the learning image. Here, forward resolution data is information that determines the location of the detection box containing the region of interest and the type of subject contained in the detection box. The types of subjects include, for example, "normal mucosa" and "polyp". Alternatively, forward resolution data can also be information that determines the type of subject captured in each pixel of the learning image.

[0146] The model here is, for example, a widely known neural network. The learning device takes the learning image as input to the neural network, performs calculations using the weighting coefficients at each time point, and thus obtains the output. The learning device calculates an error function representing the error between the output and the correct answer data, and updates the weighting coefficients to reduce this error function. For example, the widely known error backpropagation method can be applied to update the weighting coefficients. The learning device repeatedly updates the weighting coefficients using a large amount of learning data, thereby generating a fully learned model.

[0147] The image processing apparatus 3's storage unit 34 stores the learned model. This learned model includes weighting coefficients. Furthermore, the learned model may also include an inference program for performing forward calculations based on the weighting coefficients. The analysis unit 36 ​​retrieves the learned model from the storage unit 34. The analysis unit 36 ​​inputs the captured image output from the image processing unit 32 into the learned model, thereby obtaining an analysis result. As described above, the analysis result is information determining the detection box and the type of subject contained within that detection box. Alternatively, the analysis result is information determining the type of subject captured in each pixel of the input captured image. Furthermore, the analysis result may also include information indicating the accuracy of the detection box and the type of subject.

[0148] The control unit 33 controls the torsion mechanism 18 and the forward and backward mechanism 17 to scan the inner wall of the lumen through the portion of the field of view corresponding to the camera image that the analysis unit 36 ​​can analyze.

[0149] As described above in the first embodiment, scanning the inner wall of the lumen through the field of view of the camera unit can suppress the occurrence of parts that are missed without ever entering the field of view. However, even if a part enters the field of view, if it is not captured in a state where it can be analyzed, the analysis by the analysis unit 36 ​​cannot be properly performed. Specifically, the accuracy of the detection and classification processes described above is reduced, so the area of ​​interest may not be detected, or it may be incorrectly classified. In this respect, scanning is performed not through the entire field of view but through the parts that can be analyzed, thereby increasing the likelihood of being able to perform the analysis properly.

[0150] Specifically, when the area that the analysis unit 36 ​​can analyze in the captured image is defined as the analyzable area, the control unit 33 controls the torsion mechanism 18 and the forward / backward mechanism 17 to make the analyzable area corresponding to the first captured image repeat with the analyzable area corresponding to the second captured image. As described above, the first captured image is, for example, a... Figure 9 The first image is an image obtained by capturing the area shown in C3 of (B), and the second image is an image obtained by capturing the area shown in C4. In this way, it is possible to determine from the captured images whether an appropriate scan has been performed through the analyzable portion of the field of view.

[0151] For example, if it is known before scanning that a bright and suitable image can be obtained in the central region of the captured image, while a dark and unsuitable image can be obtained in the peripheral region, the storage unit 34 stores the rotation speed vθ and the movement speed vz set to make the central regions repeat each other. Then, the control unit 33 controls the torsion mechanism 18 and the forward / backward mechanism 17 according to the information stored in the storage unit 34.

[0152] However, it is also considered that if the camera image is not acquired, it is unknown which area of ​​the camera image is an analyzable area. In this case, the analyzable area is determined based on the determination result of the analyzable determination unit 37. If, based on the determined analyzable area, the analyzable areas of multiple camera images are determined to be non-overlapping, the control unit 33 may also perform a rescan.

[0153] Furthermore, when the analysis unit 36 ​​detects a region of interest from the camera image, the control unit 33 may also perform at least one of the following processes: storing information related to the region of interest and prompting the user with information related to the region of interest.

[0154] For example, the analysis performed by the analysis unit 36 ​​can be executed in real time during observation. In this case, by processing information related to the area of ​​interest and providing it to the user, the user can be assisted in observing the area of ​​interest. For example, it can prevent situations where the area of ​​interest is captured but the user misses it. Alternatively, by storing information related to the detected area of ​​interest in real time, prompts can be provided to the user at any time after the observation period ends.

[0155] Furthermore, for example, the analysis performed by the analysis unit 36 ​​can be executed after the observation is completed. In this case, by obtaining and viewing the stored information related to the area of ​​interest, the user can determine the presence or absence and severity of the area of ​​interest at any given time.

[0156] 3.2 Can it be analyzed and determined?

[0157] Next, the determination process performed by the analyzeability determination unit 37 will be explained. The analyzeability determination unit 37 determines whether the subject captured in the camera image is in an analyzeable state based on the camera image. The information output by the analyzeability determination unit 37 will also be described as analyzeability information. Hereinafter, the part of the lumen structure that is captured in an analyzeable state will be described as the analyzeable part, and the rest will be described as the non-analyzable part.

[0158] First, the meaning of "analyzable" and "unanalyzable" is investigated. As explained earlier in the first embodiment, portions that never entered the field of view of the camera unit are unanalyzable. For example, as described above, if the first and second camera images are not identical, it is determined that there are unanalyzable portions that did not enter the field of view. Alternatively, as explained later in the third embodiment, the presence or absence of such portions is determined based on the lack of information obtained about the lumen structure.

[0159] Regarding unanalyzable portions that are within the field of view but are deemed unanalyzable, consider the following two portions: First, portions located within the camera's field of view and visible in the image, but where camera conditions are poor. Second, portions located within the camera's field of view but not visible in the image.

[0160] Poor imaging conditions can be caused by factors such as a large distance between the camera and the lesion, or by shooting the lesion from an oblique angle, resulting in low resolution. Low resolution specifically refers to a very small size of the lesion in the image. In cases of poor imaging conditions, although an image is taken, the accuracy of lesion detection and malignancy determination is low, making it impossible to perform the desired analysis. Therefore, in the method of this embodiment, in cases where poor imaging conditions exist, it is determined that lesions may be missed.

[0161] Furthermore, parts that are not visible in the camera image include, for example, parts obscured by obstructions. These obstructions can be objects outside the intestines, such as food residue, bubbles, wastewater, or clamps used for hemostasis. Parts of the intestines obscured by these obstructions are not visually identifiable in the camera image; therefore, lesions located behind these obstructions may be missed. Thus, the presence of obstructions is also considered a possibility of missing lesions. Additionally, parts within the camera's field of view but not visible in the camera image include hidden portions created by luminal structures such as folds. Hidden portions include, for example, the back side of a fold. The back side refers to the surface of the fold located on the side opposite to the camera. The back side of the fold is obscured by the camera-side surface of the fold; therefore, even if it is within the field of view, it will not be captured in the camera image.

[0162] The analysis-ability determination unit 37 determines the parts of the intestine that are within the field of view of the camera unit, can be seen in the camera image, and have good camera conditions as analyzable parts, and determines the other parts as unanalyzable parts.

[0163] As can be seen from the above explanation, the following three cases can be considered regarding the unanalyzable portion. In the method of this embodiment, the unanalyzable portion can also be classified into any one of (1) to (3) below. For ease of explanation, the unanalyzable portion classified as (1) will be described as the first unanalyzable portion. Similarly, the unanalyzable portions classified as (2) and (3) will be described as the second unanalyzable portion and the third unanalyzable portion, respectively. Furthermore, when classification is not required, (1) to (3) below will only be described as unanalyzable portions. In addition, the classification is not limited to these three cases and can be further refined.

[0164] (1) The part that is within the field of view of the camera and can be seen in the camera image, but the camera conditions are poor.

[0165] (2) The part that is within the field of view of the camera but cannot be seen in the camera image.

[0166] (3) Parts that never entered the camera's field of vision.

[0167] The analysis determination unit 37 detects regions in the captured image that correspond to unanalyzable portions as unanalyzable regions. Furthermore, when refining the unanalyzable portions, the region in the image corresponding to the first unanalyzable portion is designated as the first unanalyzable region. Similarly, the region in the image corresponding to the second unanalyzable portion is designated as the second unanalyzable region. Since the third unanalyzable portion was not captured, the third unanalyzable region does not need to be considered.

[0168] Figure 13This is a flowchart illustrating the process for determining whether analysis is permissible. After the process begins, the analysis permissibility determination unit 37 first acquires a camera image from the image processing unit 32 (S21). Next, the analysis permissibility determination unit 37 determines whether analysis is permissible based on the image quality of the camera image (S22). Here, image quality specifically refers to information such as the brightness of the camera image, the camera angle, and the degree of occlusion. In a narrow sense, the degree of occlusion refers to the presence or absence of an obstruction.

[0169] Specifically, the information representing brightness is luminance information. Luminance is a weighted sum of the three pixel values ​​(RGB), and various weights can be used. Extremely bright areas in the image, such as washed-out areas, do not contain specific information about the lumen and are not suitable for analysis. Lumen information includes various information such as the surface texture of the lumen, the vascular structure on or inside the lumen, and the hue of the mucosa. Therefore, the analysis applicability determination unit 37 determines areas in the image with a brightness level above a certain threshold as unanalyzable areas. For example, the analysis applicability determination unit 37 determines areas with a brightness level above a given first brightness threshold as unanalyzable areas.

[0170] Furthermore, extremely dark areas in the image, such as blacked-out areas, do not contain specific information about the lumen and are unsuitable for analysis. Therefore, the analysis feasibility determination unit 37 determines areas in the image with brightness below a specified threshold as unanalyzable areas. For example, the analysis feasibility determination unit 37 determines areas with brightness below a given second brightness threshold as unanalyzable areas. Here, the first brightness threshold > the second brightness threshold. In addition, other information such as luminance can also be used as information representing brightness.

[0171] Furthermore, regarding areas that are whitish or blackened, there is a high probability of lost lumen information; therefore, the analysis determination unit 37 classifies such areas as the second unanalyzable area. However, depending on the threshold setting, there may be cases where lumen information remains despite low visual discernibility. Therefore, the analysis determination unit 37 may also designate areas judged as unanalyzable based on brightness as the first unanalyzable area. Alternatively, the analysis determination unit 37 may omit the classification of unanalyzable areas.

[0172] Furthermore, the analyzability determination unit 37 detects obstructions within the lumen and determines areas on the lumen surface covered by these obstructions as unanalyzable areas. Obstructions such as residue, sewage, bubbles, blood, and hemostatic clamps exhibit a different hue from the lumen surface, such as mucosa. Therefore, the analyzability determination unit 37 performs a conversion from RGB pixel values ​​to the HSV color space based on a photographic image, and determines areas in the photographic image with hue and saturation within a given range as unanalyzable areas obscured by obstructions. Alternatively, the analyzability determination unit 37 can also perform a conversion from RGB pixel values ​​to the YCrCb color space and detect obstructions based on at least one of Cr and Cb as color difference signals. Furthermore, in cases of uneven brightness, the analyzability determination unit 37 can perform the aforementioned hue determination process after performing filtering processes such as saturation correction. Saturation correction processing, for example, is gamma correction processing for each region. Furthermore, when the color and shape of the obstruction, such as a clamp used for hemostasis, are known, the ability analysis and determination unit 37 can also perform comparison processing between the sampled image of the obstruction and the camera image, thereby performing obstruction detection processing.

[0173] Furthermore, even if an area is covered by an obstruction, if the area is small enough, the likelihood of a polyp or other area of ​​interest existing beneath the obstruction is low. Therefore, the analysis feasibility determination unit 37 can also define areas larger than a predetermined size within the area covered by the obstruction as unanalyzable areas. This size can be either the size on the image or the actual size of the lumen. The conversion from image size to actual size can be performed based on optical characteristic information of the lens, imaging element, etc., and distance information to the subject. The optical characteristic information is known in the design. The distance information can be obtained using a distance sensor or calculated from a stereoscopic image using a stereo camera. Alternatively, the distance information can be obtained using the calculation results of the lumen structure information described later in the third embodiment. As described later, in the calculation processing of the lumen structure information, the three-dimensional position of the front end 11 and the three-dimensional position of the feature points are estimated, thus allowing the determination of the distance from the front end 11 to a given pixel on the captured image based on the estimation results. Furthermore, the analysis feasibility determination unit 37 can also calculate the distance information based on the brightness of the captured image. In this case, the distance to the bright area is determined to be closer, and the distance to the dark area is determined to be farther.

[0174] In addition, the lumen information will be lost in areas where there are obstructions, so the analysis determination unit 37 determines the area as the second unanalyzable area.

[0175] Furthermore, the analysis feasibility determination unit 37 determines whether analysis is possible based on the camera angle of the subject. The camera angle here refers, for example, to the angle formed by the line connecting the front end 11 and the subject and the normal direction of the subject's surface. For example, when the front end of the insertion part is directly facing the subject, the camera angle is a small value close to 0°. On the other hand, when the optical axis is along the length of the cavity, the camera angle of the cavity wall is a value larger than 0°. When the camera angle is large, the subject is photographed from an oblique direction; therefore, the size of the subject in the image is very small, and information such as fine structural details may be lost.

[0176] The analysis-probability determination unit 37 can, for example, obtain the calculation results of the lumen structure information and calculate the camera angle of each subject in the image. In this case, the analysis-probability determination unit 37 determines the area where the camera angle is above a given angle threshold as an unanalyzable area. Alternatively, the analysis-probability determination unit 37 can also determine the camera angle based on distance information. For example, when the camera angle is large, the distance to the subject changes drastically within a small area on the image. Therefore, the analysis-probability determination unit 37 can also determine the degree of change of distance information in a given area containing the processing target pixel, and determine that the camera angle is large when the degree of change is large. The distance information can be calculated based on various information such as the brightness of the image. For example, the analysis-probability determination unit 37 can also divide the image into multiple regions and determine the camera angle based on the brightness distribution of each region.

[0177] In addition, the analysis determination unit 37 determines the area with a large camera angle as the first unanalyzable area.

[0178] The above explains brightness, degree of occlusion, and camera angle as criteria for judging image quality. Figure 13 In step S22, the analysis-feasibility determination unit 37 may use all of these criteria for determination. For example, the analysis-feasibility determination unit 37 may designate areas that are determined to be unanalyzable under at least one of the determination criteria of brightness, occlusion degree, and camera angle as unanalyzable areas. However, the analysis-feasibility determination unit 37 may also use only a portion of the determination criteria of brightness, occlusion degree, and camera angle to determine whether analysis is possible.

[0179] Next, the analysis determination unit 37 detects whether there is a hidden part, and then determines whether the analysis is possible (S23). Figure 14 This is an example of a photographic image showing the presence of wrinkles. For example... Figure 14As shown, in cases where there are hidden portions, such as folds, due to the surface structure of the intestine that are not captured by the camera, the portion SA, which is not illuminated by the lighting light, is captured. The brightness of the shadowed portion SA decreases progressively compared to other portions. Therefore, when the brightness difference between adjacent pixels or adjacent pixel regions is greater than a predetermined brightness value, the analyzeability determination unit 37 determines that a hidden portion exists. For example, the analyzeability determination unit 37 determines a given area containing the shadowed portion SA as an unanalyzable area.

[0180] More specifically, the verifiability analysis determination unit 37 obtains information representing the brightness of the captured image. The information representing brightness is, for example, the brightness mentioned above. Then, if the difference in brightness values ​​between two adjacent pixels within a specified pixel area in the image is greater than or equal to a specified value, or if there are dark striped areas, the verifiability analysis determination unit 37 determines the target area as an unanalyzable area.

[0181] Alternatively, the verifiability analysis determination unit 37 may also obtain distance information obtained using a distance sensor or the like. In this case, if the difference in distance between two adjacent pixels is greater than a predetermined value, or if there is a discontinuous portion of the distance change, the verifiability analysis determination unit 37 will determine the area of ​​the object as an unanalyzable area.

[0182] In addition, the analysis determination unit 37 determines the area that is determined to have hidden parts caused by wrinkles, etc., as the second unanalyzable area.

[0183] Next, the analysis capability determination unit 37 determines whether analysis is possible based on the region size (S24). Through the processing in steps S22 and S23, a determination result is obtained for each pixel of the camera image, indicating whether it can be analyzed or not. The analysis capability determination unit 37 sets consecutive pixels determined to be analyzeable as an analyzeable region. Similarly, the analysis capability determination unit 37 sets consecutive pixels determined not to be analyzeable as an unanalyzable region.

[0184] If the size of the analyzable region is below a given size threshold, the analysis-capability determination unit 37 changes the analyzable region to an unanalyzable region. Here, size can be, for example, the size on the image, or it can be the area. The area on the image is, for example, the total number of pixels contained in the area that is the object. Even if there is no image quality problem, if the area that is the object is extremely small on the image, the area of ​​interest is not captured in a sufficiently large size, making proper analysis difficult. Therefore, by excluding areas with an area below a certain threshold from the analyzable region, analysis-capability can be appropriately determined. Furthermore, even if the area is greater than the size threshold, if the area is extremely long in both the vertical and horizontal directions, proper analysis is difficult. Therefore, the analysis-capability determination unit 37 can also change the analyzable region to an unanalyzable region if at least one of the vertical length or horizontal length of the analyzable region is below a certain threshold. In addition, the analysis-capability determination unit 37 can also perform a process of converting the size on the image to the actual size and determine analysis-capability based on the converted size.

[0185] In addition, the analysis-ability determination unit 37 determines the region that has been changed to an unanalyzable region due to its small size as the first unanalyzable region.

[0186] Next, the analysis determination unit 37 determines whether the analysis should be performed by the user (S25). The case where the analysis is not performed by the user is, for example, the case where the endoscope system 1 includes the analysis unit 36 ​​described above and the analysis is performed by that analysis unit 36. However, the analysis unit 36 ​​may also be located outside the endoscope system 1.

[0187] In the case where the analysis unit 36 ​​is omitted and the analysis is performed by the user (S25: Yes), the analysis-possibility determination unit 37 determines whether analysis is possible based on the stability of the image (S26). Here, image stability refers to the magnitude of the motion of the subject between time-series video images. Motion includes parallel movement, rotation, vibration, etc., and is generated by the relative movement between the front end 11 and the subject. Imagine a user viewing a moving image while simultaneously determining the presence or absence of a region of interest, its severity, etc. Therefore, even if a given frame contains a region that is determined to be analyzable based on image quality and region size, if the stability of the image during the period containing that frame is low, the state of the subject in the image will change drastically, making analysis difficult for the user. Therefore, the analysis-possibility determination unit 37 determines the image stability based on the time-series image containing the video image to be processed, and if the motion is greater than a certain threshold, changes the analyzable region contained in the video image to an unanalyzable region. Furthermore, the motion determination unit 37 can also determine the motion quantities of parallel movement, rotation, and vibration individually, and can also summarize them to calculate a single motion quantity, comparing the calculated motion quantity with a threshold. Regarding methods for calculating the motion quantity, various methods such as motion vectors and optical flow are known, and these methods can be widely applied in this embodiment. The magnitude of the motion quantity can also be determined based on the actual size or the size of the appearance on the camera image.

[0188] In addition, the analysis determination unit 37 determines the region that is determined to be unanalyzable due to large movement as the first unanalyzable region.

[0189] On the other hand, even when the user does not perform analysis (S25: No), appropriate analysis can still be performed even when the image stability is low. Therefore, the analysis-possibility determination unit 37 omits the processing in step S26.

[0190] As described above, the endoscope system 1 includes an analysis capability determination unit 37, which determines whether analysis based on the captured image is possible. This allows for appropriate determination of whether the lumen was captured in a state where analysis is feasible.

[0191] As described above, the analysis feasibility determination unit 37 can also output analysis feasibility information based on the magnitude of the motion of the subject in the captured image. Thus, for example, if the user cannot observe the subject in the image due to significant motion, it can be determined that the image is unanalyzable. For instance, even if a subject is captured in high-resolution image, if the subject continues to move continuously in the moving image, it can be determined that the image is unsuitable for analysis.

[0192] Furthermore, the analysis capability determination unit 37 outputs analysis capability information based on the image quality of the camera image. In this way, if analysis is not possible due to poor image quality, it can be determined that the image may have been missed.

[0193] Furthermore, the analyzeability determination unit 37 can also, after dividing the camera image into multiple regions based on a given reference, output analyzeability information for each region based on the size of each region. The given reference here is, for example, image quality as described above. Furthermore, the multiple regions are either analyzeable or unanalyzable regions. An analyzeable region consists of consecutive pixels determined to be analyzeable. This prevents regions too small for analysis from being classified as analyzeable regions.

[0194] Furthermore, the control unit 33 can also control the torsion mechanism 18 and the forward / reverse mechanism 17 based on available analysis information. For example, in a portion or all of the first camera image, particularly... Figure 9 If an unanalyzable region is detected in the lower region of (B) C3, the control unit 33 controls the movement speed or movement amount of the advance / retreat mechanism 17 to decrease. In this case, with Figure 9 Compared to example (B), the imaging range C4 corresponding to the second imaging area is shifted upwards, thus enabling control that easily allows for the repetition of analyzable areas in the two imaging images. In other words, it is possible to comprehensively and analytically image the lumen.

[0195] Furthermore, the control unit 33 can also perform a rescan if it determines that there are parts in the scanned portion of the lumen that the analysis unit 36 ​​cannot analyze. As described above, this rescan can be performed by suppressing the withdrawal amount of the insertion part 2b to photograph the same portion, or by inserting the insertion part 2b back inward to photograph the same portion. In this way, it is possible to suppress the generation of unanalyzable portions and to prevent omissions.

[0196] During rescanning, the control unit 33 can also control the change of scanning conditions. Specifically, the control unit 33 performs rescanning using scanning conditions different from those used when capturing an image that was determined to be unanalyzable. These scanning conditions include conditions related to the movement of the front end 11 during scanning, conditions related to the light source or optical system used when capturing the image, and conditions for image processing performed on the image.

[0197] As described above, the presence of unanalyzable portions can be attributed to inappropriate image quality or motion during recording, or to the presence of hidden structures such as wrinkles in the subject area. Therefore, even if a rescan is performed without changing the scanning conditions, unanalyzable portions may still be generated during that rescan. By changing the scanning conditions, the generation of unanalyzable portions can be suppressed.

[0198] Specifically, the control unit 33 performs at least one of the following controls as a control for changing scanning conditions: controlling the supply of gas to the cavity and the degassing of gas discharged from the cavity; controlling the removal of obstructions located inside the cavity; and controlling the change of movement based on the field of view of at least one of the torsion mechanism 18 and the advance / retreat mechanism 17.

[0199] For example, by supplying or degassing air, wrinkles and folds can be eliminated. Therefore, hidden parts can be photographed by supplying or degassing air. Alternatively, by removing obstructions, the subject located behind the obstruction can be photographed. Obstructions can be removed by washing with water or by suction. Furthermore, obstructions may sometimes adhere to the front end 11 rather than the wall of the cavity, but in this case, they can also be removed by supplying water or suction.

[0200] Furthermore, the control unit 33 changes the camera angle and the distance to the lumen by altering the movement of the field of view. This allows for imaging of the subject from a near-frontal position in a focused state, while also ensuring sufficient resolution. For example, the control unit 33 can also move the insertion part 2b in a direction perpendicular to the reference axis AX1, thereby adjusting the rotation axis. Alternatively, the control unit 33 can change the degree of bending of the curved part 12, thereby controlling the rotation radius. Alternatively, the control unit 33 can acquire distance information between the front end 11 and the lumen, and dynamically change the rotation radius based on this distance information, thereby performing a scan to track the intestinal surface. The distance information can be acquired by any method, such as using a distance sensor. Furthermore, when scanning to track the intestinal surface, it is necessary to acquire distance information and control the curved part 12 based on this distance information; therefore, to improve tracking accuracy, it is preferable to reduce the rotation speed. In addition, by changing the camera angle and the distance to the lumen, the brightness of the image can also be adjusted.

[0201] In addition, in order to improve image quality, the control unit 33 can control the amount of light from the light source, control the output level of the AGC (Auto Gain Control) for the camera signal, and control parameters for image processing such as noise reduction.

[0202] Furthermore, the control unit 33 can switch control based on whether the unanalyzable portion is within a predetermined range to be captured later. Specifically, the predetermined range to be captured later is a range closer to the anus than the current position of the front end 11. For example, the control unit 33 determines whether the unanalyzable portion is closer to the front or the back relative to the current position of the front end 11, based on information indicating the control history of the retraction mechanism 17. The information indicating the control history is, for example, the time sequence output of the encoder included in the drive unit that drives the retraction mechanism 17. Unanalyzable portions that are not within the predetermined range to be captured later are portions that are still likely to be missed; therefore, they will be referred to as missed portions below.

[0203] Even if the aforementioned third unanalyzable portion exists within the predetermined range to be captured later, it is merely that it has not been captured. By continuing scanning, it is possible to capture it in a state where it can be analyzed. Therefore, there is little need to change the scanning conditions for the third unanalyzable portion; it is sufficient to attempt to capture it using a normal scan first. Furthermore, for the first and second unanalyzable portions existing within the predetermined range to be captured later, the probability that they will enter the field of view is high, even without changing the scanning conditions. However, as mentioned above, the first and second unanalyzable portions are portions that enter the field of view but are not clearly captured in the image. Therefore, the control unit 33 changes the scanning conditions based on the factors that determine them to be unanalyzable.

[0204] Furthermore, for the first to third unanalyzable portions that are not located within the predetermined area to be photographed, it may be necessary to perform control different from normal scanning, such as inserting the portion inward. Therefore, the control unit 33 controls the insertion of the insertion part 2b and performs a rescan after changing the scanning conditions. Alternatively, the control for inserting the insertion part 2b can be automatic, using the twisting mechanism 18 and the advance / retracting mechanism 17, or it can be a prompt control that requests insertion from the user.

[0205] 4. Third Implementation Method

[0206] Furthermore, the endoscope system 1 can also acquire lumen structure information representing the lumen structure. The process for acquiring lumen structure information will now be described in detail. In addition, a method for associating the lumen structure information with the analyzable information described above in the second embodiment will also be explained.

[0207] 4.1 Acquisition and processing of lumen structure information

[0208] Figure 15 This diagram illustrates the structure of the endoscope system 1 according to this embodiment. The endoscope system 1 can also be... Figure 1 The structure shown includes a lumen structure detection device 5 and a magnetic field generating device 7.

[0209] A magnetic sensor 16 is disposed at the front end 11 of the insertion part 2b. Specifically, the magnetic sensor 16 is a detection device disposed near the imaging element 15 at the front end 11 and used to detect the position and orientation of the viewpoint of the imaging element 15. The magnetic sensor 16 has two coils 16a and 16b. For example, the two central axes of the two cylindrical coils 16a and 16b are orthogonal to each other. Therefore, the magnetic sensor 16 is a 6-axis sensor that detects the position coordinates and orientation of the front end 11. Here, orientation refers to Euler angles. The signal line 2e of the magnetic sensor 16 extends from the endoscope 2 and is connected to the lumen structure detection device 5.

[0210] A magnetic field generating device 7 generates a predetermined magnetic field, and a magnetic sensor 16 detects the magnetic field generated by the magnetic field generating device 7. The magnetic field generating device 7 is connected to the lumen structure detection device 5 via signal line 7a. The detection signal of the magnetic field is provided from the endoscope 2 to the lumen structure detection device 5 via signal line 2e. Alternatively, a magnetic field generating element can be provided at the front end 11 instead of the magnetic sensor 16, or a magnetic sensor can be provided outside the patient Pa instead of the magnetic field generating device 7, thereby detecting the position and orientation of the front end 11. Here, the position and orientation of the front end 11 are detected in real time by the magnetic sensor 16; in other words, the position and orientation of the viewpoint of the image captured by the imaging element 15 are detected in real time.

[0211] Figure 16 This is a structural example of a lumen structure detection device 5. The lumen structure detection device 5 includes: a processor 51, a storage device 52, an interface 53, a position and attitude detection unit 55, and a drive circuit 56. The various parts of the lumen structure detection device 5 are interconnected via a bus 58.

[0212] The processor 51 is a control unit that has a CPU and memory and controls the processing of various parts within the lumen structure detection device 5. The memory is a storage unit that includes ROM, RAM, etc. The ROM stores various processing programs and various data that are executed by the CPU. The CPU can read and execute various programs stored in the ROM and the storage device 52.

[0213] The storage device 52 stores a lumen structure calculation program. The lumen structure calculation program is a software program that calculates lumen structure information based on the position and orientation information of the front end 11 and the camera image. The processor 51 constitutes a lumen structure calculation unit by reading and executing the lumen structure calculation program through the CPU. This lumen structure calculation unit calculates the three-dimensional structure of the lumen based on the camera image obtained by the camera element 15 and the three-dimensional configuration of the front end 11 detected by the magnetic sensor 16.

[0214] Interface 53 outputs the lumen structure information calculated by processor 51 to image processing device 3. Interface 53 is, for example, a communication interface for communicating with image processing device 3.

[0215] Furthermore, interface 53 can also function as an image acquisition unit. The image acquisition unit is a processing unit that acquires camera images obtained from the image processing apparatus 3 at regular intervals. For example, it acquires 30 camera images from the image processing apparatus 3 within 1 second at the same frame rate as the images acquired from the endoscope 2. However, while the image acquisition unit acquires 30 camera images within 1 second, it can also acquire camera images at a longer interval than the frame rate. For example, the image acquisition unit could acquire 3 or more camera images within 1 second.

[0216] The position and attitude detection unit 55 controls the drive circuit 56 of the drive magnetic field generating device 7, causing the magnetic field generating device 7 to generate a predetermined magnetic field. The position and attitude detection unit 55 detects this magnetic field through the magnetic sensor 16 and generates position coordinates (x, y, z) and orientation (vx, vy, vz) data of the imaging element 15 based on the detection signal of the detected magnetic field. Orientation refers to Euler angles. That is, the position and attitude detection unit 55 is a detection device that detects the position and attitude of the imaging element 15 based on the detection signal from the magnetic sensor 16.

[0217] Figure 17 This is a flowchart illustrating an example of the calculation process for the lumen structure. First, the doctor performs a prescribed operation on an input device (not shown) with the tip 11 of the insertion unit 2b positioned at the anus. Based on this operation, the processor 51 sets position and orientation data from the position and orientation detection unit 55 as the reference position and orientation of the tip 11 for calculating the lumen structure (S1). For example, the doctor sets the reference position and orientation of the tip 11 at the anus location in three-dimensional space as initial values ​​while the tip 11 is in contact with the anus. The lumen structure calculated through the following process is calculated based on the reference position and orientation set here.

[0218] After setting the reference position and posture, the doctor inserts the front end 11 into the innermost part of the large intestine. From the position of the front end 11 of the insertion part 2b at the innermost part of the large intestine, while supplying air to expand the large intestine, the doctor pulls the insertion part 2b towards the anus, and while stopping the withdrawal of the insertion part 2b midway, bends the curved part 12 in various directions to observe the inner wall of the large intestine. While observing the inner wall of the large intestine, the doctor calculates the luminal structure of the large intestine.

[0219] The interface 53, serving as the image acquisition unit, acquires a captured image for each predetermined period Δt from the captured images provided by the image processing device 3 every 1 / 30th of a second (S2). The period Δt is, for example, 0.5 seconds. The CPU acquires the position and attitude information of the front end 11 output by the position and attitude detection unit 55 when the captured image is acquired (S3).

[0220] Processor 51 calculates the positional information of multiple feature points in three-dimensional space from the one camera image acquired in S2 and one or more previously acquired camera images (S4). The set of calculated positional information of multiple feature points constitutes the information of the lumen structure. As described later, the positional information of each feature point can be calculated using methods such as SLAM (Simultaneous Localization and Mapping) and SfM (Structure from Motion) based on image information, or it can be calculated using the principle of triangulation. The calculation method for the position of each feature point will be explained later.

[0221] In addition, when the first camera image is obtained, since there are no previously obtained camera images, the processing of S4 is not performed until a specified number of camera images are obtained.

[0222] The processor 51 generates or updates the lumen structure information by adding the position information of multiple feature points calculated in addition (S5).

[0223] Figure 18 This is an example of lumen structure information. The lumen structure information generated in S5 is composed of a set of more than one feature point, etc., in the region observed by endoscope 2. The lumen structure information is 3D data. Figure 18 The image shows an image of the lumen structure information as viewed from a given viewpoint. For example, when displaying lumen structure information, the user can confirm the structure of the lumen when viewed from a desired direction (360 degrees) by inputting an instruction to change the viewpoint position.

[0224] In addition, Figure 18 The example shown also considers the convex and concave structure information of the lumen. However, the lumen structure information can also be further simplified. For example, the lumen structure information can also be a cylindrical model. By assuming the lumen to be cylindrical, the processing load can be reduced. For example, in an embodiment that does not use sensors such as magnetic sensor 16, as described later, the reduction in computational load achieved by setting the shape of the lumen to cylindrical is significant. Furthermore, as a simplification method, one can envision a straight lumen without buckling, a lumen with only simple buckling, or a structure model where the length, diameter, and other dimensions of each part of a standard lumen structure are different.

[0225] The interface 53 of the lumen structure detection device 5 outputs the generated lumen structure information to the image processing device 3 (S6). Furthermore, in S6, the interface 53 can also control the display of the lumen structure information on the monitor 6. Next, the processor 51 determines whether the insertion part 2b has been removed from the patient (S7). For example, if the user has removed the insertion part 2b, they can use an input device (not shown) to indicate the end of observation. The processor 51 performs the determination shown in S7 based on this user input. If it has not been removed (S7: No), the process returns to S2.

[0226] Various methods exist for calculating the positions of feature points in S4. Several methods are described below. The processor 51 can also use methods such as SLAM and SfM to calculate the positions of feature points on multiple consecutive images.

[0227] In generating information about the lumen structure, the following bundle adjustment can be applied: using nonlinear least squares, the internal parameters, external parameters, and world coordinate point sets are optimized based on the image. For example, the estimated parameters are used to perform perspective projection transformation on the world coordinate points of multiple extracted feature points, and the parameters and world coordinate point sets are obtained in a way that minimizes the reprojection error.

[0228] The external parameters related to the front end 11 are calculated by solving the 5-point and 8-point algorithms. The positions of the feature points are calculated based on the position of the front end 11 and triangulation. The error E between the coordinates of the 3D points projected onto the image plane and the feature points based on the reprojection error is expressed by the following equation (3).

[0229]

Mathematical Formula 1

[0230]

[0231] Here, L is the number of feature points on the K images, Psj is the coordinate position of the 3D point Pi estimated on the image plane by triangulation and the parameters of the front end 11, and Pi is the coordinate position of the corresponding feature point on the image. The position coordinates of the front end 11 are calculated using the LM (Levenberg-Marquardt) method in a way that minimizes the error E of equation (3) as a function.

[0232] Figure 19 This is a flowchart of a method for calculating the position of each feature point in three-dimensional space using bundle adjustment. When the processor 51 sets the position of the anus as the initial position, it sets time t to t0 and sets the count value n of the software counter to 0 (S11).

[0233] Processor 51 acquires the camera image at time t0 and information on the position and orientation of the front end 11 (S12). The camera image is acquired from image processing device 3. Information on the position and orientation of the front end 11 is acquired from position and orientation detection unit 55.

[0234] The processor 51 determines the position and orientation of the front end 11 at the initial position, i.e., the position of the anus (S13). For example, the position (x, y, z) of the anus is determined to be (0,0,0), and the orientation (vx, vy, vz) is determined to be (0,1,0). S11 and S13 correspond to Figure 17 S1.

[0235] Processor 51 acquires the camera image at time (t0+nΔt) and information on the position and orientation of front-end 11 (S14). S12 and S14 correspond to Figure 17 S2. Additionally, the position and orientation information of the front end 11 can be corrected. For example, a Kalman filter can be used to correct the path previously traversed by the front end 11, and the position of the front end 11 can be corrected based on the corrected path.

[0236] When n becomes k, the processor 51 extracts multiple feature points from each camera image, sets the position and orientation of the front end 11 at k time points, i.e. the three-dimensional configuration of the front end 11, to be known, and calculates the position of m feature points contained in the obtained camera image through the bundle adjustment method described above (S15).

[0237] Figure 20 This is a schematic diagram illustrating the relationship between feature points on multiple consecutively acquired camera images and the position and orientation of the front end 11. Figure 20 In the diagram, the white triangle Pw represents the actual position and orientation of the front end 11, and the black triangle Pb represents the estimated position and orientation of the front end 11. The diagram shows the actual movement of the front end 11 along the solid line. The estimated movement of the front end 11 along the dashed line is also shown. As time passes, the position of the front end 11 changes, and its orientation also changes.

[0238] In addition, Figure 20 In the diagram, the white quadrilateral pw represents the actual location of the feature point, while the black quadrilateral pb represents the estimated, or calculated, location of the feature point. Feature points are, for example, parts of a camera image that have distinctive shapes and colors and are easily identifiable or tracked.

[0239] To obtain the three-dimensional structure of the large intestine's lumen, the coordinates of multiple feature points on the inner wall of the large intestine are calculated. A three-dimensional model is generated by combining or connecting these coordinates. In other words, the three-dimensional structure of the lumen is determined by the calculated positions of each feature point in three-dimensional space.

[0240] exist Figure 20 In the process, the position and attitude information of the front end 11 at each time point contains information corresponding to 6 axes, so the position and attitude information of the front end 11 at k time points contains 6k pieces of information. The position of each feature point contains information corresponding to 3 axes, so the position information of m feature points contains 3m pieces of information. Therefore, when using methods such as SLAM and SfM, the number of parameters that need to be determined is (6k+3m).

[0241] In this embodiment, as described above, a magnetic sensor 16 can be provided at the anterior end portion 11 of the endoscope 2, and the lumen structure detection device 5 can include a position and attitude detection unit 55, which acquires the position and attitude information detected by the magnetic sensor 16. In this case, 6k parameters corresponding to the position and attitude of the anterior end portion 11 are known. The optimization calculation based on the processor 51 is limited to calculating 3m parameters, thus reducing the processing load of the optimization calculation. Therefore, high-speed processing can be achieved. Furthermore, by reducing the number of parameters, the accumulation of detection errors is suppressed, thus suppressing deviations in the generated three-dimensional model structure.

[0242] Furthermore, even when the tip 11 of the insertion section 2b of the endoscope 2 is pressed against the inner wall of the lumen, immersed in dirty cleaning water, or when the camera image is jittery, making it impossible to obtain a proper continuous image, information on the position and orientation of the tip 11 can still be obtained. Therefore, even in cases where continuous images cannot be obtained, the possibility of calculating 3m parameters is increased. As a result, the robustness of the calculation of the lumen structure is improved.

[0243] return Figure 19 Continuing the explanation, processor 51 appends the newly calculated feature point position information to the generated lumen structure information and updates the lumen structure information (S16). S16 corresponds to... Figure 17 S5.

[0244] Processor 51 corrects the previously calculated position information of feature points (S17). For the previously calculated position information of the 3m newly calculated feature points, the newly calculated position information is used, for example, by averaging, to correct the previously calculated position information. Alternatively, processing in S17 can be omitted, and the previously calculated position information of each feature point can be updated using the newly calculated position information of the feature points.

[0245] After S17, processor 51 increments n by 1 (S18) and determines whether a command to end the examination has been input (S19). The command to end the examination is, for example, a pre-defined command input by the doctor to the input device after the insertion part 2b has been removed from the large intestine. When this command is input (S19: Yes), the process ends.

[0246] If no command to end the check is entered (S19: No), the process transfers to S14. As a result, the processor 51 acquires the image after a period Δt from the last acquisition time of the image (S14) and executes the processing after S14.

[0247] By performing the above processing, lumen structure information is output. It is envisioned that the lumen in this embodiment is a continuous curved surface without holes or the like, except at the ends. Therefore, in the obtained lumen structure information, it is expected that the distance between a given feature point and its nearby feature points is relatively small. In cases where there are areas with thick feature points, these areas can be determined as unanalyzable portions, for example, where the feature points are below a predetermined threshold within a relatively wide range. More specifically, these are determined to be the third unanalyzable portion among the unanalyzable portions. Furthermore, in the observation of the large intestine, the insertion part 2b is first inserted to the inside, and lumen structure information is generated while it is being withdrawn. Therefore, portions closer to the anus than the currently observed area are essentially determined as the third unanalyzable portion.

[0248] Furthermore, the unanalyzable portion here corresponds to, for example, [the following]. Figure 18 UIA. Figure 18 The diagram illustrates an example where lumen structure information is divided into two parts due to the presence of unanalyzable portions. If a sensor is provided to detect the position and orientation of the front end 11 of the insertion section 2b, the positional relationship between the divided lumen structure information can be determined even when the lumen structure information is divided. That is, the overall structure of the lumen can be estimated even when the lumen structure information is divided.

[0249] Furthermore, when the position and orientation can be detected using sensors, the processing for determining the lumen structure information is not limited to bundle adjustment. For example, the processor 51 can also determine the lumen structure information using triangulation based on two images. Specifically, the processor 51 calculates the position of feature points using triangulation based on the position and orientation information of the front end 11 and the two camera images. That is, based on the position and orientation information of the camera element 15 and the pixel position information of the feature points contained in the two camera images obtained by the camera element 15, the position information of the pixel in three-dimensional space is calculated based on triangulation, and the three-dimensional structure of the lumen is determined based on the position information of the pixel in three-dimensional space.

[0250] In addition, triangulation can be performed based on two camera images obtained at two different times, or based on two camera images obtained at the same time using a stereo camera.

[0251] Furthermore, the processor 51 can also use a photometric stereo image to calculate the position of each feature point. In this case, a plurality of illumination windows are provided at the front end 11 of the insertion part 2b. By controlling the driving of a plurality of light-emitting diodes for illumination provided in the light source device 4, the plurality of illumination lights emitted from the plurality of illumination windows can be switched and selectively emitted.

[0252] The state of the shadowed areas in an image of the subject's surface changes depending on the switching of the illumination light. Therefore, the distance to the shadowed areas on the subject's surface can be calculated based on this change. That is, the three-dimensional structure of the lumen can be determined based on a photometric stereo method using an image of the shadowed areas in a photographic image obtained by illumination from multiple selectively operating illumination units.

[0253] Furthermore, the processor 51 can also use a distance sensor to calculate the lumen structure. The distance sensor, for example, is a sensor that detects distance images using TOF (Time of Flight). The distance sensor measures distance by measuring the time of flight of light. The distance sensor is provided at the front end 11 of the insertion part 2b, and detects the distance from the front end 11 to the inner wall of the lumen for each pixel. The positional information of each point on the inner wall of the large intestine, i.e., the three-dimensional structure of the lumen, can be calculated based on the distances related to each pixel detected by the distance sensor and the position and orientation of the front end 11. Alternatively, the distance sensor can also be a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) sensor or other types of sensors. Furthermore, an illumination unit emitting a predetermined pattern of light can be provided at the front end 11, and the processor 51 can measure the distance from the front end 11 to the inner wall by projecting the pattern of light.

[0254] Furthermore, in the method of this embodiment, the structure of a sensor for position and attitude detection, such as a magnetic sensor 16, is not necessary for calculating the lumen structure information. Specifically, it can be omitted. Figure 15 The magnetic sensor 16 and the magnetic field generating device 7 are shown.

[0255] In this case, the processor 51 calculates the lumen structure information using methods such as SLAM and SfM based on multiple camera images. For example, in the example above, the processor 51 performs processing to optimize (6k+3m) parameters including the position and orientation of the front end 11.

[0256] like Figure 15 , Figure 16 As shown, the endoscope system 1 of this embodiment may also include a lumen structure information acquisition unit. This lumen structure information acquisition unit specifically corresponds to the processor 51. The lumen structure information acquisition unit determines lumen structure information representing the structure of the lumen based on the captured image. Furthermore, the endoscope system 1 may also include a position and attitude detection unit 55, which acquires position and attitude information of the subject light acquisition unit 20 relative to the lumen from a position sensor located at the front end of the insertion unit 2b inserted into the lumen. The position sensor is, for example, a magnetic sensor 16. In this case, the lumen structure information acquisition unit determines lumen structure information representing the structure of the lumen based on the position and attitude information and the captured image. Additionally, in Figure 15 and Figure 16 In this paper, an example of a lumen structure detection device 5 including a position and attitude detection unit 55 and a lumen structure information acquisition unit has been described. However, the lumen structure detection device 5 can also be integrated with the image processing device 3. Furthermore, part or all of the structure of the lumen structure detection device 5 can be implemented using cloud computing.

[0257] Even without obtaining information about the lumen structure, the insertion depth can be estimated based on the control input of the advance / retract mechanism 17, and the location of the lumen that was photographed can be roughly estimated based on the control input of the torsion mechanism 18. Here, the control input is, for example, the output of an encoder. However, since the lumen structure information is obtained from the photographed image, it is possible to estimate with high accuracy which part of the lumen was photographed and how, compared to using encoder output. Specifically, by using position and orientation information, in situations such as... Figure 18 Even when the lumen structure is cut into two or more segments as shown, the positional relationship between these segments can be estimated, thus enabling a high-precision estimation of the overall lumen structure. This allows for appropriate determination of whether a scan has been performed and the setting of scan conditions for rescanning.

[0258] 4.2 Association Processing

[0259] Figure 21 This is a diagram showing the structure of the image processing apparatus 3 according to this embodiment. Figure 21 As shown, the image processing device 3 in Figure 12 Based on the structure shown, it also includes a correlation processing unit 38 and a missed detection unit 39. However, the image processing device 3 is not limited to... Figure 21 The structure allows for various transformations, such as omitting some structural elements or adding other structural elements.

[0260] The association processing unit 38 performs a process that associates the analyzeability information obtained by the analyzeability determination unit 37 with the lumen structure information. As described above, the analyzeability determination is performed using the camera image. By associating the analyzeable or non-analyzable area on the camera image with which part of the lumen structure it is located, the part that should be rescanned can be appropriately determined. In addition, the target part for rescanning can be estimated based on the control history of the torsion mechanism 18 and the advance / retreat mechanism 17, and by matching the analyzeability information with the lumen structure information, the target part can be determined with high precision. Furthermore, it is also considered that the user inserts the insertion part 2b inward during rescanning. In this case, by displaying the association result, the user can be prompted in an easily understandable way with the specific operation for capturing the non-analyzable part.

[0261] Furthermore, in the calculation and processing of lumen structure information, the position and orientation of the anterior end 11 and the three-dimensional positions of feature points in the camera image are estimated. That is, while the camera image taken using the endoscope 2 and the lumen structure information are calculated in parallel, the correspondence between feature points on the camera image and the lumen structure has been established.

[0262] Therefore, the association processing unit 38 performs association processing between the analyzable information and the lumen structure information using the calculation results of the lumen structure information. For example, the association processing unit 38 can estimate the three-dimensional positions of points other than feature points in the camera image based on the three-dimensional positions of feature points. Therefore, by using multiple points to define the analyzable region of the camera image and estimating the three-dimensional positions of these multiple points, the analyzable portion of the lumen structure corresponding to the analyzable region is determined. Here, the multiple points are, for example, three or more points set on the outline of the analyzable region.

[0263] Alternatively, the multiple points defining the analyzable region can also be feature points used in the calculation of the lumen structure information. For example, the analysis feasibility determination unit 37 can also obtain information on feature points set in the calculation and processing of the lumen structure information, and perform analysis feasibility determination based on these feature points. For example, it can be performed for each region surrounded by three or more feature points. Figure 13 The image quality-based determination in S22 allows for the direct use of information acquired during the acquisition of lumen structure information to determine the three-dimensional positions of analyzable and non-analyzable regions. Specifically, the association processing unit 38 sets multiple feature points on multiple camera images captured at two or more time intervals. Then, the association processing unit 38 determines the correspondence between the multiple feature points on the two or more time-captured camera images, thereby associating analyzable information with the structure of the lumen.

[0264] Figure 22This is a schematic diagram illustrating the process of associating analyzable information with the lumen structure. Analyzable information is information that determines at least one of the analyzable and non-analyzable regions on a camera image. Figure 22 The example shows an elliptical analyzable region A2 and an unanalyzable region A1, but each region is, for example, a polygon defined by three or more feature points. The association processing unit 38 determines the closed region surrounded by the feature points defining the analyzable region in the lumen structure information—a set of multiple feature points whose three-dimensional positions are already determined—as an analyzable portion. For example, the portion corresponding to analyzable region A2 is determined as analyzable portion A4. Then, the association processing unit 38 determines the regions in the lumen structure that were not determined as analyzable portions as unanalyzable portions.

[0265] Alternatively, the association processing unit 38 may, while determining the analyzable portion, also define a closed region in the lumen structure information that is surrounded by feature points of a defined unanalyzable region as an unanalyzable portion. For example, the portion on the lumen corresponding to the unanalyzable region A1 may be determined as an unanalyzable portion A3. In this case, a given portion of the lumen structure that is determined to be an unanalyzable portion based on the first image may sometimes be determined to be an analyzable portion based on the second image. In cases where analyzable and unanalyzable portions overlap, the overlapping portion is determined to be an analyzable portion. This is because if it is determined to be analyzable based on at least one image, analysis can be performed with sufficient accuracy using that image.

[0266] The image processing device 3 outputs the correlation results. For example, the image processing device 3 performs processing to display the lumen structure information, showing the analyzable and non-analyzable parts in different forms, on a display unit such as a monitor 6. For example, the non-analyzable parts may also be displayed in a different color than the analyzable parts, or displayed with animations such as flashing. Figure 22 A3, A5, A6, and A7 are non-analyzable portions, displayed in a different color than the analyzable portions such as A4. Additionally, objects such as arrows and text can be displayed to improve the visual legibility of the non-analyzable portions.

[0267] Furthermore, when the unanalyzable portion is further subdivided into the aforementioned first to third unanalyzable portions, the association processing unit 38 associates the unanalyzable regions on the captured image with the lumen structure information, thereby determining the unanalyzable portion. Specifically, the portion associated with the first unanalyzable region is the first unanalyzable portion, and the portion associated with the second unanalyzable region is the second unanalyzable portion. Furthermore, the third unanalyzable portion can be detected based on the absence of lumen structure information, as described above. Additionally, if the first and second unanalyzable portions overlap, the association processing unit 38 can also determine the final association result based on the size, shape, etc., of each unanalyzable portion. In this case, the image processing device 3 performs the process of displaying the analyzable portion, the first unanalyzable portion, the second unanalyzable portion, and the third unanalyzable portion in different forms on the monitor 6, etc.

[0268] pass Figure 22 The processing shown can link the lumen structure information with the analyzability information. Furthermore, the image processing device 3 can also detect the parts of the unanalyzable portion that require the insertion part 2b to be inserted into the lumen again, i.e., the overlooked portions.

[0269] Figure 23 (A) Figure 23 (B) is a diagram illustrating the positional relationship between the anterior end portion 11 of the endoscope 2 and the unanalyzable portion. Figure 23 (A) Figure 23 In (B), B1 and B3 represent unanalyzable portions, and B2 and B4 represent the field of view of the camera unit. After inserting the insertion part 2b to the innermost part, the endoscope system 1 is used to observe the intestine while pulling the insertion part 2b out proximally. The innermost part is, for example, near the cecum, and the proximally side is the anal side. Even if unanalyzable portions exist, in cases such as Figure 23 Even if the unanalyzable portion exists near the front end 11 as shown in (A), it is possible to photograph the unanalyzable portion through a relatively simple operation. This operation could include, for example, changing the orientation of the curved portion 12 or slightly pushing the insertion portion 2b.

[0270] In contrast, Figure 23 In (B), there is an unanalyzable portion in front of the buckling. The buckling is, for example, the SD intersection. In order to observe the unanalyzable portion that is further inside the buckling, it is necessary to perform an operation that goes past the buckling and the fold.

[0271] The omission determination unit 39 in this embodiment does not... Figure 23 The unanalyzable portion shown in (A) is determined to be a missed portion, while... Figure 23The unanalyzable portion shown in (B) is determined to be a missed portion. Furthermore, if an unanalyzable portion exists closer to the front side than the current position of the front end 11, it is more likely to be observable in a subsequent scan. Therefore, the missed portion determination unit 39 does not determine unanalyzable portions closer to the front side than the current position as missed portions. In this way, unanalyzable portions that are highly likely to be unobservable unless the user performs a specific operation can be determined as missed portions.

[0272] For example, in the case of an unanalyzable portion, the omission determination unit 39 compares the position of the unanalyzable portion with the current position of the anterior end portion 11 to determine whether the unanalyzable portion is located further inward than the current position of the anterior end portion 11. For example, the omission determination unit 39 determines the inward and proximal directions based on the position information obtained from the time-series calculation of the lumen structure information. This position information can be obtained by a position and attitude detection sensor such as the magnetic sensor 16, or it can be parameters optimized using SLAM or SfM. Furthermore, a sensor related to changes in position and attitude, such as a gyroscope sensor that detects acceleration, can also determine the position and attitude by appropriately and repeatedly integrating the detection results over time; therefore, it can also be used as a position and attitude detection sensor. As described above, the lumen is at its innermost point at the start of observation, and the subsequent movement direction of the anterior end portion 11 is the proximal direction. Alternatively, if a magnetic sensor 16 or similar device can be used, the inward and proximal directions can also be determined based on the position and attitude information obtained during insertion in the inward direction. The movement direction during insertion is the inward direction.

[0273] When the unanalyzable portion is further inward than the front end 11, the omission determination unit 39 determines whether the unanalyzable portion can be photographed by operating the bending unit 12. The current position and orientation of the bending unit 12 are known, for example, based on the control data of the left-right bending operation knob 14a and the up-down bending operation knob 14b. Furthermore, the maximum bending angle of the bending unit 12 is known in the design. Therefore, the association processing unit 38 can determine whether the unanalyzable portion can be photographed by operating the bending unit 12 based on this information.

[0274] The omission determination unit 39 determines that unanalyzable portions located further inward than the front end portion 11 and deemed impossible to photograph solely through the bending portion 12 are omitted portions. Furthermore, as described above, a short-distance insertion operation without crossing the bending portion is relatively easy. Therefore, the omission determination unit 39 can determine whether to designate an unanalyzable portion as an omission based on factors such as the distance between the front end portion 11 and the unanalyzable portion, and the presence or absence of the bending portion, rather than solely based on whether the unanalyzable portion is located further inward than the front end portion 11.

[0275] As described above, the endoscope system 1 may also include a correlation processing unit 38, which correlates the analyzability information with the lumen structure based on the analyzability information and the lumen structure information. The correlation processing unit 38 determines the parts of the lumen structure that are determined to be analyzable based on at least one photographic image as analyzable parts, and determines the parts of the lumen structure other than the analyzable parts as unanalyzable parts.

[0276] According to the method of this embodiment, it is possible to correlate whether the lumen was photographed in a state where desired analyses such as lesion detection and malignancy determination can be performed with the structure of the lumen. Therefore, it is possible to appropriately determine which area in the lumen structure might have been missed. As a result, the aforementioned rescanning can be performed appropriately. Alternatively, the endoscope system 1 can also prompt the user during observation with the correlation results.

[0277] Furthermore, the endoscope system 1 can also identify unanalyzable portions that require insertion of the insertion part 2b into the lumen for observation as missed portions. This allows for the identification of portions with a high probability of being missed. For example, it can determine with high precision whether a rescan is needed and whether the insertion part 2b needs to be inserted again. Furthermore, in the event of re-insertion, it can estimate the optimal way to move the insertion part 2b.

[0278] The control unit 33 can also control the twisting mechanism 18 and the forward / backward mechanism 17 to ensure that the missed portion is within the field of view of the camera unit. This allows for automatic rescanning, specifically automatic rescanning accompanying the re-insertion of the insertion unit 2b, thus reducing the burden on physicians and others. However, the control unit 33 can also prompt the user with information for capturing the missed portion and delegate the specific operation for re-insertion to the user.

[0279] Furthermore, when observing the lumen by inserting the insertion part 2b into the inward direction and then pulling it out in the forward direction, the control unit 33 can also control the torsion mechanism 18 and the advance / retraction mechanism 17, as well as control the determination of the analyzable portion, when the insertion part 2b is pulled out in the forward direction. The control of the torsion mechanism 18 and the advance / retraction mechanism 17 is, for example, through scanning the inner wall of the lumen using the field of view of the camera unit. Various scanning and related controls can be performed when it is easy to photograph the inner wall of the lumen.

[0280] Furthermore, while this embodiment has been described in detail above, those skilled in the art will readily understand that various modifications can be made without substantially departing from the present embodiment, introducing new elements and effects. Therefore, all such modifications are included within the scope of the present invention. For example, a term described at least once in the specification or drawings, along with a more general or synonymous term, can be replaced with that different term at any point in the specification or drawings. Moreover, all combinations of this embodiment and its modifications are also included within the scope of the present invention. Furthermore, the structure and operation of the processing apparatus, endoscope system, etc., are not limited to those described in this embodiment, and various modifications can be implemented.

[0281] Label Explanation

[0282] 1: Endoscopic system; 2: Endoscope; 2a: Operating section; 2b: Insertion section; 2c: Universal cable; 2e: Signal line; 3: Image processing device; 4: Light source device; 5: Lumen structure detection device; 6: Monitor; 7: Magnetic field generating device; 7a: Signal line; 8: Bed; 11: Front end; 12: Bending section; 12a: Bending section; 12b: Bending section; 12c: Bending section; 13: Flexible tube section; 14: Bending operation component; 14a: Left and right bending operation knob; 14b: Up and down bending operation knob; 14c: Fixing knob; 15: Camera element; 16: Magnetic sensor; 16a, 16b: Coil; 17: Advance and retraction mechanism; 17a: Advance and retraction Roller; 18: Torsion mechanism; 18a: Rotating roller; 18b: Rotation mechanism; 18c: Transparent part; 19: Drive unit; 20: Subject light acquisition unit; 20a: Focusing lens; 20b: Actuator; 21: Illumination lens; 22: Light guide; 31: Image acquisition unit; 32: Image processing unit; 33: Control unit; 34: Storage unit; 35: Focus control unit; 36: Analysis unit; 37: Failure analysis determination unit; 38: Correlation processing unit; 39: Missed view determination unit; 51: Processor; 52: Storage device; 53: Interface; 55: Position and attitude detection unit; 56: Drive circuit; 58: Bus; AX1: Reference axis; Pa: Patient; SA: Hidden part.

Claims

1. An endoscope system, characterized in that, The endoscope system includes: The insertion part is inserted into the lumen; A subject light acquisition unit is provided in the insertion unit to acquire light from the subject, i.e., subject light. A camera unit is provided in the insertion part, which takes pictures based on the light of the subject, thereby obtaining a camera image within the field of view; A torsion mechanism that rotates the light-acquiring part of the subject; An advance / retract mechanism that moves the insertion part in the insertion direction or the withdrawal direction; and The control unit controls the torsion mechanism and the forward / reverse mechanism, thereby controlling the movement of the field of view of the camera unit. The torsion mechanism has a mechanism that allows the front end of the insertion part to rotate about the reference axis when the axis of the insertion part is set as the reference axis. The control unit controls the torsion mechanism and the forward / reverse mechanism to scan the inner wall of the lumen through the field of view. The control unit controls the rotational speed of the torsion mechanism and the moving speed or amount of movement of the forward and backward mechanism, so that the field of view of the camera in a given frame overlaps with the field of view of the camera in the next frame in the circumferential and length directions of the lumen. Thus, the first and second consecutive images captured by the camera in a time sequence have overlapping portions. When the axis of the insertion part is set as the reference axis, the control unit performs the scanning by combining the movement of the insertion part along the reference axis using the advance and retraction mechanism and the periodic rotation of the subject light acquisition part in the circumferential direction of the cavity using the torsion mechanism.

2. The endoscope system according to claim 1, characterized in that, The image captured by the camera at a given time interval is designated as the first image. The second image is the image captured by the camera unit after the first image is captured and after the control unit performs the following control: the control is a combination of control that rotates the subject light acquisition unit approximately one revolution around the reference axis and control that moves the insertion unit in the direction along the reference axis using the advance and retraction mechanism.

3. The endoscope system according to claim 1, characterized in that, The endoscope system includes an analysis unit that analyzes the captured images. The control unit controls the torsion mechanism and the forward / backward mechanism to scan the inner wall of the lumen through the portion of the field of view of the camera unit that the analysis unit can analyze.

4. The endoscope system according to claim 2, characterized in that, The endoscope system includes an analysis unit that analyzes the captured images. When the region in the captured image that the analysis unit can analyze is designated as the analyzable region... The control unit controls the torsion mechanism and the forward / backward mechanism so that the analyzable region in the first camera image and the analyzable region in the second camera image have overlapping portions.

5. The endoscope system according to claim 3, characterized in that, When the analysis unit detects a region of interest based on the camera image... The control unit performs at least one of the following processes: storing information related to the area of ​​interest and prompting the user with the information related to the area of ​​interest.

6. The endoscope system according to claim 1, characterized in that, The endoscopic system includes an analysis-capability determination unit, which outputs analysis-capability information indicating whether the camera image is in an analyzeable state based on the camera image. The control unit controls the torsion mechanism and the forward / backward mechanism based on the feasibility analysis information.

7. The endoscope system according to claim 6, characterized in that, The feasibility analysis and determination unit outputs feasibility analysis information based on the magnitude of the motion of the subject in the camera image.

8. The endoscope system according to claim 6, characterized in that, The feasibility analysis and determination unit outputs feasibility analysis information based on the image quality of the camera image.

9. The endoscope system according to claim 1, characterized in that, By performing rotation based on the torsion mechanism and movement based on the advance / retreat mechanism, the control unit controls the subject light-acquiring unit to move in a spiral shape, or performs control to alternate between rotation based on the torsion mechanism and movement based on the advance / retreat mechanism.

10. The endoscope system according to claim 1, characterized in that, When the control unit controls the torsion mechanism and the forward / backward mechanism to scan the inner wall of the lumen through the field of view of the camera unit, it performs at least one of the following two controls as a control to maintain the state of the lumen: control of supplying gas into the lumen and degassing gas out of the lumen, and control of adjusting the position of the subject.

11. The endoscope system according to claim 1, characterized in that, If the control unit determines that there is a portion of the scanned portion in the cavity that has not entered the field of view of the camera unit, it performs a rescan and controls the change of scanning conditions.

12. The endoscope system according to claim 3, characterized in that, If the control unit determines that there is a portion in the cavity that the analysis unit cannot analyze within the portion that has been scanned, it performs a rescan and controls the change of scanning conditions.

13. The endoscopic system according to claim 11 or 12, characterized in that, The control unit executes at least one of the following controls as a control to change the scanning conditions: Control the supply of gas to the cavity and the degassing of gas discharged from the cavity; Control over removing obstructions located inside the lumen; and Control of changes to the movement based on the field of view of at least one of the torsion mechanism and the forward / backward mechanism.

14. The endoscope system according to claim 1, characterized in that, The endoscope system includes: The lumen structure information acquisition unit calculates lumen structure information representing the structure of the lumen based on the camera image; The analysis-ability determination unit outputs analysis-ability information indicating whether the camera image is in an analyzable state based on the camera image. as well as The association processing unit associates the analyzability information with the lumen structure information based on the analyzability information and the lumen structure information. The association processing unit determines the analyzable portion and the non-analyzable portion, wherein the analyzable portion is the part of the structure of the lumen that is determined to be analyzable based on at least one of the camera images, and the non-analyzable portion is the part of the structure of the lumen other than the analyzable portion.

15. The endoscope system according to claim 14, characterized in that, The endoscope system includes a position and attitude detection unit that obtains position and attitude information of the subject light acquisition unit relative to the lumen from a position sensor located at the front end of the insertion unit inserted into the lumen. The lumen structure information acquisition unit calculates the lumen structure information based on the position and orientation information and the camera image.

16. The endoscopic system according to claim 14, characterized in that, The endoscope system includes a missed view determination unit, which determines that unanalyzable portions that can only be observed by inserting the insertion part into the inner side of the lumen are missed views.

17. The endoscope system according to claim 1, characterized in that, After inserting the insertion part into the lumen, and then observing the lumen by pulling the insertion part forward, The control unit controls the torsion mechanism and the forward / backward mechanism at least when the insertion part is pulled out in the forward direction.

18. An endoscope, characterized in that, The endoscope comprises: The insertion part is inserted into the lumen; A subject light acquisition unit is provided in the insertion unit to acquire light from the subject, i.e., subject light. A camera unit is provided in the insertion part, which takes pictures based on the light of the subject, thereby obtaining a camera image within the field of view; An advance / retract mechanism that moves the insertion part in the insertion direction or the withdrawal direction; and A torsion mechanism that rotates the light-acquiring part of the subject using a mechanism that allows the front end of the insertion part to rotate about the reference axis when the axis of the insertion part is set as the reference axis. The rotational speed of the torsion mechanism and the moving speed or amount of movement of the forward / backward mechanism are controlled so that the field of view of the camera in a given frame overlaps with the field of view of the camera in the next frame in the circumferential and longitudinal directions of the lumen, thereby creating a time-series overlap between the first and second consecutive images captured by the camera. When the axis of the insertion part is set as the reference axis, the control unit performs scanning by combining the movement of the insertion part along the reference axis using the advance and retraction mechanism and the periodic rotation of the subject light acquisition part in the circumferential direction of the cavity using the torsion mechanism.

19. An endoscope, characterized in that, The endoscope comprises: The insertion part is inserted into the lumen; The camera unit captures images based on light from the subject, i.e., subject light, thereby obtaining a camera image within its field of view; Torsion mechanism; as well as The forward / reverse mechanism moves the insertion part in the insertion direction or the withdrawal direction. When the axis of the insertion part is set as the reference axis... The advance / retreat mechanism includes an advance / retreat roller that moves the insertion part in a direction corresponding to the reference axis, and a drive unit that drives the advance / retreat roller. The torsion mechanism includes a rotating roller that rotates the insertion part about the reference axis, and a drive unit that drives the rotating roller. The rotational speed of the torsion mechanism and the moving speed or amount of movement of the forward / backward mechanism are controlled so that the field of view of the camera in a given frame overlaps with the field of view of the camera in the next frame in the circumferential and longitudinal directions of the lumen, thereby creating a time-series overlap between the first and second consecutive images captured by the camera. By combining the movement of the insertion part along the reference axis using the advance and retraction mechanism and the periodic rotation of the insertion part in the circumferential direction of the cavity using the torsion mechanism, the inner wall of the cavity is scanned using the field of view of the camera unit.

20. The endoscope according to claim 19, characterized in that, The subject light acquisition part, which is provided in the insertion part and acquires the subject light, is rotated by the torsion mechanism.

21. The endoscope according to claim 19, characterized in that, A portion of the advance and retraction rollers contacts the insertion portion. The insertion part moves in the insertion direction or the withdrawal direction by rotating the advance and retraction rollers.

22. The endoscope according to claim 19, characterized in that, A portion of the rotating roller contacts the insertion portion. The insertion part rotates in the opposite direction to the rotating roller as it is rotated by the rotating roller.

Images