Processing device, endoscope system, and method of processing captured image

Abstract

The processing device (9) includes: an image acquisition unit (31) that acquires a camera image of the inside of the lumen; a lumen structure information acquisition unit (33) that acquires lumen structure information representing the structure of the lumen; an analysis-ability determination unit (32) that outputs analysis-ability information based on the camera image, indicating whether the camera image is in an analysis-ability state; and an association processing unit (34) that associates the analysis-ability information with the structure of the lumen based on the analysis-ability information and the lumen structure information.

Description

Technical Field

[0001] This invention relates to processing devices, endoscope systems, and methods for processing photographic images. Background Technology

[0002] In the past, endoscopes have been widely used in the medical and industrial fields. For example, in the medical field, doctors can perform endoscopic examinations by inserting the endoscope into the body of the patient and viewing the images displayed on a camera.

[0003] To understand unobserved areas during colonoscopy, a method for constructing a 3D model of the intestine based on dynamic images captured by the endoscope has been studied. For example, Non-Patent Literature 1 discloses a method for generating a map of the colonic surface using a cylindrical model.

[0004] Existing technical documents

[0005] Non-patent literature

[0006] Non-patent document 1: Mohammad Ali Armin et al, "Automated visibility map of the internal colon surface from colonoscopy video", International Journal of Computer Assisted Radiology and Surgery, 2016, Volume 11, Issue number 9, p.1599-1610 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] By using the method described in Non-Patent Literature 1 to construct a 3D model of the intestine as a criterion for judging the completion of observation, it can be considered that the omission of lesions can be suppressed. However, even if the 3D model of the intestine is constructed, it is difficult to perform appropriate analysis based on the image if the image quality is poor.

[0009] According to some aspects of the present invention, a processing device, an endoscope system, and a method for processing photographic images can be provided, which can output information for appropriately determining omissions in luminal structures.

[0010] Methods for solving problems

[0011] One aspect of the present invention relates to a processing apparatus, comprising: an image acquisition unit that acquires a camera image of the interior of a lumen; a lumen structure information acquisition unit that acquires lumen structure information representing the structure of the lumen; an analyzeability determination unit that, based on the camera image, outputs analyzeability information indicating whether the camera image is in an analyzeable state; and an association processing unit that, based on the analyzeability information and the lumen structure information, associates the analyzeability information with the structure of the lumen.

[0012] Other aspects of the present invention relate to an endoscope system comprising: a camera unit for capturing images of the interior of a lumen; an image acquisition unit for acquiring images captured by the camera unit; a lumen structure information acquisition unit for acquiring lumen structure information representing the structure of the lumen; an analysis availability determination unit for outputting analysis availability information indicating whether the captured images are in an analyzable state based on the captured images; and an association processing unit for associating the analysis availability information with the structure of the lumen based on the analysis availability information and the lumen structure information.

[0013] Another aspect of the present invention relates to a method for processing a camera image, comprising the following steps: acquiring a camera image showing the interior of a lumen; acquiring lumen structure information representing the structure of the lumen; outputting analyzeability information indicating whether the camera image is in an analyzeable state based on the camera image; and associating the analyzeability information with the structure of the lumen based on the analyzeability information and the lumen structure information. Attached Figure Description

[0014] Figure 1 This is a structural example of an endoscope system.

[0015] Figure 2 This is an example of the structure of an endoscope.

[0016] Figure 3 This is an example of the structure of a processing device.

[0017] Figure 4 This is a structural example of a lumen structure detection device.

[0018] Figure 5 This is a flowchart illustrating the acquisition and processing of information about the lumen structure.

[0019] Figure 6 This is an example of information about the structure of the lumen.

[0020] Figure 7 This is a flowchart illustrating the process of obtaining and processing information about the cavity structure using bundle adjustment.

[0021] Figure 8 This is a schematic diagram illustrating the relationship between multiple feature points and the position and orientation of the front end.

[0022] Figure 9 (A) and (B) are variations of sensors for detecting position and attitude.

[0023] Figure 10 This is a flowchart illustrating whether analysis, judgment, and processing are possible.

[0024] Figure 11 These are examples of photographic images where there are hidden parts caused by folds, etc.

[0025] Figure 12 This is an example of whether the correlation between analytical information and lumen structure information can be analyzed.

[0026] Figure 13 (A) and (B) are examples of the positional relationship between the front end and the unanalyzable part.

[0027] Figure 14 This is an example of displaying an image.

[0028] Figure 15 Other structural examples of processing devices. Detailed Implementation

[0029] The following describes this embodiment. Furthermore, the embodiments described below do not unduly limit the scope of the claims. Also, not all structures described in this embodiment are necessarily essential technical features of the present invention.

[0030] 1. System Structure Example

[0031] During endoscopic examinations using an endoscopic system, it is crucial to minimize the omission of areas of interest. Furthermore, an area of ​​interest is an area that, for the user, has a relatively higher priority for observation than other areas. When the user is a physician performing diagnosis or treatment, the area of ​​interest might correspond to the region where a lesion is imaged. However, if the physician wants to observe vesicles or debris, the area of ​​interest could also be the region where such a vesicle or debris is imaged. In other words, the object of the user's attention varies depending on the purpose of the observation, but during the observation, the area that has a relatively higher priority for the user than other areas becomes the area of ​​interest.

[0032] The following description uses an example where the endoscopic system is used to observe systems within a living organism, and the object of observation is the large intestine. That is, in this embodiment, the lumen is narrowly defined as the intestine. However, the method of this embodiment can also target lumens other than the intestine. For example, the digestive tract other than the large intestine can be used as the object, or the lumen structure of other parts of the organism can be used. Furthermore, the endoscopic system can also be an industrial endoscope used to observe lumen-like components. Additionally, the following description uses an example where the area of ​​interest is a lesion, but as mentioned above, the area of ​​interest can be extended beyond the lesion.

[0033] Previously, it was difficult to accurately determine the precise location and movement of the endoscope within the lumen. More specifically, it was difficult to accurately determine the relationship between the position and orientation of the insertion tip and the lumen. Furthermore, for this reason, it was also difficult to determine the specific imaging conditions and range of the lumen to be imaged. Therefore, it was difficult to accurately determine whether any parts of the lumen had been missed, and it was difficult to quantify any omissions. In contrast, as disclosed in Non-Patent Document 1, a method for estimating a 3D model of the intestine based on 2D photographic images has been presented. In the method of Non-Patent Document 1, it is possible to associate a given photographic image with which part of the intestine was captured. Furthermore, in the method of this embodiment, as long as the photographic image can be associated with the lumen structure, various modifications can be implemented as described later.

[0034] To prevent lesions from being missed, the lesion needs to be captured in at least one photographic image. That is, to prevent missed views, it is important to comprehensively photograph the entire inner surface of luminal structures such as the intestine. For example, in the method of Non-Patent Document 1, missed views can be prevented by making the completion of the construction of a 3D model the end condition for observation using the endoscopic system.

[0035] However, in examinations using endoscopic systems, it is crucial to be able to perform the desired analysis based on the captured images. For example, if the area of ​​interest is a lesion, the analysis could involve detecting the lesion from the captured image or classifying the lesion according to its malignancy. Therefore, in this embodiment, in addition to the condition of capturing the lesion in the captured image, the condition of using the lesion in an analyzable state is also used to determine any missed areas.

[0036] To reliably image lesions that may be present on the surface of the intestine, it is important to include the entire desired area of ​​the intestine within the camera's field of view without omission. The camera's field of view represents a given space determined by the direction of the camera's optical axis and the field of view angle. For example, the camera's field of view is a pyramidal or conical space with the position corresponding to the camera as its vertex, and the camera's optical axis passing through the center of the vertex and the base. By aligning the camera's optical axis towards or close to the direction in which the lesion exists, the lesion can be captured within the field of view.

[0037] However, it should be noted that even if a lesion is within the field of view of the camera, it may still be missed. There are two possible scenarios where a lesion might be missed: first, although it is within the field of view and can be seen in the image, there may be areas where the imaging conditions are poor; second, although it is within the field of view, there may be areas that are not visible in the image.

[0038] Poor imaging conditions refer to situations where the resolution is low, such as due to the distance between the camera and the lesion, or because the lesion is being photographed from an oblique angle. Low resolution specifically means that the lesion is very small in the image. Although images are taken of areas with poor imaging conditions, the accuracy of lesion detection or malignancy determination is low, making it impossible to perform the desired analysis. Therefore, in the method of this embodiment, in the presence of areas with poor imaging conditions, it is determined that lesions may be missed.

[0039] Furthermore, parts that are not visible in the camera image include, for example, parts obscured by obstructions. These obstructions can be objects outside the intestines, such as food residue, bubbles, wastewater, or clamps used for hemostasis. Parts of the intestine obscured by these obstructions are not visually identifiable in the camera image, and lesions located behind these obstructions may be missed. Therefore, the presence of obstructions is also considered a possibility of missing lesions. Additionally, parts that are within the camera's field of view but not visible in the camera image include hidden portions created by luminal structures such as folds. A hidden portion is, for example, the back side of a fold. The back side refers to the side of the fold opposite to the camera. The back side of the fold is obscured by the camera-side surface of the fold, and therefore will not be captured in the camera image even if it is within the field of view.

[0040] In this embodiment, the portion of the intestine within the field of view of the camera, visible in the image, and with good imaging conditions is defined as an analyzable portion, while other portions are defined as unanalyzable portions. In other words, in this embodiment, the criterion for determining whether a given region of the luminal structure has been missed uses not only whether the given region has been captured in an image, but also whether it is in an analyzable state. The specific processing for distinguishing between analyzable and unanalyzable portions will be explained later.

[0041] Furthermore, as can be seen from the above explanation, the unanalyzable portion can be considered as the following three types. In the method of this embodiment, the unanalyzable portion can also be classified into any one of (1) to (3) below. For example, when displaying the unanalyzable portion, the processing device 9 performs processing to display (1) to (3) below in different ways. For ease of explanation, the unanalyzable portion classified as (1) will be described as the first unanalyzable portion. Similarly, the unanalyzable portions classified as (2) and (3) will be described as the second unanalyzable portion and the third unanalyzable portion, respectively. In addition, when classification is not required, any one of (1) to (3) below will be described only as an unanalyzable portion. The specific processing for classification will be explained later. In addition, the classification is not limited to the following three types and can be further subdivided.

[0042] (1) Although it is located within the field of view of the camera and can be seen in the camera image, it is a part with poor camera conditions.

[0043] (2) The part that is located within the field of view of the camera but cannot be seen in the camera image.

[0044] (3) Parts that never enter the camera's field of vision.

[0045] The processing device 9 of this embodiment determines, for example, whether a subject can be analyzed based on an image captured by the camera unit of an endoscope system. Then, the processing device 9 associates the determination result with lumen structure information representing the lumen structure. For example, the processing device 9 performs a process of mapping regions on the image determined to be analyzable based on the image onto the intestinal structure. This allows it to determine which parts of the lumen structure are determined to be analyzable and which are determined to be unanalyzable. For example, the processing device 9 can also display the result of the association processing to indicate parts of the lumen structure that were not captured when analysis was possible. This indication can be given during observation using the endoscope system or after the observation is completed. Here, "observation" refers to the state of continuous observation of the subject using the endoscope system, specifically, the state of continuous examination and diagnosis. "After observation" refers to the completion of the examination and diagnosis.

[0046] Figure 1 This is a structural diagram of an endoscope system 1, which is an example of a system including the processing device 9 of this embodiment. The endoscope system 1 includes: an endoscope 2, an image processing device 3, a light source device 4, a lumen structure detection device 5, a monitor 6 as a display device, a magnetic field generating device 7, and the processing device 9. A physician can use the endoscope system 1 to perform an endoscopic examination of the large intestine of a patient Pa lying supine on a bed 8.

[0047] However, the structure of endoscope system 1 is not limited to Figure 1 For example, the processing device 9 and the image processing device 3 can be integrated. Additionally, as described later, the processing device 9 and the lumen structure detection device 5 can also be integrated. Furthermore, the lumen structure detection device 5 can also include a magnetic field generating device 7. Alternatively, the lumen structure detection device 5 can be structured without using the magnetic sensor 16; in this case, the magnetic field generating device 7 can be omitted. Furthermore, when using previously acquired lumen structure information, the lumen structure detection device 5 can also be omitted. Moreover, the endoscope system 1 can be simplified by omitting... Figure 1 Various deformations of the structure shown, including some structural elements, additional structural elements, etc.

[0048] In addition, Figure 1 The illustration shows an example where the image processing device 3, the lumen structure detection device 5, and the processing device 9 are located near the endoscope 2, but this is not a limitation. For example, some or all of these devices may be constructed using a server system or the like that can be connected via a network. In other words, the processing device 9 and the like can also be implemented using cloud computing. The network here may be a private network such as an intranet or a public communication network such as the Internet. Furthermore, the network may be wired or wireless.

[0049] Figure 2 This is a perspective view of endoscope 2. Endoscope 2 includes: an operating part 2a, a flexible insertion part 2b, and a universal cable 2c containing signal lines, etc. Endoscope 2 is a tubular insertion device into a body cavity through which the tubular insertion part 2b is inserted. A connector is provided at the front end of the universal cable 2c, through which endoscope 2 can be detachably connected to the light source device 4 and the image processing device 3. Here, endoscope 2 is an endoscope that can be inserted into the large intestine. Furthermore, although not shown, a light guide is inserted through the universal cable 2c, and endoscope 2 allows illumination light from the light source device 4 to be emitted from the front end of the insertion part 2b through the light guide.

[0050] like Figure 2 As shown, from the front end to the base end of the insertion part 2b, the insertion part 2b has: a front end portion 11, a bendable bending portion 12, and a flexible tube portion 13. The insertion part 2b is inserted into the lumen of the patient Pa, which is the subject. The base end portion of the front end portion 11 is connected to the front end portion of the bending portion 12, and the base end portion of the bending portion 12 is connected to the front end portion of the flexible tube portion 13. The front end portion 11 of the insertion part 2b is the front end portion of the endoscope 2, and is a relatively rigid front end portion.

[0051] The bending section 12 can be bent in a desired direction according to the operation of the bending operation component 14 provided on the operation section 2a. The bending operation component 14 includes, for example, a left-right bending operation knob 14a and an up-down bending operation knob 14b. When the bending section 12 is bent to change the position and orientation of the front end 11 and capture the observation area in the subject's body within the field of view, illumination light is shone on the observation area. The bending section 12 has multiple bending blocks connected along the length axis of the insertion section 2b. Therefore, while the doctor presses the insertion section 2b into or pulls the insertion section 2b out of the large intestine, the bending section 12 is bent in various directions, thereby enabling observation of the patient Pa's large intestine.

[0052] To bend the bending section 12, the left-right bending operation knob 14a and the up-down bending operation knob 14b pull and loosen the operating line inserted into the insertion section 2b. The bending operation component 14 also has a fixing knob 14c, which fixes the position of the bent section 12 after bending. In addition to the bending operation component 14, the operation section 2a is also equipped with various operation buttons such as a release button and an air / water supply button.

[0053] The flexible tube section 13 is flexible and bends according to external force. The flexible tube section 13 is a tubular component extending from the operating section 2a.

[0054] Furthermore, an imaging element 15, serving as an imaging device, is provided at the front end 11 of the insertion part 2b. The observation area inside the large intestine, illuminated by the illumination light from the light source device 4, is captured by the imaging element 15. That is, the imaging element 15, located at the front end 11 of the insertion part 2b, constitutes an imaging unit for capturing images of the subject. The imaging signal obtained by the imaging element 15 is supplied to the image processing device 3 via the signal line in the universal cable 2c. Furthermore, the position of the imaging element 15 is not limited to the front end 11 of the insertion part 2b. For example, the imaging element 15 can be positioned further from the front end 11 by guiding the light from the subject.

[0055] The image processing device 3 is a video processor that performs prescribed image processing on the received camera signal to generate a camera image. The image signal of the generated camera image is output from the image processing device 3 to the monitor 6, and the real-time camera image is displayed on the monitor 6. The doctor performing the examination can insert the front end 11 of the insertion part 2b into the anus of the patient Pa and observe the large intestine of the patient Pa.

[0056] A magnetic sensor 16 is disposed at the front end portion 11 of the insertion portion 2b. Specifically, the magnetic sensor 16 is disposed near the imaging element 15 at the front end portion 11 and is a detection device for detecting the position and orientation of the viewpoint of the imaging element 15. The magnetic sensor 16 has two coils 16a and 16b. For example, the two central axes of the two cylindrical coils 16a and 16b are orthogonal to each other. Therefore, the magnetic sensor 16 is a 6-axis sensor that detects the position coordinates and orientation of the front end portion 11. Here, orientation refers to Euler angles. The signal line 2e of the magnetic sensor 16 extends from the endoscope 2 and is connected to the lumen structure detection device 5.

[0057] A magnetic field generating device 7 generates a predetermined magnetic field, and a magnetic sensor 16 detects the magnetic field generated by the magnetic field generating device 7. The magnetic field generating device 7 is connected to the lumen structure detection device 5 via signal line 7a. The detection signal of the magnetic field is supplied from the endoscope 2 to the lumen structure detection device 5 via signal line 2e. Alternatively, instead of the magnetic sensor 16, the magnetic field generating element can be placed at the front end 11, and instead of the magnetic field generating device 7, the magnetic sensor can be placed outside the patient Pa, thereby detecting the position and orientation of the front end 11. Here, the position and orientation of the front end 11 are detected in real time by the magnetic sensor 16; in other words, the position and orientation of the viewpoint of the image captured by the imaging element 15 are detected in real time by the magnetic sensor 16.

[0058] The light source device 4 is a light source device capable of emitting normal light for normal light observation mode. Furthermore, when the endoscope system 1 has a special light observation mode in addition to the normal light observation mode, the light source device 4 selectively emits normal light for the normal light observation mode and special light for the special light observation mode. The light source device 4 emits either the normal light or the special light as illumination light depending on the state of the switching switch provided in the image processing device 3 for switching observation modes.

[0059] Figure 3 This is an example of the structure of the processing device 9. The processing device 9 includes: an image acquisition unit 91, a feasibility analysis determination unit 92, a lumen structure information acquisition unit 93, and a correlation processing unit 94. Additionally, the processing device 9 may also include a missed view determination unit 95 and a guidance processing unit 96. However, the processing device 9 is not limited to this. Figure 3 The structure. For example, it can be like... Figure 15 Other structures can be added as shown, or some structures can be omitted. For example, the omission determination unit 95 and the guidance processing unit 96 are not necessary structures and can be omitted.

[0060] The image acquisition unit 91 acquires camera images from the image processing device 3. The image processing device 3, for example, can supply camera images every 1 / 30th of a second. The image acquisition unit 91 outputs some or all of these images to the applicability analysis and determination unit 92, etc. Furthermore, the processing in the processing device 9 is not limited to being performed in parallel with observation. For example, the image processing device 3 performs the process of storing the camera images in a given memory. The image acquisition unit 91 can also perform the process of reading the camera images from the memory after the observation using the endoscope system 1 has ended.

[0061] The analysis feasibility determination unit 92 performs analysis feasibility information based on the camera image. The specific processing details will be explained later. The analysis feasibility information is, for example, information used to determine analyzable and non-analyzable regions, where analyzable regions are areas in the camera image determined to be analyzable, and non-analyzable regions are areas in the camera image determined to be non-analyzable. However, the analysis feasibility determination unit 92 can also determine whether analysis is possible for the entire camera image, and various variations can be implemented for the analysis feasibility information.

[0062] The lumen structure information acquisition unit 93 acquires lumen structure information representing the structure of the lumen as the object of observation. For example, such as Figure 3 As shown, the lumen structure information acquisition unit 93 acquires the lumen structure information detected by the lumen structure detection device 5, which is installed outside the processing device 9. However, the lumen structure information acquisition unit 93 can also be used as follows: Figure 15 As will be explained later, the detection and processing of lumen structure information is performed based on camera images and other methods.

[0063] The association processing unit 94 performs a process that maps the analyzable information to the lumen structure. For example, the association processing unit 94 performs a process that determines the analyzable parts and the non-analyzable parts of the lumen structure.

[0064] The omission determination unit 95 detects omissions in the unanalyzable section that have a high probability of being missed. Furthermore, the omission determination unit 95 can also provide a notification for detected omissions. Details regarding omissions will be explained later.

[0065] The guidance processing unit 96 provides guidance to the user based on whether the information can be analyzed, the correlation results between the information can be analyzed and the lumen structure information, or the detection results of any missed parts. This guidance is a prompt to encourage the user to perform a given operation. Specific guidance details will be explained later.

[0066] Furthermore, each part of the processing device 9 is composed of the following hardware. Specifically, each part of the processing device 9 is an image acquisition unit 91, a feasibility analysis determination unit 92, a lumen structure information acquisition unit 93, and a correlation processing unit 94. Additionally, each part of the processing device 9 may include a missed detection unit 95, a guidance processing unit 96, or a component that uses... Figure 15 The position and attitude information acquisition unit 97 will be described later. The hardware can include at least one of circuitry for processing digital signals and circuitry for processing analog signals. For example, the hardware can consist of one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, integrated circuits (ICs), field-programmable gate arrays (FPGAs), etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0067] Alternatively, each part of the processing device 9 can be implemented using a processor as described below. The processing device 9 includes a memory for storing information and a processor that performs operations based on the information stored in the memory. The information may include programs and various types of data. The processor includes hardware. The processor can be various types of processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and DSP (Digital Signal Processor). The memory can be semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), or it can be a register, a magnetic storage device such as HDD (Hard Disk Drive), or an optical storage device such as an optical disc drive. For example, the memory stores instructions that can be read by a computer, and by executing these instructions by the processor, the functions of each part of the processing device 9 are implemented as processing. These instructions can be instructions that constitute a program's instruction set, or instructions that instruct the processor's hardware circuitry to perform actions. Furthermore, all or part of each part of the processing device 9 can be realized through cloud computing, and the various processes described later can be performed through cloud computing.

[0068] Furthermore, each part of the processing apparatus 9 in this embodiment can also be implemented as a program module that operates on a processor. For example, the image acquisition unit 91 is implemented as an image acquisition module. The feasibility analysis determination unit 92 is implemented as a processing module that performs feasibility analysis determination processing. The lumen structure information acquisition unit 93 is implemented as a lumen structure information acquisition module and a calculation module. The correlation processing unit 94 is implemented as a processing module that performs corresponding processing on feasibility analysis information and lumen structure information.

[0069] Furthermore, the program that implements the processing performed by each part of the processing apparatus 9 in this embodiment can be stored, for example, in an information storage device that is a computer-readable medium. The information storage device can be implemented, for example, as an optical disc, memory card, HDD, or semiconductor memory. The semiconductor memory is, for example, a ROM. The processing apparatus 9 performs various processing operations according to the program stored in the information storage device. That is, the information storage device stores a program for enabling the computer to function as each part of the processing apparatus 9. A computer is a device that includes an input device, a processing unit, a storage unit, and an output unit. Specifically, the program in this embodiment is for enabling the computer to execute... Figure 10 The procedures for each step will be explained later.

[0070] As described above, the processing apparatus 9 of this embodiment includes: an image acquisition unit 91 that acquires a camera image of the interior of the cavity; a cavity structure information acquisition unit 93 that acquires cavity structure information representing the structure of the cavity; an analyzeability determination unit 92 that outputs analyzeability information based on the camera image, indicating whether the camera image is in an analyzeable state; and an association processing unit 94 that associates the analyzeability information with the cavity structure based on the analyzeability information and the cavity structure information. The camera image is acquired, for example, by the camera unit capturing images of the interior of the cavity. Specifically, the camera unit corresponds to the camera element 15. In addition, the camera image may be the output of the camera element 15 itself, or it may be the result of processing the output. For example, the camera image may also be the processed information in the image processing apparatus 3. Specifically, the analyzeability determination unit 92 determines whether the subject captured in each region of the camera image is in an analyzeable state.

[0071] According to the method of this embodiment, it is possible to correlate whether the lumen is captured in a state where the desired analysis can be performed with the structure of the lumen, such as lesion detection or malignancy determination. Therefore, it is possible to appropriately determine whether any area in the lumen structure might be missed. For example, by displaying the correlation results to the user during observation, missed views can be suppressed. Alternatively, the correlation results can also be used for user competence evaluation. Furthermore, the insertion and removal of the insertion section 2b and the control of the bending section 12 can be performed based on the correlation results.

[0072] Furthermore, the method of this embodiment can be applied to an endoscope system 1, which includes: an imaging unit that captures images of the interior of a lumen; an image acquisition unit 91 that acquires images based on the imaging unit; a lumen structure information acquisition unit 93 that acquires lumen structure information representing the structure of the lumen; an analysis-readiness determination unit 92 that outputs analysis-readiness information based on the images, indicating whether the images are in an analysis-ready state; and an association processing unit 94 that associates the analysis-readiness information with the structure of the lumen based on the analysis-readiness information and the lumen structure information.

[0073] Furthermore, the processing performed by the processing device 9 in this embodiment can also be implemented as a camera image processing method. The camera image processing method of this embodiment includes the following steps: acquiring a camera image showing the interior of a lumen; acquiring lumen structure information representing the structure of the lumen; outputting analyzeability information indicating whether the camera image is in an analyzeable state based on the camera image; and associating the analyzeability information with the structure of the lumen based on the analyzeability information and the lumen structure information.

[0074] 2. Details of the processing

[0075] Each process executed in the system including the processing apparatus 9 of this embodiment will be described in detail. Hereinafter, examples of obtaining lumen structure information based on camera images, obtaining analyzable information, and correlating lumen structure information with analyzable information during observation will be described. However, as a variation, the timing of execution of each process is not limited to this, as will be described later.

[0076] 2.1 Acquisition and processing of lumen structure information

[0077] First, the processing performed in the lumen structure detection device 5 will be explained. Additionally, if using... Figure 15 As will be explained later, the inspection and processing of the lumen structure can also be performed in the processing device 9.

[0078] 2.1.1 Example of the structure of the lumen structure detection device

[0079] Figure 4 This is an example of the structure of the lumen structure detection device 5. The lumen structure detection device 5 includes: a processor 51, a storage device 52, an interface 53, an image acquisition unit 54, a position and attitude detection unit 55, and a drive circuit 56. The various parts of the lumen structure detection device 5 are interconnected via a bus 58.

[0080] The processor 51, comprising a CPU and a memory, is the control unit that controls the processing of various parts within the cavity structure detection device 5. The memory includes storage units such as ROM and RAM. The ROM stores various processing programs and data executed by the CPU. The CPU can read and execute various programs stored in the ROM and storage device 52.

[0081] The storage device 52 stores a lumen structure calculation program. This program is a software program that calculates lumen structure information based on the position and orientation information of the front end 11 and the camera image. The CPU reads and executes the lumen structure calculation program, and the processor 51 constitutes a lumen structure calculation unit. This unit calculates the 3D structure of the lumen based on the camera image obtained by the camera element 15 and the 3D configuration of the front end 11 detected by the magnetic sensor 16.

[0082] Interface 53 outputs the lumen structure information calculated by processor 51 to processing device 9. Interface 53 is, for example, a communication interface for communicating with processing device 9.

[0083] The image acquisition unit 54 is a processing unit that acquires camera images obtained in the image processing apparatus 3 at a certain period. For example, it acquires 30 camera images from the image processing apparatus 3 within 1 second at the same frame rate as the image acquired from the endoscope 2. Furthermore, while the image acquisition unit 54 acquires 30 camera images within 1 second, it can also acquire camera images at a period longer than the frame rate. For example, the image acquisition unit 54 can also acquire 3 or more camera images within 1 second.

[0084] The position and attitude detection unit 55 controls the drive circuit 56 of the drive magnetic field generating device 7, causing the magnetic field generating device 7 to generate a predetermined magnetic field. The position and attitude detection unit 55 detects this magnetic field through the magnetic sensor 16, and generates position coordinates (x, y, z) and orientation (vx, vy, vz) data of the imaging element 15 based on the detection signal of the detected magnetic field. The orientation represents Euler angles. That is, the position and attitude detection unit 55 is a detection device that detects the position and attitude of the imaging element 15 based on the detection signal from the magnetic sensor 16.

[0085] 2.1.2 Processing flow

[0086] Figure 5This is a flowchart illustrating an example of the calculation process for the lumen structure. First, with the tip 11 of the insertion unit 2b positioned at the anus, the doctor performs a prescribed operation on an input device (not shown). Based on this operation, the processor 51 sets the position and orientation data from the position and orientation detection unit 55 as the reference position and orientation of the tip 11 for calculating the lumen structure (S1). For example, the doctor sets the reference position and orientation of the tip 11 at the anus location in 3D space as initial values ​​while the tip 11 is in contact with the anus. The lumen structure calculated in the following processes is based on the reference position and orientation set here.

[0087] After establishing the reference position and posture, the doctor inserts the anterior end 11 into the innermost part of the large intestine. Starting with the anterior end 11 of the insertion part 2b located at the innermost part of the large intestine, while supplying air to expand the large intestine, the doctor pulls the insertion part 2b to move it towards the anus, stopping midway to withdraw the insertion part 2b and bending the curved part 12 in various directions to observe the inner wall of the large intestine. While the doctor is observing the inner wall of the large intestine, the luminal structure of the large intestine is calculated.

[0088] The image acquisition unit 54 acquires images at predetermined periods Δt from the images supplied by the image processing device 3 every 1 / 30th of a second (S2). The period Δt is, for example, 0.5 seconds. The CPU obtains the position and orientation information of the front end 11 output by the position and attitude detection unit 55 when acquiring the image (S3).

[0089] Processor 51 calculates the positional information of multiple feature points in 3D space from one camera image acquired in S2 and one or more previously acquired camera images (S4). The set of calculated positional information of multiple feature points constitutes the information of the lumen structure. As described later, the positional information of each feature point can be calculated using methods such as SLAM (Simultaneous Localization and Mapping) and SfM (Structure from Motion) based on image information, or it can be calculated using the principle of triangulation. The calculation method for the position of each feature point will be explained later.

[0090] In addition, when the first camera image is obtained, since there are no previously obtained camera images, the S4 process is not performed until the specified number of camera images are obtained.

[0091] The processor 51 creates or updates the lumen structure information by adding the calculated position information of multiple feature points, etc. (S5).

[0092] Figure 6 This is an example of lumen structure information. The lumen structure information created in S5 consists of a set of one or more feature points, etc., in the region observed through endoscope 2. The lumen structure information is 3D data. Figure 6 This image represents the view of the lumen structure from a given viewpoint. For example, when displaying lumen structure information, the user can confirm the structure of the lumen when viewed from a desired 360-degree direction by inputting an instruction to change the viewpoint position.

[0093] In addition, Figure 6 The illustration also considers the convex and concave structure information of the lumen. However, the lumen structure information can also be further simplified. For example, the lumen structure information can also be a cylindrical model. By assuming the lumen to be cylindrical, the processing load can be reduced. For example, in an embodiment that does not use sensors such as magnetic sensor 16, as described later, the reduction in computational load achieved by setting the shape of the lumen to cylindrical is significant. In addition, as a simplification method, one can envision a straight lumen without bends, a lumen with only simple bends, or a structural model where the length, diameter, and other dimensions of each part of a standard lumen structure are different.

[0094] The interface 53 of the lumen structure detection device 5 outputs the generated lumen structure information to the processing device 9 (S6). Additionally, in S6, the interface 53 can also control the display of the lumen structure information on the monitor 6. Next, the processor 51 determines whether the insertion part 2b has been removed from the patient (S7). For example, if the user has removed the insertion part 2b, they can use an input device (not shown) to indicate the end of observation. The processor 51 performs the determination shown in S7 based on this user input. If it has not been removed ("No" in S7), the process returns to S2.

[0095] There are various methods for calculating the positions of feature points, etc., in S4. Several methods are explained below. The processor 51 can also use methods such as SLAM and SfM to calculate the positions of feature points on multiple consecutive images.

[0096] In generating information about the lumen structure, bundle adjustment can be applied. Bundle adjustment uses nonlinear least squares to optimize internal parameters, external parameters, and world coordinate point sets based on the image. For example, using the estimated parameters, perspective projection transformation is performed on the world coordinate points of multiple extracted feature points, and the parameters and world coordinate point sets are obtained in a way that minimizes the reprojection error.

[0097] The external parameters associated with the front end 11 are calculated using 5-point and 8-point algorithms. The positions of the feature points are calculated based on the position of the front end 11 and triangulation. The error E between the coordinates of the 3D points projected onto the image plane and the feature points based on the reprojection error is expressed by the following equation (1).

[0098] [Mathematical Expression 1]

[0099]

[0100] Here, L is the number of feature points on the K images, Psj is the coordinate position of the 3D point Pi estimated on the image plane by triangulation and the parameters of the front end 11, and Pi is the coordinate position of the corresponding feature point on the image. The position coordinates of the front end 11 are calculated using the LM (Levenberg-Marquardt) method in a way that minimizes the error E of Equation (1) as a function.

[0101] Figure 7 This is a flowchart of a method for calculating the position of each feature point in 3D space using bundle adjustment. When the position of the anus is set as the initial position, the processor 51 sets time t to t0 and sets the count value n of the software counter to 0 (S11).

[0102] Processor 51 acquires the camera image at time t0 and information on the position and orientation of the front end 11 (S12). The camera image is acquired from image processing device 3. Information on the position and orientation of the front end 11 is acquired from position and orientation detection unit 55.

[0103] The processor 51 determines the position and orientation of the front end 11 at the initial position, i.e., the position of the anus (S13). For example, the position (x, y, z) of the anus is determined to be (0, 0, 0), and the orientation (vx, vy, vz) is determined to be (0, 1, 0). S11 and S13 correspond to Figure 5 S1.

[0104] Processor 51 acquires the camera image at time (t0+nΔt) and information on the position and orientation of front-end 11 (S14). S12 and S14 correspond to Figure 5 S2. Additionally, the position and orientation information of the front end 11 can be corrected. For example, a Kalman filter can be used to correct the path previously traversed by the front end 11, and based on this corrected path, the position of the front end 11 can be corrected in the past.

[0105] When n becomes k, the processor 51 extracts multiple feature points from each camera image, sets the position and orientation of the front end 11 at k time points, i.e. the 3D configuration of the front end 11, to be known, and calculates the position of m feature points contained in the obtained camera image through the bundle adjustment method described above (S15).

[0106] Figure 8 This is a schematic diagram illustrating the relationship between feature points on multiple consecutively acquired camera images and the position and orientation of the front end 11. Figure 8 In the diagram, the white triangle Pw represents the actual position and orientation of the front end 11, and the black triangle Pb represents the estimated position and orientation of the front end 11. The diagram shows the actual movement of the front end 11 along the solid line. The estimated movement of the front end 11 along the dashed line is also shown. As time progresses, the position and orientation of the front end 11 change.

[0107] In addition, Figure 8 In the diagram, the white quadrilateral pw represents the actual location of the feature point, while the black quadrilateral pb represents the estimated or calculated location of the feature point. Feature points are, for example, parts of a camera image that have distinctive shapes and colors and are easily identifiable or tracked.

[0108] To obtain the 3D lumen structure of the large intestine, the coordinates of multiple feature points on the inner wall of the large intestine are calculated. A 3D model is then generated by combining these coordinates or by connecting them. In other words, the 3D structure of the lumen is determined by the calculated positions of each feature point in 3D space.

[0109] exist Figure 8 In the process, the position and attitude information of the front end 11 at each time point contains information corresponding to 6 axes, so the position and attitude information of the front end 11 at k time points contains 6k pieces of information. The position of each feature point contains information corresponding to 3 axes, so the position information of m feature points contains 3m pieces of information. Therefore, when using methods such as SLAM and SfM, the number of parameters that need to be determined is (6k+3m).

[0110] In this embodiment, as described above, a magnetic sensor 16 may also be provided at the anterior end portion 11 of the endoscope 2, and the lumen structure detection device 5 may include a position and attitude detection unit 55 that acquires position and attitude information detected by the magnetic sensor 16. In this case, 6k parameters corresponding to the position and attitude of the anterior end portion 11 are known. The optimization calculation performed by the processor 51 is limited to calculating 3m parameters, thus reducing the processing load of the optimization calculation. Therefore, high-speed processing can be achieved. Furthermore, since the accumulation of detection errors is suppressed by reducing the number of parameters, deviations in the generated 3D model structure can be suppressed.

[0111] Furthermore, even when the tip 11 of the insertion section 2b of the endoscope 2 is pressed against the inner wall of the lumen, immersed in dirty cleaning fluid, or when there is image jitter that prevents the acquisition of a proper continuous image, information on the position and orientation of the tip 11 can still be obtained. Therefore, even in cases where continuous images are not available, the possibility of calculating 3m parameters is increased. As a result, the robustness of the calculation of the lumen structure is improved.

[0112] return Figure 7 Continuing the explanation, processor 51 adds the newly calculated feature point position information to the already created lumen structure information, updating the lumen structure information (S16). S16 corresponds to... Figure 5 S5.

[0113] Processor 51 corrects the previously calculated position information of feature points (S17). For the previously calculated position information of the 3m newly calculated feature points, the newly calculated position information is used, for example, by averaging, to correct the previously calculated position information. Alternatively, processing in S17 can be omitted, or the position information of the newly calculated feature points can be used to update the previously calculated position information of each feature point.

[0114] After S17, processor 51 increments n by 1 (S18) and determines whether a test completion instruction has been input (S19). A test completion instruction is, for example, a pre-defined instruction input by the doctor to the input device after the insertion part 2b has been removed from the large intestine. When this instruction is input ("Yes" in S19), the process ends.

[0115] If no instruction to end the check is entered ("No" in S19), the process transfers to S14. As a result, the processor 51 obtains the image after a period Δt from the last time the image was acquired (S14) and executes the processing after S14.

[0116] By performing the above processing, lumen structure information is output. It is envisioned that the lumen in this embodiment is a continuous curved surface without holes or the like, except at the ends. Therefore, in the obtained lumen structure information, it is desirable that the distance between a given feature point and its nearby feature points is relatively small. If there are areas with thick feature points, these areas can be determined as unanalyzable portions. For example, areas with thick feature points are those where the feature points are below a predetermined threshold within a certain wide range. More specifically, these areas are determined to be the third unanalyzable portion mentioned above. Furthermore, in the observation of the large intestine, the insertion part 2b is first inserted to the inside, and lumen structure information is generated while it is being withdrawn. Therefore, portions closer to the anus than the currently observed area are essentially determined as the third unanalyzable portion.

[0117] Additionally, the unanalyzable parts here include, for example, those related to... Figure 6 The UIA correspondence. In Figure 6 The diagram illustrates an example where the lumen structure information is divided into two parts due to the presence of unanalyzable portions. If a structure is provided with a sensor for detecting the position and orientation of the front end 11 of the insertion section 2b, the positional relationship between the divided lumen structure information can be determined even when the lumen structure information is divided in this way. That is, the entire lumen structure can be estimated even when the lumen structure information is divided.

[0118] Furthermore, hidden portions, such as the back of a fold, which are visible in the field of view but not captured by the structure of the cavity (e.g., unevenness), are also determined to be unanalyzable portions because their feature points are coarse. However, since these hidden portions are within the field of view of the camera unit, they are equivalent to the second type of unanalyzable portion mentioned above. Therefore, when processing to distinguish between the second and third types of unanalyzable portions, the processing device 9 can, for example, determine whether a portion is a hidden portion based on the size of the portion with coarse feature points, the shape of its surrounding portion, etc. Alternatively, the processing device 9 can perform image processing based on the captured image to detect the presence or absence of hidden portions. (This will be used later.) Figure 11 The details of image processing are explained.

[0119] 2.1.3 Variations related to obtaining information about the lumen structure

[0120] <Variations related to sensors>

[0121] In addition, in the above description, the magnetic sensor 16 is used as a position sensor for detecting the position and orientation of the front end portion 11 of the insertion portion 2b, but the position and orientation of the front end portion 11 can also be detected by other devices.

[0122] Figure 9 Figure (A) illustrates a method for detecting the position and orientation of the anterior end portion 11 using an endoscope 2 with a shape sensor 81 and a sensor that detects the insertion amount and the amount of torsion. The shape sensor 81 is disposed inside the insertion portion 2b over the entire area from the base to the anterior end. The shape sensor 81 is, for example, an optical fiber sensor that uses an optical fiber to detect the amount of bending based on the curvature of a specific location.

[0123] The insertion / torsion sensor 82 is disposed near the anus and has a cylindrical shape with a hole through which the insertion part 2b can pass. An encoder for detecting the axial insertion amount of the insertion part 2b and an encoder for detecting the rotational amount of the insertion part 2b about its axis are disposed on the inner circumferential surface of the hole of the insertion / torsion sensor 82. Therefore, using the shape sensor 81 and the insertion / torsion sensor 82, the position and orientation of the anterior end portion 11 can be estimated based on the insertion amount and torsion of the insertion part 2b, with the position of the anus as a reference.

[0124] Alternatively, the shape sensor 81 may not utilize optical fiber, but instead have a magnetic sensor 16 provided at the front end 11, and multiple magnetic sensors 16 arranged at predetermined intervals within the insertion part 2b, thereby detecting the shape of the insertion part 2b. Figure 9 (B) is a perspective view of an endoscope equipped with multiple magnetic sensors 16 inside the insertion section 2b. Figure 9 The position information of the multiple magnetic sensors 16 shown in (B) can be used to calculate the shape of the insertion part 2b.

[0125] Alternatively, a modification can be implemented in which a magnetic sensor 16 is provided at the front end 11 and disposed within the insertion part 2b. Figure 9 The shape sensor 81 shown in (A) can thus grasp the overall shape of the insertion part 2b in addition to the position and orientation of the front end 11.

[0126] <Variations related to feature point location calculation>

[0127] Furthermore, the above explanation described a bundle adjustment method where the position and orientation of the insertion unit's front end (x, y, z, vx, vy, vz) are all known and 3m parameters are optimized. However, the processor 51 can also set a portion of (x, y, z, vx, vy, vz) as known and calculate the 3D position of each feature point through bundle adjustment optimization. In this case, compared to the case where all 6k+3m parameters are optimized, the calculation accuracy of the 3D position of each feature point is improved, and the optimization computation time is shortened.

[0128] Furthermore, when the position and orientation can be detected using sensors, the processing for determining the lumen structure information is not limited to bundle adjustment. For example, the processor 51 can also determine the lumen structure information using triangulation based on two images. Specifically, the processor 51 calculates the position of feature points using triangulation based on the position and orientation information of the front end 11 and the two camera images. That is, based on the position and orientation information of the camera element 15 and the pixel position information of the feature points contained in the two camera images obtained by the camera element 15, the position information of the pixels in 3D space is calculated based on triangulation, and the 3D structure of the lumen is determined based on the position information of the pixels in 3D space.

[0129] In addition, triangulation can be based on two camera images obtained at two different times, or it can be based on two camera images obtained at the same time using a stereo camera.

[0130] Alternatively, the processor 51 can also use a photometric stereo image to calculate the position of each feature point. In this case, a plurality of illumination windows are provided at the front end 11 of the insertion part 2b. By controlling the driving of a plurality of light-emitting diodes for illumination provided in the light source device 4, it is possible to switch and selectively emit multiple illumination lights emitted from the plurality of illumination windows.

[0131] The state of the shadowed areas in an image of the subject's surface changes due to the switching of the illumination light. Therefore, based on this change, the distance to the shadowed areas on the subject's surface can be calculated. That is, the 3D structure of the lumen can be determined based on a photometric stereo method using an image of the shadowed areas in a photographic image obtained from illumination by multiple selectively operated illumination units.

[0132] Alternatively, the processor 51 can also use a distance sensor to calculate the lumen structure. The distance sensor, for example, is a sensor that detects distance images using TOF (Time of Flight). The distance sensor measures distance by measuring the time of flight of light. The distance sensor is located at the front end 11 of the insertion part 2b, and detects the distance from the front end 11 to the inner wall of the lumen for each pixel. Based on the distances related to each pixel detected by the distance sensor and the position and orientation of the front end 11, the positional information of each point on the inner wall of the large intestine, i.e., the 3D structure of the lumen, can be calculated. Furthermore, the distance sensor can also be a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) sensor or other types of sensors. Alternatively, an illumination unit emitting a predetermined pattern of light can be provided at the front end 11, and the processor 51 can perform distance measurement from the front end 11 to the inner wall by projecting the pattern of light.

[0133] <Examples of sensors for detecting position and attitude omitted>

[0134] Furthermore, in the method of this embodiment, the structure of a sensor for position and attitude detection, such as a magnetic sensor 16, is not necessary for calculating the lumen structure information. Specifically, it can be omitted. Figure 1 The magnetic sensor 16 and magnetic field generating device 7 shown may be omitted. Figure 9 The shape sensor 81 shown in (A) etc.

[0135] In this case, the processor 51 calculates the lumen structure information based on multiple camera images using methods such as SLAM and SfM. For example, in the example above, the processor 51 performs processing to optimize (6k+3m) parameters including the position and orientation of the front end 11.

[0136] 2.2 Can it be analyzed and judged?

[0137] Next, the determination process performed by the analyzeability determination unit 92 will be explained. The analyzeability determination unit 92 determines whether the subject captured in the camera image is analyzable based on the camera image.

[0138] Figure 10 This is a flowchart illustrating the process for determining whether analysis is permissible. When this process begins, the permissibility determination unit 92 first acquires a camera image from the image acquisition unit 91 (S21). Next, the permissibility determination unit 92 determines whether analysis is permissible based on the image quality of the camera image (S22). Here, image quality specifically refers to information representing the brightness, camera angle, and degree of occlusion of the camera image. In a narrow sense, the degree of occlusion refers to the presence or absence of obstructions.

[0139] The information representing brightness is specifically luminance information. Luminance is a weighted sum of the three RGB pixel values, and various weights can be used. Extremely bright areas in the image, such as washed-out areas, do not contain specific information about the lumen and are not suitable for analysis. Lumen information includes various information such as the surface texture of the lumen, the vascular structure on or inside the lumen, and the hue of the mucosa. Therefore, the analysis applicability determination unit 92 determines areas in the image with a brightness level above a certain threshold as unanalyzable areas. For example, the analysis applicability determination unit 92 determines areas with a brightness level above a given first brightness threshold as unanalyzable areas.

[0140] Furthermore, extremely dark areas in the image, such as blackened areas, do not contain specific information about the lumen and are unsuitable for analysis. Therefore, the analysis feasibility determination unit 92 determines areas in the image with brightness below a specified threshold as unanalyzable areas. For example, the analysis feasibility determination unit 92 determines areas with brightness below a given second brightness threshold as unanalyzable areas. Here, the first brightness threshold is greater than the second brightness threshold. Alternatively, other information such as luminance can be used as information representing brightness.

[0141] Furthermore, regarding areas that appear whitish or blackish, since the possibility of information loss from the lumen is high, the analysis-determination unit 92 classifies these areas as the second unanalyzable area. The second unanalyzable area is the area on the image containing the aforementioned second unanalyzable portion. However, depending on the threshold setting, there are also cases where lumen information remains even though visual recognition is low. Therefore, the analysis-determination unit 92 may also classify areas judged as unanalyzable based on brightness as the first unanalyzable area. The first unanalyzable area is the area on the image containing the aforementioned first unanalyzable portion. Alternatively, the analysis-determination unit 92 may omit the classification of unanalyzable areas.

[0142] Furthermore, the analyzability determination unit 92 detects obstructions within the lumen and determines areas on the lumen surface covered by these obstructions as unanalyzable areas. These obstructions include residue, sewage, bubbles, blood, and hemostatic clamps. Residue includes feces, undigested food residue, etc. These obstructions have a different hue from the lumen surface, such as the mucous membrane. Therefore, the analyzability determination unit 92 performs a transformation from RGB pixel values ​​to the HSV color space based on a photographic image, determining areas in the photographic image where the hue and saturation are within a given range as unanalyzable areas obscured by obstructions. Alternatively, the analyzability determination unit 92 can also perform a transformation from RGB pixel values ​​to the YCrCb color space, detecting obstructions based on at least one of Cr and Cb as color difference signals. Furthermore, in cases of uneven brightness, the analyzability determination unit 92 can perform the aforementioned hue determination process after performing filtering processes such as saturation correction. Saturation correction processing includes, for example, gamma correction processing performed on each region. In addition, when the color and shape of the obstruction, such as a clamp used for hemostasis, are known, the detection determination unit 92 can also detect the obstruction by comparing a sample image of the obstruction with a camera image.

[0143] Furthermore, even if an area is covered by an obstruction, if the area is small enough, the likelihood of a polyp or other area of ​​interest existing beneath the obstruction is low. Therefore, the analysis feasibility determination unit 92 can also consider areas larger than a predetermined size within the area covered by the obstruction as unanalyzable areas. This size can be either the size on the image or the actual size of the lumen. The transformation from image size to actual size can be performed based on optical characteristic information of the lens, imaging element, etc., and distance information to the subject. The optical characteristic information is known in the design. The distance information can be obtained using the aforementioned distance sensor or calculated based on a stereo image obtained using a stereo camera. Alternatively, the distance information can be obtained using the calculation results of the lumen structure information. As described above, in the calculation processing of the lumen structure information, the 3D position of the front end 11 and the 3D position of the feature points are estimated, thus allowing the determination of the distance from the front end 11 to a given pixel on the captured image based on the estimation results. Additionally, the analysis feasibility determination unit 92 can also calculate the distance information based on the brightness of the captured image. In this case, the areas that are considered bright are closer, and the areas that are considered dark are farther away.

[0144] In addition, information about the lumen is lost in areas where there are obstructions, so the analysis determination unit 92 determines that the area is the second unanalyzable area.

[0145] Furthermore, the analysis feasibility determination unit 92 determines whether analysis is possible based on the camera angle of the subject. Here, the camera angle refers, for example, to the angle between the straight line connecting the front end 11 and the subject and the normal direction of the subject's surface. For example, when the front end of the insertion part is directly facing the subject, the camera angle becomes a small value close to 0°. On the other hand, when the optical axis is along the length of the cavity, the camera angle of the cavity wall becomes a value larger than 0°. With a large camera angle, the subject is photographed from an oblique direction, therefore the size of the subject in the image becomes very small, potentially resulting in the loss of information such as fine structures.

[0146] The analysis-probability determination unit 92 can, for example, obtain the calculation results of the lumen structure information and calculate the camera angle of each subject in the image. In this case, the analysis-probability determination unit 92 determines the area with a camera angle of a given angle threshold or higher as an unanalyzable area. Alternatively, the analysis-probability determination unit 92 can also determine the camera angle based on distance information. For example, when the camera angle is large, the distance to the subject changes drastically within a narrow range on the image. Therefore, the analysis-probability determination unit 92 can also determine the degree of change of distance information in a given area containing the processing target pixel, and determine that the camera angle is large when the degree of change is large. As described above, distance information can be calculated based on various information such as the brightness of the image. For example, the analysis-probability determination unit 92 can also divide the image into multiple regions and determine the camera angle based on the brightness distribution of each region.

[0147] Furthermore, the analysis determination unit 92 determines the area with a large camera angle as the first unanalyzable area.

[0148] The above explains brightness, degree of occlusion, and camera angle, which are the criteria for judging image quality. Figure 10 In step S22, the analysis-feasibility determination unit 92 may use all of these criteria for determination. For example, the analysis-feasibility determination unit 92 may designate areas that are determined to be unanalyzable under at least one of the determination criteria of brightness, occlusion degree, and camera angle as unanalyzable areas. However, the analysis-feasibility determination unit 92 may also use only some of the determination criteria of brightness, occlusion degree, and camera angle to determine whether analysis is possible.

[0149] Next, the analysis determination unit 92 determines whether the analysis is possible by detecting the presence or absence of the hidden part (S23). Figure 11 This is an example of a photographic image showing the presence of wrinkles. For example... Figure 11 As shown, in cases where there are hidden portions, such as folds, due to the surface structure of the intestine that are not captured by the camera, the portion SA, which is not illuminated and becomes a shadow, is captured. The brightness of the shadow portion SA decreases progressively compared to other portions. Therefore, when the brightness difference between adjacent pixels or adjacent pixel regions is greater than a predetermined brightness value, the analyzeability determination unit 92 determines that a hidden portion exists. For example, the analyzeability determination unit 92 determines a given area, including the shadow portion SA, as an unanalyzable area.

[0150] More specifically, the verifiability analysis determination unit 92 obtains information representing the brightness of the captured image. The information representing brightness is, for example, the brightness mentioned above. Then, if the difference in brightness values ​​between two adjacent pixels within a specified pixel area of ​​the image is greater than or equal to a specified value, or if dark striped areas exist within a specified pixel area of ​​the image, the verifiability analysis determination unit 92 determines the area of ​​the object as an unanalyzable area.

[0151] Alternatively, the verifiability analysis determination unit 92 may also use a distance sensor or the like to obtain distance information. In this case, if the difference in distance between two adjacent pixels is greater than a predetermined value, or if there are discontinuous portions of distance variation, the verifiability analysis determination unit 92 will determine the area of ​​the object as an unanalyzable area.

[0152] In addition, the analysis determination unit 92 determines the area that has been determined to have hidden parts caused by wrinkles, etc., as the second unanalyzable area.

[0153] Next, the analysis capability determination unit 92 determines whether analysis is possible based on the region size (S24). Through the processing in steps S22 and S23, an analysis capability determination result is obtained for each pixel of the camera image, either analysis capability or analysis capability. The analysis capability determination unit 92 sets consecutive pixels that have been determined to be analysis capability as an analysisable region. Similarly, the analysis capability determination unit 92 sets consecutive pixels that have been determined to be analysis capability as an analysis non-analyzable region.

[0154] If the size of the analyzable region is below a given size threshold, the analysis-capability determination unit 92 changes the analyzable region to an unanalyzable region. Here, the size can be, for example, the size on the image or the area on the image. The area on the image refers to, for example, the total number of pixels contained in the region that is the object. Even if there is no image quality problem, if the area of ​​the object is extremely small on the image, it is difficult to perform proper analysis because the area of ​​interest is not captured in sufficient size. Therefore, by excluding regions with an area below a certain threshold from the analyzable region, it is possible to appropriately determine whether analysis is possible. Furthermore, even if the area is larger than the size threshold, it is difficult to perform proper analysis if the region is extremely long in either the vertical or horizontal direction. Therefore, the analysis-capability determination unit 92 can also change the analyzable region to an unanalyzable region if at least one of the following conditions is met: the vertical length of the analyzable region is below a certain threshold, or the horizontal length is below a certain threshold. Additionally, the analysis-capability determination unit 92 can also perform a process of converting the size on the image to the actual size and determine whether analysis is possible based on the converted size.

[0155] Furthermore, the analysis-ability determination unit 92 determines the region that has been changed to an unanalyzable region due to its small size as the first unanalyzable region.

[0156] Next, the analysis determination unit 92 determines whether the analysis should be performed by the user (S25). The case where the analysis is not performed by the user is equivalent to the case where the analysis is performed by the processing device 9, etc. System-based analysis refers, for example, AI diagnosis using a learned model. Furthermore, various methods are known for generating and utilizing learned models for detection tasks (such as subject detection) and classification tasks (such as classification of severity), and these methods can be widely applied in this embodiment. Therefore, a detailed description of AI diagnosis is omitted. Additionally, the analysis system can be installed separately from the processing device 9.

[0157] When the user performs analysis ("Yes" in S25), the analysis feasibility determination unit 92 determines whether analysis is possible based on the image stability (S26). Image stability here refers to the magnitude of motion of the subject between time-series video images. Motion includes parallel movement, rotation, vibration, etc., which are generated by relative movement between the front end 11 and the subject. Imagine a user observing a moving image while determining the presence or absence of a region of interest, its severity, etc. Therefore, even if a given frame contains a region deemed analyzable based on image quality and size, if the image stability during the period containing that frame is low, the state of the subject in the image will change drastically, making analysis difficult for the user. Therefore, the analysis feasibility determination unit 92 determines the image stability based on the time-series image including the video image being processed, and if the motion exceeds a certain threshold, changes the analyzable region contained in the video image being processed to an unanalyzable region. Furthermore, the motion determination unit 92 can determine the individual motion quantities of parallel movement, rotation, and vibration, or it can combine them to calculate a single motion quantity and compare the calculated motion quantity with a threshold. The magnitude of the motion quantity can also be determined based on the actual size or the size it appears to be in the camera image. Regarding methods for calculating the motion quantity, various methods such as motion vectors and optical flow are known, and these methods can be widely applied in this embodiment.

[0158] Furthermore, the analysis determination unit 92 determines the region that is deemed unanalyzable due to its large movement as the first unanalyzable region.

[0159] On the other hand, even if the user does not perform analysis ("No" in S25), appropriate analysis can still be performed even when the image stability is low. Therefore, the analysis-possibility determination unit 92 omits the processing in step S26.

[0160] Furthermore, the analysis capability determination unit 92 can also output analysis capability information based on whether the camera image can be analyzed by a computer. Using such a determination criterion, by inputting a camera image that has been determined to be analyzeable into a computer, appropriate analysis results can be obtained. "Analysis capability by a computer" means, for example, that the analysis result obtained by the computer has a desired accuracy or higher.

[0161] More specifically, the camera image can also be an image of a living organism. The analyzeability determination unit 92 outputs analyzeability information based on whether a computer can perform classification or detection processing. Classification processing involves classifying the living organism image into any category, while detection processing involves detecting a region of interest from the living organism image. Specifically, if the computer's detection or classification accuracy is at least as expected, the analyzeability determination unit 92 determines that classification or detection processing can be performed on the living organism image. Furthermore, the category here can represent either normal or abnormal, or it can represent the type or degree of malignancy of a lesion.

[0162] The term "computer" here broadly encompasses devices that perform various processing functions, including input, storage, computation, control, and output. A computer can be a PC (Personal Computer), a server system, a smartphone, tablet, or other portable terminal device. Furthermore, the processing performed by a computer can be achieved through distributed processing across multiple devices, such as using cloud computing. A computer includes a processor and memory. It may also include user interfaces such as a keyboard, pointing device, and touch panel; output interfaces such as a monitor or speakers; and communication interfaces such as display communication chips. The computer performing analysis can be a processing device 9, an image processing device 3, or other devices. For example, Figure 1 Any of the devices shown can also function as an image diagnostic device for analysis and processing, or it can be combined with... Figure 1 The image diagnostic devices shown are arranged separately. Furthermore, the computer can also operate according to the instructions of the software that performs analysis processing. For example, the computer performing analysis processing can be implemented such that the memory stores analysis programs for performing the detection processing, analysis processing, etc., described below, and the processor performs its functions according to these analysis programs. Alternatively, analysis processing can be performed in hardware using an FPGA or similar device. As described above, the computer in this embodiment broadly includes devices that perform electronic processing, and various modifications can be made to the specific devices. Furthermore, various modifications can also be made to the hardware and software structures used to enable the computer to perform analysis.

[0163] For example, consider using machine learning methods such as deep learning to construct learned models such as detectors and classifiers. It is known that inference accuracy is high when the parameters in the image used for inference are similar to the parameters in the image used for learning, and low when the parameters deviate significantly. Specifically, the image used for inference is the camera image in this embodiment. Hereinafter, the image used for learning will be referred to as the learning image. The parameters here include, for example, brightness, occlusion level, camera angle, and region size.

[0164] For example, when learning using only training images of a certain degree of brightness, even inputting dark images may not yield sufficient inference accuracy. Given these points, it is possible to determine "whether it can be analyzed by a computer" based, for example, whether the parameters in the camera image and the training image are of the same degree. More specifically, the brightness distribution is pre-calculated for multiple training images used in the learning phase. Then, the aforementioned brightness threshold is set based on the brightness distribution. Thresholds that serve as criteria for determining occlusion level and camera angle are also set in the same manner.

[0165] Alternatively, data can be prepared that maps verification images to positive solution labels representing the presence or absence and type of regions of interest in those verification images. The verification images include camera images of various image qualities. By comparing the output of the model when the verification images are input into the model after training with the positive solution labels, the estimation accuracy for each image quality can be determined. In this way, the image quality required to obtain the desired estimation accuracy can be determined. The analysis-ability determination unit 92 outputs analysis-ability information based on whether the image quality of the camera image is at or above the determined image quality.

[0166] Furthermore, as described above, the analysis suitability determination unit 92 can also output analysis suitability information based on the magnitude of motion of the subject in the captured image. Thus, for example, if the user may not be able to observe the subject in the image due to significant motion, it can be determined that analysis is not possible. For instance, even if a subject is captured at high image quality, if the subject continues to move continuously in the moving image, it can be determined that analysis is unsuitable.

[0167] Furthermore, the analysis capability determination unit 92 outputs analysis capability information based on the image quality of the camera image. In this way, if analysis is not possible due to poor image quality, it can be determined that the image may have been missed.

[0168] Furthermore, the analyzeability determination unit 92 can also output analyzeability information for each region based on the size of each region after dividing the camera image into multiple regions. The analyzeability determination unit 92 divides the camera image into multiple regions based on image quality and hidden parts, as described above. Multiple regions refer to analyzable regions or non-analyzable regions. An analyzable region consists of consecutive pixels determined to be analyzable. This prevents regions too small for analysis from being classified as analyzable regions.

[0169] 2.3 Association Processing

[0170] Next, the correlation processing will be explained. In addition, the processing for missing information determination and the estimation processing for analyzable proportions will also be explained below.

[0171] 2.3.1 Association

[0172] As described above, camera images are used to determine whether analysis is possible. However, to prevent omissions in this embodiment, it is necessary to associate the analyzable or non-analyzable regions on the camera image with the location within the lumen structure. This is because even if the non-analyzable region is indicated on the camera image, it is difficult to know the relationship between the current position and orientation of the front end 11 and the position and orientation of the front end 11 used to capture the non-analyzable region in an analytical state, making it difficult for the user to understand the specific operation. In particular, when the non-analyzable region deviates from the camera image due to the operation of the insertion part 2b, it is difficult to grasp the position and orientation of the front end 11 and the positional relationship between the non-analyzable portion based on the camera image.

[0173] Furthermore, in the calculation and processing of lumen structure information, the position and orientation of the anterior end 11 and the 3D positions of feature points in the camera image are estimated. That is, while the camera image taking using the endoscope 2 and the calculation of lumen structure information are performed in parallel, the correspondence between feature points on the camera image and the lumen structure has been established.

[0174] Therefore, the association processing unit 94 performs association processing between the analyzable information and the lumen structure information using the calculation results of the lumen structure information. For example, the association processing unit 94 can estimate the 3D positions of points other than feature points in the camera image based on the 3D positions of feature points. Therefore, by defining the analyzable region of the camera image using multiple points and estimating the 3D positions of these multiple points, the analyzable portion of the lumen structure corresponding to the analyzable region is determined. Here, multiple points are, for example, three or more points set on the outline of the analyzable region.

[0175] Alternatively, the multiple points defining the analyzable region can also be feature points used in the calculation of the lumen structure information. For example, the analysis feasibility determination unit 92 can also obtain information on feature points set in the calculation and processing of the lumen structure information in advance, and perform analysis feasibility determination based on these feature points. For example, Figure 10 The image quality-based determination in S22 can also be performed for each region surrounded by three or more feature points. In this way, by directly utilizing the information used in obtaining the lumen structure information, the 3D positions of analyzable and non-analyzable regions can be determined.

[0176] Figure 12 This is a schematic diagram illustrating the process of associating analyzable information with the lumen structure. Analyzable information is information that determines at least one of the analyzable and non-analyzable regions on a camera image. Figure 12 The example shows an elliptical analyzable region A2 and an unanalyzable region A1, but each region is, for example, a polygon defined by three or more feature points. The association processing unit 94 determines the closed region in the lumen structure information that is surrounded by the feature points defining the analyzable region as the analyzable part, based on the set of multiple feature points whose 3D positions are determined. For example, the part corresponding to the analyzable region A2 is determined as the analyzable part A4. Then, the association processing unit 94 determines the regions in the lumen structure that were not determined as analyzable parts as unanalyzable parts.

[0177] Alternatively, the association processing unit 94 may determine, while identifying the analyzable portion, a closed region in the lumen structure information surrounded by feature points defining an unanalyzable region as an unanalyzable portion. For example, the portion on the lumen corresponding to the unanalyzable region A1 may be determined as unanalyzable portion A3. In this case, sometimes a given portion of the lumen structure that was determined to be unanalyzable based on the first image may be determined to be analyzable based on the second image. In cases where analyzable and unanalyzable portions overlap, the overlapping portion is determined to be analyzable. This is because if it is determined to be analyzable based on at least one image, analysis can be performed with sufficient accuracy using that image.

[0178] The processing device 9 outputs the correlation results. For example, the processing device 9 performs the process of displaying the lumen structure information, in which the analyzable and non-analyzable parts are displayed in different ways, on a display unit such as a monitor 6. For example, the non-analyzable parts may be displayed in a different color than the analyzable parts, or they may be displayed with animations such as flashing. Figure 12 A3, A5, A6, and A7 are non-analyzable portions, which are displayed in a different color than the analyzable portions such as A4. Additionally, displaying objects such as arrows or text can further enhance the visual recognizability of these non-analyzable portions.

[0179] Furthermore, when the unanalyzable portion is subdivided into the aforementioned first to third unanalyzable portions, the association processing unit 94 determines the unanalyzable portion by associating the unanalyzable region on the camera image with the lumen structure information. Specifically, the portion associated with the first unanalyzable region is the first unanalyzable portion, and the portion associated with the second unanalyzable region is the second unanalyzable portion. Additionally, the third unanalyzable portion can be detected based on the lack of lumen structure information, as described above. Furthermore, if the first and second unanalyzable portions overlap, the association processing unit 94 can also determine the final association result based on the size, shape, etc., of each unanalyzable portion. In this case, the processing device 9 performs the process of displaying the analyzable portion, the first unanalyzable portion, the second unanalyzable portion, and the third unanalyzable portion on the monitor 6, etc., in different ways.

[0180] As described above, the association processing unit 94 sets multiple feature points on multiple camera images captured at two or more times. Then, the association processing unit 94 determines the correspondence between the multiple feature points on the camera images captured at two or more times to determine whether information can be analyzed by associating the lumen structure. For example, the association processing unit 94 obtains information used in the calculation and processing of lumen structure information; specifically, the association processing unit 94 obtains information on the feature points set for each camera image, the correspondence between feature points in multiple camera images, etc. Thus, it is possible to associate regions on the camera images with the lumen structure.

[0181] The association processing unit 94 can also define the analyzable region (or analyzable area) on the camera image using three or more feature points, thereby determining the arrangement of the analyzable region within the lumen. Additionally, the association processing unit 94 can also define the unanalyzable region (or non-analyzable region) on the camera image using three or more feature points, thereby determining the arrangement of the non-analyzable region within the lumen. Furthermore, the association processing unit 94 can perform both of these processes. In this way, the analyzable and non-analyzable portions of the lumen structure can be determined as a set of feature points.

[0182] Furthermore, the region defined by three or more feature points can also be deformed between the shape in the camera image captured at the first moment and the shape in the camera image captured at the second moment.

[0183] The lumen can deform over time. For example, the intestine is capable of expansion and contraction, and its shape can change depending on the state of insufflation, the insertion state of the insertion part 2b, etc. Therefore, the shape of a given region may deform between the first and second time points. In this regard, by establishing a deformable correlation between the two images, even in the case of lumen deformation, it is possible to appropriately correlate analyzable information with the lumen structure. For example, an image alignment method known as "conformal registration technique" can be applied in this embodiment.

[0184] Furthermore, the association processing unit 94 can also determine the parts of the lumen structure that are determined to be analyzable based on at least one photographic image as analyzable parts, and determine the parts of the lumen structure other than the analyzable parts as unanalyzable parts. In this way, when multiple images of the lumen structure are taken with overlapping portions, it is possible to appropriately determine which part remains as an unanalyzable part.

[0185] 2.3.2 Judgment of Missed Views

[0186] pass Figure 12 The processing shown can link the lumen structure information with the analyzability information. Furthermore, the processing device 9 can also detect overlooked portions, which are unanalyzable portions that require the insertion part 2b to be inserted again into the lumen.

[0187] Figure 13 (A) Figure 13 (B) is a diagram illustrating the positional relationship between the anterior end portion 11 of the endoscope 2 and the unanalyzable portion. Figure 13 (A) Figure 13 In (B), B1 and B3 represent unanalyzable portions, and B2 and B4 represent the field of view of the imaging unit. Observation of the intestine using the endoscope system 1 is performed by inserting the insertion part 2b to its innermost position, while simultaneously withdrawing the insertion part 2b proximal to the anterior side. The innermost position is, for example, near the cecum, and the proximal side is the anal side. Even if unanalyzable portions exist, in cases such as... Figure 13 Even if the unanalyzable portion exists near the front end 11 as shown in (A), it is possible to photograph the unanalyzable portion through a relatively simple operation. This operation includes, for example, changing the orientation of the curved portion 12 or slightly pressing the insertion portion 2b.

[0188] In contrast, Figure 13In (B), there is an unanalyzable portion in front of the bend. The bend is, for example, an SD junction. To observe the unanalyzable portion further inward than the bend, an operation must be performed that goes beyond the bend or fold.

[0189] In this embodiment, the omission detection unit 95 does not... Figure 13 The unanalyzable portion shown in (A) is determined to be a missed portion, while... Figure 13 The unanalyzable portion shown in (B) is determined to be a missed portion. Furthermore, if an unanalyzable portion exists at a position closer to the front than the current position of the front end 11, it is highly likely to be observable in subsequent scans. Therefore, the missed portion determination unit 95 does not determine unanalyzable portions closer to the front than the current position as missed portions. In this way, unanalyzable portions that are highly likely to be unobservable unless the user performs a specific operation can be determined as missed portions.

[0190] For example, in the presence of an unanalyzable portion, the omission determination unit 95 determines whether the unanalyzable portion is located further inward than the current position of the anterior end 11 by comparing the position of the unanalyzable portion with the current position of the anterior end 11. For example, the omission determination unit 95 determines the inward and proximal directions based on the position information obtained from the time-series calculation of the lumen structure information. This position information can be obtained by a position and attitude detection sensor such as the magnetic sensor 16, or it can be parameters optimized using SLAM or SfM. Furthermore, sensors related to changes in position and attitude, such as gyroscope sensors that detect acceleration, can also be used as position and attitude detection sensors because the position and attitude can be determined by appropriately and repeatedly integrating the detection results over time. As described above, the observation begins at the innermost part of the lumen, and the subsequent movement direction of the anterior end 11 is the proximal direction. Alternatively, if the magnetic sensor 16 or the like can be used, the inward and proximal directions can be determined based on the position and attitude information obtained during insertion in the inward direction. The movement direction during insertion is the inward direction.

[0191] If the unanalyzable portion is located further inward than the front end 11, the omission determination unit 95 determines whether the unanalyzable portion can be photographed by operating the bending unit 12. The current position and orientation of the bending unit 12 are known, for example, based on control data from the left / right bending operation knob 14a and the up / down bending operation knob 14b. Furthermore, the maximum bending angle of the bending unit 12 is known by design. Therefore, the association processing unit 94 can determine, based on this information, whether the unanalyzable portion can be photographed by operating the bending unit 12.

[0192] The omission determination unit 95 determines that unanalyzable portions located further inward than the front end portion 11, and deemed impossible to capture by the bending portion 12 alone, are omissions. Furthermore, as mentioned above, a short-distance pressing operation without crossing the bend is relatively easy. Therefore, the omission determination unit 95 can determine whether to classify an unanalyzable portion as an omission based on factors such as the distance between the front end portion 11 and the unanalyzable portion, and the presence or absence of the bend, rather than solely based on whether the unanalyzable portion is located further inward than the front end portion 11.

[0193] As described above, the lumen is the intestine. After the endoscope insertion part 2b is inserted into the intestinal tract, it is pulled out proximally while taking images of the intestine for diagnostic purposes. Typically, the endoscope insertion part 2b is inserted from the anus into the cecum, or from the anus into the deepest part that can be inserted in front of the cecum. The processing device 9 includes a missed detection unit 95. Based on the position and orientation information of the tip 11 of the insertion part 2b relative to the intestine, the missed detection unit 95 determines the following unanalyzable portions as missed portions: portions that have been determined to be unobservable without further inward insertion of the insertion part 2b. For example, such as... Figure 3 As shown, the omission determination unit 95 obtains the correlation result between the analysis availability information and the lumen structure information from the correlation processing unit 94, and obtains the position and orientation information of the front end 11 from the lumen structure detection device 5. Alternatively, it can also use... Figure 15 As will be explained later, the processing device 9 includes a position and attitude information acquisition unit 97, and a miss determination unit 95 acquires the position and attitude information of the front end 11 from the position and attitude information acquisition unit 97.

[0194] This allows us to identify sections that are highly likely to be missed. For example, by alerting users to these missed sections, we can appropriately reduce the likelihood of them being missed.

[0195] Furthermore, the processing is the same when the unanalyzable parts are classified into unanalyzable parts 1 to 3, etc. That is, the parts among the unanalyzable parts 1 to 3 that cannot be easily observed by operation are all determined to be missed parts. In this case, the missed part determination unit 95 can also classify the missed parts by designating the missed parts caused by the first unanalyzable part as the first missed part, the missed parts caused by the second unanalyzable part as the second missed part, and the missed parts caused by the third unanalyzable part as the third missed part. Alternatively, the missed part determination unit 95 can omit the classification of missed parts.

[0196] As described above, the processing device 9 of this embodiment can also associate the results of the analyzability information and the omission determination with the lumen structure information. Specifically, the processing device 9 performs processing to determine which part of the lumen structure is an analyzable part, an unanalyzable part, or an omission part. In this way, information obtained from multiple perspectives, such as whether an image was taken, whether it is analyzable, and whether the possibility of omission is high, can be associated with the lumen structure.

[0197] 2.3.3 Analyzable proportions

[0198] Additionally, the processing device 9 can also perform processing to determine the analyzable ratio, which represents the proportion of the analyzable portion relative to the overall lumen structure. The analyzable ratio indicates the extent to which the lumen structure was imaged under analytical conditions, and therefore can be used as an indicator to quantify missed views. By displaying the analyzable ratio during observation using the endoscope 2, the user can be informed of the progress of the observation.

[0199] For example, the lumen structure information is a set of triangles defined by three feature points, i.e., a polygon model. The surface area of ​​the lumen structure is the sum of the areas of the multiple triangles contained in the polygon model. Alternatively, the analyzable portion of the lumen structure can be defined as a polygon surrounded by three or more feature points, as described above, and the area of ​​the polygon can be calculated based on the area of ​​one or more triangles. Furthermore, the lumen structure information is not limited to a polygon model; those skilled in the art will readily understand that the surface area can be calculated based on a 3D model.

[0200] The correlation processing unit 94 calculates the analyzable ratio based on the following formula (2). The analyzable ratio is, for example, a number between 0 and 1. Alternatively, the analyzable ratio can be expressed as a percentage by multiplying the right side of the following formula (2) by 100. In this way, the progress of the inspection being performed can be appropriately determined. Ideally, the inspection can be appropriately terminated by setting the analyzable ratio to 100%. The surface area of ​​the entire lumen is specifically the sum of the area of ​​the analyzable portion and the area of ​​the non-analyzable portion. In addition, the correlation processing unit 94 can also calculate the omission ratio using the following formula (3). According to the following formulas (2) and (3), the difference between the analyzable ratio and the omission ratio lies in whether the analyzable portion or the non-analyzable portion is considered, and either one can be used.

[0201] Analyzable proportion = (area of ​​analyzable portion) / (surface area of ​​the entire lumen)...(2)

[0202] Omission rate = (area of ​​unanalyzable portion) / (surface area of ​​the entire lumen)...(3)

[0203] Furthermore, in the presence of a third unanalyzable portion, no lumen structure information is constructed for that portion. Therefore, it is sometimes difficult to estimate the area of ​​this third unanalyzable portion. This is particularly true in examples where the observation suggests an analyzable proportion, assuming that lumen structure information closer to the anterior side than the anterior end 11 has not been obtained.

[0204] In this case, the correlation processing unit 94 can also determine the analyzable proportions by estimating the lumen structure. For example, the correlation processing unit 94 can obtain a standard intestinal model with average size and shape in advance, use the standard intestinal model to supplement the unanalyzable parts, and thereby estimate the overall structure of the lumen. The standard intestinal model can be used for all patients, or multiple models can be prepared according to age and gender. In addition, if information related to the patient's inherent intestine has been obtained in advance using CT (computed tomography) or MRI (magnetic resonance imaging), this information can also be used. Furthermore, the standard intestinal model can also be used to supplement the second unanalyzable parts, such as hidden parts formed by folds.

[0205] For example, the standard intestinal model here is set to the same state as when observed using endoscope 2. Imagine that the intestine is expanded by inflating it during observation. When using a model without inflating it as the standard intestinal model, the degree of expansion and contraction of the intestine will change significantly between the constructed lumen structure information and the standard intestinal model, making it difficult to accurately determine the analyzable proportions. Therefore, the association processing unit 94 uses information representing the intestine under inflated conditions as the standard intestinal model.

[0206] However, by using a standard intestinal model to supplement the unanalyzable portions, the luminal structure can be simplified. For example, a cylindrical model omitting structures such as folds can be used as the standard intestinal model. That is, the analyzable proportions only need to have appropriate accuracy as quantitative indicators of what was missed, simplifying subtle individual differences, subtle fold structures, etc.

[0207] Furthermore, the simplification is not limited to the standard intestinal model. For example, if a detailed 3D model is obtained as information on the lumen structure, the analyzable scale can be determined by simplifying the lumen structure information to a cylindrical model, etc.

[0208] Furthermore, the above examples illustrate the calculation of the analyzable ratio and the omission ratio based on the entire lumen, but the method of this embodiment is not limited to this. For example, when the analyzable ratio is obtained during observation, the portion of the lumen where the observation is highly likely to have been completed can also be considered as the object, and the analyzable ratio can be calculated as shown in Equation (4). In this case, the ratio of the area of ​​the analyzable portion to the surface area of ​​the lumen in the portion further inward than the front end 11 of the insertion part 2b is called the analyzable ratio. The omission ratio is also calculated as shown in Equation (5). In this case, it is expected that the analyzable ratio is 1 or the omission ratio is 0 when there is no omission, so it is possible to evaluate whether the previous examination was performed appropriately. In addition, considering the case of re-insertion, the analyzable ratio can also be determined based on the position of the front end 11 when it is moved to the closest side to the anus during a series of observations. In addition, in this case, a standard intestinal model can also be used.

[0209] Analyzable proportion = (area of ​​analyzable portion) / (surface area of ​​the lumen portion with a high probability of completion)...(4)

[0210] Omission rate = (area of ​​unanalyzable parts) / (surface area of ​​the cavity section with a high probability of completion)...(5)

[0211] As described above, the correlation processing unit 94 can also estimate at least one of the area of ​​the analyzable portion and the area of ​​the non-analyzable portion. This allows for the output of information suitable for prompting the user, or information suitable for evaluating the user's abilities. Furthermore, the correlation processing unit 94 can also estimate at least one of the shape of the analyzable portion and the shape of the non-analyzable portion. The shapes of the analyzable and non-analyzable portions change depending on the user's operation of the front end 11. Therefore, the shapes of the analyzable and non-analyzable portions can also be used as information for evaluating the progress of observation and the user's abilities. Additionally, the shape estimation here includes estimation using a standard intestinal model, as described above, and simplified processing of the analyzable portion.

[0212] Furthermore, as described above, the correlation processing unit 94 determines at least one of the analyzable ratio and the omission ratio. The analyzable ratio can be information representing the area ratio of the analyzable portion relative to the entire inner surface of the lumen, as shown in equation (2) above, or it can be information representing the area ratio of the analyzable portion relative to the portion of the lumen that the correlation processing unit 94 determines has been observed, as shown in equation (4) above. Similarly, the omission ratio can be information representing the area ratio of the non-analyzable portion relative to the entire inner surface of the lumen, as shown in equation (3) above, or it can be information representing the area ratio of the non-analyzable portion relative to the portion of the lumen that the correlation processing unit 94 determines has been observed, as shown in equation (5) above. Furthermore, the entire inner surface of the lumen here is not limited to strictly including the entire inner surface of the lumen.

[0213] This allows for the appropriate quantification of missed observations. Analyzable proportions and missed observation proportions are useful in indicating the degree of completion of observations and in evaluating user capabilities.

[0214] Furthermore, the correlation processing unit 94 can also determine at least one of the analyzable ratio and the overlooked ratio by simplifying the shape of the unanalyzable portion. This reduces the computational load on the analyzable ratio, etc. As mentioned above, there are cases where no information about the lumen structure is available for the unanalyzable portion; in such cases, the shape of the unanalyzable portion is estimated. As a result, the area of ​​the unanalyzable portion used in the calculation of the analyzable ratio, etc., contains a certain degree of error. That is, considering complex shapes for the unanalyzable portion is less meaningful given the difficulty in performing high-precision calculations. By simplifying the shape of the unanalyzable portion, the processing load can be effectively reduced. Additionally, as mentioned above, the shape of the analyzable portion can also be simplified.

[0215] Furthermore, since the lumen is the intestine, the correlation processing unit 94 can also determine at least one of the analyzable proportion and the overlooked proportion based on the reference shape of the intestine. The reference shape here is, for example, the shape represented by the standard intestinal model described above. In this way, even when the shape of the intestine changes due to factors such as the state of air delivery, it is possible to estimate the appropriate shape and area.

[0216] The baseline shape of the intestine here corresponds to the shape of the intestine when the insertion part 2b of the endoscope 2 is inserted into the intestinal side and then withdrawn in the proximal direction. Lumen structure information is calculated based on images obtained from photographs of the intestine during withdrawal in the proximal direction. Specifically, the intestine during withdrawal refers to the intestine in a distended state due to insufflation. Thus, the state of the intestine represented by the baseline shape corresponds to the state of the intestine determined by the lumen structure information, allowing for the appropriate determination of an analyzable scale.

[0217] 2.4 Prompt Handling

[0218] 2.4.1 Display

[0219] The processing device 9 can also prompt the user by displaying the result of associating the analyzeability information with the lumen structure information on the monitor 6, etc. For example, when the analyzeability information is obtained during observation, the processing device 9 performs processing to associate the camera image, the analyzeability information, and the lumen structure information and displays the result on the monitor 6.

[0220] Figure 14 This is an example of a display image shown on a monitor 6, etc. C1 represents the displayed image, C2 represents the first area which is part of the displayed image, and C3 represents the second area which is another part of the displayed image. For example, as... Figure 14 As shown, the processing device 9 performs processing to display a camera image in a first region of the monitor 6 and to display lumen structure information in a second region different from the first region. Additionally, although in Figure 14 While some details have been omitted, the correlation results between the analyzable information and the lumen structure information can also be displayed in the second area. The correlation result between the analyzable information and the lumen structure information is a 3D model, and the viewpoint can be changed based on user input. Furthermore, the lumen structure information is not limited to a 3D model; it can also be a 2D image obtained by unfolding the cylinder. In this case, the correlation result between the analyzable information and the lumen structure information is also a 2D image. Additionally, the correlation results as a 3D model and as a 2D image can be displayed simultaneously.

[0221] In this way, by comparing the structure of the lumen, it can be indicated which parts of the lumen can be analyzed and which cannot, thereby making it easier for doctors and others to determine which parts have been missed. In addition, by identifying the unanalyzable parts that need to be re-inserted as missed parts, information can be provided to the doctor and a warning can be issued.

[0222] 2.4.2 Guidance

[0223] In addition, such as Figure 3 As shown, the processing device 9 may also include a guidance processing unit 96. When information indicating that at least a portion of the captured image is unanalyzable is output as analysis information, the guidance processing unit 96 provides guidance to improve the visibility of the subject in the captured image. For example, the guidance processing unit 96 provides guidance to improve image quality. This guidance may be guidance indicating changes in the distance between the subject and the front end 11, guidance indicating air or water supply, guidance indicating changes in the camera angle, or guidance indicating the front end 11 to remain stationary to suppress shaking. The guidance processing unit 96 performs the process of displaying text indicating the above adjustments on the monitor 6.

[0224] Furthermore, the guidance processing unit 96 can also provide specific adjustment instructions. For example, the guidance processing unit 96 can indicate the specific location of obstructions or the direction of water delivery. Alternatively, the guidance processing unit 96 can also indicate the insertion and removal amount of the insertion part 2b used to change the camera angle, the bending direction of the bending part 12, the bending angle, etc. This encourages the user to operate the device so that shooting can be performed in a state that allows for analysis.

[0225] Furthermore, the guidance processing unit 96 can also guide the user to capture the missed portion in an analyzeable state when a missed portion exists. For example, the guidance processing unit 96 can display the positional relationship between the currently captured field of view and the missed portion on the camera image. For example, it can display a guidance object such as an arrow indicating which direction the field of view should be moved in (up, down, left, right). The direction of movement can also include forward and backward. This encourages the user to perform the operation of capturing the missed portion. Furthermore, the guidance object is not limited to being displayed on the camera image. For example, the guidance processing unit 96 can also control the display of the camera image in a portion of the display area of ​​the monitor 6, etc., and display the guidance object in an area outside the camera image in the display area. The guidance object is, for example, a... Figure 14 The arrow-shaped object shown in OB. Figure 14 The guide object is displayed in an area of ​​the image that is different from both the first and second regions.

[0226] 3. Variations

[0227] 3.1 Variations related to obtaining information about the lumen structure

[0228] The above describes an example of obtaining lumen structure information in a lumen structure detection device 5, which is different from the processing device 9. However, the method of this embodiment is not limited to this, and the lumen structure information can also be calculated in the processing device 9.

[0229] Figure 15 This is a diagram showing other structures of the processing device 9. (Example) Figure 15 As shown, the processing device 9 includes, in addition to Figure 3 In addition to the structure shown, it also includes a position and attitude information acquisition unit 97 and a drive circuit 98. The position and attitude information acquisition unit 97 and... Figure 4 The position and attitude detection unit 55 corresponds to the drive circuit 98 and the drive circuit 56. That is, the position and attitude information acquisition unit 97 controls the drive circuit 98 of the drive magnetic field generating device 7 to generate a predetermined magnetic field. By detecting the magnetic field through the magnetic sensor 16, the position and attitude information acquisition unit 97 generates position coordinates and orientation data of the front end 11.

[0230] The lumen structure information acquisition unit 93 calculates lumen structure information based on the camera image from the image acquisition unit 91 and the position and attitude information from the position and attitude information acquisition unit 97. That is, Figure 15 The information acquisition unit 93 of the lumen structure is used to obtain information about the lumen structure. Figure 4 The processor 51 has the same processing capability.

[0231] However, in the description of the lumen structure detection device 5, as explained above, the sensor used to detect position and attitude information is not limited to the magnetic sensor 16 provided at the front end 11, and can be implemented... Figure 9 (A) Figure 9 The variations illustrated in (B) are also possible. Furthermore, in the method of this embodiment, position and attitude detection sensors such as the magnetic sensor 16 can be omitted. In this case, the lumen structure information acquisition unit 93 performs optimization processing on (6k+3m) parameters, including the position and attitude of the front end 11, based, for example, on a camera image.

[0232] As described above, the lumen structure information acquisition unit 93 can also determine the lumen structure information based on the camera image. Thus, the processing device 9 can perform the processing of determining the lumen structure information, the processing of determining whether the information can be analyzed, and the processing of mapping the whether the information can be analyzed to the lumen structure.

[0233] In addition, such as Figure 15 As shown, the processing device 9 may also include a position and attitude information acquisition unit 97, which acquires position and attitude information of the front end 11 relative to the cavity from a sensor provided on the insertion part 2b inserted into the cavity. The cavity structure information acquisition unit 93 calculates the cavity structure information based on the camera image and the position and attitude information acquired by the position and attitude information acquisition unit 97.

[0234] In this way, by using sensors such as magnetic sensor 16 to obtain the position and attitude information of the front end 11, it is possible to calculate the high-precision 3D model structure of the cavity at high speed.

[0235] 3.2 Real-time processing and post-event processing

[0236] The above illustrates an example of obtaining and analyzing luminal structural information and related data in real time during observation. In this situation, it is necessary to quickly alert the possibility of missed lesions. When this occurs... Figure 13 In cases where a section is missed, as shown in (B), re-insertion is required to observe the missed section, thus placing a heavy burden on both the doctor and the patient. It is useful to suppress the occurrence of missed sections by quickly notifying the doctor of the presence of unanalyzable sections.

[0237] Therefore, high-speed processing is important when performing various procedures during observation. For example, SLAM is preferred when calculating luminal structure information during observation. SLAM processes the acquired images sequentially, thus offering high real-time performance. This allows for prompting of doctors or other personnel when unanalyzable portions are encountered, thus encouraging confirmation of these unanalyzable portions. Furthermore, in the event of missed portions, confirmation of these missed portions can be prompted before the end of the observation, thus preventing final omissions. When using the endoscopic system 1 for observation, the user needs to prepare by restricting food intake and taking laxatives. Even if re-insertion is required, the burden on the patient is reduced compared to performing a second observation.

[0238] On the other hand, the aforementioned processing can also be performed after the observation is complete. In this case, even if the processing takes several minutes to several hours, it is not a problem. Therefore, in each processing step, accuracy can be prioritized over speed. For example, methods such as SfM can also be used to obtain information about the lumen structure. In SfM, the entire set of images acquired can be used as input, thus improving the estimation accuracy of the lumen structure compared to SLAM.

[0239] Furthermore, given the post-observation information on the acquisition of lumen structure and its correlation with analyzable information, a missed observation was identified after the observation period ended. Therefore, further observation is required to analyze this missed observation. The post-observation correlation results can be used, for example, to quantify the missed observation. Specifically, the correlation results can be used to determine the user's ability and to perform user training. Specific indicators can include the aforementioned analyzable proportion and missed observation proportion.

[0240] 3.3 Variations related to the timing of obtaining information on the lumen structure

[0241] Furthermore, the above has described an example of performing the processing to obtain the lumen structure information, the output of the analyzability information, and the correlation processing on the same observation using the endoscope system 1. For example, when processing is performed in real time during observation, when a camera image is acquired, the output of the analyzability information based on the camera image and the calculation of the lumen structure information based on the camera image are performed in parallel, and the analyzability information is correlated with the lumen structure information. In addition, when the object is a dynamic image stored in a memory or the like after the observation is completed, the output of the analyzability information and the calculation of the lumen structure information are also performed based on the dynamic image.

[0242] However, the method of this embodiment is not limited to this; lumen structure information can also be obtained before the observation of the object to which the output of the analyzable information is processed. For example, when the same patient is the subject, lumen structure information can be obtained in advance based on video images taken in previous observations. Then, based on the video images taken in this observation, the output of analyzable information and the association of the analyzable information with the lumen structure are performed.

[0243] However, while lumen structure information may be a set of feature points, these feature points are defined based on past photographic images. Since no feature points are defined on the photographic images used to calculate analyzability information, it is impossible to directly correlate analyzability information with the lumen structure.

[0244] Therefore, the correlation processing unit 94 can also obtain high-precision position and attitude information based on sensors such as the magnetic sensor 16. At this time, the conditions for setting the reference point of the coordinate system are the same as in past observations. Specifically, the reference point is common at the anus, and the user's posture is also common. Thus, if the past reference point and the current reference point can be correlated, the positional relationship between the anterior end 11 and the intestine in this observation can be estimated. That is, the correlation processing unit 94 can determine, based on the position and attitude information of the anterior end 11, which part of the intestine represented by the lumen structure information corresponds to the subject captured in the camera image. Therefore, the correlation processing unit 94 can correlate the analytical information obtained through this observation with the lumen structure information obtained through past observations.

[0245] Alternatively, the correlation processing unit 94 can perform matching processing between past and current camera images. This allows it to determine which position in the current camera image corresponds to a feature point set in the past image. Consequently, it becomes possible to determine the position of the subject captured in the current camera image on the tubular structure.

[0246] Alternatively, information about the lumen structure can be obtained using diagnostic devices such as CT, MRI, or ultrasound. In this case, it is preferable that the correlation processing unit 94 obtains high-precision position and orientation information based on sensors such as the magnetic sensor 16. By determining the positional relationship between the reference point of the position and orientation information detected by the sensor and the lumen structure information obtained using CT or similar methods, the positional relationship between the anterior end 11 and the intestine can be estimated with high precision. As a result, the position of the subject captured in the photographic image on the lumen structure can be determined.

[0247] 3.4 Control based on association results

[0248] Furthermore, the above description illustrates examples of using the correlation results between analyzable information and lumen structure information to provide prompts to the user or to evaluate the user's capabilities. Additionally, as an example of providing prompts to the user, guidance provided by the guidance processing unit 96 was described. However, the method of this embodiment is not limited to this; the correlation results can also be used for the control of the endoscope 2.

[0249] For example, the endoscope 2 includes a motor (not shown) and a drive unit that drives the motor. Based on the control of the motor, the endoscope 2 automatically inserts and removes the insertion part 2b and bends the bending part 12. When using such an endoscope 2, the processing unit 9 outputs a correlation result to the control unit that controls the drive unit. The control unit can be located in the image processing unit 3 or in the processing unit 9. The control unit controls the drive unit based on the correlation result. For example, if a missed portion is detected, the control unit controls the insertion part 2b to be inserted into a position where the missed portion can be captured, and controls the bending part 12 to bend. Additionally, the control unit can also perform controls to improve image quality. Controls to improve image quality can include, for example, controlling the position and orientation of the front end 11, or controlling the supply of water or air.

[0250] Furthermore, while this embodiment has been described in detail above, those skilled in the art will readily understand that various modifications can be made without substantially departing from the inventive content and effects of this embodiment. Therefore, all such modifications are included within the scope of this invention. For example, a term described at least once with a broader or synonymous term in the specification or drawings can be replaced with that different term anywhere in the specification or drawings. Additionally, all combinations of this embodiment and its modifications are also included within the scope of this invention. Moreover, the structure and operation of the processing device, endoscope system, etc., are not limited to those described in this embodiment, and various modifications can be implemented.

[0251] Symbol Explanation

[0252] 1: Endoscopic system; 2: Endoscope; 2a: Operating unit; 2b: Insertion unit; 2c: Universal cable; 2e: Signal line; 3: Image processing device; 4: Light source device; 5: Lumen structure detection device; 6: Monitor; 7: Magnetic field generating device; 7a: Signal line; 8: Bed; 9: Processing device; 11: Front end; 12: Bending section; 13: Flexible tube section; 14: Bending operation component; 14a: Left and right bending operation knob; 14b: Up and down bending operation knob; 14c: Fixing knob; 15: Camera element; 16: Magnetic sensor Device; 16a, 16b: Coil; 51: Processor; 52: Storage device; 53: Interface; 54: Image acquisition unit; 55: Position and posture detection unit; 56: Drive circuit; 58: Bus; 81: Shape sensor; 82: Insertion / torsion sensor; 91: Image acquisition unit; 92: Feasibility analysis and determination unit; 93: Lumen structure information acquisition unit; 94: Processing unit; 95: Missed view determination unit; 96: Guidance processing unit; 97: Position and posture information acquisition unit; 98: Drive circuit; Pa: Patient; SA: Hidden part.

Claims

1. A processing apparatus, characterized in that, include: The image acquisition unit acquires camera images of the inside of the lumen; A lumen structure information acquisition unit acquires lumen structure information representing the structure of the lumen; The analyzeability determination unit determines whether the camera image can be analyzed based on the camera image and the first determination criterion, and outputs analyzeability information indicating whether the camera image is in an analyzeable state; The association processing unit performs processing based on the analyzability information and the lumen structure information to associate the analyzability information with the structure of the lumen, and determines the analyzable part and the non-analyzable part in the structure of the lumen. as well as The omission determination unit compares the position and orientation information of the front end of the insertion part inserted into the lumen with the position of the unanalyzable part, and determines the unanalyzable part as an omission part based on the comparison result and the second determination criterion.

2. The processing apparatus according to claim 1, characterized in that, The feasibility analysis determination unit outputs feasibility analysis information based on whether the camera image can be analyzed by a computer.

3. The processing apparatus according to claim 2, characterized in that, The camera images are images of living organisms. The dossier determination unit outputs dossier information based on whether the computer can perform classification processing or detection processing. The classification processing is the process of classifying the organism image into any category, and the detection processing is the process of detecting the region of interest in the organism image.

4. The processing apparatus according to claim 1, characterized in that, If the magnitude of the motion of the subject in the video image is below a predetermined value and the motion is maintained for a certain period of time, the analysis-capability determination unit determines that the video image is analyzable.

5. The processing apparatus according to claim 1, characterized in that, When the image quality of the camera image is above a predetermined benchmark value, the analysis-capability determination unit determines that the camera image is analyzable.

6. The processing apparatus according to claim 1, characterized in that, After dividing the camera image into multiple regions, the analyzeability determination unit determines that the camera image is analyzable when the size of each region in the multiple regions is above a predetermined value.

7. The processing apparatus according to claim 1, characterized in that, The lumen structure information acquisition unit calculates the lumen structure information based on the camera image.

8. The processing apparatus according to claim 1, characterized in that, The processing device includes a position and attitude information acquisition unit, which acquires the position and attitude information of the front end of the insertion part relative to the lumen from a sensor disposed on the insertion part. The lumen structure information acquisition unit calculates the lumen structure information based on the position and orientation information and the camera image.

9. The processing apparatus according to claim 1, characterized in that, The association processing unit sets multiple feature points on multiple camera images captured at two or more times, and determines the correspondence between the multiple feature points on the camera images captured at two or more times, thereby associating the analyzable information with the structure of the lumen.

10. The processing apparatus according to claim 9, characterized in that, The association processing unit performs at least one of the following processes: By using three or more of the aforementioned feature points to define the analyzable region on the camera image, the configuration of the analyzable region within the lumen can be determined. as well as By using three or more of the aforementioned feature points to define the areas on the camera image that are determined to be unanalyzable, i.e., unanalyzable areas, the configuration of the unanalyzable areas in the lumen is determined.

11. The processing apparatus according to claim 10, characterized in that, The region defined by three or more of the aforementioned feature points is capable of deformation between the shape in the camera image captured at the first moment and the shape in the camera image captured at the second moment.

12. The processing apparatus according to claim 1, characterized in that, The association processing unit determines the analyzable part of the structure of the cavity that is determined to be analyzable based on at least one of the camera images, and determines the unanalyzable part of the structure of the cavity that is not analyzable.

13. The processing apparatus according to claim 1, characterized in that, The association processing unit estimates at least one of the area of ​​the analyzable portion, the shape of the analyzable portion, the area of ​​the non-analyzable portion, and the shape of the non-analyzable portion.

14. The processing apparatus according to claim 13, characterized in that, The correlation processing unit performs a process to determine at least one of the analyzable ratio and the missed ratio. The analyzable ratio is information representing the area ratio of the analyzable portion relative to the entire inner surface of the lumen, or information representing the area ratio of the analyzable portion relative to the portion of the lumen that was determined to have been observed. The omission ratio is information representing the area ratio of the unanalyzable portion relative to the entire inner surface of the lumen, or information representing the area ratio of the unanalyzable portion relative to the portion of the lumen that was determined to have been observed.

15. The processing apparatus according to claim 14, characterized in that, The association processing unit simplifies the shape of the unanalyzable portion, thereby determining at least one of the analyzable proportion and the missed proportion.

16. The processing apparatus according to claim 14, characterized in that, The lumen is the intestine. The correlation processing unit determines at least one of the analyzable proportion and the missed proportion based on the reference shape of the intestine.

17. The processing apparatus according to claim 1, characterized in that, When the lumen is the intestine, and the insertion part is inserted into the intestinal tract and then pulled out proximally while imaging the intestine, The omission determination unit determines the unanalyzable portion as the omission portion based on the position and orientation information of the front end of the insertion part relative to the intestine. This omission portion is determined to be the unanalyzable portion that cannot be observed unless the insertion part is inserted further inward.

18. The processing apparatus according to claim 1, characterized in that, The processing device includes a guidance processing unit that, in the presence of the missed portion, guides the capture of the missed portion in the analyzable state.

19. An endoscope system, characterized in that, include: The camera unit takes pictures of the inside of the tube; An image acquisition unit acquires images captured by the camera unit; A lumen structure information acquisition unit acquires lumen structure information representing the structure of the lumen; The analyzeability determination unit determines whether the camera image can be analyzed based on the camera image and the first determination criterion, and outputs analyzeability information indicating whether the camera image is in an analyzeable state; The association processing unit performs processing based on the analyzability information and the lumen structure information to associate the analyzability information with the structure of the lumen, and determines the analyzable part and the non-analyzable part in the structure of the lumen. as well as The omission determination unit compares the position and orientation information of the front end of the insertion part inserted into the lumen with the position of the unanalyzable part, and determines the unanalyzable part as an omission part based on the comparison result and the second determination criterion.

20. A method for processing camera images, characterized in that, Includes the following steps: Obtain video images of the inside of the lumen; Obtain lumen structure information representing the structure of the lumen; Based on the camera image and the first determination criterion, it is determined whether the camera image can be analyzed, and analysis-capability information indicating whether the camera image is in an analyzable state is output; The process involves associating the analyzability information with the lumen structure information on the structure of the lumen, and determining the analyzable and non-analyzable portions of the lumen structure. as well as The position and orientation information of the front end of the insertion part inserted into the lumen is compared with the position of the unanalyzable part. Based on the comparison result and the second determination criterion, the determined unanalyzable part is determined to be the missed part.

Images