Image processing device, image processing method, and image processing program

Abstract

This image processing device comprises a processor. The processor: records time-series images acquired by an endoscope during insertion into a subject (S1); associates a reference image acquired by the endoscope during extraction from the subject with one of the time-series images on the basis of the degree of similarity between the reference image and each of the time-series images (S2); generates subsequent information on the basis of the one associated image (S3); and displays the subsequent information together with the reference image (S4). The subsequent information includes information of the subject within an image predicted to be acquired after the reference image by the endoscope during extraction.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Conventionally, techniques for assisting a doctor in making a diagnosis during an endoscopy examination have been proposed (see, for example, Patent Document 1). Patent Document 1 describes automatically detecting a region of interest such as a lesion in an endoscopic image and displaying the region of interest in the endoscopic image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the field of view of an endoscope is limited, the information obtained from the current endoscopic image during observation is limited. In order to more effectively assist a doctor in making a diagnosis, it is desirable to present additional information regarding the examination target to the doctor. The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program that can present additional information regarding an examination target and thereby more effectively assist in making a diagnosis during an endoscopy examination.

Means for Solving the Problems

[0005] One aspect of the present invention is an image processing device comprising a processor, the processor records a time-series of images acquired by an endoscope during insertion into an object under examination, associates the reference image with one image in the time-series of images based on the similarity between the reference image and each of the time-series of images, the reference image is an image acquired by the endoscope during withdrawal from the object under examination, generates subsequent information based on the associated image, the subsequent information includes information about the object under examination in images predicted to be acquired by the endoscope during withdrawal after the reference image, and displays the subsequent information together with the reference image.

[0006] Another aspect of the present invention is an image processing method comprising recording a time-series of images acquired by an endoscope during insertion into an object to be examined, associating the reference image with one of the time-series images based on the similarity between the reference image and each of the time-series images, the reference image being an image acquired by the endoscope during withdrawal from the object to be examined, generating subsequent information based on the associated image, the subsequent information including information about the object to be examined in images predicted to be acquired by the endoscope during withdrawal after the reference image, and displaying the subsequent information together with the reference image.

[0007] Another aspect of the present invention is an image processing program that causes a processor to perform a process including recording a time series of images acquired by an endoscope during insertion into an object under examination, associating the reference image with one of the time series of images based on the similarity between the reference image and each of the time series of images, the reference image being an image acquired by the endoscope during withdrawal from the object under examination, generating subsequent information based on the associated one image, the subsequent information including information about the object under examination in images predicted to be acquired by the endoscope during withdrawal after the reference image, and displaying the subsequent information together with the reference image. [Effects of the Invention]

[0008] According to the present invention, additional information regarding the subject of examination can be presented, thereby more effectively supporting the physician's diagnosis during endoscopic examination. [Brief explanation of the drawing]

[0009] [Figure 1] This is an overall configuration diagram of an image processing device and an endoscope system according to one embodiment. [Figure 2] This diagram illustrates the insertion and removal of an endoscope during a colonoscopy. [Figure 3] This diagram illustrates the relationship between the position and time of images acquired during a colonoscopy. [Figure 4A] This is a flowchart of an example of an image processing method according to the first embodiment. [Figure 4B] This is a flowchart of another example of the image processing method according to the first embodiment. [Figure 5A] This is a functional block diagram of an example of a processor in an image processing apparatus according to the first embodiment. [Figure 5B] This is a functional block diagram of another example of the processor of the image processing apparatus according to the first embodiment. [Figure 5C] This is a functional block diagram of another example of the processor of the image processing apparatus according to the first embodiment. [Figure 5D] This is a functional block diagram of another example of the processor of the image processing apparatus according to the first embodiment. [Figure 6] This diagram illustrates the relationship between a reference image and an image selected based on similarity from the time-series images being inserted. [Figure 7A] This figure shows an example of subsequent information and displayed images. [Figure 7B] This figure shows other examples of subsequent information and displayed images. [Figure 7C] This figure shows other examples of subsequent information and displayed images. [Figure 8] This diagram illustrates the positional relationship between the time-series images being inserted and the 3D data. [Figure 9A]It is a flowchart of an example of an image processing method according to the second embodiment. [Figure 9B] It is a flowchart of another example of the image processing method according to the second embodiment. [Figure 10A] It is a functional block diagram of an example of a processor of an image processing apparatus according to the second embodiment. [Figure 10B] It is a functional block diagram of another example of a processor of an image processing apparatus according to the second embodiment. [Figure 10C] It is a functional block diagram of another example of a processor of an image processing apparatus according to the second embodiment. [Figure 10D] It is a functional block diagram of another example of a processor of an image processing apparatus according to the second embodiment. [Figure 10E] It is a functional block diagram of another example of a processor of an image processing apparatus according to the second embodiment.

Mode for Carrying Out the Invention

[0010] (First Embodiment) An image processing apparatus, an image processing method, and an image processing program according to the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows an endoscope system 100 to which an image processing apparatus 1 according to the present embodiment is applied. The endoscope system 100 includes an endoscope 6, an image processing apparatus 1 that processes an image acquired by the endoscope 6, and a display unit 7.

[0011] The endoscope 6 is a flexible endoscope for the digestive tract, and in the present embodiment, it is a colonoscope as shown in FIG. 2. The endoscope 6 acquires an image inside the colon A, which is the inspection target, by an imaging element 6a, and outputs the image to the image processing apparatus 1. The display unit 7 is a known display such as a liquid crystal display.

[0012] A typical colonoscopy consists of two steps: inserting the endoscope 6 into the colon A, and then withdrawing the endoscope 6 from the colon A. Specifically, the endoscope 6 is inserted from the anus B, through the sigmoid colon, descending colon, transverse colon, and ascending colon to the cecum C, and then withdrawn from the cecum C to the anus B via the same route as insertion. Generally, the physician carefully observes the images while withdrawing the endoscope 6 to diagnose the presence or absence of lesions.

[0013] The endoscopic system 100 may further include a sensor 8 that detects the insertion distance of the endoscope 6 into the colon A. The insertion distance is the length of the endoscope 6 from the anus B to the tip of the endoscope 6. The sensor 8 is, for example, attached to the anus B and detects the amount of advancement and retraction of the endoscope 6. The sensor 8 may be of any other known type.

[0014] Figure 3 shows the relationship between the position and time of images acquired by the endoscope 6 within the colon A. The position of the time-series images within the colon A during insertion changes in the direction of progression from the sigmoid colon to the cecum C, while the position of the time-series images within the colon A during withdrawal changes in the direction of regression from the cecum C to the sigmoid colon. Therefore, the time-series images during insertion and the time-series images during withdrawal are arranged in opposite directions with respect to time. Also, the images during insertion and withdrawal are similar to each other. The image processing device 1 generates subsequent information D1, D2, and D3 (described later) using the images during insertion, and displays the subsequent information D1, D2, and D3 together with the images during withdrawal on the display unit 7, thereby supporting the physician's diagnosis (see Figures 7A to 7C).

[0015] The image processing device 1 comprises a processor 2 such as a central processing unit, a memory 3 such as RAM (Random Access Memory), a storage unit 4, and an input / output unit 5. The storage unit 4 is a computer-readable, non-temporary recording medium, such as a hard disk drive or ROM (read-only memory). The storage unit 4 stores an image processing program 4a that causes the processor 2 to execute the image processing method described later. The input / output unit 5 has a known input / output interface for images and is connected to the endoscope 6 and the display unit 7.

[0016] Next, the image processing method according to this embodiment, which is executed by the image processing device 1, will be described. As shown in Figure 4A, the image processing method includes the steps of: S1 recording the time-series images being inserted; S2 acquiring first deformation information of the images being inserted; S3 associating the reference image being removed with one of the images in the time-series images being inserted; S4 acquiring second deformation information of the reference image; S5 generating subsequent information D1; and S6 displaying the reference image together with the subsequent information D1.

[0017] Processor 2 determines whether the endoscope 6 is currently being inserted or withdrawn, executes steps S1 to S2 while the endoscope 6 is being inserted, and executes steps S3 to S6 while the endoscope 6 is being withdrawn. For example, Processor 2 determines that the endoscope 6 is being inserted based on the insertion distance detected by sensor 8 increasing over time, and determines that the endoscope 6 is being withdrawn based on the insertion distance decreasing over time.

[0018] Figure 5A shows the functional units 11-17 of the processor 2 that execute the image processing method. In step S1, the first image acquisition unit 11 acquires images of the endoscope being inserted, which are input to the image processing device 1 from the endoscope 6 through the input / output unit 5, and records the images sequentially in the image recording unit 9. As a result, the image recording unit 9 records a time-series of images of the endoscope being inserted. The image recording unit 9 is a memory 3, a storage unit 4, or any other recording medium.

[0019] Next, in step S2, the first deformation information generation unit 12 acquires first deformation information between the time-series images being inserted. The first deformation information is information about the movement of the subject between the two images, for example, a motion vector between the two images. From the first deformation information, it is possible to estimate the relative positional relationship of the subject between the time-series images.

[0020] In one example, the first deformation information generation unit 12 receives an image from the first image acquisition unit 11 and encodes the image into an MPEG (Moving Picture Experts Group) file. The first deformation information is a motion vector between the current image and the previous image, calculated during the encoding process. The first deformation information generation unit 12 records the encoded image in the image recording unit 9, associating it with the first deformation information.

[0021] Figure 5B shows another example of the processor 2. In this other example, the first deformation information generation unit 12 may acquire the first deformation information after the image being inserted has been recorded in the image recording unit 9. In this case, step S2 in Figures 4A and 4B may be performed at any time before step S4, and an unprocessed image may be recorded in the image recording unit 9. Steps S2 and S4 only need to be completed before step S5 is executed.

[0022] Next, in step S3, the second image acquisition unit 13 acquires the image being withdrawn, which is input from the endoscope 6 through the input / output unit 5 to the image processing device 1. Subsequently, the similar image extraction unit 14 sets the acquired current image (time t=tj) Itj as the reference image, and associates the reference image with one of the inserted images Itj' based on the similarity between the reference image Itj and each of the time-series images being inserted.

[0023] Specifically, the similar image extraction unit 14 evaluates the similarity between the reference image Itj and each image being inserted that is recorded in the image recording unit 9, and extracts the single image Itj' with the highest similarity. As a result, the reference image Itj is associated with the single most similar image being inserted, Itj'. Hereinafter, the extracted image Itj' will be referred to as a similar image. As shown in Figure 3, the similar image Itj' is an image whose position within the colon A is the same as or nearly the same as the reference image Itj.

[0024] The inserted image may include deformation of the subject, as well as rotation and scaling of the image relative to the reference image Itj. Therefore, in evaluating similarity, it is preferable to use a similarity evaluation method that includes non-rigid deformation of the subject, such as SIFT (Scale-Invariant Feature Transform) which uses features within the image.

[0025] Next, in step S4, the second deformation information generation unit 15 takes the reference image Itj and the similar image Itj' as input and obtains second deformation information, which is deformation information between these two images. The second deformation information is, for example, a motion vector between the two images Itj and Itj'.

[0026] Next, in step S5, the subsequent information generation unit 16 selects at least one image from the time-series images being inserted based on the similar image Itj'. The selected image is located near the similar image Itj' and on the anal B side of the similar image Itj', and is typically the image Itj'-t1 acquired before the similar image Itj', as shown in Figure 3. Therefore, as shown in Figure 6, the selected image Itj'-t1 includes the field of view of the reference image Itj and the surrounding region. The subsequent information generation unit 16 generates subsequent information D1 from the selected image Itj'-t1 using the first deformation information and the second deformation information.

[0027] Subsequent information D1 is information about the colon A in the image that is expected to be acquired by the endoscope 6 during withdrawal after the reference image Itj. Specifically, as shown in Figure 7A, subsequent information D1 is an image of the region surrounding the field of view of the reference image Itj, which is positioned around the reference image Itj and is shown as a hatched region in Figure 7A. In other words, subsequent information D1 is an image of the region that will appear in the image acquired immediately after the reference image Itj and includes information about the colon A that is not present in the reference image Itj. For example, the subsequent information generation unit 16 extracts the area surrounding the field of view of the reference image Itj from the selected image Itj'-t1, and performs processing such as deformation, rotation, and scaling on the extracted area based on the first deformation information and the second deformation information to generate subsequent information D1 aligned with the reference image Itj.

[0028] Next, in step S6, the display image generation unit 17 generates a display image F in which the reference image Itj is accompanied by subsequent information D1. The display image F includes the reference image Itj and a region E that is located outside the reference image Itj and surrounds it, with region E consisting of the subsequent information D1. The display image generation unit 17 outputs the display image F to the display unit 7 for display.

[0029] Thus, in a colonoscopy, the endoscope 6 is inserted and withdrawn along the same route, and the physician diagnoses the presence and condition of a lesion based on the images taken during withdrawal. According to this embodiment, subsequent information D1 is generated using the image taken during insertion before removal, and the subsequent information D1 is displayed on the display unit 7 together with the current image Itj. Therefore, based on the subsequent information D1, the physician can predict in advance the region of the colon A that will appear in the image later. In this way, by presenting the physician with additional subsequent information D1 in addition to the current image Itj, the physician's diagnosis can be effectively supported, thereby reducing the chance of overlooking lesions.

[0030] In this embodiment, as shown in Figure 5C, the processor 2 may further include a lesion detection unit 18 as a functional unit for detecting lesions in each image being inserted. As shown in Figure 4B, the lesion detection unit 18 uses known CADe (Computer Aided Detection) techniques to detect lesions from each image being inserted, for example, based on texture and color (step S7). Information on the location of the lesions may be recorded, for example, in the image recording unit 9 in association with the image. Step S7 may be performed at any time before step S5.

[0031] In this case, the subsequent information generation unit 16 may generate other subsequent information using the information on the location of the detected lesion. Figures 7B and 7C show other examples of subsequent information. The subsequent information D2 and D3 in Figures 7B and 7C are indicators placed around the reference image Itj to show the location of the lesion. In Figure 7B, subsequent information D2 is an arrow indicating the location of the lesion that will appear in the image after the reference image Itj. In Figure 7C, subsequent information D3 is a marker displayed at the location of the lesion, superimposed on subsequent information D1. This subsequent information D2 and D3 informs the user that a lesion will appear in the image later, along with its location.

[0032] The subsequent information generation unit 16 generates subsequent information D2 and D3 aligned with the reference image Itj by performing processing such as deformation, rotation, and scaling on the lesion region based on the first deformation information and the second deformation information. The subsequent information is not limited to the subsequent information D1, D2, and D3 described above, but may take other forms.

[0033] In this embodiment, the similar image extraction unit 14 may associate the reference image Itj with one image Itj' being inserted based on insertion distance in addition to similarity. As shown in Figure 5D, during insertion and withdrawal of the endoscope 6, the insertion distance detected by the sensor 8 is recorded in the ancillary information recording unit 10 in association with a timestamp. The ancillary information recording unit 10 is a memory 3, a storage unit 4, or any other recording medium. Each image acquisition unit 11 and 13 acquires an image with a timestamp.

[0034] The similar image extraction unit 14 obtains the insertion distance of the reference image Itj from the supplementary information recording unit 10 based on the timestamp, then selects one or more images currently being inserted that have the same or close insertion distance as the reference image Itj based on the timestamp, and evaluates the similarity between the reference image Itj and each selected image. This reduces the time required to match the reference image Itj compared to evaluating the similarity between the reference image Itj and all images being inserted. Alternatively, after the matching between insertion and observation is completed, images can be selected to evaluate similarity using timestamp information.

[0035] (Second Embodiment) Next, an image processing apparatus, an image processing method, and an image processing program according to a second embodiment of the present invention will be described with reference to the drawings. This embodiment differs from the first embodiment in that it uses three-dimensional (3D) data of the large intestine A. In this embodiment, the configurations that differ from the first embodiment will be described, and the same reference numerals will be used for components common to both embodiments, and their descriptions will be omitted.

[0036] The image processing apparatus 1 according to this embodiment includes a processor 2, memory 3, storage unit 4, and input / output unit 5, similar to the first embodiment, and is applied to an endoscope system 100. As shown in Figure 8, the 3D data G is data of the three-dimensional shape of the entire colon A, acquired in advance by a 3D imaging device before the colonoscopy, for example, 3D CT image data. The 3D data G is stored in the memory unit 4 or other predetermined recording medium before the colonoscopy.

[0037] In this embodiment, the image processing program 4a causes the processor 2 to execute the image processing method shown in Figure 9A. The image processing method of this embodiment includes the steps of: acquiring 3D data G in step S11; recording the time-series images being inserted in step S12; associating the reference image being removed with one of the time-series images being inserted in step S13; generating subsequent information D1 in step S14; and displaying the reference image together with the subsequent information D1 in step S15. Processor 2 executes steps S11-S12 during insertion of endoscope 6 and steps S13-S15 during withdrawal of endoscope 6.

[0038] Figure 10A shows the functional units 11, 13, 16, 17, 19-22 of the processor 2 that executes the image processing method. In step S11, the 3D data acquisition unit 19 acquires 3D data G from the storage unit 4 or a predetermined recording medium, and subsequently, the shape data acquisition unit 20 acquires the three-dimensional shape of the large intestine A from the 3D data G.

[0039] The next step S12 includes the step S121 of calculating the positional relationship between each of the time-series images being inserted and the 3D data G, and the step S122 of generating 3D data with image information by attaching the time-series images being inserted to the 3D data G. Therefore, the time-series images being inserted may be recorded as part of the 3D data with image information.

[0040] Specifically, the first image acquisition unit 11 acquires images of the inserted endoscope 6 that are input to the image processing device 1 through the input / output unit 5. Subsequently, as shown in Figure 8, the first alignment unit 21 aligns the inserted image to the corresponding position in the 3D data G (step S121) and attaches the image to the corresponding position in the 3D data G (step S122). For example, the first alignment unit 21 may determine the corresponding position of each image by comparing each image with each position in the 3D data G. Alternatively, the first alignment unit 21 may use known techniques such as SLAM (Simultaneous Localization and Mapping) to reconstruct a 3D model of the colon A from multiple images and attach the reconstructed 3D model to the 3D data G. During the insertion of the endoscope 6, the above process is repeated, thereby generating 3D data with image information, onto which the texture of the colon A in the time-series images during insertion is applied.

[0041] Next, in step S13, the second image acquisition unit 13 acquires the current image Itj being withdrawn, which is input from the endoscope 6 through the input / output unit 5 to the image processing device 1. Subsequently, the second alignment unit 22 sets the current image Itj as a reference image and associates the reference image Itj with one of the images being inserted based on the similarity between the reference image Itj and each image being inserted in the 3D data with image information. The image to be associated is the similar image Itj' with the highest similarity to the reference image Itj, as in the first embodiment.

[0042] Next, in step S14 Then, the subsequent information generation unit 16 selects a portion of the 3D data with image information based on the similar image Itj' and generates subsequent information D1 from the selected portion. The selected portion is in the vicinity of the similar image Itj' and is closer to the anus B than the similar image Itj'. As shown in Figure 7A, the subsequent information D1 is an image of the region surrounding the field of view of the reference image Itj, which is placed around the reference image Itj. Next, in step S15, the display image generation unit 17 generates a display image F and outputs it to the display unit 7, similar to step S6 of the first embodiment.

[0043] Thus, according to this embodiment, subsequent information D1 is generated using the image taken during insertion before removal, and the subsequent information D1 is displayed on the display unit 7 together with the current image Itj. Therefore, based on the subsequent information D1, the physician can predict in advance the region of the colon A that will appear in the image later. In this way, by presenting the physician with additional subsequent information D1 in addition to the current image Itj, the physician's diagnosis can be effectively supported, thereby reducing the chance of overlooking lesions.

[0044] In this embodiment, as shown in Figures 10B and 10C, the processor 2 may further include a lesion detection unit 18 as a functional unit that detects lesions using CADe. In this case, the subsequent information generation unit 16 may generate subsequent information D2, D3 (see Figures 7B and 7C) indicating the location of the lesions, similar to the first embodiment.

[0045] The lesion detection unit 18 may detect lesions in each image being inserted (see Figure 10B), or it may detect lesions in the 3D data with image information (see Figure 10C), or it may detect lesions in both each image being inserted and the 3D data with image information. Therefore, step S16 for detecting lesions may be performed at any time before step S14 (see Figure 9B). In the case of Figure 10B, the location of each lesion is recorded on an arbitrary recording medium in association with the image. The first alignment unit 21 generates 3D data with image information including lesion information by assigning lesion information to the corresponding location in the 3D data G based on the recorded location.

[0046] The images and 3D data G from the endoscope 6 have advantages and disadvantages, respectively, in detecting lesions. For example, images from endoscope 6 can detect lesions with minimal irregularities, such as small polyps less than 5 mm in size, based on color and texture. On the other hand, 3D data G has difficulty detecting lesions with minimal irregularities, such as polyps less than 5 mm in size. Therefore, according to the configurations in Figures 10B and 10C, lesions with minimal irregularities can be detected with high accuracy.

[0047] Furthermore, because the field of view of the endoscope 6 is limited, there may be missing areas in the time-series images taken during insertion that are not captured. Lesions located in these missing areas cannot be detected by CADe. On the other hand, 3D data G, such as a 3D CT image, includes the entire colon A. Therefore, according to the configuration in Figure 10C, in addition to being able to detect lesions with small irregularities using the endoscope, lesions in areas that are difficult to capture with the endoscope can also be detected with high accuracy.

[0048] As shown in Figure 10D, the lesion detection unit 18 may detect lesions in the 3D data G. For example, the lesion detection unit 18 detects lesions from the three-dimensional shape of the colon A acquired by the shape data acquisition unit 20 using known techniques. The location of each detected lesion is recorded together with the 3D data G in the storage unit 4 or a predetermined recording medium.

[0049] In the configuration shown in Figure 10D, the subsequent information generation unit 16 may select a portion of the 3D data G based on the similar image Itj' and generate subsequent information D1, D2, and D3 from the selected portion of the 3D data G. The selected portion is in the vicinity of the similar image Itj' and is closer to the anus B than the similar image Itj'. In this case, the first alignment unit 21 determines the position of each image being inserted in the 3D data G, but does not necessarily generate 3D data with image information. The second alignment unit 22 may determine a similar image Itj' based on the similarity between the reference image Itj and each image being inserted recorded in the image recording unit 9.

[0050] The lesions in the 3D data G may be detected by a user, such as a physician, instead of by the processor 2. In this case, the user registers the location of the lesion in the 3D data G by, for example, placing a marker on the location of the lesion, and records the 3D data G with the registered lesion locations on the storage unit 4 or a predetermined recording medium.

[0051] In this embodiment, the second alignment unit 22 may associate the reference image Itj with one of the images being inserted based on insertion distance in addition to similarity. As shown in Figure 10E, during insertion and withdrawal of the endoscope 6, the insertion distance detected by the sensor 8 is recorded in the supplementary information recording unit 10 in association with a timestamp. Each image acquisition unit 11 and 13 acquires an image with a timestamp.

[0052] The second alignment unit 22 obtains the insertion distance of the reference image Itj from the supplementary information recording unit 10 based on the timestamp, then selects one or more images being inserted with the same or close insertion distance as the reference image Itj based on the timestamp, and evaluates the similarity between the reference image Itj and each selected image. This reduces the time required to match the reference image Itj compared to evaluating the similarity between the reference image Itj and all images being inserted. Alternatively, after the matching between insertion and observation is completed, images can be selected to evaluate similarity using timestamp information.

[0053] Although embodiments and modifications of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the spirit of the invention. For example, the image processing apparatus and image processing method of the present invention can be applied not only to the large intestine but also to other examination subjects where the insertion and removal routes of the endoscope are the same, such as tubular examination subjects. [Explanation of symbols]

[0054] 1 Image Processing Device 2 processors 4a Image processing program 8 sensors A. Large intestine (to be examined) D1, D2, D3 Subsequent Information F Display Image ITJ reference image G 3D data

Claims

1. Equipped with a processor, The processor, The time-series images acquired by the endoscope during insertion into the area to be examined are recorded. Based on the similarity between the reference image and each of the time-series images, the reference image is associated with one of the time-series images, and the reference image is an image acquired by the endoscope during withdrawal from the object being examined. Subsequent information is generated based on the aforementioned associated image, and this subsequent information includes information about the subject of examination in an image that is expected to be acquired by the endoscope during removal after the reference image, An image processing device that displays the aforementioned subsequent information together with the reference image.

2. The aforementioned processor, By attaching the time-series images being inserted to the previously acquired 3D data of the object to be inspected, 3D data with image information is generated. The aforementioned reference image is associated with one of the time-series images attached to the three-dimensional data with image information, The image processing apparatus according to claim 1, comprising selecting a portion of the three-dimensional data with image information based on one associated image, and generating the subsequent information from the selected portion of the three-dimensional data.

3. The image processing apparatus according to claim 1, wherein the processor selects a portion of the three-dimensional data of the object to be inspected that has been acquired in advance based on the associated image, and generates the subsequent information from the selected portion of the three-dimensional data.

4. The aforementioned processor, First deformation information, which is deformation information between the time-series images being inserted, is obtained. Second deformation information, which is deformation information between the aforementioned reference image and the associated image, is obtained. The image processing apparatus according to claim 1, comprising: selecting at least one image in the time-series images based on the associated image; and generating the subsequent information from the selected at least one image using the first deformation information and the second deformation information.

5. The image processing apparatus according to claim 1, wherein the processor further associates the reference image with one image in the time-series image based on the insertion distance of the endoscope.

6. The image processing apparatus according to claim 1, wherein the subsequent information is an image of the region surrounding the field of view of the reference image, which is arranged around the reference image.

7. The processor further detects lesions in the time-series images, The image processing apparatus according to claim 1, wherein the subsequent information is an indicator arranged around the reference image to show the location of the lesion.

8. The processor further detects lesions within the three-dimensional data with image information, The image processing apparatus according to claim 2, wherein the subsequent information is an indicator arranged around the reference image to show the location of the lesion.

9. The processor further detects lesions within the three-dimensional data, The image processing apparatus according to claim 3, wherein the subsequent information is an indicator arranged around the reference image to show the location of the lesion.

10. The time-series images acquired by the endoscope during insertion into the area to be examined are recorded. Based on the similarity between the reference image and each of the time-series images, the reference image is associated with one of the time-series images, and the reference image is an image acquired by the endoscope during withdrawal from the object being examined. Subsequent information is generated based on the aforementioned associated image, and this subsequent information includes information about the subject of examination in an image that is expected to be acquired by the endoscope during removal after the reference image, An image processing method that includes displaying the aforementioned subsequent information together with the reference image.

11. The time-series images acquired by the endoscope during insertion into the area to be examined are recorded. Based on the similarity between the reference image and each of the time-series images, the reference image is associated with one of the time-series images, and the reference image is an image acquired by the endoscope during withdrawal from the object being examined. Subsequent information is generated based on the aforementioned associated image, and this subsequent information includes information about the subject of examination in an image that is expected to be acquired by the endoscope during removal after the reference image, The subsequent information is displayed together with the reference image. An image processing program that causes a processor to perform a process that includes the following.

Images