Image processing device, image processing method, and image processing program
The image processing device addresses the limited field of view in endoscopy by associating and displaying subsequent information with reference images, enhancing diagnostic accuracy by providing additional context and predicting the region of interest during endoscopy.
Patent Information
- Application Number
- PCT/JP2023/018554
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-21
AI Technical Summary
The limited field of view of an endoscope during endoscopy restricts the information available for diagnosis, making it difficult for doctors to effectively diagnose lesions, as conventional techniques only provide information from the current image without additional context.
An image processing device that records and processes a time series of images during endoscope insertion and removal, associating reference images with similar images from the insertion phase to generate subsequent information, which is then displayed alongside the current image, providing additional context and predicting the region of interest during removal.
This solution enhances diagnostic support by presenting additional information, reducing the likelihood of overlooking lesions by predicting the region of interest in subsequent images, thus improving the accuracy of endoscopy diagnosis.
Smart Images

Figure JP2023018554_21112024_PF_FP_ABST
Abstract
Description
Image processing device, image processing method, and image processing program
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program.
[0002] Conventionally, a technique for assisting a doctor in making a diagnosis during an endoscopic examination has been proposed (see, for example, Patent Document 1). Patent Document 1 describes a technique for automatically detecting a region of interest, such as a lesion, in an endoscopic image and displaying the region of interest in the endoscopic image.
[0003] Patent No. 7125484
[0004] Since the field of view of an endoscope is limited, the information that can be obtained from the current endoscopic image during observation is limited. In order to more effectively support a doctor's diagnosis, it is desirable to present additional information about the examination target to the doctor. The present invention has been made in consideration of the above-mentioned circumstances, and an object of the present invention is to provide an image processing device, an image processing method, and an image processing program that can present additional information about the examination target, thereby more effectively supporting a diagnosis during an endoscopic examination.
[0005] One aspect of the present invention is an image processing device that includes a processor that records a time series of images acquired by an endoscope while it is inserted into an object to be examined, and matches the reference image with one image in the time series of images based on the similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope while it is being removed from the object to be examined, and generates subsequent information based on the matched image, the subsequent information including information about the object to be examined in an image that is predicted to be acquired by the endoscope during removal after the reference image, and displays the subsequent information together with the reference image.
[0006] Another aspect of the present invention is an image processing method that includes recording a time series of images acquired by an endoscope while it is being inserted into an object to be examined, matching the reference image with one image in the time series of images based on the similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope while it is being removed from the object to be examined, generating subsequent information based on the matched image, the subsequent information including information about the object to be examined in an image that is predicted to be acquired by the endoscope during removal after the reference image, and displaying the subsequent information together with the reference image.
[0007] Another aspect of the present invention is an image processing program that causes a processor to perform a process including recording a time series of images acquired by an endoscope while it is being inserted into an object to be examined, matching the reference image with one image in the time series of images based on the similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope while it is being removed from the object to be examined, generating subsequent information based on the matched image, the subsequent information including information about the object to be examined in an image that is predicted to be acquired by the endoscope during removal after the reference image, and displaying the subsequent information together with the reference image.
[0008] According to the present invention, it is possible to provide additional information about an object to be examined, thereby more effectively supporting a doctor's diagnosis during an endoscopic examination.
[0009] 1 is a diagram illustrating the overall configuration of an image processing device and an endoscopic system according to an embodiment. FIG. 1 is a diagram illustrating the insertion and removal of an endoscope in a colonoscopy examination. FIG. 2 is a diagram illustrating the relationship between the position and time of images acquired during a colonoscopy examination. FIG. 2 is a flowchart of an example of an image processing method according to the first embodiment. FIG. 3 is a flowchart of another example of an image processing method according to the first embodiment. FIG. 4 is a functional block diagram of an example of a processor of an image processing device according to the first embodiment. FIG. 5 is a functional block diagram of another example of a processor of an image processing device according to the first embodiment. FIG. 6 is a functional block diagram of another example of a processor of an image processing device according to the first embodiment. FIG. 7 is a functional block diagram of another example of a processor of an image processing device according to the first embodiment. FIG. 8 is a diagram illustrating the relationship between a reference image and an image selected from time-series images during insertion based on similar images. FIG. 9 is a diagram illustrating an example of subsequent information and a displayed image. FIG. 10 is a diagram illustrating another example of subsequent information and a displayed image. FIG. 11 is a diagram illustrating the positional relationship between time-series images during insertion and 3D data. FIG. 12 is a flowchart of an example of an image processing method according to the second embodiment. FIG. 13 is a flowchart of another example of an image processing method according to the second embodiment. FIG. 14 is a functional block diagram of another example of a processor of an image processing device according to the second embodiment. 10 is a functional block diagram of another example of the processor of the image processing device according to the second embodiment;FIG. 11 is a functional block diagram of another example of the processor of the image processing device according to the second embodiment;FIG.
[0010] First Embodiment An image processing device, an image processing method, and an image processing program according to a first embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows an endoscope system 100 to which an image processing device 1 according to this embodiment is applied. The endoscope system 100 includes an endoscope 6, an image processing device 1 that processes images acquired by the endoscope 6, and a display unit 7.
[0011] The endoscope 6 is a flexible endoscope for the digestive system, and in this embodiment is a colonoscope, as shown in Fig. 2. The endoscope 6 acquires images of the inside of the large intestine A, which is the object of examination, using an image sensor 6a, and outputs the images to the image processing device 1. The display unit 7 is a known display such as a liquid crystal display.
[0012] A typical colonoscopy consists of two steps: a step of inserting an endoscope 6 into the large intestine A, and a step of subsequently removing the endoscope 6 from the large intestine A. Specifically, the endoscope 6 is inserted from the anus B, passing through the sigmoid colon, descending colon, transverse colon, and ascending colon to the cecum C, and then removed from the cecum C to the anus B via the same route as when it was inserted. Generally, a doctor carefully observes the image while removing the endoscope 6 to diagnose the presence or absence of a lesion.
[0013] The endoscope system 100 may further include a sensor 8 that detects the insertion distance of the endoscope 6 into the large intestine A. The insertion distance is the length of the endoscope 6 from the anus B to the tip of the endoscope 6. The sensor 8 is attached to the anus B, for example, and detects the amount of advancement and retraction of the endoscope 6. The sensor 8 may also be of another known type.
[0014] FIG. 3 shows the relationship between time and position within the large intestine A of images acquired by the endoscope 6. The position within the large intestine A of the time-series images during insertion changes in the direction of progression from the sigmoid colon to the cecum C, while the position within the large intestine A of the time-series images during removal changes in the direction of retreat from the cecum C to the sigmoid colon. Therefore, the time-series images during insertion and the time-series images during removal are arranged in reverse order with respect to time. Furthermore, the images during insertion and the images during removal are similar to each other. The image processing device 1 generates subsequent information D1, D2, and D3 (described below) using the images during insertion, and displays the subsequent information D1, D2, and D3 on the display unit 7 together with the images during removal, thereby assisting the physician in making a diagnosis (see FIGS. 7A to 7C ).
[0015] The image processing device 1 includes a processor 2 such as a central processing unit, a memory 3 such as a RAM (Random Access Memory), a storage unit 4, and an input / output unit 5. The storage unit 4 is a computer-readable non-transitory recording medium such as a hard disk drive or a ROM (Read-Only Memory). The storage unit 4 stores an image processing program 4a for causing the processor 2 to execute an image processing method described below. The input / output unit 5 has a known image input / output interface and is connected to an endoscope 6 and a display unit 7.
[0016] Next, a description will be given of an image processing method according to this embodiment, which is executed by the image processing device 1. As shown in Fig. 4A , the image processing method includes step S1 of recording time-series images during insertion, step S2 of acquiring first deformation information for the images during insertion, step S3 of associating a reference image during removal with one image in the time-series images during insertion, step S4 of acquiring second deformation information for the reference image, step S5 of generating subsequent information D1, and step S6 of displaying the reference image together with the subsequent information D1.
[0017] The processor 2 determines whether the endoscope 6 is currently being inserted or removed, and executes steps S1 to S2 while the endoscope 6 is being inserted, and executes steps S3 to S6 while the endoscope 6 is being removed. For example, the processor 2 determines that the endoscope 6 is being inserted based on the fact that the insertion distance detected by the sensor 8 increases over time, and determines that the endoscope 6 is being removed based on the fact that the insertion distance decreases over time.
[0018] 5A shows functional units 11 to 17 of the processor 2 that execute the image processing method. In step S1, the first image acquisition unit 11 acquires images during insertion that are input from the endoscope 6 to the image processing device 1 through the input / output unit 5, and records the images in order in the image recording unit 9. As a result, a time series of images during insertion is recorded in the image recording unit 9. The image recording unit 9 is the memory 3, the storage unit 4, or any other recording medium.
[0019] Next, in step S2, the first deformation information generator 12 acquires first deformation information between the images in the time series being inserted. The first deformation information is information about the movement of the subject between the two images, such as a motion vector between the two images. From the first deformation information, it is possible to estimate the relative positional relationship of the subject between the images in the time series.
[0020] In one example, the first deformation information generation unit 12 receives an image from the first image acquisition unit 11 and encodes the image into an MPEG (Moving Picture Experts Group) file. The first deformation information is a motion vector between the current image and the previous image calculated in the encoding process. The first deformation information generation unit 12 records the encoded image in the image recording unit 9 in association with the first deformation information.
[0021] 5B shows another example of the processor 2. In another example, the first deformation information generator 12 may acquire the first deformation information after the image being inserted is recorded in the image recording unit 9. In this case, step S2 in FIGS. 4A and 4B may be executed at any timing before step S4, and the unprocessed image may be recorded in the image recording unit 9. Steps S2 and S4 may be completed before step S5 is executed.
[0022] Next, in step S3, the second image acquisition unit 13 acquires images during removal that are input from the endoscope 6 to the image processing device 1 through the input / output unit 5. Subsequently, the similar image extraction unit 14 sets the acquired current (time t=tj) image Itj as a reference image, and associates the reference image with one image Itj' during insertion based on the similarity between the reference image Itj and each of the images in the time series during insertion.
[0023] Specifically, the similar image extracting unit 14 evaluates the similarity between the reference image Itj and each of the images during insertion recorded in the image recording unit 9, and extracts the image Itj' with the highest similarity. As a result, the reference image Itj is associated with the most similar image during insertion Itj'. Hereinafter, the extracted image Itj' is referred to as a similar image. As shown in FIG. 3, the similar image Itj' is an image located at the same or approximately the same position in the large intestine A as the reference image Itj.
[0024] The image being inserted may include deformation of the subject as well as rotation and scaling of the image relative to the reference image Itj. Therefore, in the evaluation of similarity, it is preferable to use a similarity evaluation method that includes non-rigid deformation of the subject, such as SIFT (Scale-Invariant Feature Transform) that uses features within the image.
[0025] Next, in step S4, the second deformation information generator 15 receives the reference image Itj and the similar image Itj' as input and acquires second deformation information, which is deformation information between these two images. The second deformation information is, for example, a motion vector between the two images Itj and Itj'.
[0026] Next, in step S5, the subsequent information generation unit 16 selects at least one image from the time-series images being inserted based on the similar image Itj'. The selected image is an image located closer to the similar image Itj' and closer to the anus B than the similar image Itj', and is typically an image Itj'-t1 acquired before the similar image Itj', as shown in FIG. 3. Therefore, as shown in FIG. 6, the selected image Itj'-t1 includes the field of view of the reference image Itj and its surrounding area. The subsequent information generation unit 16 generates subsequent information D1 from the selected image Itj'-t1 using the first deformation information and the second deformation information.
[0027] The subsequent information D1 is information about the large intestine A in an image predicted to be acquired by the endoscope 6 being removed after the reference image Itj. Specifically, as shown in FIG. 7A , the subsequent information D1 is an image of the region surrounding the field of view of the reference image Itj, arranged around the reference image Itj, and is shown as a hatched region in FIG. 7A . That is, the subsequent information D1 is an image of the region that will appear in an image acquired immediately after the reference image Itj, and includes information about the large intestine A that is not present in the reference image Itj. For example, the subsequent information generator 16 extracts the region surrounding the field of view of the reference image Itj from the selected image Itj'-t1, and performs processing such as deformation, rotation, and scaling on the extracted region based on the first deformation information and the second deformation information, thereby generating subsequent information D1 that is aligned with the reference image Itj.
[0028] Next, in step S6, the display image generation unit 17 generates a display image F in which the subsequent information D1 is added to the reference image Itj. The display image F includes the reference image Itj and an area E that is disposed outside the reference image Itj and surrounds the reference image Itj, and the area E is made up of the subsequent information D1. The display image generation unit 17 outputs the display image F to the display unit 7 to display it.
[0029] Thus, during a colonoscopy, the endoscope 6 is inserted and removed along the same path, and the physician diagnoses the presence and condition of a lesion based on images taken during removal. According to this embodiment, subsequent information D1 is generated using images taken during insertion before removal, and the subsequent information D1 is displayed on the display unit 7 together with the current image Itj. Therefore, the physician can predict in advance the region of the colon A that will appear in the image later, based on the subsequent information D1. In this way, by presenting the physician with additional subsequent information D1 in addition to the current image Itj, the physician's diagnosis can be effectively supported, thereby reducing the risk of overlooking a lesion.
[0030] In this embodiment, as shown in FIG. 5C, processor 2 may further include, as a functional unit, lesion detection unit 18 that detects lesions in each image during insertion. As shown in FIG. 4B, lesion detection unit 18 uses known computer-aided detection (CADe) technology to detect lesions from each image during insertion, for example, based on texture and color (step S7). Information on the location of the lesion may be recorded in association with the image, for example, in image recording unit 9. Step S7 may be performed at any timing before step S5.
[0031] In this case, the subsequent information generator 16 may generate other subsequent information using information on the position of the detected lesion. Figures 7B and 7C show other examples of subsequent information. The subsequent information D2 and D3 in Figures 7B and 7C are displays arranged around the reference image Itj and indicate the position of the lesion. The subsequent information D2 in Figure 7B is an arrow indicating the position of the lesion that will appear in an image following the reference image Itj. The subsequent information D3 in Figure 7C is a marker displayed at the position of the lesion and is displayed superimposed on the subsequent information D1. Such subsequent information D2 and D3 can notify the user that a lesion will appear in the image later, along with its position.
[0032] The subsequent information generator 16 performs processing such as deformation, rotation, and scaling on the lesion area based on the first deformation information and the second deformation information to generate subsequent information D2 and D3 that is aligned with the reference image Itj. The subsequent information is not limited to the above-described subsequent information D1, D2, and D3, and may be in other forms.
[0033] In this embodiment, the similar image extractor 14 may associate the reference image Itj with one image Itj' during insertion based on the insertion distance in addition to the similarity. As shown in Fig. 5D, during insertion and removal of the endoscope 6, the insertion distance detected by the sensor 8 is associated with a timestamp and recorded in the collateral information recording unit 10. The collateral information recording unit 10 is the memory 3, the storage unit 4, or any other arbitrary recording medium. Each image acquisition unit 11, 13 acquires an image with a timestamp.
[0034] The similar image extraction unit 14 acquires the insertion distance of the reference image Itj from the incidental information recording unit 10 based on the timestamp, then selects one or more inserted images with insertion distances equal to or close to the insertion distance of the reference image Itj based on the timestamp, and evaluates the similarity between the reference image Itj and each selected image. This reduces the time required to associate the reference image Itj compared to evaluating the similarity between the reference image Itj and all inserted images. Alternatively, once the association between the insertion time and the observation time has been completed, the images for which similarity is evaluated may be selected using timestamp information.
[0035] Second Embodiment Next, an image processing device, an image processing method, and an image processing program according to a second embodiment of the present invention will be described with reference to the drawings. This embodiment differs from the first embodiment in that it uses three-dimensional (3D) data of the large intestine A. In this embodiment, configurations that differ from the first embodiment will be described, and configurations that are common to the first embodiment will be assigned the same reference numerals and will not be described again.
[0036] Similar to the first embodiment, the image processing device 1 according to this embodiment includes a processor 2, a memory 3, a storage unit 4, and an input / output unit 5, and is applied to an endoscope system 100. As shown in Fig. 8 , 3D data G is data of the three-dimensional shape of the entire large intestine A acquired in advance by a three-dimensional imaging device before a colonoscopy, and is, for example, three-dimensional CT image data. The 3D data G is stored in the storage unit 4 or another predetermined recording medium before the colonoscopy.
[0037] 9A. The image processing method of this embodiment includes step S11 of acquiring 3D data G, step S12 of recording chronological images during insertion, step S13 of associating a reference image during removal with one of the chronological images during insertion, step S14 of generating subsequent information D1, and step S15 of displaying the reference image together with the subsequent information D1. The processor 2 executes steps S11 to S12 while the endoscope 6 is inserted, and executes steps S13 to S15 while the endoscope 6 is removed.
[0038] 10A shows functional units 11, 13, 16, 17, and 19 to 22 of the processor 2 that executes the image processing method. In step S11, the 3D data acquisition unit 19 acquires 3D data G from the storage unit 4 or a predetermined recording medium, and then the shape data acquisition unit 20 acquires the three-dimensional shape of the large intestine A from the 3D data G.
[0039] The next step S12 includes step S121 of calculating the positional relationship between each of the time-series images being inserted and the 3D data G, and step S122 of generating 3D data with image information by attaching the time-series images being inserted to the 3D data G. Therefore, the time-series images being inserted may be recorded as a part of the 3D data with image information.
[0040] Specifically, the first image acquisition unit 11 acquires images during insertion that are input from the endoscope 6 to the image processing device 1 through the input / output unit 5. Next, as shown in FIG. 8 , the first alignment unit 21 aligns the images during insertion with corresponding positions in the 3D data G (step S121) and assigns the images to the corresponding positions in the 3D data G (step S122). For example, the first alignment unit 21 may determine the corresponding positions of each image by comparing each image with each position in the 3D data G. Alternatively, the first alignment unit 21 may reconstruct a 3D model of the colon A from multiple images using a known technique such as SLAM (Simultaneous Localization and Mapping) and assign the reconstructed 3D model to the 3D data G. The above process is repeated during insertion of the endoscope 6, thereby generating 3D data with image information to which the texture of the colon A in the time-series images during insertion is attached.
[0041] Next, in step S13, the second image acquisition unit 13 acquires the current image Itj during removal, which is input from the endoscope 6 to the image processing device 1 through the input / output unit 5. Subsequently, the second alignment unit 22 sets the current image Itj as a reference image and associates the reference image Itj with one of the images being inserted based on the similarity between the reference image Itj and each image being inserted in the 3D data with image information. As in the first embodiment, the image to be associated is the similar image Itj' that has the highest similarity to the reference image Itj.
[0042] Next, in step S14, the subsequent information generation unit 16 selects a portion of the 3D data with image information based on the similar image Itj' and generates subsequent information D1 from the selected portion. The selected portion is a portion near the similar image Itj' and closer to the anus B than the similar image Itj'. The subsequent information D1 is an image of the area surrounding the field of view of the reference image Itj, as shown in FIG. 7A . Next, in step S15, the display image generation unit 17 generates a display image F and outputs it to the display unit 7, similar to step S6 in the first embodiment.
[0043] Thus, according to this embodiment, subsequent information D1 is generated using images acquired during insertion before removal, and the subsequent information D1 is displayed on the display unit 7 together with the current image Itj. Therefore, based on the subsequent information D1, the doctor can predict in advance the area of the large intestine A that will appear in the image later. In this way, by presenting the doctor with additional subsequent information D1 in addition to the current image Itj, the doctor's diagnosis can be effectively supported, thereby reducing the chance of overlooking a lesion.
[0044] 10B and 10C, processor 2 may further include, as a functional unit, lesion detector 18 that detects lesions using CADe. In this case, subsequent information generator 16 may generate subsequent information D2 and D3 (see FIGS. 7B and 7C) indicating the positions of lesions, similar to the first embodiment.
[0045] The lesion detection unit 18 may detect lesions in each image during insertion (see FIG. 10B ), in the 3D data with image information (see FIG. 10C ), or in both the images during insertion and the 3D data with image information. Therefore, step S16 for detecting lesions may be performed at any timing before step S14 (see FIG. 9B ). In the case of FIG. 10B , the position of each lesion is recorded on any recording medium in association with the image. The first alignment unit 21 generates 3D data with image information including the lesion information by adding lesion information to the corresponding position in the 3D data G based on the recorded position.
[0046] The images of the endoscope 6 and the 3D data G each have advantages and disadvantages when it comes to detecting lesions. For example, the images of the endoscope 6 can detect lesions with little unevenness, such as small polyps less than 5 mm, based on color, texture, and the like. On the other hand, it is difficult to detect lesions with little unevenness, such as polyps less than 5 mm, using the 3D data G. Therefore, the configurations of Figures 10B and 10C can accurately detect lesions with little unevenness.
[0047] Furthermore, because the field of view of the endoscope 6 is limited, missing areas may occur in the time-series images taken during insertion. Lesions present in the missing areas cannot be detected by CADe. Meanwhile, 3D data G, such as 3D CT images, includes the entire large intestine A. Therefore, the configuration of FIG. 10C not only enables the endoscope to detect lesions with small irregularities, but also enables accurate detection of lesions in areas that are difficult to capture with an endoscope.
[0048] 10D , lesion detection unit 18 may detect lesions within 3D data G. For example, lesion detection unit 18 may use known techniques to detect lesions from the three-dimensional shape of large intestine A acquired by shape data acquisition unit 20. The position of each detected lesion is recorded together with 3D data G in storage unit 4 or a predetermined recording medium.
[0049] In the configuration of FIG. 10D , the subsequent information generator 16 may select a portion of the 3D data G based on the similar image Itj′ and generate subsequent information D1, D2, and D3 from the selected portion of the 3D data G. The selected portion is a portion near the similar image Itj′ and closer to the anus B than the similar image Itj′. In this case, the first positioning unit 21 determines the position of each image during insertion in the 3D data G, but does not necessarily generate 3D data with image information. The second positioning unit 22 may determine the similar image Itj′ based on the similarity between the reference image Itj and each image during insertion recorded in the image recording unit 9.
[0050] The lesion in the 3D data G may be detected by a user such as a doctor instead of the processor 2. In this case, the user registers the position of the lesion in the 3D data G, for example, by attaching a marker to the position of the lesion in the 3D data G, and records the 3D data G with the registered position of the lesion in the storage unit 4 or a predetermined recording medium.
[0051] In this embodiment, the second alignment unit 22 may associate the reference image Itj with one image taken during insertion based on the insertion distance in addition to the similarity. As shown in Fig. 10E, during insertion and removal of the endoscope 6, the insertion distance detected by the sensor 8 is associated with a timestamp and recorded in the incidental information recording unit 10. Each image acquisition unit 11, 13 acquires an image with a timestamp.
[0052] The second alignment unit 22 acquires the insertion distance of the reference image Itj from the incidental information recording unit 10 based on the timestamp, then selects one or more currently inserted images with insertion distances equal to or close to the insertion distance of the reference image Itj based on the timestamp, and evaluates the similarity between the reference image Itj and each selected image. This reduces the time required to associate the reference image Itj compared to evaluating the similarity between the reference image Itj and all currently inserted images. Alternatively, once the association between the insertion time and the observation time has been completed, the images for which similarity is evaluated may be selected using timestamp information.
[0053] While the embodiments and modifications of the present invention have been described above, the present invention is not limited to these, and modifications can be made as appropriate without departing from the spirit of the present invention. For example, the image processing device and image processing method of the present invention can be applied to endoscopic examination of other examination objects, such as tubular examination objects, in addition to the large intestine, where the insertion and withdrawal path of the endoscope is the same.
[0054] 1 Image processing device 2 Processor 4a Image processing program 8 Sensor A Large intestine (test object) D1, D2, D3 Subsequent information F Display image Itj Reference image G Three-dimensional data
Claims
1. An image processing device comprising: a processor, which records a time series of images acquired by an endoscope during insertion into an object of examination; matches the reference image with one image in the time series of images based on a similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope during removal from the object of examination; generates subsequent information based on the one matched image, the subsequent information including information of the object of examination in an image predicted to be acquired by the endoscope during removal after the reference image; and displays the subsequent information together with the reference image.
2. The image processing device according to claim 1, wherein the processor generates three-dimensional data with image information by attaching the time series image being inserted to previously acquired three-dimensional data of the object of inspection, associates the reference image with one image in the time series of images attached to the three-dimensional data with image information, selects a portion of the three-dimensional data with image information based on the associated image, and generates the subsequent information from the selected portion of the three-dimensional data.
3. The image processing device according to claim 1, wherein the processor selects a portion of previously acquired three-dimensional data of the object to be inspected based on the one associated image, and generates the subsequent information from the selected portion of the three-dimensional data.
4. The image processing device of claim 1, wherein the processor: obtains first transformation information which is transformation information between images in the time series being inserted; obtains second transformation information which is transformation information between the reference image and the one associated image; selects at least one image in the time series based on the one associated image; and generates the subsequent information from the at least one selected image using the first transformation information and the second transformation information.
5. The image processing device of claim 1, wherein the processor further corresponds the reference image to an image in the time series of images based on an insertion distance of the endoscope.
6. The image processing apparatus according to claim 1, wherein the subsequent information is an image of a region around the reference image and disposed around the reference image's field of view.
7. The image processing device of claim 1, wherein the processor further detects a lesion in the time series of images, and the subsequent information is a representation arranged around the reference image and indicating the location of the lesion.
8. The image processing device according to claim 2, wherein the processor further detects a lesion within the three-dimensional data with image information, and the subsequent information is a display arranged around the reference image and indicating the position of the lesion.
9. The image processing device of claim 3, wherein the processor further detects a lesion within the three-dimensional data, and the subsequent information is a representation disposed around the reference image and indicating the location of the lesion.
10. An image processing method comprising: recording a time series of images acquired by an endoscope during insertion into an object to be examined; matching the reference image with one image in the time series of images based on a similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope during removal from the object to be examined; generating subsequent information based on the one matched image, the subsequent information including information of the object to be examined in an image predicted to be acquired by the endoscope during removal after the reference image; and displaying the subsequent information together with the reference image.
11. An image processing program that causes a processor to execute a process including: recording a time series of images acquired by an endoscope during insertion into an object to be examined; matching the reference image with one image in the time series of images based on a similarity between the reference image and each of the images in the time series, the reference image being an image acquired by the endoscope during removal from the object to be examined; generating subsequent information based on the one matched image, the subsequent information including information of the object to be examined in an image predicted to be acquired by the endoscope during removal after the reference image; and displaying the subsequent information together with the reference image.
Citation Information
Patent Citations
Endoscopic system
JP2012170774A
Endoscope device, endoscope operation method, and program
WO2019220848A1
Endoscope system and method for operating same
WO2022014077A1
