Image processing device, method of operating the image processing device, and image processing program

Abstract

To perform image registration by highly accurate positioning of a whole subject and to correct a movement of visual field and deformation of the subject correctly.SOLUTION: An image processing apparatus includes a processor. The processor is configured to: set a plurality of tracking points uniformly arranged in a first image that is one of time series of images; track each of the tracking points from the first image to a second image, and determine the success or failure in tracking each of the tracking points, the second image being an image at a different time from the first image among the time series of images; estimate an image plane of the second image by perspective projection transformation using the tracking points tracked successfully; correct positions of the tracking points failed to be tracked on the second image by back projection to the image plane; and perform image registration on the second image based on the plurality of tracking points to generate a registration image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Conventionally, blood flow assessment using a fluorescent contrast agent such as ICG (indocyanine Green) has been widely known and practiced (see, for example, Patent Document 1). In an organ, the fluorescence intensity from the fluorescent contrast agent gradually increases with blood flow. Blood flow is evaluated based on Tmax, which is the time from the rise of the fluorescence intensity to the maximum intensity.

[0003] On the other hand, a technique for tracking a non-rigid object in an image is known (see, for example, Patent Document 2). Patent Document 2 describes a system and method for modeling non-rigid motion of an object using thin-plate spline (TPS) transformation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of Invention

Problem to be Solved by the Invention

[0005] In blood flow assessment, the fluorescence intensity at the same position of an organ is measured over a certain period of time (for example, 1 minute to 2 minutes). However, living organs are deformed by pulsation and the like. Further, when an organ is observed with a handheld endoscope, the field of view of the endoscope shifts due to variations in the distance and angle between the endoscope and the organ. Therefore, it is difficult to continue measuring the fluorescence intensity at the same position.

[0006] One technique for measuring the same location on a subject is to track a region of interest (ROI) set within the image. In this method, the areas where an ROI can be set are limited to areas with distinctive features such as surgical instruments and organs, making it difficult to track an arbitrary location. Furthermore, the longer the tracking time, the higher the probability of tracking failure; for example, it is difficult to continuously track an ROI for a long period of time, such as 1 to 2 minutes. Another technique for tracking subjects is the image deformation method described in Patent Document 2. In this method, multiple control points set within the image are tracked, and the image is deformed based on these control points. However, similar to ROI, it is difficult to continuously track control points over a long period of time. [Means for solving the problem]

[0007] One aspect of the present invention is an image processing apparatus for processing a time-series image of a subject, comprising a processor, which sets a plurality of uniformly arranged tracking points in a first image which is one of the time-series images, tracks each of the tracking points between the first image and a second image, determines whether the tracking of each of the tracking points is successful, the second image is an image of a different time from the first image in the time-series image, and uses the tracking points that were successfully tracked. Project the second image onto the first image using perspective projection. The image plane of the second image is estimated by perspective projection transformation, and the position of the tracking point that failed to track on the second image is determined. From the first image above This image processing device generates a registration image by correcting the second image by back projection onto the image plane and performing image registration on the second image based on the plurality of tracking points to align the second image with the first image.

[0008] Another aspect of the present invention involves processing time-series images of a subject. How to operate an image processing device And, The image processing device includes a processor, and the processor isMultiple uniformly arranged tracking points are set within a first image, which is one of the time-series images. Each tracking point is tracked between the first image and the second image, and the success or failure of tracking each tracking point is determined. The second image is an image from a different time point in the time-series images than the first image, and the tracking points that were successfully tracked are used. Project the second image onto the first image using perspective projection. The image plane of the second image is estimated by perspective projection transformation, and the position of the tracking point that failed to track on the second image is determined. From the first image above The process includes generating a registration image by correcting the second image by back projection onto the image plane and performing image registration on the second image based on the plurality of tracking points to align the second image with the first image. How to operate an image processing device That is the case.

[0009] Another aspect of the present invention is an image processing program for processing a time-series of images of a subject, wherein a plurality of uniformly arranged tracking points are set in a first image which is one of the time-series images, each of the tracking points is tracked between the first image and a second image, the success or failure of tracking each of the tracking points is determined, the second image is an image from the time-series of images at a different time than the first image, and the tracking points that were successfully tracked are used Project the second image onto the first image using perspective projection. The image plane of the second image is estimated by perspective projection transformation, and the position of the tracking point that failed to track on the second image is determined. From the first image above This image processing program causes a computer to perform image registration on the second image based on the plurality of tracking points, correcting the second image by back projection onto the image plane and aligning the second image with the first image, thereby generating a registration image. [Brief explanation of the drawing]

[0010] [Figure 1] This is an overall system configuration diagram relating to one embodiment of the present invention. [Figure 2] This is a block diagram of an image processing device according to one embodiment of the present invention. [Figure 3] This is a flowchart of the image processing method performed by the image processing device. [Figure 4A] This is a flowchart of step S1 for setting tracking points. [Figure 4B] This is a flowchart of step S2 for tracking tracking points. [Figure 4C] This is a flowchart of step S9 for analyzing time-series registration images. [Figure 5A] This is a diagram showing a target image on which a plurality of tracking points are set. [Figure 5B] This is a diagram showing a source image that has elapsed in time from the target image. [Figure 5C] This is a diagram showing a registration image of a fluorescence image paired with the source image of Fig. 5B. [Figure 6A] This is a diagram showing tracking points on the source image. [Figure 6B] This is a diagram showing the tracking points on the source image after correcting the positions of the tracking points that failed to be tracked. [Figure 7] This is a diagram for explaining perspective projection transformation. [Figure 8] This is a diagram showing an example of time-varying data of fluorescence intensity obtained from time-series registration images. [Figure 9] This is a table showing an example of the relationship between a subject and a surgical procedure, and tracking conditions. [Mode for Carrying Out the Invention]

[0011] Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to an embodiment of the present invention will be described with reference to the drawings. Figure 1 shows an example of a system 100 to which an image processing apparatus 1 according to the present embodiment is applied. The system 100 is an endoscope system including an endoscope 2, an observation apparatus 3, a measurement apparatus 4, and a display apparatus 5, and the image processing apparatus 1 is mounted on the measurement apparatus 4. The observation apparatus 3 and the measurement apparatus 4 mutually transmit and receive signals such as video signals and control signals. The measurement apparatus 4 is connected to servers such as a physical server 6 and a cloud server 7, and analysis results generated by the measurement apparatus 4 and the image processing apparatus 1 may be stored in the servers 6 and 7. The measurement apparatus 4 may be integrated with the observation apparatus 3.

[0012] The endoscope 2 irradiates a subject in a living body with white light and excitation light simultaneously or alternately, and photographs the subject illuminated with the white light and the subject illuminated with the excitation light simultaneously or alternately. Thereby, the endoscope 2 chronologically acquires and outputs pairs of a white light image and a fluorescent image at the same time or substantially the same time. The subject is an organ in a living body to which a fluorescent contrast agent such as ICG is administered for blood flow evaluation. The white light image is an image obtained by capturing reflected white light from the subject, and the fluorescent image is an image obtained by capturing fluorescence from the fluorescent contrast agent excited by the excitation light.

[0013] Figure 2 shows a block diagram of the image processing apparatus 1. The image processing apparatus 1 includes an input unit 11, a processor 12 such as a central processing unit, a memory 13, a storage unit 14, and an output unit 15. The input unit 11 has a known input interface and is connected to the observation apparatus 3. Pairs of the white light image and the fluorescent image output from the endoscope 2 are chronologically input to the image processing apparatus 1 through the input unit 11. The output unit 15 has a known output interface and is connected to the display apparatus 5. The pairs of the white light image and the fluorescent image are chronologically output from the output unit 15 to the display apparatus 5 and displayed on the display apparatus 5.

[0014] The memory 13 is composed of a volatile storage device such as a RAM, and is used as a work area for the processor 12. The storage unit 14 consists of a computer-readable non-temporary recording medium, such as ROM, flash memory, or a hard disk drive. The storage unit 14 stores an image processing program 14a that causes the processor 12 to execute an image processing method. The processor 12 processes pairs of white light images and fluorescence images that are input to the image processing device 1 in chronological order according to the image processing program 14a and analyzes the blood flow of the subject S.

[0015] Next, we will describe the image processing method performed by processor 12. The processor 12 executes an image processing method in response to a start trigger. The start trigger is input to the image processing device 1, for example, by the user operating an input device 8 such as a switch provided on the measuring device 4. The user inputs the start trigger to the image processing device 1 after administering a fluorescent contrast agent to the body.

[0016] As shown in Figure 3, the image processing method according to this embodiment includes the steps of: S1 setting tracking points P1, P2, ..., Pn in a target image A; S2 tracking each tracking point Pi (i=1,2, ...,n) between the target image A and the source image B; S3 determining the success or failure of tracking each tracking point Pi; S4 estimating the image plane of the source image B; S5 correcting the position of tracking points Pi that failed to be tracked; S6 generating a registration image of the source image B; S7 updating the tracking template for tracking tracking points Pi that failed to be tracked; S8 storing the registration image C; S9 analyzing blood flow using the registration image C; S10 creating analysis results; and S11 outputting the analysis results.

[0017] As shown in Figure 5A, the target image (first image) A is one of the images of the subject S input to the image processing device 1 in time series, and in this embodiment it is a white light image. As shown in Figure 5B, the source image (second image) B is an image of the subject S input to the image processing device 1 in time series, taken at a different time than the target image A, and in this embodiment it is a white light image and a fluorescence image.

[0018] The processor 12 sets a plurality of tracking points P1, P2, ..., Pn, which are uniformly arranged in a predetermined array pattern, within the target image A (step S1). Specifically, as shown in Figure 4A, the processor 12 determines the first white light image after the input of a start trigger to be the target image A (step S101). Then, as shown in Figure 5A, the processor 12 determines the positions of the multiple tracking points P1, P2, ..., Pn to be the intersections of grid lines arranged at equal intervals in the vertical and horizontal directions of the target image A (step S103). Thus, the multiple tracking points P1, P2, ..., Pn are arranged in a square with equal intervals in the vertical and horizontal directions.

[0019] The white light image includes a solid black mask region E outside the field of view D of the endoscope 2, which includes the subject S. Preferably, the processor 12 detects the mask region E in the target image A, for example, based on color (step S102), and sets a number of tracking points P1, P2, ..., Pn over all or substantially all of the other region excluding the mask region E (step S103). In Figure 5A, the small white circles indicate the tracking points. The processor 12 may uniformly set multiple tracking points P1, P2, ..., Pn within a measurement area F set within the target image A. The measurement area F preferably includes the entirety or a wide area of ​​the subject S within the target image A, and may be set by the user.

[0020] After step S1, the processor 12 tracks each tracking point Pi between the target image A and the source image B by template matching (step S2). The source image B used for tracking is a white light image taken some time after the target image A. When the endoscope 2 is handheld, it is difficult to keep the endoscope 2 in a constant position relative to the subject S for a long period of time, and the field of view of the endoscope 2 moves three-dimensionally relative to the subject S. In addition, the subject S, which is an organ in a living body, is deformed by pulsation, etc. Therefore, as shown in Figure 5B, the subject S in source image B may be displaced relative to the subject S in target image A.

[0021] In a specific example, as shown in Figure 4B, the processor 12 creates a template for each tracking point Pi from the target image A (step S22). Each template is a small region within the target image A and contains one tracking point Pi. Next, the processor 12 determines the tracking points P1, P2, ..., Pn in the source image B by template matching using templates (step S24). Specifically, the processor 12 searches for the matching region in the source image B that is most similar to each template, and determines the position in the matching region corresponding to the tracking point Pi in each template as the tracking point Pi in the source image B.

[0022] The processor 12 may perform image processing such as structure enhancement and noise reduction on the target image A and source image B before steps S22 and S24 (steps S21 and S23). By using the processed target image A and source image B in steps S22 and S24, the accuracy of tracking can be improved. In step S24, the processor 12 may track each tracking point Pi using a method other than template matching, for example, optical flow.

[0023] Tracking of the tracking point Pi may fail due to the aforementioned shift in the field of view, deformation of the subject S, and occlusion. The likelihood of tracking failure increases, especially when tracking for extended periods, such as 1 to 2 minutes. After step S2, the processor 12 determines the success or failure of tracking each tracking point Pi based on the similarity between the template and the matching region in the source image B. Specifically, as shown in Figure 6A, the processor 12 determines that tracking was successful if the similarity is greater than or equal to a predetermined first threshold Th1, and determines that tracking failed if the similarity is less than the threshold Th1. In Figures 6A and 6B, white circles indicate successful tracking points, and black circles indicate unsuccessful tracking points.

[0024] The ZNCC (Zero-means Normalized Cross Correlation) formula shown below may be used as the similarity measure. ZNCC is an evaluation value of image similarity calculated using normalized cross-correlation. The closer the ZNCC is to 1, the higher the similarity. The threshold Th1 is, for example, 0.8.

number

[0025] After step S3, as shown in Figure 7, the processor 12 estimates the image plane α of source image B by fluoroscopic projection transformation using only the success points (step S4). The fluoroscopic projection transformation is performed based on the following equation using the intrinsic and extrinsic parameters of the camera of endoscope 2.

number

[0026] (X,Y,Z) are 3D coordinates in the world coordinate system Σw, and (u,v) are coordinates on the image plane α of the source image. cx,cy are the camera principal points (usually the center of the image), and fx,fy are the focal lengths expressed in pixels. A matrix containing the intrinsic parameters cx,cy,fx,fy is either an intrinsic parameter matrix or a camera matrix, and if the focal lengths fy,fy are invariant, the camera matrix does not change. The matrix containing the external parameters r and t is an external parameter matrix, which is a translational rotation matrix, and transforms the coordinates (X, Y, Z) into the camera coordinate system Σcamera. The external parameter matrix represents the camera's motion relative to a static environment, or the rigid body motion of the subject relative to the camera. Estimating the image plane α of source image B means determining the external parameter matrix, which in turn determines the camera's position and orientation at the time source image B was acquired.

[0027] After step S4, as shown in Figure 6B, the processor 12 corrects the position of the failure point on the source image B by backprojecting the failure point onto the estimated image plane α (step S5). Specifically, the processor 12 uses the matrix and camera position estimated in step S4 to backproject the failure point in the target image A onto the image plane α, and corrects the position of the failure point on the source image B to the backprojected position. As a result, the tracking point Pi that failed to track returns to the appropriate position on the source image B, specifically at the intersection of the grid lines.

[0028] After step S5, if the similarity falls below a predetermined second threshold Th2, the processor 12 updates the template for failure points to a small region within the target image A containing the corrected tracking point Pi (step S7). The second threshold Th2 is a value less than or equal to the first threshold Th1. For example, if the first threshold Th1 is 0.9, the second threshold Th2 is a value in the range of 0.8 to 0.9. Therefore, if the tracking is successful, the processor 12 uses the same template for the next tracking. On the other hand, if the tracking fails, the processor 12 uses the updated template for the next tracking. This updates the template in response to changes in the color and texture of the subject S, and even after a tracking failure, the updated template can still be used to track the tracking point Pi.

[0029] After step S5, the processor 12 generates a registration image C of source image B by performing image registration on source image B based on multiple tracking points P1, P2, ..., Pn (step S6). The source image B on which image registration is performed is a fluorescence image that is paired with the white light image used as source image B in steps S2 to S5. Image registration is a process that aligns the subject S in source image B with the subject S in target image A by deforming source image B. As shown in Figure 5C, image registration generates a registration image C in which the positional shift of the subject S in source image B relative to target image A due to the movement of the field of view and the deformation of the subject S is corrected or reduced. In Figure 5C, the hatched areas indicate the fluorescence areas.

[0030] Image registration is performed using a TPS (Thin Plate Spline) transformation with tracking points P1, P2, ..., Pn as control points. Specifically, the processor 12 sets multiple tracking points P1, P2, ..., Pn within the fluorescence image. The positions of the tracking points P1, P2, ..., Pn within the fluorescence image are the same as the positions of the tracking points P1, P2, ..., Pn within the white light image after the positions of failure points have been corrected. Next, the processor 12 generates a registration image C by non-linearly distorting the fluorescence image so that the tracking point Pi within the fluorescence image coincides with the corresponding tracking point Pi within the target image A. After step S6, the processor 12 stores the generated registration image C in memory 13 or storage unit 14 (step S8).

[0031] Since TPS conversion is computationally intensive, other methods, such as image registration using perspective projection, may be used. Furthermore, the processor 12 may perform various image processing operations on each source image B, which is a fluorescence image, such as area gain correction, shading correction, and conversion from RGB to IR, and then perform image registration on the fluorescence image after image processing.

[0032] The processor 12 repeats steps S1 to S8 each time a new pair of white light images and fluorescence images is input to the image processing device 1, thereby accumulating a time-series registration image C in the memory 13 or storage unit 14. The processor 12 terminates the processing in steps S1 to S8 in response to the termination trigger (YES in step S12) and proceeds to the next step S9. For example, the termination trigger may be input to the image processing device 1 by the user operating the input device 8, or the termination trigger may be used when a predetermined measurement time has elapsed since the start trigger.

[0033] The processor 12 analyzes blood flow using the accumulated time-series registration images C (step S9). Specifically, as shown in Figure 4C, the processor 12 samples the fluorescence intensity at each location in the time-series registration image C (step S91). The processor 12 may also sample the average fluorescence intensity of each region (e.g., 8 pixels × 8 pixels) in the registration image C. This provides data on the time evolution of fluorescence intensity for each location or region of the subject S in the target image A, as shown in Figure 8.

[0034] Next, the processor 12 calculates an evaluation value of blood flow for each position of subject S in the target image A using fluorescence intensity data (step S93). An example of an evaluation value is Tmax, the length of time from start time T1 to maximum time T2. Start time T1 is the time when the fluorescence intensity rises, and maximum time T2 is the time when the fluorescence intensity is at its maximum. Another example of an evaluation value is I, the integral value of fluorescence intensity over the measurement time. The processor 12 may perform various data processing on the fluorescence intensity data, such as noise reduction, smoothing using a moving average, and normalization of fluorescence intensity (step S92), and use the processed data in step S93. Next, the processor 12 creates a color map that represents the magnitude of the evaluation values ​​for each position or region of the subject S within the target image A using color (step S94).

[0035] , After step S9, the processor 12 creates at least one analysis result for display on the display device 5 (step S10). The analysis result is, for example, an overlay image in which a color map is superimposed on the target image A. The analysis result may be just a color map, or it may be a graph showing the time change of fluorescence intensity at any position in the target image A. The analysis result may be an image showing blood flow information at each position of the subject S. The blood flow information may be, for example, time Tmax, or time-averaged blood flow calculated from the integral value I. After step S10, the processor 12 outputs the analysis results to the display device 5 and displays them on the display device 5 (step S11). Based on the analysis results, the user can recognize the blood flow at each location of the subject S.

[0036] Thus, according to this embodiment, a time-series registration image C is generated from a time-series source image B acquired after the target image A. Each registration image C is an image in which the position of the subject S has little to no error relative to the target image A. By using such a time-series registration image C, the fluorescence intensity at the same position of the subject S can be measured over a certain period of time, and the blood flow at each position of the subject S can be accurately analyzed.

[0037] Furthermore, according to this embodiment, the multiple tracking points P1, P2, ..., Pn used for image registration are automatically set within the target image A by the processor 12. In other words, the user does not need to set tracking points such as ROI (region of interest).

[0038] Furthermore, according to this embodiment, after tracking of tracking point Pi fails, the position of tracking point Pi on the source image B is corrected by back projection, and then the position-corrected tracking point Pi is tracked using the updated template. This allows all tracking points P1, P2, ..., Pn to be tracked over a long period of time (e.g., 1 to 2 minutes) without the tracking breaking down midway. In order to accurately align source image B with target image A using image registration, it is necessary to accurately track tracking points P1, P2, ..., Pn across the entire subject S within target image A. If the position of failure point Pi is not corrected, it will be impossible to accurately track a tracking point Pi that has failed to track once, resulting in a decrease in the accuracy of image registration.

[0039] Furthermore, in conventional tracking methods that track feature points as tracking points, the locations where tracking points can be set are limited to the locations of characteristic structures of the treatment instrument 9 and the subject S, and it is not possible to set tracking points at arbitrary locations within the image. Therefore, the locations where blood flow can be analyzed are also limited to locations where feature points can be set. In contrast, according to this embodiment, multiple tracking points P1, P2, ..., Pn are uniformly set within the target image A, regardless of the presence or absence of features. This makes it possible to track the entire subject S within the target image A, and to perform high-precision positioning of the entire subject S by image registration. Furthermore, this makes it possible to analyze the blood flow of the entire subject S within the target image A.

[0040] Furthermore, according to this embodiment, only tracking points that failed to track are corrected in position, while tracking points on the source image B that were successfully tracked are used for image registration without any positional correction. Therefore, as shown in Figure 6B, the tracking points that were successfully tracked have slight positional shifts due to the movement of the field of view and the deformation of the subject S. By performing image registration using such tracking points, the movement of the field of view and the deformation of the subject S can be corrected even more accurately.

[0041] Furthermore, the endoscope 2 can move not only perpendicular to the optical axis and rotate around the optical axis, but also in the forward and backward direction along the optical axis, and the field of view can be greatly enlarged or reduced by movement in the forward and backward direction. Five-axis image stabilization, which is installed in digital cameras and the like, can correct the movement and rotation of the field of view due to movement perpendicular to the optical axis and rotation around the optical axis, but it cannot correct the enlargement or reduction of the subject S. According to this embodiment, perspective projection transformation estimates not only the movement and rotation of the image plane α of the source image B in the direction along the image plane α, but also the distance from the camera position to the image plane α. Therefore, the positional shift of the tracking point Pi on the source image B due to the scaling of the subject S can also be corrected.

[0042] Furthermore, tracking of tracking points within the mask region E is unnecessary, and the size and shape of the mask region E differ depending on the endoscope 2. According to this embodiment, the amount of computation and time required for tracking can be reduced by automatically detecting the mask region E within the target image A and setting tracking points P1, P2, ..., Pn in the area excluding the mask region E.

[0043] In the above embodiment, the processor 12 may acquire information about the subject S or surgical procedure and change the tracking conditions for tracking points based on the information. The tracking conditions include at least one of the following: a threshold Th1 (success value) for determining the success or failure of tracking, a threshold Th2 (update determination value) for updating the template, and the arrangement of tracking points. The arrangement of tracking points includes at least one of the following: the spacing or number of tracking points in the horizontal and vertical directions of the target image A, and the size of the template. Figure 9 shows an example of the relationship between the subject S and the surgical procedure and the conditions. In this case, the image processing device 1 may include an acquisition unit for acquiring the aforementioned information. The acquisition unit is, for example, an input device 8, and the user can input at least one of the type of subject S and the surgical procedure to the image processing device 1 using the input device 8.

[0044] In the above embodiment, the image processing device 1 processes time-series white light images and fluorescence images that are input to the image processing device 1 in real time. However, instead, it may process time-series white light images and fluorescence images acquired in the past. For example, the image processing device 1 may process a video of the subject S recorded in the memory 13 or storage unit 14.

[0045] In the above embodiment, the source image was assumed to be an image acquired after the target image, but instead, it may be an image acquired before the target image. For example, when generating a time-series registration image C using recorded video, one white light image at an arbitrary time can be selected as the target image, and white light images and fluorescence images at other time points can be used as source images.

[0046] In the above embodiment, the image processing device 1 analyzes blood flow using the registration image C of the fluorescence image, but steps S9 to S10 may be omitted. In this case, the image processing device 1 may display the time-series registration image C on the display device 5 instead of the analysis results. By viewing the time-series registration image C, the user can recognize the precise time changes in fluorescence intensity at each position of the subject S and accurately evaluate the blood flow at each position.

[0047] In the above embodiment, the processor 12 generates a registration image C of a fluorescence image, but instead of this, or in addition thereto, it may generate a registration image of a white light image. In this case, instead of or in addition to steps S9 to S11, the processor 12 may perform analysis of the subject S using the registration image of the white light image.

[0048] In the above embodiment, the processor 12 sets a plurality of tracking points P1, P2, ..., Pn arranged in a square within the target image A. However, the arrangement of the plurality of tracking points P1, P2, ..., Pn within the target image A is not limited to this and may be in other arrangements. For example, the plurality of tracking points P1, P2, ..., Pn may be geometrically arranged at each vertex of any polygon. Furthermore, while it is preferable for the plurality of tracking points P1, P2, ..., Pn to be arranged regularly at equal intervals, they may also be arranged irregularly at unequal intervals.

[0049] Although embodiments and modifications of the present invention have been described above, the scope of the present invention is not limited thereto, and various improvements are possible without departing from the spirit of the invention. For example, the target image and source image may be of a type other than a white light image and a fluorescence image. Also, the target image and source image may be an image other than an endoscopic image, and the image processing device 1 may be applied to a system other than the endoscopic system 100. [Explanation of Symbols]

[0050] 1 Image processing device 5 Display device 12 processors 14. Memory Unit (Recording Medium) Pi tracking point S Subject A Target image (first image) B. Source image (second image) C Registration Image E Mask Area α Image plane

Claims

1. An image processing device that processes time-series images of a subject, Equipped with a processor, The processor is, Multiple tracking points arranged uniformly are set within the first image, which is one of the time-series images. Each of the aforementioned tracking points is tracked between the first image and the second image, and the success or failure of tracking each of the aforementioned tracking points is determined, wherein the second image is an image from a different time point in the time series than the first image. The image plane of the second image is estimated by a perspective projection transformation that projects the tracked points, which were successfully tracked, onto the first image from the second image. The position of the tracking point that failed to track on the second image is corrected by back projection from the first image onto the image plane. An image processing apparatus that generates a registration image by performing image registration on the second image based on the plurality of tracking points to align the second image with the first image.

2. The aforementioned processor further, Obtain information regarding the subject or surgical procedure, Based on the aforementioned information, set the conditions for tracking the tracking point. The image processing apparatus according to claim 1, wherein the condition includes at least one of the following: a first threshold for determining the success or failure of the tracking based on the similarity between a template representing the region for each tracking point in the first image and a matching region in the second image that is similar to the template; a second threshold for updating the template to the corrected region of the tracking point in the second image when the similarity falls below a second threshold that is smaller than the first threshold; and the arrangement of the plurality of tracking points.

3. The aforementioned time-series images include a time-series white light image of the subject, The image processing apparatus according to claim 1, wherein the processor tracks the plurality of tracking points using the white light image as the first image and the second image, estimates the image plane, and corrects the position of the tracking points that failed to be tracked.

4. The aforementioned time-series images further include time-series fluorescence images of the subject, The aforementioned processor, The image registration is performed using the fluorescence image as the second image. The image processing apparatus according to claim 3, which analyzes the blood flow of the subject based on the fluorescence intensity of the time-series registration images.

5. The aforementioned processor further, An overlay image is generated in which the blood flow analysis results are superimposed on the first image. The image processing apparatus according to claim 4, which causes the overlay image to be displayed on a display device.

6. The image processing apparatus according to claim 3, wherein the processor detects a mask region in the first image and sets the plurality of tracking points in the region excluding the mask region.

7. A method for operating an image processing device that processes time-series images of a subject, The image processing device includes a processor, The processor, Multiple tracking points arranged uniformly are set within the first image, which is one of the time-series images. Each of the aforementioned tracking points is tracked between the first image and the second image, and the success or failure of tracking each of the aforementioned tracking points is determined, wherein the second image is an image from a different time point in the time series than the first image. The image plane of the second image is estimated by a perspective projection transformation that projects the tracked points, which were successfully tracked, onto the first image from the second image. The position of the tracking point that failed to track on the second image is corrected by back projection from the first image onto the image plane. A method for operating an image processing device, comprising generating a registration image by performing image registration on the second image based on the plurality of tracking points to align the second image with the first image.

8. An image processing program that processes time-series images of a subject, Multiple tracking points arranged uniformly are set within the first image, which is one of the time-series images. Each of the aforementioned tracking points is tracked between the first image and the second image, and the success or failure of tracking each of the aforementioned tracking points is determined, wherein the second image is an image from a different time point in the time series than the first image. The image plane of the second image is estimated by a perspective projection transformation that projects the tracked points, which were successfully tracked, onto the first image from the second image. The position of the tracking point that failed to track on the second image is corrected by back projection from the first image onto the image plane. An image processing program that causes a computer to generate a registration image by performing image registration on the second image based on the plurality of tracking points to align the second image with the first image.

Images