Method for determining the image position of a marker point in an image of an image sequence

Abstract

Method for determining the image position of a marker point (3) in an image of an image sequence, characterized by the following steps: defining (S2) a defined marker point (3) in a first image (1) of the image sequence; determining (S4) a transformation at least between corresponding sub-areas of the first image (1) and a second image (4) of the image sequence; transforming (S5) at least the sub-area of ​​the second image (4) using the determined transformation, such that the transformed sub-area of ​​the second image (4') corresponds to the first image (1) in orientation and scaling; checking (S12) whether the image position of the marker point (3) lies within the second image (4), wherein the image position of the marker point (3) is first transferred (S11) to the transformed sub-area of ​​the second image (4') and then transferred to the second image (4) using the transformation;if the image position of the marker point (3) lies within the second image (4): Locating (S6) the marker point (3) in the transformed sub-area of ​​the image (4') and transferring (S7) the localized marker point into the second image (4) using the specified transformation.;

Description

Method for determining the image position of a marker point in images of an image sequence. In known methods, at least one point is marked in a first image. Such a marked point is subsequently referred to as a marker point. Then, corresponding image components are determined in the first image of the image sequence and in a further image of the sequence. This can be done using known object recognition, edge detection, or similar image processing techniques. However, such methods have the disadvantage that identifying the pixel or image components in the second image is difficult if, for example, the recording has been rotated or has changed significantly in some other way. This can happen, for example, during endoscopy when rapid, jerky movements are performed or the lighting changes drastically. In another method, a transformation is determined that converts the image components of the first image into the image components of the second image. Using this transformation, it is common practice to transform the marked image point into the second image. However, a disadvantage of this method is the low accuracy in identifying the marked point, especially with non-planar objects. Thomas Gross; Navya Amin; Marvin C. Offiah; Susanne Rosenthal; Nail El-Sourani; Markus Borschbach: Optimization of endoscopic video stabilization by local motion exclusion. In: Proceedings of the 9th International Conference on Computer Vision Theory and Applications (VISAPP; 2014-3), January 5-8, 2014, pp. 64-72. - ISBN 978-989-758-009-3. This work focuses primarily on the development of stabilization algorithms for endoscopic videos, which exhibit the distortions typical of endoscopy. The focus is on optimizing the motion detection method, which is the most important step in developing a video stabilization algorithm. DE 10 2009 017135 A1 describes a method for video data analysis in which a background image transformation caused by the camera's own movement is first determined for discrete images of a sequence recorded by the camera. To compensate for the camera's own movement, a background-stabilized image sequence is then generated from the discrete images by compensating for the background image transformation determined for each image. Using this background-stabilized image sequence, pixels associated with moving objects are then detected. DE 102 46 355 A1 describes a method for processing a three-dimensional image data set, wherein the three-dimensional image data set is processed into a data set for two-dimensional image reproduction. Furthermore, the invention relates to devices for performing the necessary calculations and / or for reproducing the images. The invention is particularly well suited for medical applications in endoscopy, especially virtual colonoscopy. DE 11 2004 001 861 B4 describes a system and a method for analyzing locally deformable motion and for accurately tracking the motion of an object while isolating the object's local motion from its global motion. The object is viewed in a sequence of images, and these sequences are sampled to identify object image areas and background image areas. The motion of at least one of the identified background image areas is estimated to identify those background image areas affected by global motion. The motion of several background image areas is combined to measure the global motion within that image frame. The measured global motion in the object image areas is compensated to measure the object's local motion, and the object's local motion is tracked. PURGATHOFER, Werner: Geometric Transformations. Computer Graphics 1, Text Sheet 02 Vs. 10, TU Wien. URL: https: / / www.cg.tuwien.ac.at / courses / CG1 / textblaetter / 02%20Geometräch%20Transformationen.pdf describes geometric transformations in general. The object of the invention is therefore to create a method of the aforementioned type that allows for the reliable identification of previously defined marking points. This problem is solved according to the invention by a method having the features of claim 1. In particular, it is advantageous that the marked point(s) are not used, or not necessarily used, to determine the transformation. This allows even pixels with few distinctive characteristics compared to the other elements of the scene to be easily identified in the second image. It has been found that identifying the corresponding pixel to such image markers is difficult when, for example, the image rotates or otherwise changes significantly. Another advantage is that the marker point is clearly visible even in non-consecutive images. This is particularly beneficial for fast or jerky movements. Furthermore, the identity of the marker points is retained, so that marker points that are in a similar environment and / or close to each other are not confused. When determining the transformation, position information from a position sensor or similar auxiliary signals may also be taken into account. In principle, the process can be performed on a sub-area of ​​an image, which can increase the speed of the process. In the simplest case, the sub-area comprises the entire image. In one embodiment, the method is further characterized by the pictorial representation of the transferred marker point in the second image. This representation can be achieved, for example, by a border, a color, or some other highlighting. In one embodiment, the method is further characterized by the output of the image position, in particular coordinates, of the transferred marker point. In one implementation, setting a marker point is done manually. In particular, a user can enter the marker points on a screen by touch. It is particularly advantageous to define a marker point in a still image of the image sequence. This allows an interesting point to be selected and marked calmly and with great precision. In one implementation, a geometric transformation with multiple degrees of freedom is used to determine the transformation. Such a matrix transformation allows for the quick and easy calculation of a transformation between the first image and a second image, especially when the camera position changes significantly. It is particularly advantageous to use a matrix transformation with eight degrees of freedom. This allows scaling, rotation, translation, and perspective changes, such as shearing, to be taken into account and detected. Additional advantages include a more accurate reconstruction of the reference image and thus a more precise localization of the marker point in the transformed image. Furthermore, this method can be used to compensate for effects caused by a rolling shutter of the image sensor during image capture. The localization of the marker point in the transformed image is performed using one of numerous known methods. Such a method could, for example, be an algorithm for object recognition and / or feature detection, such as edge or corner detection. In one implementation, the first image is transformed, but in this case the feature description for the marked pixel is recalculated for each subsequent image, which is computationally intensive. It is therefore particularly advantageous if the transformation is carried out on the second image. According to the invention, the method includes checking, prior to localization, whether the image position of the marker point lies within the second image. This eliminates the need for the time-consuming localization process if it cannot be successful. According to the invention, for verification purposes, the image position of the marker point is first transferred to the transformed second image and then transformed back into the original second image. This is possible with relatively little computational effort, so that it can be estimated whether a more complex calculation is necessary. An advantageous approach involves checking for a transformation before localization. If not, this could mean, for example, that the image now contains a different scene. This could be caused by a significant camera movement. In one embodiment of the invention, an error message or other warning is issued to a user if the marker point would be outside the image. One implementation involves checking, particularly during localization, whether the similarity with which a corresponding marker point is found is sufficient. This prevents, in the case of a hidden pixel in the subsequent image, any other point in the image that is the next best corresponding pixel from being selected. Similarity can be determined, for example, using descriptors. In particular, a threshold value for similarity can be defined. In one implementation, however, the first image does not move along with the subsequent images. This means that the first image remains constant, at least for a certain number of subsequent images, so that the second image and each subsequent image are compared to this first image. Thus, the transformation is determined between the first image and a second image, then between the first and a third image, and so on. This allows for a more precise localization of the marker in the subsequent image. In an alternative embodiment of the invention, after each successful identification of a marker point in a second image, this second image is used as a reference, i.e., the first image, for the subsequent third image of the image sequence. This means that the transformation is always determined from one image to the next. In addition to individual points as markers, multiple marker points, especially geometrically related ones, can also be used in a single implementation. These can, for example, indicate a specific geometric extent of the object under investigation, such as a tumor. In addition to the method, a device for image processing with at least one means for carrying out the method according to the invention is also part of the invention. The invention is explained in more detail below with reference to an advantageous embodiment and the accompanying drawings. It shows: Fig. 1: a flowchart of a method according to the invention, Fig. 2: a flowchart of a method according to the invention with defect detection, Fig. 3: a first image with a marker point, Fig. 4: a second image, Fig. 5: the transformed second image with a localized marker point, Fig. 6: the second image with the transformed marker point, Fig. 7: a second image showing a different scene, Fig. 8: a second image in which the marker point is outside the field of view, and Fig. 9: a second image in which the marker point is obscured. Figure 1 shows a flowchart of a method according to the invention. The method can be implemented, for example, in a video controller of an endoscope or another image processing unit, in particular in an FPGA. In a first step S1, a first image is read in. A first image 1 is shown, for example, in Fig. 3. This first image 1 shows, for example, various tissue structures 2. In a marking step S2, a marker point 3 is defined in the first image 1, which is to be tracked continuously. Of course, multiple marker points can also be defined. For the sake of simplicity, only one marker point is shown in each of the following. In step S3, an nth image is read in. Figure 4 shows such an nth image 4. This nth image 4 is rotated relative to the first image 1. The position of the first image 1 is shown here with a dashed line. In a further step S4, a transformation is determined that converts the rotated image 4 into the original image 1. To find a suitable transformation, a matrix transformation with several unknowns, particularly eight unknowns, can be used. By solving the transformation equations, rotation, translation, scaling, and perspective changes, for example, can be taken into account. After the transformation has been determined, in the next step S5 the nth image 4 is transformed using this transformation. The result of the transformation is shown as an example in Fig. 5. The transformed image 4' now corresponds to the first image 1 in orientation and scaling. In a localization step S6, the marker point 3' is now located in the transformed image 4'. This localization can be performed using known search algorithms, for example, for object detection. The search can be restricted to a limited search area 5, which is defined around the position of the marker point 3 in the first image 1. In a transformation step S7, the found marker point 3' is now transformed into the nth image 4. Finally, in a rendering step S8, the marker point 3 can be displayed in the nth image 4, or, for example, its coordinates can be output. This places marker point 3 in the nth image 4 exactly at the position originally defined in the first image 1, as shown in Fig. 6. The process is then repeated with the n+1th image. Typically, the images are taken from a video sequence. Ideally, the image signals are processed so quickly that the marker can be tracked in the live video signal. It is particularly useful if the first image is retained for the ongoing image sequence, so that all further images of the image sequence are each related to this first image. Alternatively, the first image can also be reset after a certain time and / or a certain number of elapsed images, or set to the last second image of the period or the last image with a visible marker point. When applying the method, for example in endoscopy, various error situations can occur that make tracking the marker point impossible. Figure 2 shows a flowchart of a method according to the invention with corresponding error detection and error handling. The method is based on the method of Figure 1, although for the sake of simplicity some steps are not shown here. Here too, in step S1, a first image 1 is loaded. In a transformation control step S9, it is checked whether a transformation has been found that converts an nth image 4 into the first image 1. If not, a scene error step S10 detects an error in the camera scene. This is particularly the case if there has been a large change in the camera position. Such a situation is illustrated, for example, in Fig. 7. Here, the image shows different tissue structures, which is why a transformation is not possible. A corresponding error message is displayed in a message step S16. This can also include the display of an error symbol 6, as shown in Fig. 7. In addition, an error message can be displayed in plain text. If so, in a transfer step S11 the marking points 3 marked in the first image 1 are transferred to the transformed nth image 4' and converted into the nth image 4 using the transformation. In a plausibility step S12, it is checked whether the transmitted marker points 3 lie within a valid image area, in particular within the nth image 4. If not, in an edge error step S13, it is detected that at least one marker point 3 lies outside the image area of ​​the nth image 4. The message step S16 then follows. Fig. 8 shows such a situation, in which the marker point 3 lies outside the visible image area. If so, the localization step S6 follows. In a similarity test step S14, it is now checked whether the similarity between the determined point and the first image 1 is sufficiently high. A threshold value for the similarity can be defined here. If not, a point error step S15 detects that the marker point is indeed within the valid image area but is nevertheless not visible, particularly because it is obscured. Figure 9 shows such a situation, where the marker point 3 is obscured, for example, by a medical device 7. The message step S16 then follows. If so, the transfer step S7 and the display step S8 follow. Reference symbol list 1 First image 2 Tissue structures 3 Marker point 3' Transformed marker point 4 nth image 4' Transformed nth image 5 Search area 6 Error symbol 7 Medical device S1 - S16 Procedure steps

Claims

Method for determining the image position of a marker point (3) in an image of an image sequence, characterized by the following steps: defining (S2) a defined marker point (3) in a first image (1) of the image sequence; determining (S4) a transformation at least between corresponding sub-areas of the first image (1) and a second image (4) of the image sequence; transforming (S5) at least the sub-area of ​​the second image (4) using the determined transformation, such that the transformed sub-area of ​​the second image (4') corresponds to the first image (1) in orientation and scaling; checking (S12) whether the image position of the marker point (3) lies within the second image (4), wherein the image position of the marker point (3) is first transferred (S11) to the transformed sub-area of ​​the second image (4') and then transferred to the second image (4) using the transformation;if the image position of the marker point (3) lies within the second image (4): Locating (S6) the marker point (3) in the transformed sub-area of ​​the image (4') and transferring (S7) the localized marker point into the second image (4) using the specified transformation.; Method according to claim 1, characterized by the pictorial representation (S8) of the transferred marker point (3) in the second image (4) and / or output of the image position, in particular coordinates, of the transferred marker point (3). Method according to claim 1 or 2, wherein the setting of a marker point (3) is done manually and / or wherein the setting of a marker point (3) is done in a still image of the image sequence. Method according to one of the preceding claims, wherein a geometric transformation with several, in particular eight, degrees of freedom is used to determine (S4) a transformation. Method according to one of the preceding claims, wherein an object tracking and / or feature detection algorithm is used to locate (S6) the marking point (3). Method according to one of the preceding claims, characterized in that before localization (S6) it is checked (S9) whether a transformation has been found. Method according to one of the preceding claims, characterized by the additional method step of checking (S14), in particular during localization (S6), whether the similarity with which a corresponding pixel is found is sufficient, in particular wherein a threshold for the similarity is defined. Method according to one of the preceding claims, characterized in that the first image (1) does not move along, but remains constant for a certain number of subsequent images, so that each of the second images (4) is compared with this first image (1). Method according to one of the preceding claims, characterized in that several geometrically related marking points (3) are used. Device for image processing comprising at least one means for carrying out the method according to one of the preceding claims.

Images