Method for determining the image position of a marked point in an image of an image sequence

Abstract

A method for determining the image position of a marker point (3) in an image of an image sequence has the following method steps: specifying (S2) the marker point in a first image (1) of the image sequence; determining (S4) a transformation between at least corresponding partial regions of the first image (1) and a second image (4) of the image sequence; transforming (S5) at least a partial region of the first image (1) or a partial region of the second image (4) by means of the determined transformation; locating (S6) the marker point (3) in the transformed partial region of the image (4'); and converting (S7) the located marker point into the second image (4) by means of the determined transformation.

Description

Technical Field

[0001] A method for determining the image position of a marked point in an image of an image sequence. Background Art

[0002] In known methods, at least one point is marked in a first image. Such a marked point is also referred to hereinafter as a marked point. Subsequently, the image components corresponding to the points are determined in the first image of the image sequence and another image of the image sequence. This can be achieved by known object recognition or edge recognition or similar image processing.

[0003] Of course, such a method has the disadvantage that when the imaging unit rotates or otherwise undergoes a significant change, for example, the recognition of the image points or image components in the second image becomes difficult.

[0004] This may occur, for example, during endoscopic examinations during rapid, jerky movements or when there are strong changes in illumination.

[0005] In another method, a transformation for transforming the image components of the first image into the image components of the second image is determined. By means of this transformation, it is common to transform the marked image points into the second image. However, the disadvantage here is the low accuracy in recognizing the marked points, especially in the case of uneven objects. Summary of the Invention

[0006] Therefore, the object of the present invention is to implement a method of the type mentioned above, which allows reliable recognition of predefined marked points.

[0007] This object is solved by the method according to the invention.

[0008] The method according to the invention comprises the following method steps:

[0009] Specify a marked point in the first image of the image sequence;

[0010] Determine a transformation between at least corresponding partial regions of the first image and the second image of the image sequence;

[0011] Transform at least a partial region of the first image or a partial region of the second image by means of the determined transformation;

[0012] Locate the marked point in the transformed partial region of the image; and

[0013] Convert the located marked point into the second image by means of the determined transformation.

[0014] In particular, it is advantageous not to consider or not necessarily consider the one or more marked points for determining the transformation.

[0015] This is because, for example, image points having less typical characteristics compared to the rest of the image can also be easily recognized in the second image. That is, it has been proven that it is difficult to recognize image points corresponding to such image markers when, for example, the image rotates or otherwise undergoes strong changes.

[0016] Another advantage is that the marker points can be reliably recognized even in discontinuous images. This is particularly advantageous during fast or jerky movements.

[0017] Furthermore, the identity of the acquired marker points is maintained, so that marker points in similar environments and / or marker points close to each other are not confused either.

[0018] When determining the transformation, the position information of the position sensor or similar auxiliary signals may perhaps also be considered together.

[0019] In principle, the method can be implemented on a partial region of the image, whereby the speed of the method can be increased. In the simplest case, the partial region includes the entire image.

[0020] In one embodiment, the method is further characterized in that the transformed marker points are visually (bildliche) displayed in the second image. This display can be achieved, for example, by adding an edge, coloring, or other emphasis.

[0021] In one embodiment, the method is further characterized in that the image position, in particular the coordinates, of the transformed marker points are output.

[0022] In one embodiment, the marker points are specified manually. In particular, the user can mark the marker points on the screen by means of a touch input.

[0023] It is particularly advantageous to specify the marker points in the still images of the image sequence. In this way, the points of interest can be selected and marked statically and with great precision.

[0024] In one embodiment, a geometric transformation with multiple degrees of freedom is used to determine the transformation. Such a matrix transformation allows for a fast and simple calculation of the transformation between the first image and the second image. In particular, even when the camera position changes very strongly.

[0025] It is particularly advantageous to use a matrix transformation with eight degrees of freedom. In this way, scaling, rotation, translation, and perspective changes, such as shear, can be considered and recognized. An additional advantage is that the reference image is reconstructed more precisely and thus the marker points are located more precisely in the transformed image.

[0026] Furthermore, in this way, for example, the effects caused by the rolling shutter of the image sensor during image capture can be compensated.

[0027] Based on a large number of known methods, the positioning of marker points in the transformed image is achieved. Such methods can be, for example, algorithms for object recognition and / or feature detection, such as for edge recognition or corner recognition.

[0028] In one embodiment, the first image is transformed. However, in this case, for each individual subsequent image, the feature description for the marked image points is recalculated, which is computationally expensive.

[0029] Therefore, it is particularly advantageous to perform the transformation on the second image.

[0030] One advantageous embodiment includes: before positioning, checking whether the image position of the marker point is within the second image. Thereby, for example, the costly positioning can be aborted when the positioning is not successful.

[0031] For the check, first the image position of the marker point is transferred into the transformed second image and transformed back into the original second image by means of the transformation. This can be achieved with relatively little computational effort, so that it can be evaluated whether costly calculations are required.

[0032] One advantageous embodiment includes: before positioning, checking whether the transformation has already been found. If not, this can, for example, indicate that the image currently contains other scenes. This can be caused, for example, by strong camera movement.

[0033] In one embodiment of the present invention, when the marker point will be outside the image, an error report or other warning is output to the user.

[0034] One embodiment includes: verifying, especially during positioning, whether the similarity is sufficient, and using the similarity to find the corresponding marker point. This prevents any other points in the image from being selected in the case where the image points in the subsequent image are occluded, and the other points are suboptimal corresponding image points.

[0035] The similarity can be determined, for example, via descriptors. For this purpose, a threshold can be defined especially for the similarity.

[0036] However, in one embodiment, the first image does not change along. This means that the first image remains unchanged for at least a certain number of subsequent images, so that the second image and each subsequent image are compared with this first image. That is, the transformation is determined between the first image and the second image, and then between the first image and the third image, etc. Thereby, more accurate positioning of the markers in the subsequent images can be achieved.

[0037] In an alternative embodiment of the present invention, after each successful identification of a fiducial point in the second image, the second image is used as a reference for a subsequent third image of the image sequence, i.e., as the first image. This means that the transformation from one image to the subsequent image is always determined.

[0038] In addition to using individual points as fiducials, in one embodiment, a plurality of consecutive, in particular geometrically consecutive, fiducial points can also be used. These fiducial points can be, for example, specific geometric extents of a research object, such as a tumor.

[0039] In addition to the method described above, a device for image processing is also part of the present invention, which device has at least one mechanism for implementing the method according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further explained below with reference to the accompanying drawings by means of advantageous embodiments.

[0041] Wherein:

[0042] Figure 1 shows a flowchart of a method according to the present invention;

[0043] Figure 2 shows a flowchart of a method according to the present invention with error recognition;

[0044] Figure 3 shows a first image with fiducial points;

[0045] Figure 4 shows a second image;

[0046] Figure 5 shows the transformed second image, which has the fiducial points located;

[0047] Figure 6 shows the second image with the transformed fiducial points;

[0048] Figure 7 shows a second image, which shows different views;

[0049] Figure 8 shows a second image in which the fiducial points are outside the visible area; and

[0050] Figure 9 shows a second image in which the fiducial points are covered. DETAILED DESCRIPTION

[0051] Figure 1A flowchart of a method according to the present invention is shown. The method can be implemented, for example, in a video controller of an endoscope or other image processing unit, especially in an FPGA.

[0052] In a first step S1, a first image is read in. The first image 1 is shown, for example, in Figure 3 . The first image 1 shows, for example, different tissue structures 2. In a marking step S2, a marking point 3 is specified in the first image 1, which marking point should be continuously tracked. Of course, multiple marking points can also be specified. For simplicity, only one marking point is shown correspondingly below.

[0053] In step S3, at this time, the nth image is read in. Such an nth image 4 is shown in Figure 4 . The nth image 4 is rotated relative to the first image 1. The position of the first image 1 is shown here by a dashed line.

[0054] In the subsequent step S4, at this time, the transformation for converting the rotated image 4 into the first image 1 is determined. To find a matching transformation, for example, a matrix transformation with multiple unknowns, especially eight unknowns, can be used. By solving the transformation equation, rotations, translations, scalings, and perspective changes can be taken into account in this way, for example.

[0055] After the transformation is determined, in the next step S5, the nth image 4 is transformed with the aid of this transformation. The result of the transformation is shown exemplarily in Figure 5 . The transformed image 4' is now equal to the first image 1 in terms of orientation and scaling.

[0056] At this time, in a positioning step S6, the marking point 3' is positioned in the transformed image 4'. This positioning can be achieved with the aid of known search algorithms, for example, search algorithms for object recognition.

[0057] Here, the search can be restricted to a defined search region 5, which is defined around the position of the marking point 3 in the first image 1.

[0058] In a conversion step S7, at this time, the found marking point 3' is transformed into the nth image 4.

[0059] Finally, in a display step S8, the marking point 3 can be displayed in the nth image 4, or coordinates can be output, for example. Thereby, the marking point 3 is exactly at the original position defined in the first image 1 in the nth image 4, as shown in Figure 6 .

[0060] The process is then repeated with the (n + 1)th image. The images are usually taken from a video sequence. It is especially preferably to process the image signals so quickly that the marking can be tracked together in a real-time video signal.

[0061] It is particularly expedient if a first image is retained for a continuous image sequence, so that all further images of the image sequence are respectively associated with this first image.

[0062] Alternatively, in principle, the first image can also be reset after a certain time and / or after a certain number of images have passed, or the first image can be set to the second to last image of the time period or the last image with a visible marking point.

[0063] When the method is used, for example, in endoscopy, various error situations can occur which make tracking of the marking points impossible. Figure 2 A flow chart of a method according to the invention with corresponding error detection and error handling is shown. The method is based on Figure 1 A method in which some steps are not shown here for reasons of simplicity.

[0064] Here too, the first image 1 is first loaded in step S1 . In a transformation check step S9 , it is verified whether a transformation has been found which converts the nth image 4 into the first image 1 .

[0065] If not, then in a picture error step S10 an error in the camera's picture is detected. This is the case in particular when a large change in the camera position occurs. Figure 7 In this case, the image shows a different tissue structure, so that a transformation is not possible. In a reporting step S16, a corresponding error report is displayed. The error report can also include an embedding of an error symbol 6, as exemplified in Figure 7 In addition, error messages can be displayed in plain text.

[0066] If yes, in a transfer step S11 the marking point 3 marked in the first image 1 is transferred to the transformed n-th image 4 ′ and converted into the n-th image 4 by means of the transformation.

[0067] In a plausibility step S12 , it is checked whether the marking point transferred in this way lies within the valid image area, in particular within the nth image 4 .

[0068] If not, then in the edge error step S13 it was detected that at least one marking point 3 was outside the image area of ​​the nth image 4. Here too, a reporting step S16 follows. Figure 8 A situation is shown in which the marking point 3 is outside the visible image area.

[0069] If so, then the subsequent step here is the positioning step S6. In the similarity test step S14, at this time it is verified whether the similarity to the first image 1 is large enough for the point to be determined. A threshold value can be defined here for the similarity.

[0070] If not, then in the point error step S15 it is recognized that, although the marked point is in a valid image area, it is not visible, in particular because the marked point is covered. Figure 9 This shows a situation in which the marked point 3 is covered by the medical device 7, for example. The subsequent step is the reporting step S16.

[0071] If so, then the subsequent steps are the transformation step S7 and the display step S8.

[0072] List of reference signs

[0073] 1 First image

[0074] 2 Tissue structure

[0075] 3 Marked point

[0076] 3‘ Transformed marked point

[0077] 4 nth image

[0078] 4‘ Transformed nth image

[0079] 5 Search area

[0080] 6 Error symbol

[0081] 7 Medical device

[0082] S1–S16 Method steps

Claims

1. A method for determining the image position of a marked point (3) in an image of an image sequence, characterized in that The following method steps: Specify (S2) marker points in a first image (1) of the image sequence; Determine (S4) a transformation between at least corresponding partial regions of the first image (1) and a second image (4) of the image sequence; Transform (S5) at least a partial region of the second image (4) by means of the determined transformation; Locate (S6) the marker points (3) in the transformed partial region of the transformed second image (4'); and Convert (S7) the located marker points by means of the determined transformation into the original second image (4).

2. The method according to claim 1, wherein Visually display (S8) the converted marker points (3) in the original second image (4), and / or output the image position of the converted marker points (3).

3. The method according to claim 2, wherein The image position is configured as coordinates.

4. The method according to any one of claims 1 to 3, wherein, The specifying of the marker points (3) is carried out manually, and / or the specifying of the marker points (3) is carried out in a still image of the image sequence.

5. The method according to any one of claims 1 to 3, wherein To determine (S4) the transformation, a geometric transformation with a plurality of degrees of freedom is used.

6. The method according to claim 5, wherein To determine (S4) the transformation, a geometric transformation with eight degrees of freedom is used.

7. The method according to any one of claims 1 to 3, wherein To locate (S6) the marker points (3), an algorithm for object tracking and / or feature detection is used.

8. The method according to any one of claims 1 to 3, characterized in that, Before the locating (S6), check (S12) whether the image position of the marker points (3) is within the second image (4), and / or check (S9) whether a transformation has been found before the locating (S6).

9. The method according to claim 8, wherein First, transfer (S11) the image position of the marker points (3) into the transformed second image (4') and convert it into the original second image (4) by means of the transformation.

10. The method according to any one of claims 1 to 3, characterized in that The following additional method step, verify (S14) whether the similarity is sufficient and use the similarity to find corresponding image points.

11. The method according to claim 10, wherein, Carry out the verification (S14) during the locating (S6).

12. The method according to claim 10, wherein, Define a threshold for the similarity.

13. The method according to any one of claims 1 to 3, characterized in that The first image (1) is not changed together, but remains unchanged for a specific number of subsequent images, so that each second image (4) is compared with this first image (1).

14. The method according to any one of claims 1 to 3, characterized in that Use a plurality of geometrically consecutive marker points (3).

15. An apparatus for image processing, the apparatus having at least one means for carrying out the method according to any one of claims 1 to 14.

Images