Delay compensation for image processing devices

Abstract

The invention relates to delay compensation for an image processing device. In order to assign overlay data to a frame of an original image stream (210), past frames in a selection result of frames of the original image stream are considered, which have already undergone image processing. A current frame of the original image stream (210) is compared to the past frames contained in the selection result of frames and one of the past frames is identified which is most similar to the current frame. Overlay data from the identified past frame is selected and assigned to the current frame. Thus, the current frame can be presented with the assigned overlay data selected from the most similar past frame without waiting for the result of computationally expensive and time-consuming image processing of the current frame.

Description

Technical Field

[0001] This invention relates to image processing, and more particularly to image processing apparatus, medical imaging devices for processing images, and methods for processing images. More specifically, this invention relates to image processing of images presented to a user together with overlay data determined by an image processing algorithm. Background Technology

[0002] For various purposes and applications, image frames from the original image stream are processed to obtain overlay data for the frames allocated to the original image stream, and then presented to the operator or user along with the overlay data from the original image stream frames. This method can generally be referred to as augmented imaging.

[0003] For example, in the field of medical imaging equipment, interventional guidance applications can be used to display the position of devices such as catheter tips or stents relative to the human body. In such applications, information about the device can be generated by image processing algorithms and presented to the operator / user along with the raw image stream obtained from an image source such as X-rays. The added information can be called overlay data. Overlay data can highlight or emphasize the outline of the device, or it can be any other information obtained from frames of the raw image stream. For example, in cardiac applications, the stent position relative to the vessel wall can be visually enhanced.

[0004] Overlay data is typically generated by image processing algorithms. However, determining or generating overlay data takes time, so there may be a time delay between capturing frames of the original image stream and presenting these frames and the assigned overlay data to the user. Summary of the Invention

[0005] Therefore, it may be necessary to reduce the time delay between capturing the raw image stream and presenting the frames and overlay data of the raw image stream to the operator / user.

[0006] The object of the invention is achieved through the subject matter of the independent claims; further embodiments are included in the dependent claims. It should be noted that the aspects of the invention described below are also applicable to image processing apparatus, medical imaging devices for processing images, and methods for processing images.

[0007] According to the present invention, an image processing apparatus is provided. The image processing apparatus includes a processor, a data storage device, an input interface, and an output interface. The input interface provides the processor with a raw image stream having multiple frames. The output interface provides an output image stream to a display.

[0008] The processor is configured to: receive an original image stream; process frames of the original image stream using an image processing algorithm to obtain a processed image stream, wherein the processed image stream includes the original image stream and overlay data generated by the image processing algorithm containing additional information for comparison with the original image stream; provide the frames of the processed original image stream together with the corresponding overlay data to the data storage; generate a selection result of past frames of the original image stream that have undergone image processing algorithm processing; compare the current frame of the original image stream with at least some frames in the selection result of past frames and determine a similarity value for the current frame and at least some frames in the selection result of past frames; identify matching frames in the selection result of frames based on the similarity value with the current frame; allocate overlay data from the matching frames to the current frame; and add the current frame and the allocated overlay data from the matching frames to the output image stream.

[0009] This provides an output image stream containing overlay data that is allocated to the captured original image stream. At least in the first stage, the overlay data of the current frame is the result of a similarity comparison between the current frame of the original image stream and past frames of the original image stream that have undergone image processing.

[0010] From the past frames of the original image stream, select matching frames that are relatively similar to the current frame. Then, select the overlay data of the similar frame and assign the selected overlay data to the current frame.

[0011] Matching or similar frames identified from the selection results of past frames are considered close visual substitutes for the current frame. Therefore, it can be assumed that the overlay data of the close visual substitutes will also match the current frame. In this example, the matching frame is the most similar frame, specifically the frame from the selection results of the past frame with the highest similarity value. In other words, in this example, the matching frame is the past frame most similar to the current frame.

[0012] Therefore, advantageously, instead of waiting for the computationally expensive and time-consuming image processing algorithms to generate overlay data for the current frame, overlay data from similar past frames can be presented together with the current frame. The operation of comparing the current frame of the original image stream with past frames of the original image stream requires far less computational power than processing the current frame through image processing algorithms. Therefore, the overlay data selected from similar past frames can be presented to the operator / user much earlier than the results of image processing algorithms, preferably (near) real-time. Thus, the time delay in presenting frames of the original image stream along with the overlay data is eliminated or at least significantly reduced.

[0013] In other words, overlay data from already processed past frames is added to the output image stream along with the current image. A similarity or matching algorithm, which is less computationally expensive than image processing algorithms, is used to compare the current frame with a limited number of past frames. When a matching past frame is identified, the overlay data for that matching past frame is added to the output image stream along with the current frame, for example, to be displayed to a physician. As a result, the time delay between the output image stream and the original image stream is eliminated or at least significantly reduced.

[0014] This is beneficial, for example, in image-guided medical procedures where image streams are used to provide real-time feedback to the physician. In this case, what the physician is doing is guided by live images on a monitor (e.g., fluoroscopic images from an interventional X-ray system). For this, it is necessary to display the live images in real time without any processing delay.

[0015] According to an embodiment, the frame selection result has a predetermined size and includes multiple past frames corresponding to the predetermined size. In this embodiment, the processor can be configured to determine the frame selection result of the original image stream by applying a sliding window algorithm to past frames of the original image stream processed by the image processing algorithm. Alternatively, selection can be based on physiological parameters, such as the phase of the current frame within a cardiac or respiratory cycle.

[0016] In general, the frame selection results consist of frames from the original image stream that have already undergone image processing algorithms to generate relevant overlay data. These processed frames are then selected as a basis for comparison with the current frame. In the example, the sliding window algorithm can then select some of the most recent frames from those processed by the image processing algorithms. This approach can result in a relatively high similarity between the selected frames and the current frame.

[0017] In the example, the predetermined number of past frames should be small enough that the comparison between the current frame and each past frame in the selection results is still computationally feasible in real time.

[0018] According to an embodiment, the processor is configured to generate overlay data based on frames of the original image stream and / or quantity values ​​derived from frames of the original image stream.

[0019] According to an embodiment, the processor is configured to add an index of the similarity values ​​of the determined matching frames to the output image stream.

[0020] In this embodiment, the operator is provided with information about the similarity between the current frame and the most similar past frame. The degree of similarity can indicate the accuracy of the overlay data of past frames presented together with the current frame.

[0021] According to an embodiment, the processor is configured to compare the similarity value of the matched frames with a similarity threshold. Therefore, for example, if the similarity value is less than the similarity threshold, only the current frame is added to the output image stream; conversely, if the similarity value is equal to or greater than the similarity threshold, the superimposed data of both the current frame and the matched past frames are added to the output image stream.

[0022] Therefore, if the similarity value is less than a predetermined similarity threshold, overlay data from past frames will not be selected to be presented to the operator / user along with the current frame. The underlying principle is to avoid presenting overlay data from past frames that differ to some extent from the current frame.

[0023] According to an embodiment, the processor is configured to compare overlay data of a first frame of the original image stream, to which the frame-based selection result is assigned, with overlay data generated from an image processing algorithm applied to the first frame.

[0024] In this embodiment, the overlay data assigned to the first frame based on a similarity comparison with past frames is compared with overlay data generated by an image processing algorithm based on the first frame. This method is able to compare the correspondence between the initially assigned overlay data and the finally determined overlay data. The operator / user of the image processing device or a medical imaging apparatus including such an image processing device can use this information to decide whether to use a time-delayed overlay data estimation procedure (i.e., identifying overlay data based on the similarity between the current frame and past frames), or whether the operator / user prefers to wait for the overlay data actually generated by the time-consuming image processing algorithm.

[0025] According to the present invention, a medical imaging apparatus for processing images is also provided. The medical imaging apparatus includes an image sensor, a display, and an image processing device as described in one or more embodiments above and below. The image sensor is communicatively coupled to the image processing device to provide a raw image stream to the image processing device, and the display is communicatively coupled to the image processing device to receive an output image stream from the image processing device.

[0026] According to an embodiment, the imaging system is an X-ray system, specifically an interventional X-ray system capable of providing live X-ray images during image-guided medical interventions. Alternatively, the imaging system may be an echocardiogram device, an ultrasound device, a photoacoustic imaging device, etc.

[0027] An image sensor can be any device suitable for capturing images according to the technology of a specific imaging system. The image sensor can be a device specifically designed for a particular imaging system technology. However, an image sensor can provide a raw image stream according to known standards, thus standardizing the interface between the image sensor and the image processing device and eliminating the need to adapt to different imaging technologies. Nevertheless, the image processing device can include a dedicated interface for communicatively coupling to different image sensors.

[0028] A display can be any type of device suitable for visually displaying information to a human operator / user. For example, a display can be a monitor or multiple monitors, a touchscreen, or a head-mounted display.

[0029] Of course, the principles of image processing equipment apply to medical imaging devices.

[0030] According to the present invention, a method for image processing is also provided. The method includes the following steps: receiving an original image stream having multiple frames; processing the frames of the original image stream using an image processing algorithm to obtain a processed image stream, wherein the processed image stream includes the original image stream and overlay data generated by the image processing algorithm containing additional information for comparison with the original image stream; providing the processed frames of the original image stream together with the corresponding overlay data to a data storage device; generating a selection result of past frames of the original image stream that have undergone image processing algorithm processing; comparing the current frame of the original image stream with at least some frames in the selection result of past frames; determining a similarity value for the current frame and at least some frames in the selection result of past frames; identifying matching frames in the selection result of frames based on the similarity value with the current frame; allocating the overlay data from the selection result of the identified frames to the current frame; and adding the current frame and the allocated overlay data from the matching frames to an output image stream.

[0031] The functional features described above with reference to embodiments of the image processing apparatus can be implemented as method steps for processing images.

[0032] Furthermore, a computer program unit is provided. The computer program unit is configured to control an image processing apparatus as described herein, which, when executed by a processing unit, is adapted to perform the method steps of the method for processing images described herein.

[0033] Finally, a computer-readable medium on which computer program units are stored is provided.

[0034] According to one aspect, the apparatus and method described herein relate to latency compensation for image processing devices, specifically for systems used in image-guided medical procedures. To assign overlay data to frames of an original image stream, past frames within the selected frame set of the original image stream, which have undergone image processing by an image processing algorithm, are considered. The current frame of the original image stream is compared with past frames contained in a sliding window, and the past frame most similar to the current frame is identified. Overlay data from the identified past frame is selected and assigned to the current frame. Thus, the current frame can be presented with the assigned overlay data selected from the most similar past frame without waiting for the computationally expensive and time-consuming image processing of the current frame.

[0035] These and other aspects of the invention will become apparent and will be illustrated from the embodiments described below. Attached Figure Description

[0036] Exemplary embodiments of the present invention will now be described with reference to the following accompanying drawings:

[0037] Figure 1 An image processing apparatus according to an embodiment is illustrated schematically;

[0038] Figure 2 An imaging apparatus according to an embodiment is illustrated schematically;

[0039] Figure 3 The illustration schematically shows the original image stream and the time delay generated by the image processing algorithm applied to the frames of the original image stream;

[0040] Figure 4 This illustration shows how overlay data from past frames is assigned to the current frame of the original image stream;

[0041] Figure 5 This illustration shows how overlay data from past frames is assigned to the current frame of the original image stream;

[0042] Figure 6 The illustration shows a comparison between the overlay data of the assigned frame and the overlay data of the generated frame;

[0043] Figure 7 The steps of a method for processing an image according to an embodiment are illustrated schematically. Detailed Implementation

[0044] Certain embodiments will now be described in more detail with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same elements, even in different drawings. Items defined in the description, such as detailed structures and elements, are provided to aid in a comprehensive understanding of the exemplary embodiments. Furthermore, well-known functions or structures are not described in detail, as they would obscure the embodiments with unnecessary detail. Also, expressions such as “at least one” are used before the list of elements to modify the entire list of elements, rather than a single element in the list.

[0045] Figure 1 An embodiment of the image processing device 10 is illustrated schematically.

[0046] Device 10 includes a processor 20, a data memory 30, an input interface 40, and an output interface 50. Input interface 40 is configured to provide a raw image stream of multiple frames to processor 20. Output interface 50 is configured to provide an output image stream to a display.

[0047] The processor 20 is configured to receive an original image stream, process frames of the original image stream to obtain a processed image stream, wherein the processed image stream includes the original image stream and overlay data containing additional information compared with the original image, the overlay data being generated by an image processing algorithm.

[0048] The original image stream and the generated overlay data are provided to the data storage 30. The processor 20 is also configured to subsequently generate a selection result of past frames of the original image stream, compare the current frame of the original image stream with at least some frames in the selection result of past frames, determine a similarity value for the current frame and at least some frames in the selection result of frames, and identify matching frames based on the similarity value.

[0049] Finally, the processor 20 is configured to allocate overlay data from matching past frames that have already been processed using image processing algorithms, and to add the current frame and the allocated overlay data to the output image stream.

[0050] One advantage of this approach is that it compensates for the latency introduced by computationally demanding and / or time-consuming image processing algorithms applied to frames of an image stream.

[0051] For example, if additional information (i.e., overlay data) needs to be generated from a frame and presented into an image stream, this process takes time. Depending on the image processing algorithm, the time delay introduced by the image processing stream can be significant. For some applications, specifically for medical image-guided interventions, a time delay of tens of milliseconds may already be undesirable or unacceptable, as it causes what the practitioner sees on the monitor to be delayed compared to what they are actually doing, thus hindering the execution of the intervention or procedure.

[0052] Delay compensation is accomplished by comparing the current image frame of the real-time image stream (i.e., the original image stream) with past frames of the same real-time image stream that have undergone image processing algorithms to determine the overlay data. The overlay data can be the result of the image processing algorithm, where the result is presented to the individual frames of the real-time image stream.

[0053] In diagnostic imaging procedures, additional information is incorporated into the raw image stream, such as overlay image data of enhanced catheter tips or stents, to make visual examination of the displayed image frames easier and more comfortable. In this example, moving the catheter tip introduces a delay between the tip movement and the processed image stream, as incorporating catheter tip-related information into the raw image stream is computationally expensive. Due to this delay, the displayed image of the catheter tip or stent, along with the enhancement overlay, no longer accurately represents the actual situation.

[0054] The raw image stream consists of multiple frames. The most recent frame is called the current frame. However, in order to render additional information into the current frame, a certain amount of processing time is required, which introduces a time delay.

[0055] The raw image stream originates from an image source, such as a fluorescence fluoroscopy imaging device like an X-ray device, an ultrasound device, etc.

[0056] At least some or all of the past frames of the original image stream and the allocated overlay data are temporarily stored in data memory 30. Consider comparing the selected frames of the original image stream with the current frame of the original image stream. Comparing two images, i.e., comparing the current frame of the original image stream with at least one, several, or every selected frame of the past frames of the original image stream, requires significantly less computation time than image processing operations. The current frame of the original image stream is compared with at least one, several, or every selected frame of the past frames of the original image stream. Therefore, multiple image comparison operations are performed, where the corresponding image comparison operation is applied to two images, i.e., the current frame and a past frame. For at least some of these image comparison operations, similarity values ​​are calculated for pairs of comparison frames. Then, the pair of frames with the highest similarity is selected to identify the past frame most similar to the current frame. Based on the identified most similar past frame, overlay data is selected from the data memory to be presented in that (past) frame, and also presented in the current frame of the original image stream. The additional information presented in the frames of the original image stream is called overlay data. The frames and overlay data of the raw image stream are presented to the operator of the imaging device.

[0057] For example, in cardiac applications, this method helps to visually enhance the catheter tip relative to the vessel wall with a very small time delay. The visually enhanced optical features are superimposed onto frames of the original image stream to highlight or emphasize the characteristics of the original image stream frames. This composition of optical features is called the processed image stream, which consists of frames of the original image stream and the added optical features.

[0058] The comparison of the current frame of the original image stream with past frames of the original image stream can be accomplished using any applicable image comparison and / or image similarity determination algorithm. For example, to calculate a similarity value, at least some of the structures, lines, shapes, and brightness present in those frames can be taken into account. For instance, a similarity value can be calculated by summing the squared differences between the intensity values ​​of the frames. It is assumed that the image processing results for similar frames are similar. This is why, if the current frame is (to some extent) identical or similar to the past frames, additional information presented in the past frames can be presented in the current frame.

[0059] In one embodiment, comparisons can be performed using only selected portions or regions of past frames. For example, a selected region surrounding the overlay data is considered for comparison with the current frame. The selected region surrounding the overlay data can be defined as the region in the past frame where the overlay data is located. The selected region is larger than the overlay data. Furthermore, during the comparison process, an image transformation, such as the offset between the region of interest of a past image and the current frame, can be determined. This transformation can also be applied to the overlay data when overlay data from past frames is assigned to the current frame.

[0060] In other words, in the example, the current frame is compared with the selection results of past frames that have already undergone image processing and been assigned to overlay data, similar past frames are identified, such as the one most similar to the current frame, and the output image stream is composed based on the overlay data of the current frame and the selected past frames.

[0061] Figure 2 An exemplary medical imaging apparatus 100 is shown. The imaging apparatus 100 includes an image sensor 60 and an image processing device 10 (e.g., see reference 100). Figure 1The image sensor 60 is communicatively coupled to the image processing device 10 to provide a raw image stream to the image processing device. Specifically, the image sensor 60 provides the raw image stream to an input interface 40 of the image processing device 10, and the input interface provides the raw image stream to a processor 20. For example, the image sensor 60 can be coupled to the image processing device 10 via a wired or wireless connection. In one embodiment, the image sensor 60 and the image processing device 10 can be connected via a data transmission network comprising multiple segments, some of which may be wired segments and others may be wireless segments. The display 70 is communicatively coupled to the image processing device 10 to receive an output image stream from the image processing device. For the communicative coupling of the display 70 and the device 10, the same principles as described with reference to the coupling of the image sensor 60 and the image processing device 10 are applied.

[0062] It should be noted that Figure 2 The imaging system 100 may include an image processing device 10 described with reference to any embodiments or examples contained herein.

[0063] According to an embodiment, the imaging system 100 is an X-ray system, an echocardiogram device, an ultrasound device, a photoacoustic imaging device, etc.

[0064] Image sensor 60 can be any device suitable for capturing images according to the technology of a specific imaging system. Image sensor 60 can be a device specifically designed for a certain imaging system technology. However, image sensor 60 can provide a raw image stream according to known standards, so that the interface between image sensor 60 and image processing device 10 is standardized and does not need to be adapted to different imaging technologies. Nevertheless, image processing device can include dedicated interfaces to communicatively couple to different image sensors.

[0065] The principles of image processing implemented by the image processing device 10 disclosed in this article, as well as the methods for image processing, are referenced. Figures 3 to 6 To be further described.

[0066] Figure 3A raw image stream 210 is shown, comprising multiple frames numbered from 1 to 17 according to the time they were captured and / or provided to an image processing device. In other words, the first frame is provided first, followed by the second frame after a certain time, and so on. When an image processing algorithm is applied to each frame to generate overlay data for the frames, this operation takes a certain amount of time, depending on the complexity of the image processing algorithm. The time required to complete image processing for a single frame is indicated by a time delay 205. The processed image is the output of the image processing algorithm, and the overlay data is assigned to at least some or all of the frames of the raw image stream. The overlay data is represented by circles within each frame. However, it should be noted that the overlay data can be any data generated based on the frames of the raw image stream. Examples of overlay data are provided above and below with reference to other examples and embodiments.

[0067] Figure 4 An original image stream with 11 frames is shown, where the 11th frame is the most recent frame, i.e., the current frame. In this embodiment, it is assumed that frames 1 to 5 of the original image stream 210 have undergone image processing by an image processing algorithm, and that overlay data for frames 1 to 5 is available, such that frames 1 to 5 with allocated overlay data correspond to the processed image 220.

[0068] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to determine the selection result of frames in the original image stream by applying a sliding window algorithm to past frames of the original image stream that have already undergone the image processing algorithm.

[0069] The frame selection corresponds to a sliding window 207, which in this example contains five frames (frames 1 to 5) of the processed image 220. The current frame of the original image stream 210, i.e., frame 11, is compared with at least one, some, or all of the frames contained in the sliding window 207 to determine the most similar frame within the sliding window. For example, if it is determined that the fourth frame of the sliding window 210 is most similar to the current frame 11, then the overlay data of the fourth frame is selected and allocated and rendered into the current frame. Figure 4 In the image, frame 11 (the current frame) and frame 4 (the past frame most similar to the current frame) are shown with dashed lines for illustrative purposes only, and to indicate the two frames that are most similar to each other.

[0070] At least some or every frame in the selected frames of the original image stream is compared with the current frame of the original image stream. The probability of finding similar frames may increase when comparing frames captured approximately at the same time or close in time. Therefore, once a frame has been processed, it is input into the frame selection results as the latest frame that has undergone image processing. All frames in the current image stream that have undergone image processing can be referred to as the processed image stream. Therefore, the frames in the processed image stream are those frames for which the image processing algorithm has completed the identification of overlay data.

[0071] In the example, the most recent frame of the processed image stream enters the selection result, and the most past frame of the processed image stream exits the selection result. In other words, the selection result thus comprises a sliding window of processed frames, which functions like a buffer that implements the first-in, first-out principle.

[0072] In one embodiment, other parameters known to affect image similarity can be used to select past frames included in the sliding window. For example, in a periodic process, frames are captured at the same or nearly the same time point as the reference periodic process. Specifically, if the current frame is captured from a human heart and during the end-diastolic phase, the sliding window could include past frames captured near or during the end-diastolic phase of a previous heartbeat cycle. Similarly, if the current frame was acquired at another time within the cardiac cycle, the selection of past frames captured at or approximately the same time is conceivable.

[0073] Figure 5 Exemplary examples show Figure 4 The scene following the one shown. Figure 5 In the example shown, the original image stream 210 contains 12 frames, where frame 12 is the current frame. At this point in time, more than one frame may have undergone image processing by an image processing algorithm, resulting in a processed image 220 containing six frames, i.e., frames 1 to 6. The sliding window 207 is shifted one position to the right to include the five most recent frames, i.e., frames 2 to 6. The current frame 12 is compared with frames 2 to 6 within the sliding window to identify the most similar frame (in this embodiment: frame 6), then overlay data is selected from the most similar past frames, and the selected overlay data is presented in frame 12. If it is possible to... Figure 5 As seen in the image, the sliding window contains the most recent frame of the processed image. Figure 5 In the diagram, frame 12 (the current frame) and frame 6 (the past frame most similar to the current frame) are shown with dashed lines for illustrative purposes only, and to indicate that these two frames are most similar to each other.

[0074] In reference Figure 4and Figure 5 In the described embodiment, the sliding window 207 is five frames in size. However, it should be noted that the size of the sliding window may differ from this size and may also change during operation of the image processing device.

[0075] In one embodiment and reference Figure 1 The image processing device 10, processor 20, is configured to adjust the number of frames within the selected frame result. Therefore, the size of the sliding window is adjusted, i.e., increased or decreased.

[0076] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to generate overlay data based on frames of the original image stream and / or quantity values ​​derived from frames of the original image stream.

[0077] Overlay data can highlight elements contained in frames of the original image data or add derived quantities to overlay frames (e.g., heart volume or heart size) and display them together with the frames. For example, the entire body of a structure can be marked, or the algorithm can process frames of the original image stream and output the quantity values ​​without marking individual markers in the frames that produce the quantity values. Furthermore, in one embodiment, overlay data can be generated based on contours contained in frames of the original image stream.

[0078] In one embodiment and reference Figure 1 The image processing device 10, with processor 20 configured to process multiple frames of the original image stream 210 in parallel, thereby obtaining a corresponding frame of the processed image stream 220 from each processing operation executed in parallel.

[0079] Assuming the processing time for each frame of the original image stream is approximately constant, when the processing time exceeds the time between two subsequent frames of the original image stream, the processor must be able to process multiple frames in parallel to avoid extending the processing time of each individual frame. For example, using an algorithm with a frame rate of 20 to 25 frames per second and a processing time of 2 seconds per frame to calculate the processed frames of the image stream, the processor must be able to process 50 frames in parallel to avoid extending the processing time of each individual frame.

[0080] The processor can process at least some or every frame of the original image stream according to the principles described herein. However, with reduced computing power, the processor can process every second frame (or every third frame or a higher number of frames) of the original image stream.

[0081] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to add the similarity value of the identified frame from the selected result of the frame with the highest similarity value to the output image stream.

[0082] The selection of previously processed frames may not include frames that are absolutely identical to the current frame. Therefore, it may be helpful for the operator of the imaging device to understand the similarity level between the current frame and the most similar frames identified in the past selection results. This similarity level can indicate how accurately the overlay data of past frames fits the current frame. The similarity value or percentage value indicating the similarity between the current frame and the selected past frames can be displayed on display 70 along with the frames of the output image stream.

[0083] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to compare the similarity value of the identified frame with the highest similarity value with a similarity threshold and generate an output image stream to indicate whether the similarity value is less than or equal to or greater than the similarity threshold.

[0084] This allows defining a similarity threshold that indicates the minimum required similarity to generate a minimal, reliable output image stream based on past frames. In this embodiment, the device indicates whether the similarity between the current frame and the most similar past frame (in the frame selection result) is higher or lower than the similarity threshold. If the similarity value is lower than the similarity threshold, the output may be unreliable for continuing, and the operator can choose to wait for the result of the image processing algorithm.

[0085] For example, similarity values ​​and / or symbols can be displayed to indicate whether the similarity value is less than or greater than the similarity threshold.

[0086] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to stop adding overlay data to the current frame if the similarity value is less than a similarity threshold.

[0087] In this embodiment, when the similarity is too low, overlay data will not be added to the current frame so that the operator of the image device will not be misled by potentially inaccurate overlay data.

[0088] In one embodiment and reference Figure 1 The image processing device 10, processor 20 is configured to compare overlay data of the first frame of the original image stream, which is assigned a frame-based selection result, with overlay data generated from the image processing algorithm applied to the first frame.

[0089] Figure 6 Is Figure 4 This embodiment is described in more detail based on the illustrated embodiment. See also... Figure 4As described, frame 11 of the original image stream 210 was initially assigned overlay data of a fourth frame, which was the most similar frame from the sliding window 207 when frame 11 was captured or provided by the image sensor. However, Figure 6 This shows a scene at a later time when frame 11 of the original image stream 210 has been processed by a (time-consuming) image processing algorithm. Figure 6 At the moment shown, the actual processing result of frame 11 is available, and at this time, the overlay data initially allocated to frame 11 (see...) Figure 4 It can overlay data generated from image processing algorithms (see...) Figure 6 This comparison allows for the determination of the accuracy of the initially assigned overlay data. Figure 6 In the middle, frame 17 is shown with a dashed line for illustrative purposes only and to indicate that this frame is the current frame.

[0090] In other words, this embodiment allows comparison of the current image frame of the original image stream, along with overlay data derived from the selection results of past frames (referred to as the estimated output), with the result of the current image frame being processed by an image processing algorithm (referred to as the processed output). The accuracy of the estimated output depends on the degree of similarity between the current frame and the frames in the selection results of past frames. If the similarity level between the past frames and the current frame is low, the overlay data selected from the past frames and assigned to the current frame may also have low accuracy. On the other hand, the processing results of the image processing algorithm are highly accurate. Therefore, taking into account the overlay data generated from the image processing algorithm, this embodiment allows the accuracy of the initially assigned overlay data to be determined.

[0091] More specifically, the first frame of the original image stream is assigned overlay data from past frames that meet similarity criteria. This initially assigned overlay data can be referred to as the estimated overlay data. However, this first frame undergoes image processing, and at some future time, the results of the image processing become available. The processing results contain overlay data calculated based on the first frame. The calculated overlay data (the result of the image processing algorithm) and the estimated overlay data (selected from the most similar past frames) are then compared to each other to determine the similarity between these different overlay data.

[0092] Further reference Figures 3 to 6 This paper describes a method for compensating for visual delays introduced by image analysis algorithms when their results are rendered into real-time perspective.

[0093] This method describes a general approach and apparatus that allows the use of computationally expensive image processing algorithms or algorithms hosted in cloud computing centers without introducing significant additional latency in the feedback loop. Although transmitting images to the cloud may introduce significant latency, the display of frames and processing results remains synchronized and in real time.

[0094] Therefore, an additional fast algorithm (an algorithm that compares the current frame with processed past frames) calculates the image similarity to the current frame for at least some or every frame in the sliding window 207 (temporally returned) of frames. The sliding window is positioned in such a way that the processing result is available for all frames in the sliding window 207. In the example, the most similar frame within the sliding window can be considered the best visual substitute for the current frame, so this paper assumes that the corresponding processing result is also close to the processing of the current frame (although the actual processing result of the current frame is not yet available). For the current frame, the substitute processing result is superimposed on the original stream (original image stream), and the original frame, along with the superimposed data from selected past frames, is added to the output image stream.

[0095] When the current frame has already been processed (later), the processor compares the initially assigned overlay data (time-returned) with the current true result. If the comparison reveals a significant error, an indication can be given to the operator on the display and / or the assignment of overlay data from past frames to the corresponding current frame can be completely disabled.

[0096] As an application example, during interventional X-rays, the operator can interactively change the position of the catheter tip / stent while visually inspecting the displayed image frames. In many intervention-guided applications, not only are image frames displayed, but also additional information extracted from the images simultaneously (overlay data). For example, in cardiac applications, stent placement can be visually enhanced relative to the vessel wall being acted upon. Dedicated processing algorithms can attenuate background noise and unwanted anatomical structures in the real-time image stream.

[0097] The additional processing in image processing algorithms used to compute overlay data typically introduces some delay between tip movement and the output image stream. Only delays on the order of a few hundred milliseconds are acceptable; otherwise, the catheter tip movement and image information may no longer be synchronized, making it difficult for the operator to correlate the effects of the tip movement with changes in the visual image content.

[0098] The method described herein allows the use of computationally expensive image processing algorithms and / or transmission channels (e.g., a connection between image sensor 60 and image processing device 10 on one hand, and a connection between image processing device 10 and display 70 on the other) without introducing significant latency in the feedback loop. Although the overall data transmission and processing are on the order of seconds per frame, the display of frames and processing results remains synchronized and in real time.

[0099] exist Figure 4 In the process, the segmentation result of frame 11 (the result of the image processing algorithm) is unknown, so the segmentation result of frame 4 is used and assigned to frame 11. However, after acquiring frame 17 (see...), Figure 6 The actual segmentation result of frame 11 is known. For example... Figure 6 As shown, the alternative overlay data of frame 4 and the actual segmentation result of frame 11 can now be compared and quantitatively characterized to identify and indicate the accuracy of the initially assigned overlay data from frame 4 to frame 11. This indicates how close the overlay data of frame 4 is to the actual calculated overlay data of frame 11. This step can also be performed for subsequent frames, and the moving average can be presented to the operator / user as a retrospective quality measurement or error measurement in the display along with the current frame and overlay data. Based on the presented information, the operator can determine how well the similarity algorithm performed several frames ago, and if the measurement is below a certain quality threshold, the overlay data obtained from past frames will no longer be displayed, as described with reference to some embodiments. If the quality improves, the overlay data can be displayed again.

[0100] In one embodiment, the ventricles of the human heart can be used as a guiding anatomical structure for ventricular segmentation. However, the ventricles are used for illustration only. Any single-frame-based (2D or 3D) processing task is possible. Utilizing a frame rate of 20-25 frames per second and a processing time of 2 seconds per frame, the algorithm allows for parallel processing of 50 frames without increasing the processing time for each individual frame.

[0101] As described above with reference to one embodiment, the length of the sliding window can be larger, and it can also vary over time, for example, increasing or decreasing over time. The calculation of similarity measurements can be performed on the order of microseconds to meet the real-time constraints of ultrasound. Furthermore, other similarity standards can be used.

[0102] For cardiac applications, similarity can be evaluated not only frame by frame, but also based on the complete cardiac cycle. Therefore, the entire sequence of frames can be compared (in one step), or the position of guided frames in the loop can be compared frame by frame to reduce the overall size of the sliding window.

[0103] The methods described in this article can also be used in ultrasound applications. In this case, the operator can control the probe of the image-forming process without moving the catheter tip or support.

[0104] Figure 7 An embodiment of a method 300 for processing images is shown. In this embodiment, method 300 includes the following steps:

[0105] Step 310: Receive the raw image stream with multiple frames;

[0106] Step 320: Process the frames of the original image stream using an image processing algorithm to obtain a processed image stream, wherein the processed image stream includes the original image stream and overlay data generated by the image processing algorithm, the overlay data containing additional information compared with the original image stream;

[0107] Step 330: Provide the processed original image stream frames and overlay data to the data storage;

[0108] Step 340: Generate the selection results of past frames of the original image stream that have undergone image processing algorithms;

[0109] Step 350: Compare the current frame of the original image stream with at least some frames from the selected frames;

[0110] Step 360: Determine the similarity values ​​between the current frame and at least some frames in the selection results of past frames;

[0111] Step 370: Based on the similarity with the current frame, identify matching frames from the selection results of past frames;

[0112] Step 380: Assign the overlay data of the frames identified from the selection results of the frames to the current frame;

[0113] Step 390: Add the overlay data assigned from the current frame and the matched past frames to the output image stream.

[0114] for Figure 7 Method 300, illustrated schematically, is applied and referenced in [the context of the diagram]. Figures 1 to 6 The image processing device 10 and the medical imaging device 100 described herein operate on the same principles. The functional features described with reference to the image processing device 10 and the medical imaging device 100 may be implemented as additional or optional steps of method 300.

[0115] In one embodiment, the method includes the steps of: comparing the similarity value of the identified frame with the highest similarity value with a similarity threshold, and generating an output image stream to indicate whether the similarity value is less than the similarity threshold, or whether it is equal to or greater than the similarity threshold.

[0116] In another embodiment, the method includes the step of comparing overlay data of a first frame of the original image stream, which is assigned based on a frame selection result, with overlay data generated from an image processing algorithm applied to the first frame.

[0117] In another exemplary embodiment of the invention, a computer program or computer program unit is provided, characterized in that it is adapted to perform method steps of the method according to one of the foregoing embodiments on a suitable system, for example when executed by a processing unit such as processor 20.

[0118] The computer program unit can therefore be stored on the computer unit or distributed across one or more computer units, which may also be part of embodiments of the present invention. Such a computing unit can be adapted to perform or induce the execution of the steps of the described methods. Furthermore, it can be adapted to operate components of the described apparatus. The computing unit is capable of automatically operating and / or executing user commands. The computer program can be loaded into the working memory of a data processor, such as processor 20. Therefore, the data processor can be configured to perform the methods of the present invention.

[0119] Various aspects of this invention can be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer-readable storage device executable by a computer. The instructions of this invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes. The instructions can be provided as a complete executable program, a partial executable program, a modification (e.g., an update) of an existing program, or an extension (e.g., a plug-in) of an existing program. Furthermore, some processing of this invention can be distributed across multiple computers or processors.

[0120] As described above, the processing unit or processor 20, such as a controller, implements the methods as described with reference to some embodiments. The controller can be implemented in various ways, utilizing software and / or hardware, to perform a variety of desired functions. A processor is one example of a controller, employing one or more microprocessors that can be programmed using software (e.g., microcode) to perform desired functions. However, a controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware performing certain functions and processors (e.g., one or more programmed microprocessors and associated circuitry) performing other functions.

[0121] Examples of controller components that may be employed in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0122] This exemplary embodiment of the invention covers computer programs that use the invention from the outset and computer programs that convert existing programs into programs that use the invention through updates.

[0123] Furthermore, the computer program unit may be able to provide all the necessary steps to complete the process of the exemplary embodiment of the above method.

[0124] According to other exemplary embodiments of the invention, a computer-readable medium, such as a CD-ROM, is provided, wherein the computer-readable medium has computer program units stored thereon, said computer program units as described in the preceding portion. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or solid-state medium provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0125] However, computer programs can also be presented via networks like the World Wide Web and can be downloaded from such networks to the working memory of a data processor. According to other exemplary embodiments of the invention, a medium is provided for making computer program units available for download, said computer program units being arranged to perform a method according to one of the foregoing embodiments of the invention.

[0126] It should be noted that embodiments of the present invention are described with reference to different subjects. Specifically, some embodiments are described with reference to method-type claims, while others are described with reference to device-type claims. However, unless otherwise stated, those skilled in the art will understand from the above and below description that any combination of features related to different subjects, in addition to any combination of features belonging to one subject, is also contemplated and disclosed with this application. However, all features can be combined, and the synergistic effect provided is not merely a simple sum of features.

[0127] While the invention has been described in detail with reference to the accompanying drawings and the foregoing description, such description should be considered illustrative or exemplary rather than restrictive. The invention is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the dependent claims, those skilled in the art will be able to understand and implement other variations to the disclosed embodiments in practicing the claimed invention.

[0128] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude multiple. A single processor or other unit can perform the functions of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously. Any reference numerals in the claims should not be construed as limiting the scope of protection.

Claims

1. An image processing apparatus (10), comprising: Processor (20); Data storage device (30); An input interface (40) is configured to provide the processor (20) with a raw image stream (210) having multiple frames. An output interface (50) is configured to provide an output image stream to a display; The processor (20) is configured as follows: Receive the original image stream (210); The frames of the original image stream (210) are processed by an image processing algorithm to obtain a processed image stream (220), wherein the processed image stream (220) includes the original image stream and overlay data generated by the image processing algorithm, the overlay data containing additional information compared with the original image stream (210); Processed frames of the original image stream (210) with corresponding overlay data are provided to the data storage (30); Select past frames from the processed frames of the original image stream. The current frame of the original image stream (210) is compared with at least some frames in the selection results of past frames, and a similarity value is determined for the current frame and the at least some frames in the selection results of past frames; Based on the similarity value with the current frame, a matching frame in the selection result is identified, and the overlay data from the matching frame is assigned to the current frame; and The current frame and the overlay data assigned from the matching frame are added to the output image stream.

2. The image processing apparatus (10) according to claim 1, wherein, The selection result of the frame has a predetermined size and includes a plurality of past frames corresponding to the predetermined size, and the processor (20) is configured to determine the selection result of the frame of the original image stream (210) by applying a sliding window algorithm to the past frames of the original image stream (210).

3. The image processing apparatus (10) according to claim 2, wherein, The processor (20) is configured to adjust the number of frames within the selected result of the frames.

4. The image processing apparatus (10) according to any one of claims 1-3, wherein, The processor (20) is configured to generate the overlay data based on the frames of the original image stream and / or the quantity values ​​derived from the frames of the original image stream.

5. The image processing apparatus (10) according to any one of claims 1-3, wherein, The processor (20) is configured to process multiple frames of the original image stream (210) in parallel to obtain a corresponding frame of the processed image stream (220) from each processing operation performed in parallel.

6. The image processing apparatus (10) according to any one of claims 1-3, wherein, The processor (20) is configured to add an indication of the similarity value of the matching frame in the selection result of the determined frame to the output image stream.

7. The image processing apparatus (10) according to any one of claims 1-3, wherein, The processor (20) is configured to: The similarity value of the matched frame is compared with a similarity threshold.

8. The image processing apparatus (10) according to claim 7, wherein, The processor (20) is configured to stop adding the overlay data to the current frame if the similarity value is less than the similarity threshold.

9. The image processing apparatus (10) according to any one of claims 1-3, wherein, The processor (20) is configured to compare the overlay data of the first frame of the original image stream, which is assigned the frame-based selection result, with the overlay data generated by the image processing algorithm applied to the first frame.

10. A medical imaging apparatus (100) for processing images, the apparatus comprising: Image sensor (60); Display (70); as well as Image processing apparatus (10) according to any one of claims 1-9; The image sensor (60) is communicatively coupled to the image processing device (10) to provide a raw image stream (210) to the image processing device (10); and The display (70) is communicatively coupled to the image processing device (10) to receive an output image stream from the image processing device (60).

11. A method (300) for processing an image, comprising the following steps: Receive (310) a raw image stream (210) having multiple frames; The original image stream (210) is processed (320) using an image processing algorithm to obtain a processed image stream (220), wherein the processed image stream (220) includes the original image stream and overlay data generated by the image processing algorithm, the overlay data containing additional information compared with the original image stream (210); The processed frames of the original image stream (210) and the corresponding overlay data are provided to the data storage (30) (330); Select past frames from the processed frames of the original image stream; Compare the current frame of the original image stream (210) with at least some frames from the selection results of past frames (350). Determine (360) the similarity values ​​of at least some frames in the selection results for the current frame and past frames; Based on the similarity value with the current frame, identify the matching frame in the selection result of the (370) frame; Assign (380) the overlay data of the frames identified from the selection results of the frames to the current frame; and The current frame and the overlay data assigned from the matching frame are added (390) to the output image stream to be provided to the display.

12. The method of claim 11, further comprising: The similarity value of the matched frames is compared with a similarity threshold, and If the similarity value is equal to or greater than the similarity threshold, then add the current frame and the assigned overlay data, or The current frame is added only when the similarity value is less than the similarity threshold.

13. The method according to claim 11 or 12, further comprising: The overlay data of the first frame of the original image stream, which is assigned the selection result based on the frame, is compared with the overlay data generated by the image processing algorithm applied to the first frame.

14. A computer program product comprising computer program instructions for controlling an image processing apparatus (10) according to any one of claims 1 to 9, wherein, When executed by the processing unit, the computer program instructions are adapted to perform the steps of the method according to any one of claims 11 to 13.

15. A computer-readable medium storing computer program instructions for controlling an image processing apparatus (10) according to any one of claims 1 to 9, wherein, When executed by the processing unit, the computer program instructions are adapted to perform the steps of the method according to any one of claims 11 to 13.

Images