Computer-implemented method for visualizing delay times in transmission images

Abstract

The present invention relates to a method for visualizing a delay time in an image, wherein the delay time describes a time interval between processing of an image by a transmitting device at a first processing time point and processing of the image by a receiving device at a second processing time point, the method comprising the steps of: the transmitting device pre-assigning a first mark depending on the first processing time point; the transmitting device processing original image data of an image so that a portion of an image segment describing a pre-assigned portion of the original image data is replaced by the first mark to provide modified image data of the image; the modified image data is transmitted from the transmitting device to a receiving device; the receiving device pre-assigns a second mark depending on the second processing time point; the receiving device processes the modified image data so that at least a portion of the modified image data is replaced by a superposition of the second mark and the modified image data in the image segment to provide final image data of the image; and the final image data is output.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for visualizing a delay time in at least one image transmitted from a transmitting device to a receiving device, wherein the delay time describes a time interval between processing of the image by the transmitting device at a first processing time and processing of the image by the receiving device at a second processing time. The invention also relates to a device for outputting at least one image. Background Art

[0002] When image data are transmitted between a transmitting device and a receiving device, delay times occur, which can fluctuate greatly, especially in the case of transmission via a public network (e.g. via the Internet or a virtual private network set up therein, etc.). By using a sufficiently large buffer memory, corresponding delays can be compensated in purely passive image observation. However, in applications in which image transmission is to be part of the control, for example in the remote control of a robot or a tool, as is known in principle from the field of workpiece processing or telemedicine, high delays or strong fluctuations in image transmission can lead to operating errors or impair hand-eye coordination, especially if the user of the received image data does not recognize or incorrectly assesses these delays or fluctuations or the extent of the delays or fluctuations.

[0003] Therefore, when incorrect operation could lead to significant costs or even health hazards, the application of remote control of processes based on video data is only feasible at least if: sufficiently short and stable transmission times can be guaranteed, for example by using a dedicated closed network, which, however, is not available in many application cases or thus results in high costs; or if the delay time between the processing on the sending device side and the receiving device side can be detected robustly enough or at least the exceeding of a limit value can be detected robustly enough by this delay time.

[0004] However, since, for example, in some telemedicine applications, erroneous failure to detect high or fluctuating delay times can lead to significant health risks for the patient, sufficiently high initial error safety of all involved components is required when the delay times are monitored only mechanically. This can be achieved by using sufficiently initial error-safe components, so that the involved software components must comply with, for example, software safety level C of the international standard IEC 62304 and generally must not use commercial hardware and software, or by additional measures, for example by redundancy. For example, a watchdog mechanism can be used to monitor an internal time counter, which has sufficient initial error safety or is verified by additional mechanisms.

[0005] In principle, it is possible that an experienced user observing the image data stream can already recognize when longer delays occur between individual images. However, in medical imaging or other fields of application, only slight movements in the images are usually detectable over longer periods of time, so that even an experienced user cannot easily detect a short additional delay before a further image is displayed in all cases, for example. Summary of the invention

[0006] Therefore, the technical problem to be solved by the present invention is to support the user in estimating the delay time or identifying the long delay time between the image processing on the sending device side and the receiving device side, thereby, in particular, enabling the user to robustly identify fluctuations in the transmission time, thereby avoiding erroneous operations on the user side due to such delays even if these fluctuations cannot be automatically identified.

[0007] According to the invention, this technical problem is solved by a method of the type mentioned at the outset, which method comprises the following steps:

[0008] - a first identifier is predefined by the transmitting device, wherein the first identifier depends on a first processing time point,

[0009] - processing the raw image data of the image by the transmitting device such that a part of the raw image data describing a predetermined image segment is replaced by the first marking or by a superposition of the first marking with the raw image data in the image segment in order to provide modified image data of the image,

[0010] - transmitting the modified image data from the sending device to the receiving device,

[0011] - a second identifier is predefined by the receiving device, wherein the second identifier is dependent on a second processing time point,

[0012] - processing the modified image data by the receiving device such that at least a part of the modified image data in a predetermined image segment is replaced by a superposition of the second marking with the modified image data in the image segment in order to provide final image data of the image,

[0013] - Outputting the final image data via a display device in order to visualize the delay time in the image segment of the image.

[0014] In the method according to the invention, a first marking and a second marking are superimposed in the image segment. Since the first and second markings depend on the respective processing time points, the delay time between these time points and therefore in particular the transmission time between the transmitting and receiving devices, which usually at least mainly accounts for the variable part of the delay time, can be visualized to a user observing the image on the display device based on the superposition.

[0015] As will be explained in more detail later, by suitable selection of the first and second markings, in particular intuitive marking can be performed, so that, for example, a delay time above the tolerance time can be marked by inserting a red bar or circle segment and a sufficiently low delay time can be marked by inserting a green bar or circle segment, wherein the length of the bar or the angle spanned by the circle segment can, for example, indicate the intensity of exceeding or falling below the tolerance time. Thus, the user can recognize at a glance when a high delay time is currently present and, therefore, for example, remote control based on the current image should be avoided.

[0016] At the same time, by appropriately selecting the mark or the dependency of the mark on the corresponding processing time point, it can be ensured that when the image sequence or image data stream is displayed on the display device, the image display in the segment remains in motion, so that the user can, for example, directly recognize the freezing of the image, thereby avoiding erroneous operations caused by this.

[0017] Preferably, the transmitting device and the receiving device each have an internal timer, wherein in the case of modern timers it can be assumed that even over a long period of time, due to tolerances, only very small time deviations occur. Time synchronization can be performed by synchronization protocols known per se, such as NTP, SNTP, PTP, etc. Here, a pair consisting of a transmitting device and a receiving device can be synchronized directly with each other, wherein one of the devices can be used as a master for the synchronization. However, preferably, both the transmitting device and the receiving device are synchronized to at least one external time server. This is particularly advantageous when the transmitting device should cooperate with different receiving devices (and / or vice versa). As foreseeable errors, slow drifts or sudden jumps may occur in the internal timer. Both can be inherently identified as errors in the method described, and errors in time synchronization are usually identified before the image is transmitted. If the error of the internal timer will approximately compensate for the fluctuation or increase of the delay time that actually occurs, the error in the internal timer will be problematic. However, this is highly unlikely.

[0018] The described procedure for visualizing delay times also has the advantage that, apart from the initial time synchronization, the visualization itself requires communication in only one direction, namely from the transmitting device to the receiving device, and in principle, no further information other than the modified image data of the image needs to be transmitted to achieve such a visualization. Since a part of the image or image data stream is used for the visualization of the delay times, it is also easy to know which delays occurred in the case of recording the image or image data stream afterwards, so that, for example, it is also possible to record whether the user responded appropriately to the delay that occurred or whether the user responded appropriately to the delay that occurred. Since the method relies directly on the image or video data stream itself and does not require any additional components or mechanisms, the method is inherently safe.

[0019] The superposition of the first mark with the original image data and / or the superposition of the second mark with the modified image data can be an additive or subtractive superposition. For example, in the case of an additive superposition, the color value of the corresponding mark and the color value of the corresponding image data can be added. In this case, it may be particularly advantageous when the second mark only contains color information in a color channel that is not used in the first mark. For example, the first mark may only obtain non-zero entries in the green color channel, and the second mark may only obtain non-zero entries in the red color channel. If this is the case, then in the corresponding superposition, the previous values ​​in the original image or the original or modified image data in the color channel used by the corresponding mark can also be simply overwritten by the values ​​pre-given by the mark, etc.

[0020] A first mark can be pre-set so that the first mark changes periodically with the first processing time point. In addition or alternatively, a second mark can be pre-set so that the second mark changes periodically with the second processing time point. The periodicity of the first mark and the second mark change can be the same. In other words, the first and second marks used can be repeated at fixed time intervals, for example, at time intervals of hundreds of ms or several seconds. For example, when a periodicity of 1s is used, this may result in that a delay time of 200ms and a delay time of 1200ms can no longer be distinguished from a delay time of 1200ms based on the image alone. However, since very long delay times can be easily identified at least in the image sequence, this is still advantageous in the sense of intuitive accessibility of the mark. Only when the delay time changes in a jumpy manner, a superimposed representation that is essentially unchanged due to the periodicity will be generated in the image sequence, wherein the size of the change corresponds approximately to the cycle duration, and wherein the delay time before and after the jump is essentially constant. On the one hand, this is very unlikely, and this instability of image transmission can usually be identified anyway, for example by suitable markings, as will be explained in more detail below.

[0021] It is also possible to use a plurality of first and second markers which change with different periods from one another. If the first and second markers are respectively superimposed with the same period duration, the delay time can be represented with high accuracy based on the superposition of the markers with the shorter period duration, and at least when the delay time remains below the longer period, the ambiguity caused by the shorter periodicity can be eliminated based on the superposition of the markers with the longer period duration. The different first and second markers can be represented, for example, as concentric rings. Functionally, this corresponds to, for example, the use of the hour and minute hands of a clock. For example, 1s and 5s can be used as different periodicities.

[0022] The first marking may be predetermined such that the predetermined first marking template is rotated on the one hand and / or at least partially shifted on the other hand relative to the image segment according to the first processing time point. Additionally or alternatively, the second marking may be predetermined such that the predetermined second marking template is rotated on the one hand and / or at least partially shifted on the other hand relative to the image segment according to the second processing time point.

[0023] In the case of rotation of the corresponding marking template, in particular a circular, circle segment, annular or ring segment-shaped marking template can be used, which can rotate around the rotation axis of the center point of the circle. Thus, for example, a rotating circle with a color trend or saturation or brightness gradient can be used as a corresponding mark, and by superimposing two such circles or marks, a fixed rotating pattern can be formed with a fixed delay time. Therefore, based on the cessation of the rotation of the superimposed pattern, an interruption or stop of the image flow can be directly identified, and a change in the delay time leads to a change in the rotating pattern or the color distribution of the rotation. A particularly advantageous design scheme in this regard will be explained later.

[0024] If the corresponding marking is to be provided by moving the corresponding marking template according to the corresponding processing time point, for example, a rectangular image segment can be used to display the marking, and the movement can be carried out parallel to both sides of the rectangle. The movement can be carried out in particular in such a way that the entire marking template is first moved, and then the part of the marking template that is pushed out of the image segment is pushed back into the image segment from the opposite side of the image segment. Thus, a continuous, periodic movement is achieved similar to a rotation. However, due to the displacement explained, different sections of the marking template are therefore moved in different directions.

[0025] The first and / or second marking and / or the first and / or second marking template can be a linear or circular or annular or circle segment or annular segment gradient of the brightness and / or color saturation of the corresponding color or include such a gradient. Alternatively or additionally, the hue within the gradient can also be changed. The gradient can be designed quasi-continuously so that, for example, the brightness and / or color saturation and / or hue in each image point changes along the gradient path. However, it is also possible that the gradient is more coarsely resolved so that the brightness and / or color saturation and / or hue changes step by step, for example in multiple, in particular in at least three steps.

[0026] The corresponding marking or marking template can be formed completely by a corresponding gradient, in particular with the exception of a section of the marking or marking template, wherein the section has the same color (in particular white or black) all over or is marked as transparent. The marking or marking template in this section cannot in particular change the image information superimposed thereon, which can be achieved, for example, in the case of additive superposition by setting all color channels to 0 or selecting the color black.

[0027] The result of such a selection of markings or marking templates will be explained below based on an example, wherein a green gradient is used in the first marking template, whose color saturation and / or brightness increases in the direction of movement or rotation, and a red gradient is used as the second marking template, whose color saturation and / or brightness also increases in the direction of movement or rotation. By selecting the position of the saturation or brightness maximum in the corresponding gradient, it can be achieved that when the delay time or the difference in the processing time points corresponds exactly to the tolerance time, the position of the saturation or brightness maximum coincides at the same angular position in the case of rotation or coincides at the same position in the case of movement. In this case, a continuous yellow saturation or brightness gradient is generated by superposition.

[0028] If the delay time increases from this tolerance time, this will lead to a later second processing time point at the same first processing time point, and thus to a stronger shift or rotation of the second marking template. Therefore, the maximum value of the color saturation or brightness of the second marking stands out from the maximum value of the color saturation or brightness of the first marking, and thus stands out into the following range: in this range, the first marking has a low color saturation or brightness or is even black or transparent, as explained above. This produces a substantially red stripe or a substantially red circle segment with strong saturation or brightness, which notifies the user before the delay time exceeds the tolerance time and whose width depends on the degree of exceeding the tolerance time.

[0029] Apart from this red stripe, the remaining superimposed pattern forms a yellow color gradient and, in the case of correspondingly marked black or transparent segments, a pure green monochrome area remains, but in this monochrome area the color saturation or brightness is low and therefore does not stand out in color. Thus, in the case of too high a delay time, the image impression is characterized by strongly saturated red stripes or circle segments.

[0030] Conversely, if the delay time is lower than the tolerance time, the most saturated or bright area of ​​the first mark protrudes beyond the most saturated or bright area of ​​the second mark in the direction of movement or rotation, thereby generating a green circle segment or green stripe, whose width in turn shows how far the delay time is still below the tolerance time.

[0031] From the examples described, it can be seen that a very intuitive visualization of the delay time can be achieved for the user with only a low technical outlay. The same result can obviously be achieved if the direction of the color saturation or brightness trend is reversed and the selection of the green and red colors of the marking template is simultaneously swapped. Although green and red are particularly clear signal colors for visualizing a problem-free or problematic operating state, it may also be appropriate to use other colors for the corresponding gradients, for example in order to enable adaptation to the color vision deficiencies of the user.

[0032] In an alternative design, for example, a sequence of numbers or symbols can be used as the first pattern or first pattern template, and a mask can be used as the second pattern or second pattern template, by which the first pattern is completely covered outside of certain segments or sections of the second pattern. This can be achieved, for example, in the case of additive superposition when the second pattern template is black in certain areas in which the first pattern is to be recognizable and white in all other areas. In this case, for example, the delay time can be represented as a numerical value by the superimposed pattern, etc.

[0033] In the method according to the invention, an image stream of a plurality of images that follow one another in time, in particular an image stream of a medical image data acquisition, can be transmitted from a transmitting device to a receiving device, wherein for each image a visualization of the corresponding delay time is performed by superimposing a corresponding first marking and a corresponding second marking. The visualization of the delay time according to the invention is particularly suitable in the case of an image data stream, since on the one hand a change in the delay time during the image data stream can be directly detected by changing the superimposition pattern, and on the other hand, as already explained above, a freeze of the image data stream, i.e. a permanent display of the same image, can also be detected very quickly.

[0034] Medical image data acquisition can be, for example, X-ray imaging, ultrasound imaging or video imaging. The imaging can be performed by an imaging device, for example, by an imaging device arranged on a catheter, or also by a fixed acquisition device that images the patient or a part of the patient, for example, to track the catheter.

[0035] The user's operation input can be collected by the receiving device via the operating device, wherein the operation input or the control information determined by the operation input is transmitted from the receiving device to the sending device, and then the sending device controls at least one actuator according to the operation input or the control information. In principle, the method according to the present invention can be used for remote control of any process. For example, a robot or workpiece processing can be remotely controlled, or an actuator for telemedicine applications can be involved.

[0036] If the actuator is used to manipulate the interventional device in the telemedicine framework, the interaction between the interventional device and the patient cannot be part of the method according to the invention. Thus, for example, a surgical operation performed in the telemedicine framework can be performed outside the claimed method.

[0037] In many applications, the total delay including image acquisition and control should be less than 500 ms, since a larger total delay significantly affects hand-eye coordination. In some medical imaging processes, such as X-ray imaging, a delay of about 200 ms is already caused by the imaging itself, and it must be assumed that approximately the same time is required to transmit operating inputs or control information as to transmit the corresponding image, so that, for example, a tolerance time of about 150 ms should be observed, which should not be exceeded by the delay time.

[0038] In this case, for example, circles can be used as first and second marking templates, respectively, wherein one quarter of the circle is black or transparent, while the other three quarters form the corresponding color saturation or brightness gradient. A complete rotation of the corresponding circle can correspond to a change of 600 ms in the corresponding processing time point, so that the marking template can be selected, that is, assuming that the first and second processing time points are the same, the black or transparent quarter of the second marking template is offset by 90° in the opposite direction relative to the rotation of the black or transparent quarter of the first marking template, so that these black or transparent quarters are superimposed with a delay time equal to the tolerance time, resulting in the superposition of the gradient without color fringes as described above.

[0039] At least a portion of the final image data in the image segment can be evaluated by an evaluation algorithm in order to determine a measure of the delay time, wherein, when a trigger condition depending on the measure is met, a warning signal is output to a user of the receiving device and / or a parameter of an imaging device providing the image and / or a parameter according to which the control of the actuator is adjusted. In the above example, in which a saturation or brightness gradient of red and green is used as a marking template, it can be checked, for example, whether in the image segment with respect to a pixel or a minimum number of pixels a boundary value of the red color channel is exceeded and at the same time a boundary value of the color value of the green color channel is not exceeded.

[0040] Although the process according to the invention enables the user himself to directly estimate the delay time that actually exists, it may still be appropriate to additionally inform the user as a precaution that the delay time has exceeded a limit value or to adjust the parameterization of the device used. For example, in the case of high delay times, the maximum movement speed and / or the force that can be provided by the actuator controlled by the operating input can be limited, and / or an attempt can be made to influence the imaging, for example in order to achieve a higher frame rate and thus compensate for the unreasonably high delay time by reducing the time required for image acquisition. Alternatively, it may also be appropriate to reduce the frame rate of the imaging when the trigger condition is met, for example in order to reduce the radiation load in the case of X-ray imaging. This may be appropriate because in the case of high delays, for example in the case of medical interventions, one should only act very slowly anyway, so that a lower frame rate may also be sufficient.

[0041] In addition to the method according to the invention, the invention also relates to a device for outputting at least one image, the device having a transmitting device, a receiving device and an output device.

[0042] - wherein the transmitting device is configured to: predetermine a first marking, which depends on a first processing time point at which the image is processed by the transmitting device; and process raw image data of the image so that a part of the raw image data describing a predetermined image segment is replaced by the first marking or by a superposition of the first marking with the raw image data in the image segment, in order to provide modified image data of the image; and transmit the modified image data to the receiving device,

[0043] -Wherein, the receiving device is configured to: pre-set a second mark, which depends on a second processing time point at which the image is processed by the receiving device; and process the modified image data so that at least a part of the modified image data in the pre-set image segment is replaced by a superposition of the second mark and the modified image data in the image segment so as to provide final image data of the image; and control the output device so as to output the final image data, whereby, in the image segment of the image, a delay time describing the time interval between the first processing time point and the second processing time point is visualized.

[0044] The device according to the invention can be used in particular to implement the computer-implemented method according to the invention, or the computer-implemented method according to the invention can be implemented during the operation of the device according to the invention. In any case, the device according to the invention can be expanded with the features mentioned in relation to the computer-implemented method according to the invention with the advantages mentioned there, and vice versa.

[0045] The receiving device may have an operating device for capturing an operating input of a user, wherein the receiving device is configured to transmit the operating input or control information determined by the operating input from the receiving device to the transmitting device, wherein the transmitting device is associated with at least one actuator of the device, wherein the transmitting device is configured to control the actuator based on the operating input or the control information. Advantageous characteristics of such a device for remotely operating an actuator based on a transmitted image or image data stream have already been explained above with respect to the method according to the invention.

[0046] The device can be in particular a telemedical device, wherein the actuator or at least one of the actuators is used to manipulate components of the medical imaging device of the device and / or interventional devices that interact with the patient. Thus, by using the device according to the invention, for example, imaging procedures can be remotely controlled, but also, for example, telemedical surgical or interventional procedures can be performed, etc.

[0047] In the computer-implemented method according to the invention or in the device according to the invention, the transmission of the modified image data can in principle take place via any communication path. For example, a direct physical data line can be used as a communication path. However, the described method and the described device are particularly advantageous if the transmission time and / or processing time can be varied, i.e., for example, in the case of transmission in a packet broker network or due to the data encryption used. In an advantageous embodiment, the modified image data can therefore be transmitted via the Internet or via a virtual private network implemented on the Internet. Within the framework of the transmission, any authentication and / or encryption methods can be used, i.e., known methods and corresponding methods yet to be developed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Further advantages and details of the present invention are obtained from the following examples and the associated drawings. Here, schematically in the drawings:

[0049] Figure 1 shows a flow chart of an embodiment of a method according to the present invention,

[0050] Figure 2 An embodiment of the device according to the invention is shown, and

[0051] Figure 3 Shows Figure 1 Schematic diagram of a possible marking template and the overlay of the marking template with image data in the illustrated embodiment. DETAILED DESCRIPTION

[0052] Figure 1 A flow chart of a computer-implemented method for visualizing a delay time in at least one image transmitted from a transmitting device to a receiving device is shown. Figure 2 Explain the method, Figure 2 An exemplary device is schematically shown, by which the method can be implemented.

[0053] For a better understanding, before discussing the individual method steps, a brief overview of the method sequence should first be given. The central idea here is that first a first marking 6 is predefined at a first processing time 4 at which the image 3 is processed by the transmitting device 1, for example by the communication device 13 of the transmitting device 1, which first marking depends on the processing time 4 at which the processing takes place. The first marking replaces the original image data 7 of the image 3 in the image section 8 or is superimposed on it in order to provide modified image data 9, which are then transmitted to the receiving device 2 or its communication device 14. In this example, the transmission takes place via a packet broker network 15, for example the Internet or a virtual private network implemented on the Internet.

[0054] When the image 3 is processed by the receiving device 2, for example, when the image is processed by the communication device 14 of the receiving device immediately after reception, a second marker 10 is predefined, which second marker depends on the second processing time point 5 at which the processing by the receiving device 2 is performed and is superimposed with the modified image data 9 in the image segment 8 or in particular with the first marker 6 in order to provide final image data 11 output by the display device 12.

[0055] As will be referred to later Figure 3 As explained in more detail by way of example, superposition of first and second markings 6 , 10 predefined according to the respective processing times 4 , 5 enables the delay time, ie the interval between the processing times 4 , 5 , to be visualized for the user in an easily intuitively recognizable manner.

[0056] exist Figure 1 In the example shown, only a single image 3 is transmitted and modified to visualize the delay time. However, as already explained in more detail in the general part, the described process is particularly advantageous when an image data stream or a sequence of images 3 (which can be generated in particular at a constant frame rate) is to be transmitted and visualized.

[0057] exist Figure 1 In the design shown, in step S1, the image 3 to be transmitted or its original image data 7 is first captured by the imaging device 30 of the transmitting device 1. Figure 2 The device shown is, for example, a telemedicine device or generally a medical imaging image is to be transmitted, and the imaging device 30 can be, for example, an ultrasound sensor, an X-ray sensor or also a camera.

[0058] However, the described method or described device for outputting images is purely exemplary in relation to medical imaging. The described process or described device can be used generally for image transmission, in particular in combination with remote control, ie, for example, also in the field of robot or tool control.

[0059] Steps S2 and S3 serve to predetermine a first marking 6 according to a first processing time point 4 at which the image 3 is processed by the transmitting device 1 or, in this example, specifically by the communication device 13 of the transmitting device. Figure 3 As explained in more detail, in step S2, a marking template 17 is first predefined, which can be predefined as fixed, and then rotated about the rotation axis 18 by an angle which depends on the first processing time 4, such as Figure 3 As shown by arrow 19 in FIG. 1 . Since a rotation of a multiple of 360° results in the same marking 6, the first marking 6 changes periodically with the first processing time point 4. As already explained in the general part, a plurality of markings with different periodicities can also be used. However, for the sake of simplicity, the following explanation is based on the use of just one first marking and one second marking.

[0060] In step S4, the raw image data 7 of the image 3 are subsequently processed by the transmitting device 1 so that the part of the raw image data 7 which describes the predetermined image section 8 is replaced by the first marking 6. Figure 3 In the example shown, this can be easily achieved because the segment 8 is arranged in a header line of the image 3 which can record additional information 41 , such as patient data or recording parameters, so that the image 42 of the patient tissue is not changed by replacing the original image data 7 .

[0061] If no corresponding free space is available in the image 3, the first mark 6 may alternatively be superimposed with the original image data 7 in the segment 8. However, the information may remain separable, e.g. the image 42 is black and white data, while the mark 6 contains data in a color channel only, etc.

[0062] By inserting the first marker 6 , modified image data 9 are provided after step S4 , which are transmitted to the receiving device 2 or its communication device 14 in step S5 .

[0063] In steps S6 and S7, the receiving device 2 or its communication device 14 generates a second marking 10, which depends on the second processing time 5 at which the image 3 was processed by the receiving device 2 or its communication device 14. This is done in a similar way to the generation of the first marking 6 explained above. For this purpose, in step S6, first a second marking template 20 is provided, such as in Figure 3 Then in step S7, the second marking module rotates around the rotation axis 21, as in Figure 3 As is indicated by arrow 52 in FIG. 1 , wherein the rotation angle depends on the second processing time.

[0064] Subsequently, in step S8, the generated second marking 10 is superimposed on the modified image data 9 in the image section 8 and thus on the first marking 6. Thus, superimposed patterns 43 to 45 are generated, which depend on the relative orientation of the first and second markings 6, 9 and thus on the time interval or delay time between the first and second processing time points 4, 5. In step S16, the generated final image data 11 are output by correspondingly controlling the display device 12.

[0065] If a plurality of images 3 are transmitted one after another, wherein the delay time between the respective first and second processing time points 4, 5 remains essentially constant, then the same superposition pattern 43 to 45 is displayed in all these images, wherein, however, the superposition pattern is rotated about different rotation angles due to the different processing times of the images. Thus, a uniform image stream is provided with a fixed rotation superposition representation, so that freezing of the image stream can be easily recognized. Changes in the delay time, for example due to the transmission of the image data over different paths, lead to changes in the superposition pattern itself. This will be explained below according to Figure 3 The example shown is explained in more detail.

[0066] In this example, the first and second marking templates 17, 20 are circular and include gradients 22, 26 of color saturation and / or brightness of the respective colors. In the case of the marking template 17, the saturation and / or brightness of green varies within the gradient 22 from a maximum at the front edge 23 of the gradient 22 in the direction of rotation to a minimum at the rear edge 24 in the direction of rotation. In the marking template 20, the color saturation and / or brightness of red varies within the gradient 26 from a maximum at the front edge 27 in the direction of rotation to a minimum at the rear edge 28 in the direction of rotation. In the remaining segments 25, 29 of the respective marking templates 17, 20, all color channels are set to 0, so that in the case of additive superposition of the generated markings 6, 10 used, the segments 25, 29 have no influence on the image data.

[0067] The marking templates 17, 20 therefore differ from one another in their angular position and in the color of the respective gradients 22, 26. The different angular positions can be selected such that they are exactly compensated when the delay time corresponds to a predefined tolerance time.

[0068] For example, if it is assumed that the first processing time point coincides with a rotation angle of 0°, the first marking 6 will be identical to the first marking template 17, and the delay time corresponds to the rotation angle of 90° of the second marking template 20 for providing the second marking 10, so that the colorless sections 25, 29 of the resulting markings 6, 10 coincide exactly. This is exemplary with respect to Figure 3 44 is shown in the superimposed representation in FIG. Here, the circle segment 47 remains black, because the segments 25 and 29 are superimposed there, and a yellow gradient is produced in the circle segment 46, because the red and green color gradients of the marking templates 17, 20 are added there.

[0069] In contrast, when the delay time corresponds only to a rotation angle of approximately 60°, i.e., when the delay time is below the tolerance time, a superimposed representation 43 is produced. Here, the circle segment 47 continues to remain black and the circle segment 46 continues to exhibit a yellow saturation or brightness gradient. In contrast, in the circle segment 48, the color there is predetermined only by the first marking template 17, so that due to the high color saturation and / or brightness in the region of the edge 23, a green region with high color saturation and / or brightness is produced there.

[0070] In the circle segment 49, by contrast, the color is predetermined only by the second marking template 20. Thus, a red region is produced there, wherein, however, due to the lower color saturation or brightness in the region of the edge 28, the color saturation and / or brightness there is very low, so that the circle segment 49 also appears at least approximately black.

[0071] A delay time that falls below the tolerance time is thus visualized by displaying a clearly recognizable green circle segment 48 next to a circle segment 46 with a yellow gradient, wherein the angular extent of the circle segment 48 depends on the extent to which the delay time falls below the tolerance time.

[0072] Conversely, in the case where the delay time exceeds the tolerance time, a superimposed representation 45 is generated. The circle segments 46, 47 continue to be covered with a yellow saturation or brightness gradient or are black. The color in the circle segment 50 is only predetermined by the second marking template 20, so a red with high saturation or brightness is generated. The color in the circle segment 51 is only predetermined by the first pattern template 17, so a low green saturation or brightness is generated, which basically continues to appear black to the observer.

[0073] A delay time exceeding the tolerance time is therefore marked by a clearly recognizable red color segment adjoining the yellow color gradient, wherein the angular extension of the red circle segment 50 depends on the extent to which the delay time exceeds the tolerance time.

[0074] Based on the superimposed representations 43 to 45 shown in the image 3 or the image detail 8 , the user can therefore directly see how high the current delay time is or whether the current delay time is above or below a predefined tolerance time and how large the deviation from the tolerance time is.

[0075] Further advantageous embodiments of the markings 6 , 10 or of the marking templates 17 , 20 , by means of which this can also be achieved, have already been explained in the general part of the description.

[0076] In some applications, it may be advantageous to additionally automatically evaluate the delay time. Possible configurations for this are described in Figure 1 In the example shown, this is illustrated by steps S9 to S11 .

[0077] In step S9, at least a portion of the final image data 11 in the image segment 8 is evaluated by an evaluation algorithm 35 in order to determine a measure 36 for the delay time. For example, it can be evaluated whether there is a partial segment in the image segment 8 in which high values ​​are present in the red color channel and no values ​​or very low values ​​are present in the other color channels. The measure 36 for the delay time can be, for example, how many image points in the segment 8 this is the case.

[0078] In step S10 , it is then checked whether trigger condition 37 is met, which may be the case in particular if the delay time exceeds a tolerance time or exceeds a predefined limit value, or if measure 36 exceeds a limit value.

[0079] If the trigger condition 37 is not met, no additional steps are required due to the high delay time, so that in this example, steps S13 to S15 to be explained later can be executed.

[0080] If, on the other hand, the trigger condition 37 is met, a warning signal 38 can be output in step S11 to the user of the receiving device 2. For example, an acoustic, optical or tactile warning can be provided, which informs the user that a high delay time exists. This can be particularly suitable when the user is to remotely control a process based on the transmitted image or a plurality of transmitted images, as will be explained later.

[0081] Additionally or alternatively, parameters 39 of the imaging device 30 providing the image can be predefined in step S12. This can be suitable, for example, in order to increase the frame rate and thus reduce the additional delay required for imaging, in order to thereby compensate for long transmission times or the like.

[0082] Additionally or alternatively, a parameter 40 can also be adjusted, on which the control of the actuator 32, which will be explained later, depends. This can be used, for example, to reduce the force and / or the control path of the actuator remotely controlled by the user in the presence of a large delay time, in order to compensate for the greater inertia of the operator's reaction due to the delay time.

[0083] As already explained, the explained visualization of delay times within the framework of image transmission or processing may be particularly suitable if a user, to whom corresponding images 3 are provided via display device 12 , remotely controls a process based on these images 3 or image sequences.

[0084] To implement such remote control, in step S13, operating inputs 33 may first be recorded via operating means 31 of receiving device 2. Operating inputs 33 or control information 34 determined therefrom are transmitted back to transmitting device 1 in step S14 and used there to control actuator 32 in step S15.

[0085] Such a control can generally be used in the field of robotics. Exemplary application cases are remote control of mobile robots, workpiece handling or telemedicine applications. By means of the visualization of the delay times, the user always knows to what extent the current image 3 is relevant to the current state of the imaging object (e.g. a patient or a medical tool) or to what extent the current image display may no longer be up-to-date due to high delays. This information can be taken into account in the framework of the remote control. As a result, the user can generally act more cautiously in the case of longer delay times or can generally omit critical steps or any operator input in the case of very high delay times.

[0086] if Figure 2 If the device shown is, for example, a telemedicine device, the actuator 32 can control components of the medical imaging device, such as a diaphragm or a C-arm, in order to adapt the imaging to the operator's wishes. In addition or alternatively, the actuator or another actuator 32 can, for example, control an interventional device, by which a medical intervention can be remotely controlled or, for example, a catheter can be positioned.

[0087] Since in the method according to the invention and the device according to the invention the corresponding markings are predetermined independently of one another by the transmitting device 1 or the receiving device 2 according to the corresponding processing time 4, 5, at least approximate time synchronization is required between the receiving device and the transmitting device 1, 2. Although such synchronization can be carried out directly between the transmitting device and the receiving device, it is expedient to synchronize the transmitting device 1 and the receiving device 2 via at least one common time server 16, in particular when the transmitting device 1 is to cooperate with a plurality of receiving devices 2 (and / or vice versa).

[0088] Although the present invention has been illustrated and described in more detail through preferred embodiments, the present invention is not limited to the disclosed examples and those skilled in the art may derive other variations therefrom without departing from the scope of protection of the present invention.

Claims

1. A computer-implemented method for visualizing a delay time in at least one image (3) transmitted from a transmitting device (1) to a receiving device (2), wherein: The delay time describes the time interval between the processing of the image (3) by the sending device (1) at a first processing time point (4) and the processing of the image (3) by the receiving device (2) at a second processing time point (5), and the method comprises the steps of: - predefining a first identifier (6) by the transmitting device (1), wherein the first identifier (6) depends on the first processing time (4), - processing the original image data (7) of the image (3) by the transmitting device (1) so that a part of the original image data (7) describing a predetermined image segment (8) is replaced by the first marking (6) or by a superposition of the first marking (6) with the original image data (7) in the image segment (8) in order to provide modified image data (9) of the image (3), - transmitting the modified image data (9) from the transmitting device (1) to the receiving device (2), - predefining a second identifier (10) by the receiving device (2), wherein the second identifier (10) depends on the second processing time (5), - processing the modified image data (9) by the receiving device (2) such that at least a portion of the modified image data (9) in the predetermined image segment (8) is replaced by a superposition of the second marking (10) with the modified image data (9) in the image segment (8) in order to provide final image data (11) of the image, - outputting the final image data (11) via a display device (12) in order to visualize the delay time in an image section (8) of the image (3).

2. The computer-implemented method of claim 1, wherein: The first marker (6) is predetermined so that the first marker changes periodically with the first processing time point (4), and / or the second marker (10) is predetermined so that the second marker changes periodically with the second processing time point (5).

3. The computer-implemented method according to claim 1 or 2, characterized in that The first marking (6) is predetermined so that a predetermined first marking template (17) is rotated on the one hand and / or moved at least in sections on the other hand relative to the image segment (8) according to the first processing time point (4), and / or the second marking (10) is predetermined so that a predetermined second marking template (20) is rotated on the one hand and / or moved at least in sections on the other hand relative to the image segment (8) according to the second processing time point (5).

4. The computer-implemented method according to claim 1 or 2, characterized in that: The first and / or second marking (6, 10) and / or the first and / or second marking template (17, 20) are respectively linear or circular or annular or circle segment or annular ring segment gradients (22, 26) of the brightness and / or color saturation of the corresponding color or include said gradients (22, 26).

5. The computer-implemented method of claim 1 or 2, wherein: An image stream of a plurality of images (3) which follow one another in time is transmitted from the transmitting device (1) to the receiving device (2), wherein for each of the images (3) a corresponding delay time is visualized by superimposing a corresponding first marker and a corresponding second marker (6, 10).

6. The computer-implemented method of claim 1 or 2, wherein: The receiving device (2) collects a user's operation input (33) through an operating device (31), wherein the operation input (33) or control information (34) determined by the operation input (33) is transmitted from the receiving device (2) to the sending device (1), and then the sending device (1) controls at least one actuator (32) according to the operation input (33) or the control information (34).

7. The computer-implemented method of claim 1 or 2, wherein: At least a portion of the final image data (11) in the image segment (8) is evaluated by means of an evaluation algorithm (35) in order to determine a measure (36) of the delay time, wherein, when a trigger condition (37) depending on the measure (36) is met, a warning signal (38) is output to a user of the receiving device (2) and / or a parameter (39) of an imaging device (30) providing the image and / or a parameter (40) based on which an actuator (32) is controlled is adjusted, The actuator (32) is controlled according to the user's operation input or control information determined by the operation input.

8. The computer-implemented method of claim 5, wherein: The image stream is an image stream acquired by medical image data.

9. A device for outputting at least one image (3), comprising a transmitting device (1), a receiving device (2) and an output device (12), -in, The transmitting device (1) is configured to: predetermine a first mark (6), which is dependent on a first processing time point (4) at which the image (3) is processed by the transmitting device (1); and to process raw image data (7) of the image so that a part of the raw image data (7) describing a predetermined image segment (8) is replaced by the first mark (6) or by a superposition of the first mark (6) with the raw image data (7) in the image segment (8) in order to provide modified image data (9) of the image (3); and to transmit the modified image data (9) to the receiving device (2), -Wherein, the receiving device (2) is configured to: predetermine a second mark (10), which depends on a second processing time point (5) at which the image (3) is processed by the receiving device (2); and process the modified image data (9) so that at least a portion of the modified image data (9) in the predetermined image segment (8) is replaced by a superposition of the second mark (10) and the modified image data (9) in the image segment (8) in order to provide final image data (11) of the image (3); and control the output device (12) in order to output the final image data (11), whereby a delay time describing the time interval between the first and second processing time points (4, 5) is visualized in the image segment (8) of the image (3).

10. The device according to claim 9, characterized in that The receiving device (2) has an operating device (31) for collecting operating input (33) from a user, wherein the receiving device (2) is configured to transmit the operating input (33) or control information (34) determined by the operating input (33) from the receiving device (2) to the sending device (1), wherein the sending device (1) is associated with at least one actuator (32) of the device, wherein the sending device (1) is configured to control the actuator (32) according to the operating input (33) or the control information (34).

11. The device according to claim 10, characterized in that The device is a telemedical device, wherein the actuator (32) or at least one corresponding one of the actuators (32) is used to manipulate components of a medical imaging device (30) of the device and / or an interventional device that interacts with a patient.

Images