Method and system for remotely displaying images or videos, in particular medical images or videos

Abstract

A method for remotely displaying images or videos, in particular medical images or videos, comprising the steps of a) selecting a predefined set of quality parameter ranges comprising quality parameter ranges into which quality parameters of a remotely displayed image or video should fall, b) configuring, based on said selected set of quality parameter ranges, a data transfer and display system into a first configuration for remotely displaying images or videos, said system comprising a server and a client being in data connection with each other through a data exchange net, in particular the internet, said client comprising a display, c) sending via said system from the server to the client and receiving said data by the client, d) obtaining quality parameters of said data received by the client, e) comparing said quality parameters with said predefined set of quality parameter ranges, f) depending on the comparison: sending image or video data from said server via said system to said client and displaying the image or video data received by the client on said display, if the quality parameters are with a predefined tolerance value within said parameter ranges, and optionally optimizing said first configuration, or changing said first configuration, if the quality parameters are not with said predefined tolerance value within said parameter ranges and repeating the steps c) to f).

Description

[0001] METHOD AND SYSTEM FOR REMOTELY DISPLAYING IMAGES OR VIDEOS, IN PARTICULAR MEDICAL IMAGES OR VIDEOS

[0002] TECHNICAL FIELD

[0003] The invention relates to a method and a system for remotely displaying images or videos, in particular medical images or videos.

[0004] TECHNICAL BACKGROUND

[0005] In recent years, working remotely with a client computer, hereinafter simply called a "client", while being connected via a data exchange net, in particular the internet, to a server computer, hereinafter simply called a "server", has become increasingly popular.

[0006] An application of particular interest is the so-called remote desktop application, in which a program is executed on the server, while the client having input / output devices such as, e.g., keyboard, mouse, and, in particular, a display, is used only for communicating with the program and for displaying content produced by the program. A user of the client thus basically sees and interacts with the same graphical interface (the "desktop") of the program that a person working directly on the computer acting as server would see and interact with when the program is run on the server. Note that a server in this sense is simply a computer remotely accessed and can be a typical office multi-purpose microcomputer, although, for many applications, it will be a much more powerful computer like a minicomputer or even a mainframe.

[0007] Provided that the connection between the client and the server is stable, secure, and allows data exchange at a speed sufficient for the respective application, using a remote desktop for interacting with a program has many advantages. For example, it enables keeping sensitive data accessed by the program to create specific content "in house", makes it superfluous to install the program locally on the client, and enables the use of complex programs and the handling of large amounts of data which need high computational power by a client not having such computational power.

[0008] While different well-known providers offer hard- and software necessary for using remote desktop applications, most of the existing solutions are primarily designed for administrative applications. If images or videos shall be transmitted in a remote desktop application, due to bandwidth limitations of the connection some form of data compression is typically applied to the images and videos, in particular if the corresponding data is transmitted by streaming, i.e. by sending the data as a steady, continuous flow, allowing displaying to start while the rest of the data is still being received. The known compression algorithms often reduce framerate or resolution, leading to a loss in image resp. video quality, which, however, is acceptable for most administrative applications.

[0009] While some providers of remote desktop applications have recently introduced solutions that are better suited for image and video intensive applications, these solutions typically apply one of two approaches: either they refrain from using any lossy image compression, which in turn leads to a low framerate in video transmissions, or they reduce the amount of compression applied making transmitted images and videos looking better but still not suitable for applications such as the evaluation of medical images, as the properties of images and videos displayed to a user of the client then depend on many different factors, some of which may even change constantly (such as e.g. the bandwidth available for data transmission), so that the quality of the displayed images and videos in not predictable.

[0010] DISCLOSURE OF THE INVENTION

[0011] In view of the aforementioned restrictions, the invention aims at solving the problem of providing a method and a system for remotely displaying images or videos, in particular medical images or videos, which ensure that images and videos displayed to a user always fulfill certain predefined quality criteria and thus enabling e.g. remote evaluation of images and videos for example in telemedicine applications.

[0012] The problem is solved by a method according to claim 1 respectively a system according to claim 24. Advantageous embodiments are defined in the dependent claims.

[0013] The invention ensures that a user of a client can trust that images or videos shown on a display of the client meet certain quality standards and have for example not lost important information due to data compression.

[0014] Another huge advantage of the method according to the invention is that it can be used with already existing equipment (which could be called the "remote desktop infrastructure"), provided of course that the respective hardware, in particular the server, has sufficient computational resources for handling images and videos of the respective type, which, however, is nowadays typically not a problem. Using an existing infrastructure is not only cost saving and environment friendly, it has also the advantage that the technical characteristics of the components such as e.g. the properties of a display like maximum displayable luminance, contrast, color presentation capability, grayscale performance, resolution, frame rate etc. are already known. The invention so to say adds an additional "layer" on top of existing remote display infrastructure such that the quality of images and videos displayed to a user is not only improved once but facilitates a continuous quality assurance ensuring that certain quality requirements for example for medical imaging applications are are always met, which in turn allows for example to achieve medical approvals.

[0015] There exist different technical variants of client-server setups and remote desktop setups.

[0016] A first one is what typically is called a VDI solution, or a Virtual Desktop Infrastructure solution. In this variant, a complete client computer is simulated / running on the server side. This means that the server can be configured to e.g. simulate a complete client computer with specific configuration (e.g. number of CPUs and GPUs, memory, disk space, network interfaces, video outputs etc.). In that simulated client computer an operating system can be installed and inside that operating system one or more applications can be installed. In this VDI setup the video output(s) of the simulated client computer are sent over the internet to a typically lightweight real client computer, and the video output is shown on or more displays connected to that real client computer. The user interacts with the application running on the server side by means of input devices (mouse, keyboard, video, audio) that are connected to the real client computer. These inputs are forwarded to the server and provided to the operating system and application running on the server. Currently, companies such as Citrix Systems Inc. and VMware, Inc., offer VDI well-known solutions. These VDI solutions are very flexible because in principle any operating system and any application can be installed and used in a VDI setup.

[0017] In a second variant the server does not simulate an entire client computer, but instead the server is running a specific preconfigured application or service. For example, a server could run a medical viewing application that is rendering medical images for one or more client devices. At the client side, a local client application is installed that connects to the server. This local client application is operated by the user and this local client application is sending over the internet requests to the server, e.g. to render images. In some implementations of this second variant, all rendering is done on the server. In other implementations of the second variant, some (typically more complex) rendering activities are done by the server while (typically more simple) rendering activities (e.g. typically user interface elements) are done on the client side. But the concept remains the same: the user resides at the client side, and is interacting with the client based on local input devices, these input commands are sent to the server, and the server is reacting by streaming images or video to the client that is visualized to the user at the client side.

[0018] A third variant is very similar to the second variant, but everything is done in a web context (e.g. HTML / 5). In this case the server would be a web server that is generating web pages. The client is a web browser that is interpreting the web pages and visualizing these to the user of the system. Input commands of the user are forwarded to the web server by the web browser. In this variant, the web pages often contain of a mix of elements (e.g. static images, streamed video elements, user interface elements, code elements such as JavaScript). Some of these elements are generated at the server side and merely visualized by the client (e.g. streaming video encoded by the server, sent to the web browser client and visualized there), while other elements can be locally generated by the web browser / client (e.g. local user interface elements, local calculations or processing done based on code such as JavaScript sent by the server to the client).

[0019] The current invention is directly applicable to all variants of client-server setups and remote desktop setups, and for sake of simplicity, in the following it is mostly referred to remote desktop applications, which however is meant to include variants such as virtual desktop applications.

[0020] The invention makes it possible to use existing or commonly available remote desktop and other client-server solutions not only for administrative purposes, but also for image intensive imaging applications, and in particular (life) critical applications such as medical imaging applications that may require regulatory approvals. It also facilitates that highly specialized experts can examine or coexamine medical images remotely for example in emergency situations where it is impossible to bring the expert on-site, and the invention thus may drastically improve the live saving aspects of telemedicine. Further advantages and details of the invention will become apparent from the following detailed description of preferred embodiments and the drawing.

[0021] BRIEF DESCRIPTION OF T HE DRAWING

[0022] Fig. 1 shows very schematically some of the hardware components and software modules of a data transfer and display system comprising a server and a client for remotely displaying images or videos.

[0023] DESCRIPTION OF PREFERRED EMBODIMENTS

[0024] For some applications clear regulations, standards and guidelines on minimum quality requirements exist. For example, for breast cancer screening standards like the USA MQSA standard and the European EUREF standard should be met. Such minimum quality requirements typically include the visualization components and cover aspects such as (display) resolution, luminance, contrast ratio, image noise levels, etc. Although for many other applications including some medical applications no strict quality standard in the sense of a written official standard like an ISO or DIN standard exists and assessing e.g. the diagnostic value of a medical image is subjective and influenced by factors such education, skills, and experience of the assessor, experts in the respective field generally agree on what makes a good quality image, and it is hence easily possible to define in both cases (standard exists, no standard exists) a set of quality parameter ranges comprising quality parameter ranges into which objective and measurable quality parameters of a remotely displayed image or video should fall to meet the thus defined quality standard.

[0025] According to the method of the invention, in a first step a predefined set of quality parameter ranges comprising quality parameter ranges into which quality parameters of a remotely displayed image or video should fall, is selected. Obviously, the quality parameters differ from application to application. For example, if very detailed single medical pictures shall be examined (e.g. mammography images such as FFDM and tomosynthesis images), the resolution is important, whereas for examining blood flow in an organ visualized by a color ultrasound video the framerate should not fall below a certain minimum. Typically, the quality parameters will comprise values indicative of true observed resolution, true observed framerate, greyscale characteristics, color characteristics, noise characteristics, compression artifacts, contrast characteristics.

[0026] True resolution, true bit depth, true framerate can differ from set resolution, set bit, set framerate. For example, the server could be configured to have a framerate of 30 frames per second. That means that applications running on the server can generate up to 30 frames per second. Also the client can be configured to output a given number of frames per second (e.g. 60 frames per second). Purely querying the framerate settings on the client does not provide sufficient information to understand what the true framerate is that is being generated by the application and that is being visualized on the client. Indeed, even though the server may be configured to e.g. run at 30 frames per second, this does not provide any information on how many frames per second the application is truly generating: An application showing static images may only generate new images when the user is interacting with the image (e.g. panning, zooming, rotating). In this example the true framerate generated by the application could be e.g. 2 frames per second and may vary over time depending on user interactions. Even though the server may be configured for 30 frames per second, in reality only e.g. 2 frames per second are generated by the application. Also the network connection and the data transfer settings can influence the true observed framerate. For example: an application may generate 15 frames per seconds for example while browsing through a CT stack. The server could be configured for 30 frames per second. But the network connectivity and the data transfer system settings may be such that only 5 frames per second can be transmitted over the network. In that case the framerate will be reduced to 5 frames per second before transfer (e.g. by dropping frames) and only 5 frames per second will be received by the client. The client may be configured though for 60 frames per second, meaning that the client will provide 60 frames per second to the display system. In this situation the true observed frame rate is only 5 frames per seconds, even though querying the client settings and display settings would suggest that 60 frames per seconds are being generated. The true observed frame rate is one quality parameter since this makes clear how many frames generated by the application in reality are visualized to the client user. If the intended use of the system is diagnosis of brain MRI images, a radiologist typically may want to browse through a stack of MRI slices. The radiologist is inspecting each of those slices for abnormalities and clinically relevant findings. If the application running on the server is generating 10 slices per seconds (corresponding to a framerate of 10 frames per second), then it is crucial that all slices are actually transmitted, received by the client, and visualized on the display system such that the radiologist can inspect these slices in order not to miss clinically relevant features or abnormalities. Therefore, the true observed framerate is an important quality parameter and needs to be within an acceptable range. The mere framerate settings of server, client, and display have in this case rather little relevance.

[0027] Measuring true framerate is not straightforward. The fact that the server is configured for a specific framerate or refresh rate does not say anything about the how many images per second the application is generating. According to the invention, one way to measure the true number of frames per second that an application is generating is by monitoring if the output of the application has changed. If a server is configured for 25 frames per second, a program can monitor the output window of the application 25 times per second (ideally synchronized with the refresh signal of the framebuffer) and check if the new frame is different from the previous frame. This can be done by directly by comparing the pixel data, or by calculating one or more cyclic redundancy check (CRC) values over the entire or parts of the framebuffer and checking whether the CRC values have changed. A change in these CRC values indicates that the application output window has changed and that therefore the application has generated a new frame. In this way it is possible to measure how many frames per second the application is truly generating. At the client side a similar approach can be taken. If e.g. at the client side the framerate is set to 45 frames per second, a program at the client side can monitor the decoded image or video stream and determine, how many time per second the frame has changed. Again, this can be done by direct comparison of pixel data of the frame or CRC values can be used. This way of working gives information about the true framerate that the application generates, and the true framerate that the client is receiving and that is presented to the user of the client system.

[0028] Depending on the intended use, the true observed framerate of the overall system can be an important quality parameter of the data transfer and display system. This true observed framerate may for example be required to at least be some specific value (e.g. 5, 10, 15, 20, 30, 60 etc. frames per second depending on the respective intended use). It will normally be higher for applications that require a lot of interaction with content, or applications that generate a high number of frames per second.

[0029] Another important quality parameter of the data transfer and display system can be that the true observed framerate of the overall system is at least a percentage of the true framerate of the application. For example, if the application generates a true framerate of 20 frames per second, then a typical minimum quality requirement could be that the true observed framerate of the overall system (true framerate presented to the user of the client) is at least 25%, 50%, 75%, 80%, 90%, or 100% of the true framerate of the application. In this particular example it means that the true framerate of the overall system should at least be 5, 10, 15, 16, 18 or 20 frames per second. This percentage again will typically depend on the intended use and will normally be higher for applications that require a lot of interaction with content, or applications that generate a high number of frames per second.

[0030] Ideally, framerate settings should be aligned to each other. This means that ideally server frame rate settings reflect (are equal to, or multiples of) the true framerate that the application on the server is generating. At the same time, the framerate settings of the client ideally reflect (are equal to or a multiple of) the framerate setting of the server. The display framerate settings ideally are set equal to the frame setting of the client, and frame lock (frame sync) on the display should be enabled to make sure the refresh of the display is synchronized with the refresh of the client side. Similar to the difference between set and observed (true) framerate, there is also a difference between set resolution and observed (true) resolution. Data transfer codecs often use scaling where images or video are downscaled at the sending side and then upscaled again at the receiver side. Purely looking at the configured resolution of server and client side one could think there is no loss in actual resolution, while in reality the true (observed) resolution can be lower. E.g. resolution on the server side could be set to 12 Megapixel, the codec downscales to 3 Megapixel, and on the client side the codec upscales again to 12 Megapixel with resolution settings on the client side being 12 Megapixel. The set resolution in this example is 12 Megapixel, while the true (observed) resolution will be 3 Megapixel. True (observed) resolution can be determined by comparing (pixel wise) the original (application generated) image with the image being presented to the user of the client. It can also be determined by inspecting codec settings or it by means of test patterns (specific resolution test patterns) that can be inspected and from which information can be deducted on the true resolution of the system. Similar to framerate, true (observed) resolution settings can be an important quality parameter and both absolute and relative minimum required values can be set (e.g. minimum 3 Megapixel, or minimum 25% / 75% / 100% of the original application generated resolution).

[0031] Similar to framerate and resolution, also observed (true) bit depth can be different from set bit depth. The same principles as above can be applied to bit depth. True bit depth can be determined by inspecting codec settings, by comparing application generated image data with client presented image data, or by using specific bit depth test patterns (such as e.g. greyscale or color ramps).

[0032] Although not a characteristic of an image or video itself, in a preferred embodiment the predefined set of quality parameter ranges further comprises ranges into which quality parameters of the data connection between said server and said client should fall such as in particular bitrate, as this influences possible data compression (which should normally be low) and framerate (which should normally be high). Other examples of quality parameters of the data connection between said server and said client include latency of the connection, jitter, packet drop statistics, packet error statistics, whether wired or wireless connectivity is used, the collision or interference rate of wireless networks.

[0033] In another preferred embodiment the predefined set of quality parameter ranges further comprises ranges into which quality parameters of the server and / or the client should fall, such as in particular server computing capabilities, processor type, processor speed, memory size, memory speed, disk size and disk speed, presence or absence of a GPU, type and speed of the GPU (if present), network connectivity of the server, server load or usage (e.g. number of clients running on the server, current load of the CPU or GPU, etc.). These quality parameters of the server and / or the client are obtained and taken into account in at least some of steps of configuring, based on said selected set of quality parameter ranges, the data transfer and display system into said first configuration, sending via said system from the server to the client and receiving said data by the client, obtaining quality parameters of said data received by the client, comparing said quality parameters with said predefined set of quality parameter ranges, and depending on the comparison either sending image or video data from said server to said client or changing said first configuration.

[0034] In a preferred embodiment, the predefined set of quality parameter ranges is selected amongst a plurality of sets of quality parameter ranges corresponding to different intended uses. In medical regulatory context, the term "intended use" is often used to define what a solution is intended for, by whom it is to be used, what the exact functionality and purpose of the solution is. In medical applications, typical intended uses are e.g. diagnosis of lung CT, clinical review of ultrasound cardiac images, diagnosis of digital pathology images. The selection of the predefined set of quality parameter ranges based on the type of intended use may be done manually by a user or may be done automatically, either based on an indication given by a user, what he intends to do, or based on the type of an image or video requested by a user for being displayed on the display, or a medical procedure code being selected or visualized. Of course, a predefined set of quality parameter ranges is not static and may change over time for example when new intended uses arise or when it is found that for an existing intended use other parameter ranges are better suited (e.g. when new standards, regulations or guidance documents are created, or when technological advances increase the minimum quality level that is considered proper care).

[0035] According to the method of the invention, a data transfer and display system for remotely displaying images or videos comprising a server and a client being in data connection with each other through a data exchange net, in particular the internet, said client comprising a display, is configured based on the selected set of quality parameter ranges. Configuring said data transfer and display system may comprise adjusting bit depth of image or video to be displayed, framerate in case a video shall be displayed, bandwidth assigned to a specific data connection between the server and the client, priority settings assigned to a specific data connection between the server and the client, luminance setting of the display, contrast setting of the display, color setting of the display, greyscale settings of the display, framerate settings of the display, calibration settings of the display, resolution setting of the display, ICC profile setting of the display, resolution settings of the server, resolution settings of the client, color or ICC settings of the server, color or ICC settings of the client, framerate settings of the server, framerate settings of the client, contrast or HDR settings of the server, contrast or HDR settings of the client, rendering settings of the server, rendering settings of the client.

[0036] According to a preferred embodiment, different configurations that have been tested to be able to achieve image and video transmission and display with certain quality parameters are stored in a memory or database. Accordingly, the method may comprise a step of creating a look-up table with different configurations for said data transfer and display system. This allows to bring the system into a first configuration very fast and then, if needed, use this configuration to further "fine-tune" the settings.

[0037] Next, data is sent from the server to the client and received the client, quality parameters of said data received by the client are obtained and it is checked, if these quality parameters, which, as set out above, may also include quality parameters of the data connection or server or client, fall within the predefined quality parameter ranges. This data, which could also be called "test data", is typically data similar or identical to the data of the images or the video that shall actually be remotely displayed.

[0038] In a preferred embodiment, the test data sent to the client is processed by the data transfer and display system in a similar or the same way image or video data for being displayed on the display would be processed, but preferably without actually this test data being visible to the user of the client. The test data may be one of additional image or video data, in particular additional lines and / or columns of pixels in an image or video intended for being displayed, said additional lines and / or columns being outside a range of lines and columns of the display, an additional bit layer to an image or video intended for being displayed, additional color data (such as multiprimary color data) or high dynamic range data to an image or video intended for being displayed, additional 3D (e.g. stereoscopic) data to an image or video intended for being displayed, an additional header of an image or video intended for being displayed, said header carrying test patterns or additional data not intended for being displayed, an additional framebuffer data, in particular data corresponding to an additional virtual display, said additional framebuffer data not intended for being displayed.

[0039] The additional image or video data, the additional bit layer, the additional color data or high dynamic range data, the additional 3D data, the additional header and the additional framebuffer data may be a combination of test patterns and metadata such as in particular a timestamp, a frame number or a cyclic redundancy check.

[0040] Obtaining the quality parameters and comparing them with the predefined set of quality parameter ranges may be done by the client and can be performed in different ways like measuring actual physical properties or comparing the data before and after sending, which sending typically involves coding and decoding and thus e.g. (lossy) compressing the data. To facilitate this comparison, information about the the data sent may be sent to the client via a lossless channel. In other words, the client may receive information about how e.g. a test image should look via a lossless channel and compare this information with how the image created from test data actually looks. Note that this "look" may be entire virtual, i.e. the images may be compared by the client internally without showing the images to the user. The user may of course be informed about the progress by corresponding messages like "setting up system", "checking data transfer" and the like. Similar messages may also be outputted during further operation of the method, for example when the data transfer rate changes and good quality images or video may no longer be displayed. In such case, the user may also be asked to change a selected intended use. For example, if it a radiologist selects as intended use "primary diagnosis of lung CT images" and it turns out to be impossible to present such pictures with the required quality, he may be offered a selection of possible other "intended uses" for example examination "clinical review of lung CT images", as the quality requirements for a primary diagnosis are higher than those for clinical review.

[0041] Obtaining quality parameters of said data received by the client and comparing said quality parameters with said predefined set of quality parameter ranges may also be done by the server. To enable this, it may be foreseen that information about the the data received by the client is sent to the server via a lossless channel. It is also possible to distribute the tasks for example such that obtaining quality parameters of said data received by the client is done by the client and comparing said quality parameters with said predefined set of quality parameter ranges is done by the server, in case of which information about the the data received by the client may again be sent to the server via a lossless channel. If a look-up table with pre-stored configurations is used, feedback from comparing the obtained quality parameters and with the predefined set of quality parameter may be used to improve the look-up table, or more particularly the configurations stored therein in a self-learning manner.

[0042] To allow more flexibility when checking, if the quality parameters fall within the predefined quality parameter ranges, for each parameter range a predefined tolerance value may be used, which however, for certain crucial parameter range, may also be set to zero. If the quality parameters are within said parameter ranges (± said tolerance value), image or video data from said server will be sent to the client and displayed on the respective display, while the configuration may optionally be further optimized for example to bring the quality parameters of the received data to the middle of the respective parameter ranges. If the quality parameters are, even taking a tolerance value into account, not within said parameter ranges, the configuration is changed and new "test data" is send and the steps of checking, if the quality parameters of the data received by the client are within the predefined quality parameter ranges, and changing the configuration are repeated until a configuration that allows displaying images or videos data with the predefined quality is found. However, even if such configuration is found, the method according to the invention is preferably performed in such manner that the quality parameters are constantly monitored to maintain or even improve the quality by respectively adjusting the settings of the system formed by the server, the client and the display. In other words, even if the quality parameters are with a predefined tolerance value within said parameter ranges, the steps of sending "test" data, obtaining quality parameters, checking them against the allowed parameter ranges and adjusting the configuration is regularly repeated. Regularly means that the repetitions are not randomly, but according to a specific rule like for example every five seconds, every time a new image is displayed, every time a new client session is started, every time a new user starts, every time some parameter changes e.g. available bandwidth, number of active clients on the server etc. To do this, one could say that the test data is "injected" into the image or video data transmitted, and may then be processed by the "infrastructure" just as the "regular" images or videos requested by a user. In this way, any artifacts or image quality reductions that would be applied to the regular images or videos would also be applied to the injected data.

[0043] In certain applications, a user may want to examine images or videos of the same or different types in parallel. In such case the display may comprise several dedicated areas or there may be several displays, and each dedicated area respectively each display then forms part of an own "data transfer and display system" as set out above and the respective steps for improving and ensuring the quality of images and videos displayed may be performed for each such data transfer and display system. The method advantageously also allows a "multiple user" case, i.e. establishing a connection between one server and multiple clients. Each connection then forms a data transfer and display system, and each user is able to select a different intended use. In such case, the method may comprise a step of prioritizing distribution of resources between said data transfer and display systems based on at least one of intended use selected by the user of each of said systems, urgency requests by the users of said systems, user identifications of the users of said systems. For example, if the overall bandwidth is limited and a user has an emergency case like urgently examining medical images of a patient in a critical state, all or most bandwidth may be assigned to the system of that user.

[0044] Fig. 1 shows very schematically some hardware components as well as software modules of a system for remotely displaying images or videos, in particular medical images or videos, comprising a server 10 and a client 12 connectable with each other through a data exchange net 14, in particular the internet. The server 10 and the client 12, which in this example comprises three displays 16, 18, 20, are adapted to form, when the server 10 and the client 12 are connected with each other through the data exchange net 14, a data transfer and display system for performing the method as laid out above.

[0045] Besides the displays 16, 18, 20, the client 12 comprises in this example also a camera 22. Not shown are input / output devices such as keyboard, mouse etc. Client housing 24 holds components to run software modules schematically indicated in box 26 as well as GPU 28, network connector 30 and USB connector 32. GPU 30 is driven by GPU driver 34, which in turn is controlled by color management module 36 and desktop window manager 38, that form part of the client's operating system schematically indicated by box 40.

[0046] In operation, a remote desktop receiver application, which, as explained above, may be a virtual display receiver application, indicated by box 42 interacts with the operating system 40 and via an API 44 also with other components and modules of the client 12, such as in particular quality monitoring receiver 46. Virtual display receiver application 42 comprises in this embodiment a framebuffer 48, a monitoring module 50 and a decoder 52.

[0047] Server housing 54 connected to display 56 and operable via keyboard 58 holds components to run software modules schematically indicated in box 60 as well as GPU 62 and network connector 64. GPU 62 is driven by a desktop windows manager 66, that forms part of the server's operating system schematically indicated by box 68.

[0048] In operation, a remote desktop sender application, which, as explained above, may be a virtual display sender application, indicated by box 70 interacts with the operating system 68 and via an API 72 also with other components and modules of the server 10, such as in particular quality monitoring sender 74. Virtual display application 68 comprises in this embodiment an encoder 76 and a monitoring module 78. An end-user application 80 creates pixels and is composited into framebuffer (not shown) by the desktop window manager 66.

[0049] )

[0050] The virtual display receiver application 42 grabs the framebuffer and performs encoding of the pixels. The encoded pixels get transmitted through the network to the client 12 via data exchange net 14, and the encoded pixels are delivered to the The virtual display receiver application 42, which decodes the pixels and pushes them to framebuffer 48. The framebuffer 48 is composited by the desktop window manager 38 and passed to the GPU 28 to be placed on one of the displays 16, 18, 20.

[0051] Inside the virtual display sender application 70 there is some monitoring and control which is exposed via API 72. Monitoring module 78 monitors and the virtual display sender application 70 collects the monitored data. Monitoring module 78 can also monitor the operating system 68, the network and others. Similarly, on the client's 12 side monitoring module's 50 monitoring is exposed via API 44 and is interrogated by the virtual display receiver application 42. Monitoring module 50 can also monitor the displays 16, 18, 20 and other sensors (e.g. via USB). The coordination between the quality monitoring sender 74 and the quality monitoring receiver 46 may be done in the cloud by a corresponding quality monitoring backend or direct (peer-to-peer).

[0052] The system may comprise a plurality of clients 12, each comprising at least one display, each client connectable with the server 10 through the data exchange net 14 to form a respective data transfer and display system, and in that case the server 10 and the clients 12 may be adapted to perform, when the server and the clients are connected with each other through the data exchange net, the method of the multiple user case.

[0053] Within the scope of protection as defined by the appended claims, many variations and embodiments are possible. For example, the "infrastructure" may be put in a first configuration prior to selecting a certain set of parameter ranges, and certain steps may be performed not in the order in which they are given in the claims, so that the given order shall not be construed as binding. An important idea is that by continuously optimizing the remote desktop infrastructure including the display settings based on the needs for a specific intended use, in particular a medical use, continuously and in an unobtrusive way measuring key quality metrics of the remote desktop setup, the network connection, as well as the display, comparing these metrics to minimum quality requirements for the specific intended use it is possible improve and ensure the quality of images and videos remotely display in an reliable and predictable manner.

Claims

CLAIMS1 . A method for remotely displaying images or videos, in particular medical images or videos, comprising the steps of a) selecting a predefined set of quality parameter ranges comprising quality parameter ranges into which quality parameters of a remotely displayed image or video should fall, b) configuring, based on said selected set of quality parameter ranges, a data transfer and display system into a first configuration for remotely displaying images or videos, said system comprising a server and a client being in data connection with each other through a data exchange net, in particular the internet, said client comprising a display, c) sending via said system from the server to the client and receiving said data by the client, d) obtaining quality parameters of said data received by the client, e) comparing said quality parameters with said predefined set of quality parameter ranges, f) depending on the comparison: sending image or video data from said server via said system to said client and displaying the image or video data received by the client on said display, if the quality parameters are with a predefined tolerance value within said parameter ranges, and optionally optimizing said first configuration, or changing said first configuration, if the quality parameters are not with said predefined tolerance value within said parameter ranges and repeating the steps c) to f).

2. The method according to claim 1 , wherein said predefined set of quality parameter ranges further comprises quality parameter ranges into which quality parameters of the data connection between said server and said client should fall and wherein obtaining quality parameters of said data received by the client includes obtaining quality parameters of the data connection.

3. The method according to claim 1 or 2, wherein said predefined set of quality parameter ranges further comprises quality parameter ranges into which quality parameters of the server and / or the client should fall, such as in particular server computing capabilities, processor type, processor speed, memory size, memory speed, disk size and disk speed, presence or absence of a GPU, type and speed of the GPU (if present), network connectivity of the server, server load or usage, and wherein said quality parameters the server and / or the client are obtained and taken into account in at least some of steps b) to f).

4. The method according to one of claims 1 to 3, wherein said quality parameters comprise values indicative of at least one of true observed resolution, true observed framerate, color characteristics, noise characteristics, compression artifacts, luminance characteristics, contrast characteristics, latency, jitter.

5. The method according to one of claims 1 to 4, wherein the predefined set of quality parameter ranges is selected amongst a plurality of sets of quality parameter ranges corresponding to different intended uses.

6. The method according to claim 5, wherein a user selects the predefined set of quality parameter ranges based on the type of intended use.

7. The method according to claim 5, wherein the predefined set of quality parameter ranges is automatically selected based on the type of intended use indicated by a user, the type of an image or video requested by a user for being displayed on the display, or a type of a medical procedure being performed.

8. The method according to one of claims 1 to 7, further comprising a step of creating a look-up table with different configurations for said data transfer and display system, each configuration corresponding to one of said predefined sets of quality parameter ranges.

9. The method according to claim 8, further comprising updating said look-up table in a self-learning manner via feedback from the comparing performed in step e).

10. The method according to one of claims 1 to 9, wherein configuring said data transfer and display system comprises adjusting at least one of: resolution of image or video to be displayed, bit depth of image or video to be displayed, framerate in case video or a series of images shall be displayed, enabling a lossless mode of an image or video compression codec, bandwidth assigned to a specific data connection between the server and the client, bitrate of an image or video compression codec, priority settings assigned to a specific data connection between the server and the client, luminance setting of the display, contrast setting of the display, color setting of the display, calibration setting of the display, resolution setting of the display, ICC profile setting of the display, resolution settings of the server, resolution settings of the client, color or ICC settings of the server, color or ICC settings of the client, framerate settings of the server, framerate settings of the client, contrast or HDR settings of the server, contrast or HDR settings of the client, rendering settings of the server, rendering settings of the client.11 . The method according to one of claims 1 to 10, wherein the data sent in step c) is processed by the data transfer and display system in the same way image or video data for being displayed on the display would be processed, preferably without displaying the data sent in step c) on the display.

12. The method according to one of claims 1 to 11 , wherein the data sent in step c) is one of: an additional bit layer to an image or video intended for being displayed, additional color data or high dynamic range data to an image or video intended for being displayed, additional 3D (e.g. stereoscopic) image data to an image or video intended for being displayedadditional header data of an image or video intended for being displayed, said header data carrying test patterns or additional data not intended being displayed, additional framebuffer data, in particular data corresponding to an additional virtual display, said additional framebuffer data not intended for being displayed.

13. The method according to claim 12, wherein the additional image or video data, the additional bit layer, the additional color data or high dynamic range data, the additional 3D data, the additional header and the additional framebuffer data is a combination of test patterns and metadata such as in particular a timestamp, a frame number or a cyclic redundancy check.

14. The method according to one of claims 1 to 13, wherein obtaining quality parameters of said data received by the client and comparing said quality parameters with said predefined set of quality parameter ranges is done by the client.

15. The method according to claims 14, wherein information about the the data sent in step c) is sent to the client via a lossless channel.

16. The method according to one of claims 1 to 13, wherein obtaining quality parameters of said data received by the client and comparing said quality parameters with said predefined set of quality parameter ranges is done by the server.

17. The method according to claims 16, wherein information about the the data received by the client is sent to the server via a lossless channel.

18. The method according to one of claims 1 to 13, wherein obtaining quality parameters of said data received by the client is done by the client and comparing said quality parameters with said predefined set of quality parameter ranges is done by the server.

19. The method according to claims 18, wherein information about the the data received by the client is sent to the server via a lossless channel.

20. The method according to one of claims 1 to 19, wherein the steps c) to f) are regularly repeated even if the quality parameters are with a predefined tolerance value within said parameter ranges.21 . The method according to one of claims 1 to 20, further comprising displaying on the display information relating to the result of the comparison and optionally prompting a user to change a selected intended use.

22. The method according to one of claims 1 to 21 , wherein the display is comprised by a dedicated area of a display unit and / or wherein the client comprises several displays, each dedicated area respectively each display forming part of an own data transfer and display system in the sense of claim 1 , the method comprising performing the steps a) to f) for each data transfer and display system.

23. The method according to one of claims 1 to 22, wherein the server is in data connection with multiple clients, each connection forming a data transfer and display system, comprising a step of prioritizing distribution of resources between said data transfer and display systems based on at least one of intended uses selected by the user of each of said systems, urgency requests by the users of said systems, user identifications of the users of said systems.

24. A system for remotely displaying images or videos, in particular medical images or videos, comprising a server and a client connectable with each other through a data exchange net, in particular the internet, said client comprising a display, characterized in that the server, the client and the display are adapted to form, when the server and the client are connected with each other through the data exchange net, a data transfer and display system for performing the method of one of claims 1 to 22.

25. The system according to claim 24, comprising a plurality of clients, each comprising a display, each client connectable with the server through the data exchange net to form a respective data transfer and display system, characterized in that the server and the clients are adapted to perform when the server and the clients are connected with each other through the data exchange net the method of claim 23.

Images