Method and system for remotely displaying images or videos, in particular medical images or videos

Abstract

A method for remotely displaying images or videos, particularly medical images or videos, includes the following steps: a) selecting a predefined set of quality parameter ranges, the predefined set of quality parameter ranges including a range of quality parameters into which the quality parameters of the remotely displayed images or videos should fall; b) configuring a data transmission and display system for remotely displaying images or videos based on the selected set of quality parameter ranges; the system including a server and a client, the server and client being data-connected to each other via a data exchange network, particularly the Internet, the client including a display; c) sending data from the server to the client via the system, and receiving the data by the client; d) obtaining quality parameters of the data received by the client; e) comparing the quality parameters with the predefined set of quality parameter ranges; f) based on the comparison: if the quality parameters have a predefined tolerance value within the parameter range, then sending image or video data from the server to the client via the system, and displaying the image or video data received by the client on the display, and optionally optimizing the first configuration; or if the quality parameters do not have the predefined tolerance value within the parameter range, then changing the first configuration and repeating steps c) to f).

Description

Technical Field

[0001] The present invention relates to a method and system for remotely displaying images or videos, particularly medical images or videos. Technical Background

[0002] In recent years, it has become increasingly popular to work remotely with a client computer (hereinafter referred to as "client") while simultaneously connecting to a server computer (hereinafter referred to as "server") via a data exchange network (especially the Internet).

[0003] A particular focus is on so-called remote desktop applications, where a program runs on a server, while a client with input / output devices (such as a keyboard, mouse, and especially a monitor) is only used to communicate with the program and display its content. Therefore, the client user essentially sees and interacts with the program running on the server, using the same graphical interface (“desktop”) that a person working directly on the server would see and interact with. It's important to note that in this sense, the server is simply a remotely accessed computer, which could be a typical office multifunction microcomputer, although for many applications it would be a more powerful computer, like a minicomputer or even a mainframe.

[0004] If the connection between the client and server is stable, secure, and allows for data exchange at a speed sufficient for the application, then using remote desktop to interact with programs has many advantages. For example, it can keep sensitive data accessed by programs "internal" to create specific content, making the installation of programs locally on the client redundant, and it enables the use of complex programs and the processing of large amounts of data, which require high computing power from clients that do not possess such capabilities.

[0005] While various well-known providers offer the hardware and software necessary for using remote desktop applications, most existing solutions are primarily designed for management applications. When transmitting images or videos in remote desktop applications, some form of data compression is typically applied to the images and videos due to connection bandwidth limitations, especially if the data is transmitted via streaming—that is, by sending data in a stable, continuous stream, allowing display to begin while the remaining data is still being received. Known compression algorithms often reduce frame rate or resolution, resulting in image loss. However, the video quality is generally acceptable for most management applications.

[0006] While some remote desktop application providers have recently introduced solutions better suited for image and video-intensive applications, these solutions typically employ one of two approaches: either they do not use any lossy image compression, resulting in a lower frame rate in video transmission, or they reduce the amount of compression applied, making the transmitted images and videos look better, but still not suitable for applications such as medical image evaluation, because the properties of the images and videos displayed to the client user subsequently depend on many different factors, some of which may even be constantly changing (e.g., the bandwidth used for data transmission), making the quality of the displayed images and videos unpredictable. Summary of the Invention

[0007] In view of the above limitations, the present invention aims to solve the problem of providing a method and system for remotely displaying images or videos, especially medical images or videos, which ensures that the images and videos displayed to the user always meet specific predefined quality standards, thereby enabling remote evaluation of images and videos, for example, in telemedicine applications.

[0008] The problem is solved by the method according to claim 1 and the system according to claim 24, respectively. Advantageous embodiments are defined in the dependent claims.

[0009] This invention ensures that users on the client side can be confident that the images or videos displayed on the client's monitor meet certain quality standards and that, for example, no important information is lost due to data compression.

[0010] Another significant advantage of the method according to the invention is its compatibility with existing equipment (which may be referred to as "remote desktop infrastructure"), provided that the corresponding hardware, especially servers, has sufficient computing resources to process the appropriate types of images and videos; however, this is generally not a problem nowadays. Using existing infrastructure is not only cost-effective and environmentally friendly, but also benefits from the well-known technical characteristics of components, such as display properties like maximum display brightness, contrast ratio, color rendering capability, grayscale performance, resolution, frame rate, etc. The invention adds an extra "layer" to existing remote display infrastructure, thereby not only improving the quality of images and videos displayed to the user once, but also facilitating continuous quality assurance to ensure that certain quality requirements for applications such as medical imaging are always met, thus enabling, for example, medical approval.

[0011] There are different technical variations in client-server setups and remote desktop setups.

[0012] The first type is commonly referred to as a VDI solution or Virtual Desktop Infrastructure solution. In this variant, a complete client computer is simulated / run on the server side. This means the server can be configured to, for example, simulate a complete client computer with specific configurations (e.g., the number of CPUs and GPUs, memory, disk space, network interfaces, video output, etc.). An operating system can be installed on this simulated client computer, and one or more applications can be installed on that operating system. In this VDI setup, the video output of the simulated client computer is sent over the internet to a typical lightweight real client computer, and the video output is displayed on a monitor or multiple monitors connected to that real client computer. Users interact with the applications running on the server side through input devices (mouse, keyboard, video, audio) connected to the real client computer. This input is forwarded to the server and provided to the operating system and applications running on the server. Currently, companies such as Citrix Systems Inc. and VMware, Inc. offer well-known VDI solutions. These VDI solutions are very flexible because, in principle, any operating system and application can be installed and used in a VDI setup.

[0013] In the second variant, instead of simulating an entire client computer, the server runs specific, pre-configured applications or services. For example, the server might run a medical viewing application that is rendering medical images on one or more client devices. On the client side, a local client application is installed and connected to the server. This local client application is operated by the user and sends requests to the server via the internet, such as rendering images. In some implementations of this second variant, all rendering is done on the server. In other implementations of the second variant, some (often more complex) rendering activities are performed by the server, while (often simpler) rendering activities (e.g., typically user interface elements) are performed on the client. However, the concept remains the same: the user is on the client side and interacts with the client based on a local input device; these input commands are sent to the server, and the server responds by streaming images or videos to the user-visible client side.

[0014] The third variant is very similar to the second, but everything is done within a web context (e.g., HTML / 5). In this case, the server is the web server that is generating the webpage. The client is a web browser that interprets the webpage and visualizes it for the user of the system. User input commands are forwarded to the web server via the web browser. In this variant, the webpage typically contains a mixture of elements (e.g., still images, streaming video elements, user interface elements, code elements, such as JavaScript). Some of these elements are generated on the server side and are only visualized by the client (e.g., streaming video encoded by the server, sent to the web browser client, and visualized), while other elements can be generated locally by the web browser / client (e.g., local user interface elements, local calculations, or processing based on code such as JavaScript sent from the server to the client).

[0015] This invention is directly applicable to all variations of client-server setups and remote desktop setups. For the sake of brevity, the following text primarily refers to remote desktop applications; however, it is intended to include variations such as virtual desktop applications.

[0016] This invention enables the use of existing or general-purpose remote desktop and other client-server solutions not only for administrative purposes but also for image-intensive imaging applications, particularly critical applications such as medical imaging applications requiring regulatory approval. It also facilitates remote or collaborative examination of medical images by highly specialized experts; for example, in emergency situations where bringing experts to the scene is not feasible, this invention can significantly improve the life-saving aspects of telemedicine.

[0017] Further advantages and details of the invention will become apparent from the following detailed description of preferred embodiments and accompanying drawings. Attached Figure Description

[0018] Figure 1 The diagram illustrates, in a highly schematic manner, some hardware components and software modules of a data transmission and display system that includes servers and clients for remotely displaying images or videos. Detailed Implementation

[0019] For certain applications, there are clear regulations, standards, and guidelines regarding minimum quality requirements. For example, for breast cancer, screening standards such as the US MQSA standard and the European EUREF standard should be met. These minimum quality requirements typically include aspects of visualization components and coverage such as (display) resolution, brightness, contrast, and image noise level. Although for many other applications, including some medical applications, there are no strict quality standards in the sense of written official standards (such as ISO or DIN standards), and the assessment of the diagnostic value of, for example, medical images is subjective and influenced by factors such as the education, skills, and experience of the assessor, experts in various fields generally agree on what constitutes a high-quality image. Therefore, in both cases (where standards exist and where they do not), it is easy to define a set of quality parameters that includes a range of quality parameters to meet such a defined quality standard, where the objective and measurable quality parameters of remotely displayed images or videos should fall within this set of quality parameter ranges.

[0020] According to the method of the present invention, in the first step, a predefined set of quality parameter ranges is selected, including a range of quality parameters, wherein the quality parameters of the remotely displayed image or video should fall within the predefined set of quality parameter ranges. Obviously, the quality parameters vary depending on the application. For example, if a very detailed single medical image (e.g., mammogram images, such as FFDM and tomographic composite images) is to be examined, resolution is important, while for the examination of blood flow in an organ visualized via color ultrasound video, the frame rate should not be lower than a certain minimum. Typically, the quality parameters will include values ​​indicating the resolution of the actual observation, the frame rate of the actual observation, grayscale features, color features, noise features, compression artifacts, and contrast features.

[0021] The actual resolution, actual bit depth, and actual frame rate can differ from the set resolution, set bit depth, and set frame rate. For example, a server can be configured to have a frame rate of 30 frames per second. This means that an application running on the server can generate up to 30 frames per second. Furthermore, a client can be configured to output a given number of frames per second (e.g., 60 frames per second). Simply querying the frame rate settings on the client does not provide enough information to understand the actual frame rate that the application is generating and visualizing on the client. In fact, even if the server can be configured to run at, for example, 30 frames per second, this does not provide any information about the actual number of frames per second that the application is generating: when the user interacts with the image (e.g., panning, zooming, rotating), an application displaying a static image can only generate new images. In this example, the actual frame rate generated by the application could be, for example, 2 frames per second, and could vary over time depending on user interaction. Even if the server can be configured to 30 frames per second, in reality, the application only generates, for example, 2 frames per second. Network connectivity and data transfer settings also affect the actually observed frame rate. For example, an application might generate 15 frames per second, for example, while browsing a CT stack. The server could be configured to 30 frames per second. However, network connectivity and data transmission system settings may limit transmission to 5 frames per second. In this case, the frame rate is reduced to 5 frames per second before transmission (e.g., by dropping frames), and the client will only receive 5 frames per second. The client can be configured for 60 frames per second, meaning the client will supply 60 frames per second to the display system. In this case, the actual observation frame rate is only 5 frames per second, even though querying client and display settings will indicate that 60 frames per second are being generated. The actual observation frame rate is a quality parameter because it clearly indicates how many frames generated by the application are actually visualized by the client user. If the system's intended use is to diagnose brain MRI images, radiologists may typically want to review a large number of MRI slides. Radiologists are examining these slides for abnormalities and clinically relevant findings. If an application running on a server generates 10 slides per second (corresponding to a frame rate of 10 frames per second), it is crucial that all slides are actually transmitted, received, and visualized on the display system by the client so that radiologists can examine these slides to avoid missing clinically relevant features or abnormalities. Therefore, the actual observation frame rate is an important quality parameter and needs to be within acceptable limits. In this scenario, the frame rate settings of the server, client, and monitor are very insensitive.

[0022] Measuring the true frame rate is not straightforward. The fact that the server is configured to a specific frame rate or refresh rate does not indicate how many images the application generates per second. According to the invention, one method for measuring the true number of frames generated per second by an application is by monitoring whether the application's output changes. If the server is configured to 25 frames per second, the program can monitor the application's output window 25 times per second (ideally synchronized with the refresh signal of the frame buffer) and check whether the new frame differs from the previous frame. This can be achieved by directly comparing pixel data, or by calculating one or more Cyclic Redundancy Check (CRC) values ​​over the entire or partial frame buffer and checking whether the CRC values ​​change. Changes in these CRC values ​​indicate that the application's output window has changed, and therefore the application has generated new frames. In this way, it is possible to measure how many frames the application actually generates per second. A similar approach can be used on the client side. If, for example, the frame rate is set to 45 frames per second on the client side, the client-side program can monitor the decoded image or video stream and determine how many times the frames change per second. Again, this can be achieved by directly comparing the pixel data of the frames or by using CRC values. This approach provides information about the true frame rate generated by the application and the true frame rate that the client is receiving and presenting to the user of the client system.

[0023] Depending on the intended use, the actual observed frame rate of the entire system can be an important quality parameter for data transmission and display systems. For example, this actual observed frame rate may be required to be at least some specific values ​​(e.g., 5, 10, 15, 20, 30, 60 frames per second, etc., depending on their intended use). For applications that require extensive interaction with content, or applications that generate a large number of frames per second, it is usually higher.

[0024] Another important quality parameter for data transmission and display systems is the percentage of the actual observed frame rate of the entire system relative to the actual frame rate of the application. For example, if the application generates an actual frame rate of 20 frames per second, a typical minimum quality requirement could be that the actual observed frame rate of the entire system (the actual frame rate presented to the user on the client side) is at least 25%, 50%, 75%, 80%, 90%, or 100% of the application's actual frame rate. In this particular example, this means that the actual frame rate of the entire system should be at least 5, 10, 15, 16, 18, or 20 frames per second. This percentage often again depends on the intended use and is typically higher for applications that require extensive interaction with content or generate a large number of frames per second.

[0025] Ideally, frame rate settings should be aligned with each other. This means that an ideal server frame rate setting reflects (is equal to or a multiple of) the actual frame rate being generated by the application on the server. Similarly, ideally, the client frame rate setting reflects (is equal to or a multiple of) the server frame rate setting. Ideally, the display frame rate setting should be set to be equal to the client's frame rate setting, and frame locking (frame synchronization) should be enabled on the display to ensure that the display refresh is synchronized with the refresh on the client side. Similar to the difference between the set and observed (real) frame rates, there is also a difference between the set resolution and the observed (real) resolution. Data transmission codecs typically use scaling, where an image or video is scaled down on the sending side and then scaled up again on the receiving side. Simply looking at the configured resolutions on the server and client sides, one can assume there is no loss in actual resolution, while the actual (observed) resolution can be lower. For example, the server-side resolution might be set to 12 megapixels, with the codec scaled down to 3 megapixels, while on the client side, the codec might be scaled up again to 12 megapixels, and the client-side resolution might be set to 12 megapixels. In this example, the resolution is set to 12 megapixels, while the actual (observed) resolution is 3 megapixels. The actual (observed) resolution can be determined by comparing the original (application-generated) image with the image presented to the client user (pixel-wise). It can also be determined by examining codec settings or by using a test mode (a resolution-specific test mode), from which information about the system's actual resolution can be subtracted. Similar to frame rate, the actual (observed) resolution setting can be an important quality parameter, and absolute and relative minimum required values ​​can be set (e.g., a minimum of 3 megapixels, or a minimum of 25% / 75% / 100% of the original application-generated resolution).

[0026] Similar to frame rate and resolution, the observed (true) bit depth can differ from the set bit depth. The same principles described above can be applied to bit depth. The true bit depth can be determined by examining codec settings, comparing application-generated image data with client-presented image data, or by using specific bit depth test modes (e.g., grayscale or color ramp).

[0027] While not a feature of the image or video itself, in a preferred embodiment, the predefined set of quality parameter ranges also includes the range into which the quality parameters of the data connection between the server and the client should fall, such as a specific bit rate, as this affects possible data compression (which should generally be low) and frame rate (which should generally be high). Other examples of quality parameters for the data connection between the server and the client include connection latency, jitter, packet loss statistics, packet error statistics, whether a wired or wireless connection is used, and the rate of collisions or interference in the wireless network.

[0028] In another preferred embodiment, the predefined set of quality parameter ranges further includes the range into which the quality parameters of the server and / or client should fall, such as, in particular, server computing power, processor type, processor speed, memory size, memory speed, disk size and disk speed, the presence or absence of a GPU, the type and speed of the GPU (if present), the server's network connectivity, server load or utilization (e.g., the number of clients running on the server, the current load of the CPU or GPU, etc.). These quality parameters of the server and / or client are obtained, and taken into account in at least some steps, based on the selected set of quality parameter ranges, the data transmission and display system is configured to the first configuration, through which data is sent from the server to the client and received by the client, the quality parameters of the data received by the client are obtained, the quality parameters are compared with the predefined set of quality parameter ranges, and based on the comparison, image or video data is sent from the server to the client or the first configuration is changed.

[0029] In a preferred embodiment, a predefined set of quality parameter ranges is selected from multiple sets of quality parameter ranges corresponding to different intended uses. In a medical regulatory environment, the term "intended use" is generally used to define the purpose, target audience, exact function, and objective of a solution. Typical intended uses in medical applications include, for example, lung CT diagnosis, clinical review of echocardiographic images, and diagnosis of digital pathology images. The selection of the predefined set of quality parameter ranges based on the type of intended use can be done manually by the user, based on user instructions, the user's desired actions, the type of image or video requested to be displayed on the monitor, or automatically by selecting or visualizing medical procedure code. Of course, the predefined set of quality parameter ranges is not static and can change over time, for example, when new intended uses emerge, or when other parameter ranges are found to be more suitable for existing intended uses (e.g., when new standards, regulations, or guidance are created, or when technological advancements raise what is considered an appropriate minimum quality level).

[0030] According to the method of the present invention, a data transmission and display system for remotely displaying images or videos includes a server and a client, which are connected via a data exchange network, particularly the Internet. The client includes a display and is configured based on a selected set of quality parameters. Configuring the data transmission and display system may include: adjusting the bit depth of the image or video to be displayed, the frame rate of the video to be displayed, the bandwidth allocated to a specific data connection between the server and the client, the priority setting allocated to the specific data connection between the server and the client, the brightness setting of the display, the contrast setting of the display, the color setting of the display, the grayscale setting of the display, the frame rate setting of the display, the calibration setting of the display, the resolution setting of the display, the ICC profile setting of the display, the resolution setting of the server, the resolution setting of the client, the color or ICC setting of the server, the color or ICC setting of the client, the frame rate setting of the server, the frame rate setting of the client, the contrast or HDR setting of the server, the contrast or HDR setting of the client, the rendering settings of the server, and the rendering settings of the client.

[0031] According to a preferred embodiment, different configurations that have been tested to enable image and video transmission and display with specific quality parameters are stored in a memory or database. Accordingly, the method may include the step of creating lookup tables with different configurations for the data transmission and display system. This allows the system to quickly enter a first configuration and then, if necessary, further “fine-tune” the settings using this configuration.

[0032] Next, data is sent from the server to the client, and the client receives the data, obtains the quality parameters of the data received by the client, and checks the quality parameters of the data received by the client. If the quality parameters as described above also include the quality parameters of the data connection or the quality parameters of the server or the client, then it falls within the predefined quality parameter range. This data, also known as "test data," is typically similar to or identical to the data of an image or video actually displayed remotely.

[0033] In a preferred embodiment, the test data sent to the client is processed by the data transmission and display system in a manner similar to or the same as that used for processing image or video data displayed on a monitor; however, preferably, the client user does not actually see this test data. The test data can be one of the following:

[0034] - Additional image or video data, particularly additional rows and / or columns of pixels in an image or video for display, which are outside the range of rows and columns on the display.

[0035] - An additional bit layer used for displaying images or videos.

[0036] - Additional color data (e.g., multi-primary color data) or high dynamic range data used for displaying images or videos.

[0037] - Additional 3D (e.g., stereoscopic) data used for displaying images or videos,

[0038] - An additional header for displaying images or videos, which carries test patterns or additional data not used for display.

[0039] - Additional frame buffer data, especially data corresponding to the additional virtual display, which is not used for display.

[0040] The additional image or video data, the additional bit layer, the additional color data or high dynamic range data, the additional 3D data, the additional header, and the additional frame buffer data can be a combination of test modes and metadata, such as, in particular, timestamps, frame numbers, or cyclic redundancy checks.

[0041] Acquiring quality parameters and comparing them to a predefined set of quality parameter ranges can be performed by the client, and can be done in a manner different from, for example, measuring actual physical properties or comparing data before and after transmission, where transmission typically involves encoding and decoding, thus, for example, (lossy) compressed data. To facilitate this comparison, information related to the transmitted data can be sent to the client via a lossless channel. In other words, the client can receive information about, for example, how a test image should be viewed via a lossless channel and compare that information with how an image created from the test data is actually viewed. Note that this "viewing" can be entirely virtual, i.e., the client can internally compare images without displaying them to the user. The user can, of course, follow the progress through corresponding messages such as "System established," "Check data transmission," etc. Similar messages can also be output in further operations of the method, for example, when the data transmission rate changes and high-quality images or videos are no longer displayed. In this case, the user can also be asked to change the selected intended use. For example, if a radiologist selects "preliminary diagnosis of lung CT images" as the intended use, and it turns out that such images cannot be presented at the required quality, other possible "intended uses" can be offered, such as checking "clinical review of lung CT images," since the quality requirements for a preliminary diagnosis are higher than those for a clinical review.

[0042] The server can also obtain quality parameters of the data received by the client and compare these parameters with a predefined set of quality parameter ranges. To achieve this, information related to the data received by the client can be transmitted to the server via a lossless channel. Alternatively, tasks can be assigned, for example, such that the client obtains the quality parameters of the data received by the client, and the server compares these parameters with the predefined set of quality parameter ranges. In this case, information related to the data received by the client can be retransmitted to the server via a lossless channel. If a lookup table with a pre-stored configuration is used, feedback from comparing the obtained quality parameters with the predefined set of quality parameters can be used to improve the lookup table, or more specifically, to improve the configuration stored therein in a self-learning manner.

[0043] To provide greater flexibility during inspection, a predefined tolerance value can be used for each parameter range if the quality parameters are within a predefined range; however, for certain critical parameter ranges, it can also be set to zero. If the quality parameters are within the parameter range (± tolerance value), image or video data from the server is sent to the client and displayed on the corresponding monitor, and the configuration can optionally be further optimized, for example, by bringing the quality parameters of the received data to the middle of the corresponding parameter range. If the quality parameters are still not within the parameter range even considering the tolerance value, the configuration is changed and new "test data" and inspection steps are sent. If the quality parameters of the data received by the client are within the predefined quality parameter range, the configuration change is repeated until a configuration that allows the display of image or video data with the predefined quality is found. However, even if such a configuration is found, the method according to the invention is preferably performed in such a way that the quality parameters are continuously monitored to maintain or even improve the quality by adjusting the settings of the system consisting of the server, client, and monitor separately. In other words, even if the quality parameters have a predefined tolerance value within the parameter range, the steps of sending "test" data, obtaining the quality parameters, verifying them according to the allowed parameter range, and adjusting the configuration are repeated regularly. Regularity means that the repetition is not random, but according to specific rules, such as every five seconds, each time a new image is displayed, each time a new client session is started, each time a new user is started, and each time some parameters are changed, such as available bandwidth, the number of active clients on the server, etc. Therefore, it can be said that test data is "injected" into the transmitted image or video data, and can then be processed by the "infrastructure" just like a "regular" image or video requested by the user. Thus, any artifacts or image quality degradation that would be applied to regular images or videos will also be applied to the injected data.

[0044] In some applications, users may wish to inspect the same or different types of images or videos in parallel. In this case, the display may include several dedicated areas, or it may include several displays, with each dedicated area of ​​each display forming part of its own "data transmission and display system" as described above. Appropriate steps can be performed for each data transmission and display system to improve and ensure the quality of the displayed images and videos.

[0045] This method also advantageously allows for "multi-user" scenarios, i.e., establishing connections between a server and multiple clients. Each connection then forms a data transmission and display system, and each user can select a different intended use. In this case, the method may include the step of preferentially allocating resources among the data transmission and display systems based on at least one of the intended use selected by the user of each system, the user's urgent request for the system, and the user's user identifier. For example, if total bandwidth is limited and the user has an emergency, such as urgently needing to examine medical images of a critically ill patient, all or most of the bandwidth can be allocated to that user's system.

[0046] Figure 1 The diagram schematically illustrates some hardware components and software modules of a system for remotely displaying images or videos, particularly medical images or videos. The system includes a server 10 and a client 12, which can be connected to each other via a data exchange network 14, particularly the Internet. The server 10 and client 12 (in this example, including three displays 16, 18, and 20) form a data transmission and display system for performing the methods described above when the server 10 and client 12 are connected via the data exchange network 14.

[0047] In addition to displays 16, 18, and 20, client 12 in this example also includes a camera 22. Input / output devices such as a keyboard and mouse are not shown. Client housing 24 houses components to operate the software modules schematically shown in box 26, as well as GPU 28, network connector 30, and USB connector 32. GPU 30 is driven by GPU driver 34, which in turn is controlled by color management module 36 and desktop window manager 38. GPU driver 34, color management module 36, and desktop window manager 38 constitute part of the client's operating system schematically indicated by box 40.

[0048] In operation, the remote desktop receiver application, as explained above, can be a virtual display receiver application, indicated by box 42, that interacts with operating system 40 and also with other components and modules of client 12 (e.g., in particular quality monitoring receiver 46) via API 44. In this embodiment, the virtual display receiver application 42 includes frame buffer 48, monitoring module 50, and decoder 52.

[0049] Server housing 54 is connected to monitor 56 and operated via keyboard 58. Server housing 54 houses components for running the software modules schematically shown in box 60, as well as GPU 62 and network connector 64. GPU 62 is driven by desktop window manager 66, and GPU 62 and desktop window manager 66 constitute part of the operating system of the server schematically indicated by box 68.

[0050] In operation, the remote desktop transmitter application, as described above, can be the virtual display transmitter application shown in box 70, interacting with the operating system 68 and also interacting with other components and modules of server 10 (e.g., in particular the quality monitoring transmitter 74) via API 72. In this embodiment, the virtual display application 68 includes an encoder 76 and a monitoring module 78. The end-user application 80 creates pixels, which are then composited into a frame buffer (not shown) by the desktop window manager 66.

[0051] The virtual display receiver application 42 captures the frame buffer and encodes the pixels. The encoded pixels are transmitted to the client 12 via the data exchange network 14, and then sent to the virtual display receiver application 42. The virtual display receiver application 42 decodes the pixels and pushes them to the frame buffer 48. The frame buffer 48 is synthesized by the desktop window manager 38 and passed to the GPU 28 for placement on one of the displays 16, 18, and 20.

[0052] In the virtual display transmitter application 70, there are some monitoring and control functions exposed via API 72. Monitoring module 78 performs monitoring, and virtual display transmitter application 70 collects monitoring data. Monitoring module 78 can also monitor operating system 68, network, etc. Similarly, monitoring by monitoring module 50 on the client side 12 is exposed via API 44 and queried by virtual display receiver application 42. Monitoring module 50 can also monitor displays 16, 18, 20 and other sensors (e.g., experiential USB). Coordination between quality monitoring transmitter 74 and quality monitoring receiver 46 can be performed by the corresponding quality monitoring backend or directly (peer-to-peer) in the cloud.

[0053] The system may include multiple clients 12, each client 12 including at least one display. Each client can be connected to the server 10 via a data exchange network 14 to form its own data transmission and display system. In this case, when the server and clients are connected to each other via the data exchange network, the server 10 and the clients 12 can be used to perform a method for multiple user situations.

[0054] Within the scope of protection defined by the appended claims, numerous variations and embodiments are possible. For example, the "infrastructure" may be placed in a first configuration before selecting a certain set of parameter ranges, and certain steps may be performed out of order given in the claims, such that the given order should not be construed as binding. An important idea is that by continuously optimizing the remote desktop infrastructure, including display settings based on the needs of a specific intended use (especially medical use), continuously and implicitly measuring key quality metrics of the remote desktop setup, network connectivity, and display, and comparing these metrics to the minimum quality requirements of the specific intended use, it is possible to improve the quality of images and videos and ensure that images and videos are displayed remotely in a reliable and predictable manner.

Claims

1. A method for remotely displaying images or videos, in particular medical images or videos, characterized in that, Includes the following steps: a) Select a predefined set of quality parameter ranges, which includes the quality parameter ranges that the remotely displayed image or video should fall into. b) Based on a selected set of quality parameters, configure a data transmission and display system in a first configuration for remotely displaying images or videos, the system including a server and a client, the server and the client being data-connected to each other via a data exchange network, particularly the Internet, the client including a display. c) The data is sent from the server to the client via the system, and the data is received by the client; d) Obtain the quality parameters of the data received by the client. e) Compare the quality parameter with the predefined set of quality parameter ranges. f) Based on the comparison: - If the quality parameter has a predefined tolerance value within the parameter range, then image or video data is sent from the server to the client via the system, and the image or video data received by the client is displayed on the display. Optionally, the first configuration can be optimized, or... - If the quality parameter does not have the predefined tolerance value within the parameter range, then change the first configuration and repeat steps c) to f).

2. The method according to claim 1, characterized in that, wherein, The predefined set of quality parameter ranges also includes the quality parameter ranges that the quality parameters of the data connection between the server and the client should fall into. Among these, obtaining the quality parameters of the data received by the client includes obtaining the quality parameters of the data connection.

3. The method according to claim 1 or 2, characterized in that, wherein The predefined set of quality parameter ranges also includes the range of quality parameters that the server and / or client should fall into, specifically server computing power, processor type, processor speed, memory size, memory speed, disk size and speed, the presence or absence of a GPU, the type and speed of the GPU (if present), server network connectivity, server load or utilization, and Specifically, the quality parameters of the server and / or the client are obtained, and the quality parameters are considered in at least some of the steps in steps b) to f).

4. The method according to any one of claims 1 to 3, characterized in that, wherein The quality parameters include values ​​indicating at least one of the following: actual observed resolution, actual observed frame rate, color features, noise features, compression artifacts, brightness features, contrast features, latency, and jitter.

5. The method according to any one of claims 1 to 4, characterized in that, The predefined set of quality parameter ranges is selected from multiple sets of quality parameter ranges corresponding to different intended uses.

6. The method of claim 5, wherein, Users select the predefined set of quality parameters based on the type of intended use.

7. The method of claim 5, wherein, The predefined set of quality parameters is automatically selected based on the type of intended use indicated by the user, the type of image or video requested by the user to be displayed on the monitor, or the type of medical procedure in progress.

8. The method according to any one of claims 1 to 7, characterized in that, It also includes the step of: creating a lookup table for the data transmission and display system with different configurations, each configuration corresponding to one of the predefined sets of quality parameter ranges.

9. The method according to claim 8, characterized in that, Also includes: The lookup table is updated in a self-learning manner using feedback from the comparisons performed in step e).

10. The method according to any one of claims 1 to 9, characterized in that, Configuring the data transmission and display system includes adjusting at least one of the following: the resolution of the image or video to be displayed, the bit depth of the image or video to be displayed, the frame rate of the video or series of images to be displayed, enabling lossless mode of the image or video compression codec, the bandwidth allocated to a specific data connection between the server and the client, the bit rate of the image or video compression codec, the priority setting allocated to a specific data connection between the server and the client, the brightness setting of the display, the contrast setting of the display, the color setting of the display, the calibration setting of the display, the resolution setting of the display, the ICC profile setting of the display, the resolution setting of the server, the resolution setting of the client, the color or ICC setting of the server, the color or ICC setting of the client, the frame rate setting of the server, the frame rate setting of the client, the contrast or HDR setting of the server, the contrast or HDR setting of the client, the rendering setting of the server, and the rendering setting of the client.

11. The method according to any one of claims 1 to 10, characterized in that, The data sent in step c) is processed by the data transmission and display system in the same manner as the image or video data displayed on the display. Preferably, the data sent in step c) is not displayed on the display.

12. The method according to any one of claims 1-11, characterized in that, The data sent in step c) is one of the following: - An additional bit layer used for displaying images or videos. - Additional color data or high dynamic range data used for displaying images or videos. - Additional 3D (e.g., stereoscopic) image data used for displaying images or videos. - Additional header data for the image or video to be displayed, which carries test mode or additional data not used for display. - Additional frame buffer data, especially data corresponding to the additional virtual display, which is not used for display.

13. The method according to claim 12, characterized in that, The additional image or video data, the additional bit layer, the additional color data or high dynamic range data, the additional 3D data, the additional header, and the additional frame buffer data are a combination of test modes and metadata, such as, in particular, timestamps, frame numbers, or cyclic redundancy checks.

14. The method according to any one of claims 1 to 13, characterized in that, The client is responsible for obtaining the quality parameters of the data received by the client and comparing the quality parameters with the predefined set of quality parameter ranges.

15. The method according to claim 14, characterized in that, Information related to the data sent in step c) is sent to the client via a lossless channel.

16. The method according to any one of claims 1 to 13, characterized in that, The server is responsible for obtaining the quality parameters of the data received by the client and comparing the quality parameters with the predefined set of quality parameter ranges.

17. The method according to claim 16, characterized in that, Information related to the data received by the client is sent to the server via a lossless channel.

18. The method according to any one of claims 1 to 13, characterized in that, The acquisition of the quality parameters of the data received by the client is performed by the client, and the comparison of the quality parameters with the predefined set of quality parameter ranges is performed by the server.

19. The method according to claim 18, characterized in that, Information related to the data received by the client is sent to the server via a lossless channel.

20. The method according to any one of claims 1 to 19, characterized in that, Even if the quality parameter has a predefined tolerance value within the parameter range, steps c) to f) are repeated regularly.

21. The method according to any one of claims 1 to 20, characterized in that, Also includes: Information related to the results of the comparison is displayed on the monitor, and the user may be prompted to change the intended use of the selection.

22. The method according to any one of claims 1 to 21, characterized in that, The display includes a dedicated area for a display unit and / or the client includes multiple displays, each dedicated area of ​​each display forming part of its own data transmission and display system according to claim 1, the method comprising steps a) to f) for each data transmission and display system.

23. The method according to any one of claims 1 to 22, characterized in that, The server establishes data connections with multiple clients, each connection forming a data transmission and display system, including the steps of: preferentially allocating resources among the data transmission and display systems based on at least one of the intended use selected by the user of each system, an urgent request from the user of the system, and a user identifier of the user of the system.

24. A system for remotely displaying images or videos, particularly medical images or videos, comprising a server and a client, the server and the client being interconnected via a data exchange network, particularly the Internet, the client including a display, characterized in that, The server, client, and display are adapted to form a data transmission and display system for performing the method according to any one of claims 1 to 22 when the server and the client are connected to each other via the data exchange network.

25. The system according to claim 24, comprising multiple clients, each client including a display, each client being connected to the server via the data exchange network to form a corresponding data transmission and display system, characterized in that, The server and the client are adapted to perform the method according to claim 23 when the server and the client are connected to each other via the data exchange network.

Images