Server device for providing at least one graphical user interface to a client device, client device, system, method, computer program and electronically readable data carrier

Abstract

The invention relates to a server device (18) for providing at least one graphical user interface (26) to a client device (16) via a packet-based network (12). The server device (18) comprises a display adapter device (20) configured to operate a virtual display instance (28) according to a display instance configuration (44), in which a virtual display device (14) is provided. The server device (18) includes an application module (22) configured to generate character data (24) for defining the graphical user interface (26) of an application of the application module (22) and to provide this data to the display adapter device (20) for the virtual display instance (28).The display adapter device (20) is configured to provide the character data (24) for generating display data (25) for displaying the graphical user interface (26) in the virtual display device (14) of the virtual display instance (28) of a graphics card of the server device (18) and to transmit the generated display data (25) to a streaming server device (30) of the server device (18). The streaming configuration (48) specifies latency limits (50) above which the streaming server device (30) initiates a predefined action.

Description

[0001] The present invention relates to a server device for providing at least one graphical user interface to a client device via a packet-based network, a client device for receiving at least one graphical user interface from a server device via a packet-based network, a system comprising at least one server device and at least one client device, a method for operating a system, a computer program and an electronically readable data carrier.

[0002] Modern medical devices, such as mobile C-arms, have a wide range of internal functions for processing acquired 2D and 3D image data using the built-in imaging systems. However, there is a growing need to extend this basic functionality with third-party applications. This is important, for example, to minimize internal development costs for niche applications or to integrate specialized functionalities into the devices, focusing on the expertise of a third-party provider. These third-party providers are primarily medical application specialists.

[0003] This can be achieved, for example, with the so-called syngo.via OpenApps concept from Siemens Healthineers. Here, third-party applications can run on a separately provided server device, the AppHost computer, in a functionally isolated environment within the medical device. This prevents unintended interference between the medical device and the third-party application. This server device resides within the Siemens Healthineers system; that is, with this type of integration, only the third-party software runs on a dedicated server device. This separation is necessary for safety reasons, ensuring that malfunctions in the externally developed applications do not negatively impact the actual medical device within the system.

[0004] Alternatively, this can be achieved using a Network Graphical User Interface (NGI) streaming interface, which allows interactive screen content from third-party systems, such as medical navigation systems, to be streamed into the system from outside via Ethernet-based streaming. This concept is similar to remote desktop applications, but offers additional features to ensure functional safety during the transmission and display of third-party screen content, such as latency monitoring. Functional safety is relevant for many medical applications, such as medical navigation systems and / or operating room systems. In particular, latency times must not exceed certain limits. Standard remote desktop applications do not adequately provide sufficient solutions for the required functional safety.

[0005] To address these challenges, current technological approaches include the use of video distribution solutions. These solutions tap into monitor signals from medical devices with dedicated hardware outputs and display them on other devices. The advantage of these solutions is that they allow the GUI to be displayed without modifying the third-party system's software and achieve very low latency through the use of standard interfaces such as HDMI or DisplayPort. However, disadvantages include the complex additional cabling and the lack of a return channel for transmitting user interactions with the displayed GUI for remote control purposes. Achieving this functionality requires additional cables, such as USB cables, and additional hardware, such as KVM switches.Furthermore, not all external third-party systems have the necessary video outputs to provide GUI pixel data via a classic video distribution method.

[0006] Another approach is the use of "Remote Desktop" or "Terminal Server" solutions, which enable remote control of GUIs on a server by a client. In this approach, data transmission for both the GUI and user interactions occurs via a packet-based medium such as Ethernet. An advantage of this solution is its simple cabling, as only one network cable is required.

[0007] A disadvantage of existing approaches is that video distribution solutions typically have a lower bandwidth, and standard protocols do not allow for functionally reliable monitoring of the safety-critical latency of the displayed graphical user interface of the server device. While fieldbus systems like Profinet or EtherCAT exist to guarantee a response time, these solutions are expensive and, unlike Ethernet, are not readily available with many third-party systems, increasing the effort and cost of connecting such systems. Therefore, a solution that does not require additional components is desired.

[0008] The use of suitable remote desktop solutions can be achieved without providing additional hardware. However, to use appropriate remote desktop solutions, such as Network Graphical User Interface (NGI) streaming, the third-party provider of the interactive application must painstakingly and potentially error-prone integrate a provided reference code from the video server software solution into their application. This reference code enables the application to provide an interface for delivering a stream to a client device. This is followed by very complex verification and validation steps before the application provided by the third-party provider and the client device can be approved in a combined declaration. With this process, the third-party provider has no control over the latency thresholds at which the client device notifies the user if the threshold is exceeded.Instead, the value is fixed by the client device.

[0009] Additionally, this method inevitably exposes the interface, as source code must be provided. Furthermore, changes to the protocol require the third-party application's software to be extensively adapted and re-released.

[0010] Solutions for the transmission of user interfaces are known in the prior art, for example from DE 10 2014 216 887 B3.

[0011] The following challenges arise from the current state of the art and need to be addressed: The high-resolution user interface of third-party applications should be displayed on the display unit of the actual medical device with very low latency and high image quality, ideally visually lossless or physically lossless. The resolution of the display on the OpenApps PC must be dynamically adjusted to the current display area of ​​the third-party application on the imaging system screen and within the system context. The third-party application should be able to transmit image content to secondary control units (TouchUls) of the medical device to enable operation from within the sterile environment of the medical device. Third-party providers of safety-critical applications must be able to specify a maximum permissible latency for the stream and be notified if this latency is exceeded, so that they can take appropriate action.Integrating existing third-party applications into the system must be simple and require minimal effort.

[0012] The object of the present invention is therefore to provide a solution that enables the transfer of a graphical user interface of a server device to a display device of a client device, minimizing the need to adapt the application providing the graphical user interface. The solution is intended to ensure that the integration of the application on the third-party side can be carried out simply and efficiently, while simultaneously ensuring optimal performance and security for the interaction of the application with the server device.

[0013] This problem is solved by the respective subject matter of the independent claims. Advantageous further developments and preferred embodiments are the subject matter of the dependent claims.

[0014] A first aspect of the invention relates to a server device for providing at least one graphical user interface to a client device via a packet-based network. The at least one graphical user interface can, in particular, be an interactive graphical user interface. The server device comprises a display adapter device configured to operate a virtual display instance, in which a virtual display device is provided, according to a display instance configuration. The server device includes an application module configured to generate character data for defining the graphical user interface of an application of the application module and to provide this data to the display adapter device for the virtual display instance.The display adapter device is configured to use the character data to generate display data, which can then be displayed in the virtual display device of the virtual display instance on a graphics card of the server device. The display adapter device then transmits the generated display data to a streaming server device of the server device, which forwards this data in a display data stream via an associated streaming instance to a stream proxy device of the client device. The stream proxy device is configured to forward the display data stream received via the associated streaming instance to a display module of the client device, which in turn is configured to send the display data to the client device's display device.

[0015] The client device can be, in particular, a device for controlling an imaging device. The imaging device could be, for example, a computed tomography (CT) scanner or a magnetic resonance imaging (MRI) scanner. It could be, in particular, a mobile C-arm intended for intraoperative use. This may require the integration of applications for controlling devices during surgery. These could be third-party applications for real-time navigation based on 2D / 3D X-ray images. A high degree of functional safety is required for the display of a graphical user interface.The client device includes a display unit that may provide information or a graphical user interface for controlling the client device and / or the imaging device, as well as diagnostic images that may be generated by the imaging device. The client device may also include an input device that can be configured to capture user input.

[0016] The server device is a separate device from the client device, which can, for example, provide an additional application that may be related to the imaging device. The server device may, for instance, be used to control a surgical instrument. It may be necessary or advantageous to display a graphical user interface on the client device's display unit for controlling the server device and / or for displaying content provided by the server device. The server device and the client device may be connected via a packet-based network for data exchange.

[0017] To provide the graphical user interface of the server device to the client device, the server device includes a display adapter device. The display adapter device is configured to operate at least one virtual display instance. Within this virtual display instance, a virtual display device is provided. The virtual display device can simulate a physical display device, such as a monitor.

[0018] The application module of the server device is configured to generate character data for defining the graphical user interface and to provide this data to the virtual display instance of the display adapter device. The character data is data that describes the graphical user interface to be output. This character data is intended to instruct a graphics card to generate display data that can be presented by a display device.

[0019] The display adapter device is configured to transmit character data to the server device's graphics card in order to generate the display data for rendering the graphical user interface in the virtual display instance's virtual display device, based on this character data. In other words, the graphics card provides the character data along with information about the virtual display device. This information might include, for example, the refresh rate, color depth, and / or resolution of the virtual display device. The graphics card is configured to generate the display data based on this information and provide it to the display adapter device.

[0020] The display adapter device is configured to provide display data to a streaming server device on the server device. The streaming server device is configured to communicate with a stream proxy device on the client device over the packet-based network in a streaming instance associated with the virtual display device. The streaming server device is configured to forward the display data in a display data stream from the streaming instance to the stream proxy device on the client device over the packet-based network. The streaming instance can be provided, for example, via a Network Graphical User Interface (NGI) streaming interface.

[0021] The invention enables the graphical user interface of the server device to be displayed on the client device's display device via the packet-based network. A particular advantage of the invention is the provision of the display adapter device. This makes it possible to receive the drawing data from the application module and adapt the display data by providing the virtual display device – independently of any physical display device present on the third-party system. Advantageously, the server device's graphics card, or more precisely, a hardware functionality of the physical graphics card such as a 3D or AI accelerator of the server device, is used to provide the display data and redirected to the virtual display device according to the invention. That is, the physical graphics card renders the content of the virtual display device using hardware acceleration.The display data is transmitted by the server device. Therefore, it is not necessary to adapt the application module to provide the display data stream.

[0022] According to a further development of the invention, the streaming server device is configured to receive user input from the client device's stream proxy device and to provide it to the display adapter device for the virtual display instance assigned to the streaming instance. The display adapter device is configured to forward the user input to the application module of the display instance. In other words, the client device's display module is configured to receive user input from an input device. In other words, the further development provides that the system is configured not only to display the graphical user interface of the application module generated by the server device to the client device, but also to transmit user input from the client device to the server device for controlling the application module.The display module is therefore configured to receive user input provided by an input device. The input device could be, for example, a mouse, a keyboard, and / or a touch-sensitive layer of the display device through which the graphical user interface is shown. The display module is configured to provide the user input to the stream proxy device. The client device's stream proxy device is configured to transmit the user input in a user input stream over the packet-based network to the server device's streaming server device. The stream proxy device can then map the user input to the streaming instance associated with the respective display instance.

[0023] The streaming server setup can unpack the user input stream from the streaming instance and extract the user input. The streaming server setup then provides the user input to the display adapter device, which in turn forwards the user input to the virtual display instance of the display adapter device associated with the streaming instance.

[0024] The display adapter device is configured to provide user input to the application module assigned to the display instance. The application module is configured to process the received user input, thereby enabling the implementation of the entered user input. This training enables the integration of user input transmission to the application module.

[0025] According to the invention, the streaming configuration defines latency limits for the streaming instance. These latency limits relate to the display data stream and / or the input data stream of the streaming instance. Predefined actions are assigned to these latency limits, which are to be initiated by the streaming server as soon as the respective latency limit is exceeded by the latency of the streaming instance. For example, it can be provided that when one of the latency limits is exceeded, the streaming server sends a signal to the application module, which initiates a predefined action in the application module and / or informs the application module that the latency limit has been exceeded. In this case, the application module can, for example, display a warning message in the graphical user interface or initiate the output of an audible alarm message.For example, it may also be possible to deactivate certain functions provided by the application module as a preventative measure. For instance, an application module for operating a surgical instrument's navigation device may deactivate control of the surgical instrument via the client device if, due to exceeding the latency limit, it is assumed that proper operation cannot be guaranteed. This enhanced functionality offers the advantage of enabling consideration of the latency value and the initiation of latency-dependent measures.

[0026] According to a further development of the invention, the streaming server device is configured to set up input drivers and display drivers of the virtual display device of the virtual display instance.

[0027] According to a further development of the invention, the display adapter device is configured to operate several virtual display instances in a multi-instance mode according to their respective display instance configurations. This means that the display adapter device is configured to provide several virtual display instances simultaneously, each of which is defined by a separate display instance configuration. The display adapter device is also configured to provide the respective display data for displaying the respective graphical user interface in the respective virtual display device of the respective virtual display instance of the server device's graphics card.This means that the display adapter device, for each virtual display instance, instructs the graphics card to generate the necessary display data and sends it to the respective virtual display device to display the graphical user interface of that instance. Furthermore, the display adapter device is configured to transmit the display data of the respective virtual display instance to the streaming server device of the server device. The streaming server device is configured to transmit the respective display data in the display data stream, via the packet-based network, to the stream proxy device of the client device associated with the respective virtual display instance in the streaming instance assigned to that virtual display instance.This means that the streaming server receives the display data of the virtual display instances from the display adapter device and sends it to the stream proxy device of the respective client device via the associated streaming instances. It is possible to use multiple virtual display instances from virtual display devices simultaneously. Individual virtual display instances can be used to provide display data for multiple streams. In this case, a unique combination of streaming configuration and display instance configuration can be provided for each combination of stream and virtual display device. This allows for a unique set of configuration values, such as latency limits, to be set for each combination.This means that, according to the streaming configuration of one of the combinations, a warning can be issued, for example, if the latency exceeds 100ms, while according to the streaming configuration of another combination, a warning can be issued if the latency exceeds 1000ms. The first combination might be required for real-time display, while the second might be for non-real-time display. This further development offers the advantage that multiple virtual display instances can be deployed in parallel.

[0028] According to a further development of the invention, the display adapter device is configured to receive the character data from the application module for defining the graphical user interface and to provide it to several of the display adapter devices' virtual display instances. This means that the display adapter device is configured to receive the character data from the application module and make this data available to multiple virtual display instances. The display unit of the virtual display instance is configured to display the provided character data and thus present a graphical user interface to the user. By making this data available to multiple virtual display instances, the display adapter device can ensure that each instance has the necessary information to provide a correct display of the user interface.The use of virtual display instances makes it possible to host and display multiple graphical user interfaces on a single server device. By providing character data to multiple instances, the display adapter device can offer a scalable and flexible solution that can adapt to different requirements and configurations.

[0029] Overall, this further development of the invention enables efficient use of the application module's drawing data and a consistent display of the graphical user interface across multiple virtual display instances. The display adapter device is thus able to provide an adaptable and scalable solution for displaying graphical user interfaces.

[0030] According to a further development of the invention, the display adapter device is configured to receive multiple character data sets from the respective application modules for defining the respective graphical user interface and to provide these to the respective virtual display instances of the display adapter device. This means that the display adapter device is configured to receive character data from multiple application modules and forward this data to the respective responsible virtual display instance. Each application module provides its own graphical user interface, which is defined by the corresponding character data. The display adapter device is capable of supporting multiple such application modules and forwarding the respective character data to the correct virtual display instance.By using virtual display instances, the display adapter device can simultaneously host and display a multitude of graphical user interfaces, with each instance representing its own application with its own user interface. Providing the character data for the respective virtual display instances enables a flexible and scalable solution that can be adapted to various requirements and configurations. Overall, this further development of the invention enables efficient use of the character data from multiple application modules and the correct display of the respective graphical user interfaces for the responsible virtual display instances.

[0031] According to a further development of the invention, the server device comprises a scaling device. The scaling device is configured to receive the display data from one of the virtual display instances for rendering the respective graphical user interface according to the display instance configuration. The scaling device is configured to generate scaled display data for rendering at least a portion of the graphical user interface according to a scaling configuration based on the received display data. The scaling device is configured to provide the scaled display data to the streaming server device.

[0032] The display data can be scaled to scale the graphical user interface (GUI) up or down. Cropping of portions of the GUI is also possible. This scaling occurs before the scaled display data is transmitted to the client device. For example, a virtual display with a resolution of 1920x1080 pixels can be operated in clone mode alongside a physical display on the server device with a resolution of 1280x720 pixels. The scaling device then downscales the display data from 1920x1080 pixels to 1280x720 pixels. It can also crop the GUI to a portion of the image, for example, 720x720 pixels. The server device can be configured to provide only the display data requested at runtime.The advantage in further training is that no additional rendering of the scaled display data is required. Only existing display data is scaled and / or cropped, thus saving server resources.

[0033] According to a further development of the invention, the streaming server is configured to receive display data from virtual display instances and transmit this display data to the respective client devices via multiple streaming instances in a packet-based network. The streaming server is configured to receive the display data from the virtual display instance and process this data in multiple streaming instances. Each streaming instance is assigned to a specific virtual display instance and transmits the display data of that instance to the client device. Transmission occurs via a packet-based network, where the data is sent in small packets. The streaming server is capable of integrating the display data into the respective display data stream and transmitting it to the correct stream proxy of the client device.By using multiple streaming instances, the streaming server setup can process a large number of display data simultaneously and send it to the relevant client devices, thus providing an efficient and high-performance solution for transmitting display data. Overall, this further development of the invention enables reliable and fast transmission of display data over a packet-based network to the respective client devices.

[0034] According to a further development of the invention, the display adapter device is configured to receive multiple user inputs from different display modules and forward them to the responsible application module according to a prioritization specification. The display adapter device is capable of receiving and processing multiple user inputs from different display modules. Each display module provides its own graphical user interface, which is controlled by user input. The display adapter device is capable of receiving these inputs and determining, based on a prioritization specification, which inputs should be forwarded to which application module.By using a prioritization setting, the display adapter device can ensure that the correct user inputs are forwarded to the responsible application module and that the graphical user interface functions correctly.

[0035] The advantages and further developments set out above in connection with the server device according to the first aspect also apply mutatis mutandis to the client device, the system, the method, the computer program, and the electronically readable data carrier according to the invention. Accordingly, the features of the server device shown are to be regarded as features of the client device, the system, the method, the computer program, and the electronically readable data carrier.

[0036] In connection with the invention, a client device of a system for transmitting at least one graphical user interface of a server device to a display device of the client device via a packet-based network can be provided. The client device comprises a stream proxy device configured to receive display data in a display data stream via a streaming instance and forward it to a display module of the client device, which in turn is associated with a display device. The stream proxy device is configured to forward the display data stream received via the streaming instance to the display module of the client device. The display module is then configured to send this display data to the display device and thus make it visible to the user.

[0037] The client device's stream proxy is configured to receive the display data stream. The stream proxy is configured to forward the display data from the display data stream to a display module associated with the client device's display unit on which the graphical user interface is to be displayed. The display module is configured to transmit the display data to the relevant display unit for rendering the graphical user interface.

[0038] The display module can, for example, be configured to forward the display data to the client device's display unit in such a way that the graphical user interface generated by the server device is provided in a predefined display area of ​​a graphical user interface provided by the client device, or that an entire area of ​​the display unit displays the graphical user interface generated by the server device.

[0039] According to further training, the client device includes a control module. The control module is configured to generate the display instance configuration, which is used to configure the display instance. In other words, to provide the graphical user interface in the intended form, it is necessary to provide the display adapter device with the display instance configuration so that the display adapter device can then provide the display instance accordingly. The graphics card is configured to generate the display data for rendering the graphical user interface defined in the character data, according to the display instance configuration generated by the control module.The control module can, for example, be designed to change a screen resolution of the virtual display device specified in the display instance configuration, such as when the display area in which the graphical user interface is shown on the client device is changed. The control module can be located on a separate computing device from the client and server devices, or on both the client and / or server devices. This enhanced functionality offers the advantage of enabling dynamic and automatic adjustment of the display instance configuration.

[0040] According to a training module, the control module is configured to provide the streaming server setup with a streaming configuration for the streaming instance. In other words, the streaming instance's parameters are defined by the streaming configuration. The streaming configuration can include various parameters and settings, such as codec settings, bitrate, resolution, and other parameters that affect the streaming quality and performance. The streaming configuration can also specify individually configurable latency limits. The control module is configured to generate this streaming configuration and send it to the streaming server setup so that the streaming instance can be provisioned and / or adjusted accordingly.The control module can react automatically and dynamically to changes and ensure that the streaming instance is always correctly configured according to current requirements.

[0041] The advantages and further developments described above in connection with the client device also apply analogously to the server device, the system, the method, the computer program, and the electronically readable data carrier according to the invention. Accordingly, the features of the client device described above are to be considered features of the server device, the system, the method, the computer program, and the electronically readable data carrier.

[0042] A third aspect of the invention relates to a system comprising at least one server device according to the first aspect of the invention and at least one client device.

[0043] A fourth aspect of the invention relates to a method for operating a system for transmitting at least one graphical user interface of a server device to a display device of a client device via a packet-based network. The method comprises the following steps: providing a virtual display instance through a display adapter device of the server device, in which a virtual display device is provided, and sending character data to the display adapter device for the virtual display instance to define the graphical user interface of an application of the application module. The character data is then sent to a graphics card of the server device and used by the streaming server device to generate display data for rendering the graphical user interface.The streaming server provides a streaming instance associated with the virtual display device over the packet-based network and transmits the generated display data as a display data stream to the client device's stream proxy. The stream proxy receives the display data stream transmitted via the associated streaming instance and forwards it to a display module of the client device that is associated with the display device. The display module then sends the display data to the display device.

[0044] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0045] The aforementioned problem is also solved according to the invention by a computer program that can be directly loaded into the memory of a computing device. The computer program comprises programming means for executing the steps of the server device and / or the client device of the method according to the third aspect of the invention when the program is executed in the computing device.

[0046] Likewise, an electronically readable storage medium containing electronically readable control information may be present, which includes at least one described computer program (product) and is designed such that, when the storage medium is used in a computing device, it performs the steps of the server device and / or the client device of the method according to the third aspect of the invention. The storage medium may comprise a data carrier or a storage unit.

[0047] The server device and the client device can each comprise a computing unit.

[0048] A computing unit can be understood, in particular, as a data processing device containing a processing circuit. The computing unit can, in particular, process data to perform arithmetic operations. This may also include operations to perform indexed access to a data structure, such as a lookup table (LUT).

[0049] The computing unit may, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems on a chip (SoCs). The computing unit may also contain one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual cluster of computers or other units of the aforementioned type.In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more storage units.

[0050] A storage unit can be volatile data storage, for example as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data storage, for example as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), or magnetoresistive random access memory.It can be designed as MRAM (magnetoresistive random access memory) or as phase-change random access memory, PCRAM (phase-change random access memory).

[0051] Further features of the invention will become apparent from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures, may be encompassed by the invention not only in the combinations specified, but also in other combinations. In particular, the invention may also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention may encompass embodiments and combinations of features that go beyond or deviate from the combinations of features set out in the cross-references to the claims.

[0052] The invention is explained in more detail below with reference to specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be designated with the same reference numerals. The description of identical or functionally equivalent elements is not necessarily repeated with respect to different figures.

[0053] The figures show: Fig. 1 a schematic diagram of a system comprising a server device and a client device; Fig. 2 shows the components of the client device and the server device; Fig. 3 a schematic representation of a first operating mode of the system; Fig. 4 a schematic representation of a second operating mode of the system; Fig. 5 a schematic representation of a third operating mode of the system; and Fig. Figure 6 shows a schematic representation of a method for operating a system for transmitting at least one graphical user interface.

[0054] Fig. 1 a schematic representation of a system 10 comprising a server device 18 and a client device 16.

[0055] System 10 for transmitting one or more graphical user interfaces 26A, 26B of a server device 18 to a display device 46A, 46B of a client device 16 over a packet-based network 12 is shown. The server device 18 includes a display adapter device 20 configured to operate two virtual display instances 28A, 28B, each virtual display device 14A, 14B being generated by the graphics card of the server device 18.

[0056] The server device 18 has an application module 22, which is configured to generate character data 24 for defining a graphical user interface 26A, 26B of the application module 22 and to provide this data to the display adapter device 20 for the virtual display instances 28A, 28B. The display adapter device 20 is configured to transmit the character data 24 to the graphics card of the server device 18 to generate display data 25A, 25B for the virtual display device 14A, 14B of the respective virtual display instance 28A, 28B. The display adapter device 20 is configured to transmit the generated display data 25A, 25B to a streaming server device 30 of the server device 18. The respective display instance 28A, 28B is always the virtual image source for the respective graphical user interface 26A, 26B. The display module 46A, 46B is the respective sink.One of the display instances 28A, 28B can be assigned to several of the display modules 46A, 46B with respect to the respective graphical user interface 26A, 26B.

[0057] The streaming server facility 30 is configured to provide an associated streaming instance 42 for each virtual display instance 28 via a packet-based network 12 to a stream proxy facility 32 of the client device 16. The stream proxy facility 32 and the streaming server facility 30 can also encrypt the packet-based communication via the streaming instance 42. The stream proxy facility 32 and the streaming server facility 30 can be configured to authenticate themselves when the streaming instance 42 is established, for example, to allow access only for authorized partners. The streaming server facility 30 is configured to transmit the display data 25A, 25B in a respective display data stream 56A, 56B via the associated streaming instance 42A, 42B to the stream proxy facility 32 of the client device 16.

[0058] In the configuration shown, two streaming instances 42A and 42B are set up. However, there can also be one streaming instance 42 or any number of streaming instances 42. The stream proxy 32 is configured to receive the display data streams 56A and 56B received via two streaming instances 42A and 42B and to extract the display data 25A and 25B from the display data streams 56A and 56B. The stream proxy 32 is configured to forward the display data 25A and 25B to the respective display modules 34A and 34B of the client device 16. Display data 25A of a first display data stream 56A, which is assigned to the first streaming instance 42A, can be transmitted to three display modules 34A, each of which can be assigned to further display devices 46A of the client device 16.Display data 25B of a second display data stream 56B, which is assigned to the second streaming instance 42B, can be transmitted to a display module 34B, which can be assigned to a second display device 46B of the client device 16. The one display module 46B and the three display modules 46A are only exemplary values; generally, n=>1 per possible display module without a specific limit for the number of display locations of a streaming instance 42A, 42B. The display modules 34A, 34B are configured to send the display data 25A, 25B, which shows the respective graphical user interfaces 26A, 26B, to the respective display devices 46A, 46B. It can also be provided that, during operation, the graphical user interface 26A, 26B to be displayed on one of the display devices 46A, 46B is switched.In this case, for example, a request to the stream proxy device 32 can be used to switch the type of forwarded display data 25A, 25B from primary display data 25A to secondary display data 25B, so that the secondary graphical user interface 26B is displayed on the relevant display device 46A instead of the primary graphical user interface 26A, if this is useful in the workflow. Furthermore, one of the display devices 46A, 46B can also display several of the graphical user interfaces 26A, 26B simultaneously, e.g., via picture-in-picture, i.e., the primary graphical user interface 26A and the secondary graphical user interface 26B in parallel in two designated screen areas via screen compositing. In this case, the stream proxy device 32 can send both the primary display data 25A and the secondary display data 25B to the display module 34 responsible for the display device 46. Fig. Figure 1 shows the case of two different graphical user interfaces 26A and 26B. One of the display devices 46B shows one of the graphical user interfaces 26B, which is provided via a streaming instance 42B. Another of the display devices 46A shows another of the graphical user interfaces 26A, which is provided via another streaming instance 42A. Each of the streaming instances 42A and 42B is assigned a virtual display instance 28A and 28B in the display adapter device 20.

[0059] This display adapter device 20 is implemented such that it passes the character data 24 of the application module 22 to a graphics card physically present in the server device 18 in order to generate the display data 25A, 25B. This means that in a server device 18 to which no physically present display device 46 is connected, it is still possible to pass the graphics card through to an application and continue to use all the functions of the graphics card (e.g., CUDA in the case of Nvidia graphics cards). This is a function that commercially available solutions for the remote transmission of a graphical user interface 26 do not offer, or only offer to a limited extent—especially in combination with the other features of the invention, such as latency monitoring over non-deterministic transmission paths, such as Ethernet-based networks.If one or more physical display devices 46 are present on the server device 18, the acceleration provided by the graphics card for the virtual display devices 46 is also retained.

[0060] The display adapter device 20 is multi-instance capable, meaning that the display adapter device 20 can dynamically generate several secondary display devices 46A with resolutions and refresh rates differing from the primary display device 46A, in order to provide additional display devices 46A to specific applications as needed. Further parameters, such as individually adjustable latency limits, can also be individually defined for each streaming instance 42.

[0061] These secondary display devices 46A can be used, for example, to generate a secondary graphical user interface 26A in order to render content different from the primary graphical user interface 26A for secondary display devices 46A deployed as touch displays in the client device 16.

[0062] In addition to the display adapter device 20, there is also a virtual, multi-touch-capable input driver 52, which enables simultaneous interaction between the applications running on server device 18 via the application module 22 and the input devices 38 of the client device 16 (mouse, single and multi-touch screens). This input driver 52 can, if necessary, mutually lock parallel user inputs 36A, 36B from equally authorized input devices 38. The input drivers 52 can also be used to provide input devices 38 for other input modalities such as gesture control. With the appropriate input driver 52, such interactions can then also be returned from the client device 16 to the application module 22 on the server device 18 via the input data stream 58 of the streaming instance 42.The input driver 52 can receive user input 36 from the input devices 38 and forward the corresponding input to the virtual display instance 28. For example, the coordinates of a mouse pointer controlled by one of the input devices 38 can be converted according to the display instance configuration 44 such that they are converted from screen coordinates of one of the display devices 46 to screen coordinates of the virtual display device 14.

[0063] The virtual display devices 28A, 28B are managed on the server device 18 like real display devices 46A, 46B at the operating system level, making it very easy for the third party to continue using its existing technology for delivering the graphical user interface 26 and simply provide customized content on the virtual display devices 28A, 28B for the client device 16, which is then streamed to the client device 16 via NGS protocol.

[0064] Furthermore, the third-party vendor can also choose to mirror the graphical user interface 26 for the existing display device 46 of the server device 18 to the display device 46 of the client device 16 by duplicating or cloning its primary display device 46 to the virtual display device 14 at the operating system level. This results in minimal integration effort, as its actual application module 22 remains unchanged and the cloning process at the operating system level is decoupled from the application of the application module 22. During cloning, the server device 18 can also scale the graphical user interface 26 of the virtual display device 28 up or down and even crop portions of the graphical user interface 26 so that the entire content of the graphical user interface 26 is not streamed.For example, it may be intended that the display data 25 generated for the virtual display device 28, which represents the entire graphical user interface 26, is used as the basis for further display data 25, which, for example, only cover a sub-area of ​​the graphical user interface 26. The further display data 25 can then be provided to another of the display modules 34.

[0065] Furthermore, there is at least one server instance on the server device 18 (one per virtual display device 14) that works with the input driver 52 and the display driver 54 and provides encoding of the screen contents of the virtual display drivers.

[0066] The described system 10 has the following structure: The first display module 34B of the client device 16 displays the streamed image content received via Ethernet (wired or wireless) from the server device 18 on a display unit 46B of the client device 16 and receives user input 36B from existing input units 38B of the client device 16 and forwards it to the server device 18 or the respective virtual display instance 28B via network 12. The same applies to the second display module 34A of the client device 16, which is provided, for example, on a separate operator terminal of the client device 16. Both streamed data and user input 36B are routed through the stream proxy unit 32 of the client device 16. The stream proxy unit 32 can simultaneously manage multiple display data 25A, 25B from several virtual display instances 28A, 28B and interactions from several display modules 34A, 34B.

[0067] The control module 40 can run on a separate computer and / or on the client device 16 and / or on the server device 18. The control module 40 communicates the desired resolution and other streaming parameters for all display modules 34A and 34B to the streaming server device 30 running on the server device 18 via telegram through the stream proxy device 32. Alternatively, the server device 18 can also request the parameters from the control module 40 if it requires a different screen resolution or wants to reduce input from multi-touch to single-touch. Furthermore, the application module 22 can specify individual latency limits for the stream on the server device 18. If a maximum latency is exceeded, the application module 22 can be notified accordingly and can initiate appropriate measures (e.g., a warning message, an audible alarm, etc.).

[0068] The streaming server device 30 applies a predefined encoding procedure and communicates the requested parameters to the input driver 52 and the display driver 54. These then generate corresponding virtual display devices 46. Applications running on the server device 18 do not need to be adapted to this architecture – for them, the virtual display devices 46 appear like normal operating system desktops in the extended or duplicated multi-monitor setup.

[0069] The nature of the virtual display devices 46 ensures that local resources of the graphics card of the server device 18 can be properly utilized despite the absence of physical display devices 46, which would not normally be possible without this feature, at least under the operating systems 10 preferred here. The display module 34 becomes part of the user interface of the client device 16 and can display streams in various ways and scale them as needed, provided the application allows it. Furthermore, quality metrics (latency exceedance, dropped frames, checksum errors, etc.) are also reported back to the respective application module 22 via the described data connection. The application module 22 can then use this information to take further action, such as an audible warning in case of latency exceedance or a reduction in the frame rate. The response is defined by the respective provider of the application module 22 as part of their product risk assessment.This extension to include specific application measures is optional and can be defined by the third-party provider as to whether they should be used.

[0070] Fig. Figure 2 shows the components of the client device 16 and the server device 18.

[0071] System 10 comprises server device 18 and client device 16. Server device 18 and client device 16 can be interconnected via a packet-based network 12. The purpose of System 10 can be to provide an application on server device 18, which can be run in an application module 22. To enable control of the application running on server device 18 at client device 16, or to display a function of the application on client device 16, a graphical user interface 26 of the application can be provided on one or more display devices 46 of the client device 16. In the example shown, client device 16 has two display devices 46. Display modules 34 can be assigned to each display device 46.A first graphical user interface 26 of the application can be provided for the first of the display devices 46, and a second graphical user interface 26 of the application can be provided for the first of the display devices 46. The two graphical user interfaces 26 can differ in content and appearance. The content can, for example, include displayed controls or elements. The appearance can relate to a resolution, a color depth, and a refresh rate.

[0072] The application module 22 can be configured to define the content of the respective graphical user interfaces 26 in character data 24. The display adapter device 20 determines the display data 25, which describes the display of the respective graphical user interface 26 based on the character data 24. The display data 25 depends on the characteristics of the display device 46 and / or the integration of the graphical user interface 26 into a representation on the display device 46. According to the prior art, it is necessary, among other things, to configure the application module 22 in such a way that it already generates the display data 25. Thus, an adaptation of the application module 22 to the respective circumstances is required, which necessitates a considerable additional effort on the part of the third-party application provider.By providing the display adapter device 20, the task of providing the display data 25 is transferred from the application module 22 to the display adapter device 20. The application module 22 therefore only needs to provide the character data 24.

[0073] The display adapter device 20 is configured to provide the character data 24 for generating the display data 25 by means of a provision of virtual display instances 28. In the example shown, each display device 46 of the client device 16 is assigned one of the virtual display instances 28. The respective virtual display instance 28 provides a virtual display device 14. Appropriate display drivers can be assigned to the virtual display device 14. The virtual display device 14 can represent the display device 46 of the client device 16 or a portion of the display device 46 of the client device 16. The display adapter device 20 is configured to provide the character data 24 to the graphics card of the server device 18. The display instance 28 and the virtual display device 14 can be configured by a display instance configuration 44.This can be provided, for example, by a control module 40. Based on this, the graphics card can generate the display data 25 for rendering the respective graphical user interface 26 for the respective virtual display device 14. The display adapter device 20 can receive the display data 25 and provide it to a streaming server device 30. One idea is to stream the display data 25 for rendering the graphical user interface 26 to the client device 16. An advantage of this is that the graphics card of the server device 18 can be integrated.

[0074] The streaming server device 30 is configured to provide a streaming instance 42 associated with the virtual display device 14 to a stream proxy device 32 of the client device 16 via a packet-based network 12, and to transmit the display data 25 in a display data stream 56 via the associated streaming instance 42 to the stream proxy device 32 of the client device 16. The streaming server device 30 thus acts as an interface for the server device 18. The interface therefore does not need to be provided by the application module. The stream proxy device 32 can be configured, in particular, for the NGS protocol. The stream proxy device 32 can be configured to encode the display data and transmit it in the display data streams. The stream proxy device 32 is the corresponding counterpart and thus acts as the interface of the client device 16 for the packet-based network 12.The stream proxy device 32 can be configured to receive the mapping data stream and decode it accordingly for provision of the mapping data. The streaming instance 42 can be configured by a streaming configuration 48, which can be provided to the streaming server device 30 by the control module 40. The streaming configuration 48 can, for example, specify latency limits 50 for the display data stream 56 and / or the input data stream, above which a predefined action is initiated by the stream proxy device 32 and / or the streaming server device 30. The streaming configuration 48 can also define a compression level and a bandwidth to be provided for the streaming instance 42.

[0075] The stream proxy device 32 is configured to forward the mapping data to the relevant display module 34. The display modules 34 are configured to forward the display data 25 to the associated display device 46 in order to display the graphical user interface 26 through the display device 46. For interaction, an input device 38 can be provided, which can be connected to one of the display modules 34. The display module 34 can receive user input 36 from the input device 38 and provide it to the stream proxy device 32. The stream proxy device 32 can forward the user input 36 via an input data stream 58 in the streaming instance 42 to the streaming server device 30. The streaming server device 30 can provide the user input 36 to the relevant virtual display instance 28.The input drivers 52 allow the user inputs 36 to be processed and the application operated in the application module to be controlled.

[0076] System 10 can be specifically designed for the NGS protocol. The NGS (Network GUI Sharing) protocol is a System 10 for remotely controlling graphical user interfaces 26 of server devices 18 connected to client devices 16. It allows a client device 16 to display and control a graphical user interface 26 of the server device 18. Since it operates at the framebuffer level, NGS, from an architectural perspective, supports all windowing systems 10 and applications from third-party systems 10.

[0077] Multiple instances can run simultaneously on the same server device 18 to stream multiple graphical user interfaces 26 to separate ports at the same time. It is also possible to run other server instances located on different IP addresses and connected to the same stream proxy device 32 simultaneously.

[0078] The stream proxy device 32 typically runs on the client device 16 and can connect to one or more of the server devices 18 simultaneously. The stream proxy device 32 forwards the display data 25 to the display module 34 of the client device 16, which is assigned to the display device 46 for displaying the graphical user interface 26.

[0079] The NGS protocol uses various mechanisms to monitor and ensure the timely delivery of the display data 25, which describes the graphical user interface 26. These include Absolute Frame Latency (AFL), Network Bandwidth, Heartbeat Timeout, RTT, and Surveillance of Continuous Update. These mechanisms can be used to ensure that the graphical user interface 26 on the client device 16 is updated in a timely manner and that delays or failures can be detected and appropriate measures taken.

[0080] For each streaming instance 42, a version handshake is required to ensure that only compatible NGS versions are coupled. For each of the streaming server facilities 30 to which the stream proxy facility 32 is connected, a replicated port is provided on the internal network 12 of the client device 16, to which the display modules 34 can connect to receive the same (replicated) display data 25 for displaying the graphical user interface 26. The stream proxy facility 32 should also handle the arbitrage of interactions from the connected display modules 34 to ensure that an interaction on one display module 34 does not block interactions on any other display modules 34 located behind the stream proxy facility 32. Passing interactions between display modules 34 is not permitted.An important function of the stream proxy device 32 is its gatekeeper role, ensuring that no malicious traffic from the external network 12 can penetrate the internal system network 12 of the client device 16. The stream proxy device 32 must also support the handshake procedure (for authentication and encryption) to the streaming server device 30 and handle different refresh rates for the display modules 34. The stream proxy device 32 must support an algorithm for reducing the refresh rate for slower display modules 34 that do not support the incoming refresh rate, in order to prevent overloading of critical components. The display modules 34 should connect to the stream proxy device 32 via the internal TCP / IP network 12 of the client device 16.It is also possible for a display module 34 to run on the same physical / virtual system 10 as the stream proxy device 32. The display modules 34 should display the streamed graphical user interface 26 and collect user input 36 (touch commands and mouse input) to transmit it to the stream proxy device 32, which forwards the user input 36 to the streaming server device 30 for processing. In addition to mouse / touch input (single or multiple), keyboard commands or audio / haptic feedback (where applicable) can also be transmitted. The streaming server device 30 is responsible for the streamed graphical user interface 26. The display modules 34 should always scale the graphical user interface 26 to the available space of an allocated video frame area and also map the mouse / touch coordinates accordingly.To avoid distortion and ensure optimal image quality, the server device 18 must support the supported screen resolutions, e.g. 1280x720 for pixel-perfect 1:1 display.

[0081] All three communication nodes (server device 18, stream proxy device 32, and display modules 34) should also include an implementation of real-time clock (RTC) deviation calculation to measure the total transmission delay of the graphical user interface 26 and report it to the server. The server can then decide, based on the individual classification of the streamed graphical user interface, which response is appropriate for a specific transmission delay. Furthermore, the display modules 34 should have a mechanism to detect delays above a configurable threshold and inform the user that the "outdated" graphical user interface content is present due to latency exceedance. This information can be displayed, for example, by a red, semi-transparent overlay indicating that the latency limit has been exceeded.In cases of such an exceedance, mouse / touch inputs must be blocked by the display modules 34 and ongoing interactions terminated to prevent outdated ("too old") graphical user interface remote control on the CIOS system. Furthermore, the physical link speed should be monitored during connection time to avoid streaming over slow network connections, such as those caused by poor cabling, which can affect the MAC / NIC auto-negotiation process or temporarily impaired WiFi performance.

[0082] The communication nodes must be able to process the NGS protocol in its entirety and are also responsible for error handling, recovery from errors (e.g., automatic reconnection after connection loss), and displaying relevant information to the user, e.g., when the network cable is unplugged or when the network speed is below an acceptable threshold for secure communication.

[0083] The display modules 34 are divided into the standard Cios / CIARTIC TouchUls and a new communication node, the so-called MCP (Monitor Control Panel). The MCP overlays the GUI content of the display unit 46 upon request. All display modules 34 communicate with the system control via the AXCS interface and should be controlled by the TUI-IF. The NGS must be designed so that it does not cause critical delays for the (A)XCS traffic on the internal system network running in parallel. Since this network 12 does not support any protocol besides QoS (e.g., as offered by ProfiNet) or switches besides QoS (e.g., offered by a managed switch) for cost reasons, the stream proxy unit 32 must also implement a load balancing mechanism to ensure that the other system traffic is not affected by the video input.All display modules 34 should have a rectangular display area in which the streamed graphical user interface 26 is displayed and with which the user can interact. This area also contains local user messages. On the TouchUls, this display field should also contain relevant radiation values ​​(e.g., kV, mA, dose, cumulative exposure time) to enable the normal operation of the Cios system 10 (especially radiation release) in parallel with the also visible third-party GUI. A "hybrid" operation must therefore be possible. See the following screenshot as a suggestion for the design of this hybrid screen – the upper banner contains the Cios content, the lower part is reserved for the third-party content.

[0084] Fig. 3, Fig. 4 and Fig. 5 show different operating modes of system 10.

[0085] The following examples show the operation of the display adapter device 20 in three variations.

[0086] Fig. 3 a schematic representation of a first operating mode of the system.

[0087] In Fig. 3. The display unit 46 of the server device 18 shows a first graphical user interface 26A. The server device 18 generates a second graphical user interface 26B for display on a second display unit 46B of the client device 16, as well as a third graphical user interface 26C for display on a third display unit 46 of the client device 16. The display data 25B, 25C for displaying the second graphical user interface 26B and the third graphical user interface 26C are transmitted to the respective display modules 34B, 34C.

[0088] The graphical user interfaces 26A, 26B, and 26C may differ from each other. The first virtual display unit 14B and the second virtual display unit 14C operate in Extended Desktop mode.

[0089] Fig. 4 a schematic representation of a second operating mode of the system.

[0090] In Fig. Figure 4 shows the display device 46 of the server device 18 displaying the first graphical user interface 26A according to the first character data. The first graphical user interface 26A can be described in the first display data 25A. The second graphical user interface 26B can also be generated according to the first character data 24A and displayed on the first virtual display device 14A. The second graphical user interface 26B can be described in the second display data 25B.

[0091] This can be done in a clone mode, in which the graphical user interface 26A is displayed on the display unit 46A of the server device 18 and the virtual display unit 14A according to the first drawing data. Any optional scaling required can be handled by the streaming server device 30 and / or the first display modules 34A if the resolution of the display unit 46A of the server device 18 and the virtual display unit 14A are not identical. This allows display data 25A, 25B to be generated for the two display units 46A, 46B, which differ from each other.

[0092] A third, second user interface 26B can be displayed on the second display unit 46B, which is assigned to the second display module 34B. This second user interface is described by second display data 25B, which can be generated based on second character data 24B. The virtual display units 14A and 14B are operated in mixed clone and extended desktop mode.

[0093] Fig. 5 a schematic representation of a third operating mode of the system.

[0094] In Fig. In section 5, display unit 46 of server device 18 shows content. The same content can be displayed on the first virtual display unit 14A and the second virtual display unit 14B. Any necessary scaling is handled by streaming server device 30 or display modules 34A and 34B if the resolution of display unit 46 of server device 18 and virtual display units 14A and 14B are not identical. Virtual display units 14A and 14B operate in clone desktop mode.

[0095] The already known NGS method with its properties for the secure, latency-monitored transmission of safety-critical GUI content is extended by the described functions, which in this combination result in a novel transmission system.

[0096] It is proposed to install on the third-party external server device 18 a display adapter device 20, a streaming server device 30, and input drivers 52 and display drivers 54 for each graphical user interface 26 that the provider intends to provide individually for display on a display device 46A, 46B of the client device 16.

[0097] The advantage over providing source code for connecting the third-party application to the streaming network 12 is that the virtual display device 14A, input driver 52, and display driver 54, as well as the streaming server device 30, are available as binary files and, in the simplest case, do not require integration into the application module 22. This makes the actual streaming interface significantly easier to maintain and further develop, as tight integration with the application module 22 and its source code is no longer necessary. Furthermore, no internals of the streaming interface need to be disclosed. Moreover, the decoupling via the virtual display and input driver 52 and the display driver 54 ensures that the software works with any third-party application, regardless of the display technology it uses (e.g., HTML5, Qt, WPF, etc.). This massively simplifies and accelerates the onboarding of new application modules 22.

[0098] If the provider wants to provide two streaming instances 42 for a client device 16 designed as a mobile C-arm - one for displaying the graphical user interface 26A on a display device 46A that serves as the main monitor and one for displaying the graphical user interface 26B on a display device 46B that serves as a TouchUl control console, he would in this case install two of the virtual display devices 14A, 14B in respective virtual display instances 28A, 28B of the display adapter device 20 according to the respective display instance configurations 44A, 44B.

[0099] If the third-party provider wants to provide identical display data 25 for displaying the graphical user interface 26 on all display devices 46A, 46B, one virtual display instance 28A, the streaming server device 30 and the input driver 52 and the display driver 54 are sufficient.

[0100] The proposed system 10 is capable of displaying and making operable medical interactive graphical user interfaces 26A, 26B from server devices 18 via a packet-based network 12 on client devices 16. Compared to conventional systems 10, this system 10 offers additional functional blocks that extend its range of functions. These functional blocks include monitored and programmable latency, multi-display operation, virtual display instances 28 for easy integration of the server devices 18, internal routing of imaging data streams to multiple display devices 46A, 46B, and the arbitration of parallel user inputs 36A, 36B.

[0101] Fig. Figure 6 shows a schematic representation of a method for operating a system 10 for transmitting at least one graphical user interface 26.

[0102] The process may include the following steps: A first step S1 can include the provisioning of a virtual display instance 28 according to a display instance configuration 44 by a display adapter device 20 of the server device 18. A virtual display unit can be provided in the virtual display instance 28.

[0103] A second step S2 can include providing character data 24 to define the graphical user interface 26 of an application of the application module to a display adapter device 20 for a virtual display instance 28.

[0104] A third step S3 can include providing character data 24 to generate display data 25 to display the graphical user interface 26 to a graphics card of the server device 18 and transmitting the generated display data 25 to a streaming server device 30 of the server device 18.

[0105] A fourth step S4 can include the provision of a streaming instance 42 associated with the virtual display unit via a packet-based network 12 to a stream proxy facility 32 of the client device 16 by the streaming server facility 30, as well as the transmission of the display data 25 in a display data stream 56 via the associated streaming instance 42 to the stream proxy facility 32 of the client device 16 by the streaming server facility 30.

[0106] A fifth step S5 can include receiving the display data stream 56 transmitted via the associated streaming instance 42 by the stream proxy device 32 and forwarding the display data 25 to a display module 34 of the client device 16, which is assigned to the display unit.

[0107] A sixth step S6 can include sending the display data 25 to the display unit through the display module 34.

[0108] The preceding description is intended to include persons of male, female or other gender identities, regardless of the grammatical gender of a particular term.

Claims

[1] Server device (18) for providing at least one graphical user interface (26) to a client device (16) via a packet-based network (12), wherein - the server device (18) includes a display adapter device (20) configured to operate a virtual display instance (28) in which a virtual display device (14) is provided, according to a display instance configuration (44); - the server device (18) includes an application module (22) configured to generate character data (24) for defining the graphical user interface (26) of an application of the application module (22) and to provide it to the display adapter device (20) for the virtual display instance (28); - the display adapter device (20) is configured to provide the character data (24) for generating display data (25) for displaying the graphical user interface (26) in the virtual display device (14) of the virtual display instance (28) of a graphics card of the server device (18) and to transmit the generated display data (25) to a streaming server device (30) of the server device (18); and - the streaming server facility (30) is configured to provide a streaming instance (42) associated with the virtual display facility (14) to a stream proxy facility (32) of the client device (16) via a packet-based network (12), and to transmit the display data (25) in a display data stream (56) via the associated streaming instance (42) to the stream proxy facility (32) of the client device (16), characterized by, that the streaming configuration (48) specifies latency limits (50) for the streaming instance (42), above which a specified action is initiated by the streaming server setup (30). [2] Server device (18) according to claim 1, characterized by , that - the streaming server device (30) is configured to receive user input (36) from the stream proxy device (32) of the client device (16) in the streaming instance (42) and to provide it to the display adapter device (20) for the virtual display instance (28) assigned to the streaming instance (42); and - the display adapter device (20) is configured to forward the user inputs (36) to the application module (22) of the display instance (28). [3] Server device (18) according to any one of the preceding claims, characterized by, that the streaming server setup (30) is configured to set up display drivers (54) of the virtual display setup (14) of the virtual display instance (28). [4] Server device (18) according to any one of the preceding claims, characterized by , that - the display adapter device (20) is configured to operate several of the virtual display instances (28) in a multi-instance mode according to the respective display instance configurations (44); - the display adapter device (20) is configured to provide the respective display data (25) for displaying the respective graphical user interface (26) in the respective virtual display device (14) of the respective virtual display instance (28) of the graphics card of the server device (18); and to transmit the display data (25) of the respective virtual display instance (28) to the streaming server device (30) of the server device (18); and - the streaming server setup (30) is configured to transmit the respective display data (25) in a display data stream (56) via the packet-based network (12) to the stream proxy setup (32) of the respective client device (16) in the streaming instance (42) assigned to the respective virtual display device (14). [5] Server device (18) according to claim 4, characterized by , that the display adapter device (20) is configured to receive the character data (24) of the application module (22) for defining the graphical user interface (26) and to provide it to several of the virtual display instances (28) of the display adapter devices (20). [6] Server device (18) according to claim 4, characterized by, that the display adapter device (20) is configured to receive several of the character data (24) of respective application modules (22) for defining the respective graphical user interface (26) and to provide the character data (24) to the respective of the virtual display instances (28) of the display adapter devices (20). [7] Server device (18) according to claim 4, characterized by , that the server device (18) has a scaling device (60) which is configured to, - to receive the display data (25) of one of the virtual display instances (25) for displaying the relevant graphical user interface (26) according to the display instance configuration (44); - to generate scaled display data (62) to display at least part of the graphical user interface (26) according to a scaling configuration (44) based on the received display data (25); and - to provide the scaled display data (25) to the streaming server setup (30). [8] Server device (18) according to any of the preceding claims, characterized by , the streaming server facility (30) is configured to receive the display data (25) of the virtual display instance (28); and the streaming server facility (30) is configured to transmit the display data (25) in the respective display data stream (56) to the stream proxy facility (32) of the respective client device (16) in several streaming instances (42) associated with the virtual display facility (14) via the packet-based network (12). [9] System (10), characterized by , that the system (10) comprises at least one server device (18) according to one of claims 1 to 8 and at least one client device (16) for receiving at least one graphical user interface (26) from the server device (18) via a packet-based network (12), wherein - the client device (16) comprises a stream proxy device (32) configured to receive display data (25) for displaying the graphical user interface (26) in a display data stream (56) via a streaming instance (42) and forward it to a display module (34) of the client device (16) which is associated with a display device (46); and - the display module (34) is configured to send the display data (25) to the display device (46) in order to display the graphical user interface (26) on the display device (46). [10] Method for operating a system (10) for transmitting at least one graphical user interface (26) of a server device (18) according to one of claims 1 to 8 to a display device (46) of a client device (16) via a packet-based network (12), comprising the following steps: - Provision of a virtual display instance (28) according to a display instance configuration (44) by a display adapter device (20) of the server device (18) in which a virtual display device (14) is provided; - Providing character data (24) for defining the graphical user interface (26) of an application of the application module (22) to a display adapter device (20) for a virtual display instance (28); -Providing character data (24) for generating display data (25) for displaying the graphical user interface (26) to a graphics card of the server device (18) and transmitting the generated display data (25) to a streaming server device (30) of the server device (18); - Providing a streaming instance (42) associated with the virtual display device (14) via a packet-based network (12) to a stream proxy device (32) of the client device (16) by the streaming server device (30); - Transmitting the display data (25) in a display data stream (56) via the associated streaming instance (42) to the stream proxy device (32) of the client device (16) by the streaming server device (30); - Receiving the ad data stream (56) transmitted via the assigned streaming instance (42) by the stream proxy device (32) - Forwarding the display data (25) to a display module (34) of the client device (16), which is assigned to the display device (46); and - Sending the display data (25) to the display device (46) by the display module (34), characterized by, that the streaming configuration (48) specifies latency limits (50) for the streaming instance (42), above which a specified action is initiated by the streaming server setup (30). [11] A computer program that can be loaded directly into a memory of a server device (18) according to any one of claims 1 to 8, comprising program means to execute the steps of the method according to claim 10 to be carried out by the server device (18) when the program is executed in the server device (18) and / or a computer program that can be loaded directly into a memory of a client device (16), comprising program means to execute the steps of the method according to claim 10 to be carried out by the client device (16) when the program is executed in the client device (16). [12] Electronically readable data carrier with electronically readable control information stored thereon, which includes at least one computer program according to claim 11 and is designed such that, when the data carrier is used in a server device (18) according to one of claims 1 to 8, it performs the steps of the server device (18) of the method according to claim 10 and / or that, when the data carrier is used in a client device (18), it performs the steps of the client device (16) of the method according to claim 10.

Images