Wireless display sharing with dynamic resolution switching

Abstract

A video source device for wireless display sharing, comprising: an encoder designed to dynamically switch between encoding a full video frame into a first bitstream with a first resolution and a video subframe into a second bitstream with a second resolution, the second resolution being higher than the first resolution; a processing circuit arrangement designed to decide between encoding the full video frame and the video subframe based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, and / or an area-change category of the full video frame; and a transmitter designed to wirelessly transmit the first bitstream and the second bitstream to a video sink device.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] This application claims priority over preliminary US application No. 18 / 193,806, which was filed on March 31, 2023. Technical field

[0002] The aspects described herein generally concern wireless display sharing and, in particular, wireless display sharing that dynamically switches between lower and higher resolution for different change categories. background

[0003] Although wireless display systems offer advantages by eliminating the need for a wired display connection or projector for presentations, many productivity solutions do not utilize this technology due to suboptimal user experiences in terms of latency, performance, and image fidelity. Sharing high-resolution, low-latency wireless displays requires significant processing power, which can have a major impact on power consumption. It is desirable to reduce this impact on performance without compromising visual quality, especially for productivity applications such as word processing and presentation applications, which ideally exhibit high fidelity when shared with large external displays or monitors.This is particularly relevant for user input, where visual quality and higher frame rates contribute to a positive user experience. Brief description of the characters Fig. Figure 1 illustrates a schematic diagram of a wireless display sharing system according to aspects of the revelation. Fig. Figure 2 illustrates a schematic diagram of a video source / sink device according to aspects of the revelation. Fig. Figure 3 illustrates a schematic diagram of an operational overview of the system for wireless display sharing. Fig. 1 according to aspects of revelation. Fig. Figure 4 illustrates a flowchart of a procedure for wireless display release according to the disclosure. Fig. 5A and Fig. 5B illustrates time diagrams 500 of a wireless display release according to one aspect of the revelation. Detailed description

[0004] This disclosure describes a wireless display sharing system designed to reduce power consumption and latency while providing a visually lossless user experience. The system achieves this by capturing and encoding lower-resolution video images (e.g., 1920 x 1080 pixels) on the video source device side and upscaling them to a higher resolution (e.g., 4K = 3840 x 2160 pixels) on the video sink device side, which features a higher-resolution screen (monitor). The system dynamically switches between lower and higher resolutions for full-screen and partial-screen changes, respectively, to ensure a visually lossless user experience.

[0005] As a background, some wireless display systems allow a video source device to wirelessly share graphical content with a video sink device. For example, a mobile device can wirelessly transmit graphical content to a display device (such as a monitor, television, or other device) for viewing. The graphical content can originate from a variety of sources. For example, such graphical content might include screen sharing (such as displaying the mobile device's screen on the display device), audiovisual files, or data generated by an operating system or an application running within an operating system (such as a word processor, a web browser, a virtual meeting application, a video player, a video game, and the like). The transmission of graphical content from one device to another takes time, sometimes referred to as latency.In some cases, latency causes problems when displaying graphical content. For example, high latency can cause display errors (e.g., partially rendered displays / frames, distorted graphical data, and the like) and / or delays (e.g., a "frozen" screen, a screen that is unresponsive or responds slowly, and the like).

[0006] Fig. Figure 1 illustrates a schematic diagram of a wireless display sharing system 100 according to aspects of the revelation. The wireless display sharing system 100 comprises a video source device 102 that uses a wireless display pipeline to wirelessly communicate video data to a video sink device 104 according to aspects of the revelation. The video data consists of an ordered set of video frames. Each video frame is composed of a set of pixels. The video data can be stored locally on the video source device 102, stored remotely from the video source device 102 (cloud storage service, online file storage), and / or streamed from an online source (e.g., a website, an application connected via the internet to a source of the video data, an online video streaming service, an online live television streaming service).

[0007] The System 100 for wireless display sharing comprises a video source device 102, configured to wirelessly transmit a portion of a video image, and a video sink device 104, configured to wirelessly receive that portion of a video image. Generally, the video data can originate from any source on the video source device. For example, any graphical data from the video source device 102 is treated as video data for transmission to the video sink device 104. In some cases, the video data is generated by an application that plays video (e.g., a video player application, a web browser, a virtual meeting application, and the like) or by an application that generates multiple frames (e.g., a game application, a simulation application, and the like).Video data can be generated by any program running on the video source device. For example, an operating system can create a virtual desktop, the views of which can be included in the video data for transmission to the video sink device for screen sharing. As another example, a view from a word processing application can be included in the video data for transmission to the video sink device. Any source of graphical data from the video source device can be used in the video data for transmission to and display on the video sink device.

[0008] The video source device 102 comprises the wireless display pipeline, which includes a render engine 106, a display engine 108, an encoder 110, a packetizer 112, and a transmitter 114. The video sink device 104 comprises a display 116, a decoder 118, a packetizer 120, and a receiver 122. In operation, the video source device 102 processes the video data via the wireless display pipeline to enable the transmission of the video data to the video sink device 104. The video sink device 104 processes the video data received from the video source device 102 to enable the video data to be displayed on the display 116.

[0009] The video source device 102 processes the video data through the wireless display pipeline in a specific sequence. In this example, the video data through the wireless display pipeline is processed in the following order: (1) the render engine 106, (2) the display engine 108, (3) the encoder 110, (4) the packetizer 112, and (5) the transmitter 114. The render engine 106 renders the video data. The display engine 108 assembles the video data. Specifically, the display engine 108 can mix, overlay, combine, arrange, scale, or otherwise assemble pixels into a final video image for display. The encoder 110 encodes (e.g., compresses) the video data into a video compression format. Compression allows the video data to consume less storage space and / or less bandwidth for transmission. The packetizer 112 creates packets corresponding to the video data.Transmitter 114 sends the packets corresponding to the video data to video sink device 104. Upstream components process the video data and forward the processed data to a downstream component for further processing. For example, render engine 106 is upstream of display engine 108 (and display engine 108 is downstream of render engine 106), display engine 108 is upstream of encoder 110 (and encoder 110 is downstream of display engine 108), and so on. Video data is processed sequentially by each component and then forwarded downstream to an adjacent component in the wireless display pipeline.

[0010] Although the wireless display pipeline has several components in a specific order, each component is used when required for a particular wireless display scenario. There are situations where some components of the wireless display pipeline are not used. In such cases, a relative order of components is maintained, with the unused component(s) being removed from the specified order.

[0011] The video source device 102 processes the video data in a specific order to display the video data on the display 116. In this example, the video data is processed via the video sink device 104 in the following order: 1) the receiver 122, 2) the packetizer 120, 3) the decoder 118, and 4) the display 116. The receiver 122 receives the packets corresponding to the video data (from the video source device 102). The packetizer 120 unpacks the video data from the packets. The decoder 118 decodes the video data from the video compression format. The display 116 displays the video data.

[0012] Fig. Figure 2 illustrates a schematic diagram of a video source / recess device 200 (101 / 104 in Fig. 1) according to aspects of revelation.

[0013] The video source / sink device 200 comprises a processing circuit arrangement 202, a display engine 204, a memory 206, a wireless interface 228, a packetizer 238, one or more video codecs 240, and a render engine 242, each functionally coupled to the other by a data bus 226. The memory 206 comprises an operating system 208 containing code for execution by the processing circuit arrangement 202. The wireless interface 228 comprises a receiver 230, a receive (Rx) and transmit (Tx) buffer 232, and a transmitter 234. The device 200 can be used to perform the functions described herein.

[0014] The Video Source / Sink Device 200 can be any type of device designed to establish a wireless communication session with another device, such as a computer, a Personal Digital Assistant (PDA), a laptop or electronic notebook, a mobile phone, a tablet, a game console or a virtual reality (VR) or artificial reality (AR) device, or any other devices, components, elements, network elements or objects capable of initiating video, media and / or data exchange within a communication system.

[0015] Memory 206 is configured to store data used for the operations described here. Memory 206 is capable of storing software, logic, or processor instructions that are executed to perform the activities described here. Although memory 206 is illustrated as a single block for clarity, it can comprise multiple memories. For example, memory 206 can include a hard disk drive (HDD), a solid-state drive (SSD), random-access memory (RAM), a remote storage device (such as a cloud storage service or online file storage), other memories (or combinations thereof). To achieve low latency, some data can be stored in main memory (such as the HDD or static random-access memory (SRAM)), while other data is stored in RAM (where faster access maintains low latency).The processing circuit arrangement 202 can execute any type of instruction associated with the data to perform the operations detailed here.

[0016] The processor 202 can be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or any other device for executing code. The processing circuit arrangement 202 can be single-threaded or multi-threaded, meaning it can have more than one hardware thread context (or "logical processor"). The processing circuit arrangement 202 is configured to execute instruction code in a manner disclosed herein. Although the processing circuit arrangement 202 is represented as a single block, it can comprise multiple (one or more) processing elements.

[0017] The wireless interface 228 comprises the receiver 230, the Rx / Tx buffer 232, and the transmitter 234. The wireless interface 228 is used for wireless transmission and wireless reception of data (e.g., packets). The wireless interface 228 is configured to implement any wireless communication protocol. The wireless interface 228 is configured to wirelessly transmit and receive packets conforming to any wireless or cellular communication protocol, or the like. The wireless interface 228 can be a transceiver or a transmitter-receiver. As an example, the receiver 230 can comprise a radio receiver configured to receive electromagnetic waves at a high frequency (e.g.,Packets encoded therein in a wireless communication protocol); the transmitter 234 may include a radio transmitter configured to transmit electromagnetic waves at a high frequency (e.g., packets encoded therein in a wireless communication protocol). The transmitter 234 and the receiver 230 may use the Rx / Tx buffer 232 to store data to be transmitted and / or data to be received.

[0018] The Render Engine 242 renders graphical data. The Render Engine 242 writes rendered graphical data to an input buffer in one or more input buffers 210. The Render Engine 242 can generate graphical representations of data produced by a program. Such graphical representations can include, for example, images, pixels, a video frame, or a portion thereof. Many programs generate data that is displayed graphically on a screen. Such a program can include, for example, an operating system 208, an application running within an operating system (e.g., a word processor, a web browser, a virtual meeting application, a video player, a video game, and the like). Every program generates data that can be rendered for display on a screen.For example, text, icons, images, the position of icons within the program, images corresponding to the icons, the size and position of letters in the program, or any other data for which a graphical representation can be displayed. The Render Engine 242 can process data from a three-dimensional virtual reality game to render a video image based on input from an input device (e.g., rendering video images showing a user's virtual perspective based on joystick movements). The Render Engine 242 writes the rendered video image to an input buffer in one or more input buffers 210.

[0019] The display engine 204 assembles graphical data, such as video data, which is read from one or more input buffers 210 and was generated, for example, by the render engine 242. After assembling a pixel, the display engine 204 writes the pixel to the frame buffer 212 to create a video image. The display engine 204 can be implemented in hardware, software, firmware, or a combination thereof.

[0020] Video scaling by the Display Engine 204 involves converting a video resolution to a higher or lower resolution. The Display Engine 204 of the Video Source Device 102 downscales video frames to a lower resolution for transmission, and then the Display Engine 204 of the Video Sink Device 104 upscales the received video frames to the original higher resolution. The scaling process involves using algorithms to interpolate or extrapolate pixels in the original video image to insert the additional pixels required to achieve the higher resolution. The goal of video scaling is to preserve image quality and minimize distortion or artifacts in the upscaled video. Various scaling techniques can be used, such as nearest neighbor, bilinear interpolation, bicubic interpolation, and Lanczos resampling.These techniques differ in their computational complexity and their trade-offs between image quality and processing speed.

[0021] Each of the 240 video codecs can encode and decode video images based on a video compression / encoding format. The encoding and decoding operations are similar, so the same hardware can encode and decode. For example, a video codec is designed to encode an unencoded video image into an encoded video image (e.g., a bitstream). Similarly, a video codec is designed to decode an encoded video image into an unencoded video image. The 240 video codecs can be implemented in hardware, software, firmware, or a combination thereof. In operation, each of the 240 video codecs encodes and / or decodes groups of pixels (blocks), each based on a video compression standard.

[0022] The packetizer 238 is configured to generate packets corresponding to graphical data. For example, after a portion of a video image has been encoded into a bitstream by one of the video codecs 240, the packetizer 238 generates packets from the bitstream. The packetizer 238 inserts the bitstream into packets that conform to a wireless communication protocol used by the wireless interface 228. The packetizer 238 can be implemented in hardware, software, firmware, or a combination thereof.

[0023] The video device 200 may include or be functionally coupled to a display device (e.g. a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying the video data, a keyboard and a pointing device (e.g. a mouse or trackball) by which a user can provide input.

[0024] Fig. Figure 3 illustrates a schematic diagram of an operational overview 300 of the system 100 for wireless display release of Fig. 1 according to aspects of revelation.

[0025] The video source device 102 is designed to dynamically switch between transmitting a full video frame 332 and a partial video frame 334. The processing circuit arrangement 202 of the video source device 102 makes this decision based on factors such as the amount of available wireless transmission bandwidth, the number of pixels in a modified area of ​​the full video frame, the spatial complexity of a modified area of ​​the full video frame, the temporal complexity of a modified area of ​​the full video frame, and / or the area change category of the full video frame.

[0026] For full-frame video transmission, the video source device 102 renders and encodes a full video frame 332 into a first bitstream 3321 with a first resolution, performing downscaling. Full-frame video transmissions have a lower resolution, which reduces the duty cycle and consequently improves latency and power efficiency. Furthermore, the use of fewer encoded frames enables faster wireless transmission to the video sink device 104.

[0027] For video subframe transmission, the video source device 102 renders and encodes a video subframe 334 into a second bitstream 3322 with a second resolution. The second resolution is higher than the first. The video subframe transmissions 334 have a higher resolution to provide a better visual user experience for interactive tasks such as texting, menu pop-ups, and mouse click options in productivity applications.

[0028] The video sink device 104 is designed to receive the first bitstream 3321 and the second bitstream 3322, which were wirelessly transmitted from the video source device 102. The video sink device 104 decodes and upscales the first bitstream 3321 and decodes the second bitstream. The video sink device 104 produces a composite video image for display 316 based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0029] Fig. Figure 4 illustrates a flowchart of a process 400 for wireless display release in accordance with aspects of the disclosure.

[0030] In step 410, video source device 102 initiates wireless display sharing by performing a discovery process to search for available video sink devices on the same network. Video source device 102 then negotiates a connection with video sink device 104 to establish a communication channel between them. Finally, in step 420, video source device 102 receives metadata about video sink device 104.

[0031] At step 430, the processing circuit assembly 202 of the video source device 102 decides between encoding the full video frame with the first resolution for the first bitstream 3321 and encoding a video subframe with the second resolution for the second bitstream 3322. The processing circuit assembly 202 makes this decision based on factors monitored by the display engine 108. These factors include, among others, the amount of available wireless transmission bandwidth, the number of pixels in a modified area of ​​the full video frame, the spatial complexity of a modified area of ​​the full video frame, the temporal complexity of a modified area of ​​the full video frame, and / or the area-change category of the full video frame.

[0032] Once the processing circuit arrangement 202 decides between transmitting a full video frame or a partial video frame, the encoder 110 / 240 encodes the full video frame or the partial video frame with the respective resolution. If the processing circuit assembly 202 decides to transmit a full video frame, the render engine 106 / 242 renders the full video frame at step 440F, the display engine 108 / 204 downscales the full video frame to the lower resolution, and the encoder 110 / 240 encodes the downscaled full video frame into the first bitstream 3321. Alternatively, if the processing circuit assembly 202 decides to transmit a partial video frame, the render engine 106 / 242 renders the partial video frame at step 440S, and the encoder 110 / 240 encodes the rendered partial frame into the second bitstream 3322 at a higher, native display resolution, without the need for downscaling.Furthermore, encoder 110 encodes a position of the video subframe within the video full frame into the second bitstream 3322.

[0033] At step 450, the packetizer generates 112 packets corresponding to the encoded video data to be placed in a buffer queue for transmission by the transmitter 114 to the video sink device 104.

[0034] At step 460, the receiver 122 of the video sink device 104 receives the packets transmitted by the transmitter 114 of the video source device 102. And at step 470, the decoder 118 decodes the received packets.

[0035] At step 480, the processing circuit arrangement 202 of the video sink device 104 determines whether the decoded frame is based on the first bitstream 3321 with the lower resolution or the second bitstream 3322 with the higher resolution. If the decoded frame is based on the first bitstream 3321 with the lower resolution, the display engine 204 upscales the decoded frame to the higher resolution at step 480S, as described above.

[0036] In step 490, the display engine 108 / 204 of the video sink device 104 generates the composite video image with the decoded and upscaled first bitstream 3321 as a base layer and the decoded second bitstream 3322 as a superimposition on the base layer. The last video frame, transmitted at full resolution, is used as a layer of the composition and upscaled to the native resolution. All relevant video subframes that were transmitted are composited onto the single full-resolution frame and do not require scaling because they were transmitted at the higher resolution. In this way, video changes during typing appear visually lossless to the user, even if the background image does not have the same visually lossless image fidelity.

[0037] The encoder 110 / 240 encodes an initial full-frame video into the second bitstream 3322 at the second, higher resolution. During user operation, documents requiring full-frame transmission can be opened. For example, opening a word processing document in full-screen mode would be sent as a full-frame transmission. Since it typically takes a few seconds for the user to begin interacting with the document, the initial full-frame video is transmitted at the higher resolution and retained for the encoder operation.

[0038] Video frame changes can be saved for later addition or removal. For example, a context menu might appear when a user right-clicks. Because this change is sent as a video frame, the video frame is set to a previous full-frame video. Once the user clicks away, the context menu disappears. This disappearance simply involves a transmission indicating that the video frame is no longer valid, and the composition reverts to the previous full-resolution video frame.

[0039] During periods of little or no change (i.e., pixel changes are below a predetermined threshold), more complex video blocks are transmitted at the higher resolution to gradually refresh the full frame. A separate, higher-resolution frame is stored at the video sink device 140 as the last fully composed frame. The encoder 110 tracks the complexity of blocks based on their spatial and temporal complexity, which indicates how easily the pixel content can be compressed. These video subframes are sent as permanent compositions, as opposed to a context menu that can be assembled or removed from the composition.Feedback on the extent of the change and its relative complexity could be provided either by the Display Engine 108 during composition (if used) or directly by the application that registered the change.

[0040] Fig. 5A and Fig. 5B illustrates time diagrams 500 of a wireless display release according to one aspect of the revelation.

[0041] Fig. 5A and Fig. Figure 5B illustrates processing times for a video source device 102 for rendering and encoding at different resolutions. Fig.Figure 5A shows that the video source device 102 requires more than 30 ms for a 4K resolution, while it requires approximately 17 ms for a 1080 resolution. The rendering and encoding processing time for a 4K resolution is longer, resulting in a longer end-to-end time for video stream transmission to the video sink device 104. The wireless display sharing technique disclosed here dynamically switches between full-frame and partial-frame video and renders and encodes at different resolutions to improve latency and power consumption, with the difference being less perceptible to the end user.

[0042] The techniques of this revelation can also be described in the following examples.

[0043] Example 1.A video source device for wireless display sharing, comprising: an encoder designed to dynamically switch between encoding a full video frame into a first bitstream with a first resolution and a video subframe into a second bitstream with a second resolution, the second resolution being higher than the first resolution; a processing circuit arrangement designed to decide between encoding the full video frame and the video subframe based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, and / or an area-change category of the full video frame; and a transmitter designed to wirelessly transmit the first bitstream and the second bitstream to a video sink device.

[0044] Example 2. Video source device according to Example 1, wherein during a period in which the video full frame exhibits a pixel change below a predetermined threshold, the encoder is further configured to encode the video full frame into the second bitstream with the second resolution in order to refresh the video full frame for display at the video sink device with the second resolution.

[0045] Example 3. Video source device according to one of Examples 1-2, wherein the processing circuit arrangement is further configured to: decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on a quantity of available wireless transmission bandwidth.

[0046] Example 4. Video source device according to one of claims 1-3, further comprising: a display engine designed to monitor the amount of available wireless transmission bandwidth.

[0047] Example 5. Video source device according to one of Examples 1-4, wherein the encoder is further configured to encode a position of the video subframe within the video full frame into the second bitstream.

[0048] Example 6. Video source device according to one of Examples 1-5, wherein the encoder is further configured to encode an initial full video frame into the second bitstream with the second resolution.

[0049] Example 7. Method according to any one of claims 1-6, wherein the video source device is a device for virtual reality (VR) or artificial reality (AR).

[0050] Example 8. Wireless display sharing system, comprising: the video source device according to any of Examples 1-7; and a video sink device designed to receive the first bitstream and the second bitstream transmitted wirelessly, to decode the first bitstream, to upscale the decoded first bitstream from the first resolution to the second resolution, to decode the second bitstream, and to produce a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0051] Example 9. Wireless video sink device for display sharing, comprising: a receiver designed to wirelessly receive a first bitstream and a second bitstream from a video source device, the second resolution being higher than the first resolution; a decoder designed to dynamically switch between decoding the first bitstream into a full video frame at a first resolution and the second bitstream into a partial video frame at a second resolution; a scaler designed to upscale the decoded bitstream from the first resolution to the second resolution; and a display designed to show a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0052] Example 10. Video sink device according to Example 9, wherein the composite video image is generated with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer.

[0053] Example 11. Video sink device according to one of claims 9-10, wherein the video sink device is a device for virtual reality (VR) or artificial reality (AR).

[0054] Example 12. Wireless display sharing system, comprising: a video source device designed to dynamically switch between encoding a full video frame into a first bitstream of a first resolution and a partial video frame into a second bitstream of a second resolution, the second resolution being higher than the first resolution, and to wirelessly transmit the first bitstream and the second bitstream; and the video sink device according to any of Examples 9-11.

[0055] Example 13. Component of a video source device for wireless display sharing, comprising: a processing circuit arrangement; and a non-volatile, computer-readable storage medium comprising instructions which, when executed by a processing circuit arrangement, cause the processing circuit arrangement to: dynamically switch between encoding a full video frame into a first bitstream of a first resolution and a partial video frame into a second bitstream of a second resolution, the second resolution being higher than the second resolution;Deciding between encoding the full video frame and encoding the partial video frame based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, and / or an area-change category of the full video frame; and causing a transmitter to wirelessly transmit the first bitstream and the second bitstream to a video sink device.

[0056] Example 14. Component according to Example 13, wherein the instructions further cause the processing circuit arrangement to do the following: during a period of time in which the video full frame exhibits a pixel change below a predetermined threshold, encode the video full frame into the second bitstream with the second resolution in order to refresh the video full frame for display at the video sink device with the second resolution.

[0057] Example 15. Component according to one of Examples 13-14, wherein the instructions further cause the processing circuit arrangement to: decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on a quantity of available wireless transmission bandwidth.

[0058] Example 16. Component of one of Examples 13-15, wherein the instructions further cause the processing circuit arrangement to: Encode a position of the video subframe within the video full frame into the second bitstream.

[0059] Example 17. Component according to one of claims 13-16, wherein the instructions further cause the processing circuit to: encode an initial full video frame into the second bitstream with the second resolution.

[0060] Example 18. Component of a video sink device for wireless display sharing, comprising: a processing circuit arrangement; and a non-volatile, computer-readable storage medium comprising instructions which, when executed by a processing circuit arrangement, cause the processing circuit arrangement to: dynamically switch between decoding a first bitstream into a full video frame at a first resolution and a second bitstream into a partial video frame at a second resolution, wherein the first bitstream and the second bitstream were received from a video source device and the second resolution is higher than the first resolution; and upscaling the decoded first bitstream from the first resolution to the second resolution; and generating a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0061] Example 19. Component according to Example 18, wherein the instructions further cause the processor circuit arrangement to: generate the composite video image with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer.

[0062] Example 20. Video source device for wireless display sharing, comprising: an encoding means for dynamically switching between encoding a full video frame into a first bitstream with a first resolution and a video subframe into a second bitstream with a second resolution, the second resolution being higher than the first resolution; processing means for deciding between encoding the full video frame and encoding the video subframe based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, and / or an area-change category of the full video frame; and a transmitting means for wirelessly transmitting the first bitstream and the second bitstream to a video sink device.

[0063] Example 21. Video source device according to Example 20, wherein during a period in which the video full frame exhibits a pixel change below a predetermined threshold, the encoder is further configured to encode the video full frame into the second bitstream with the second resolution in order to refresh the video full frame for display at the video sink device with the second resolution.

[0064] Example 22. Video source device according to one of Examples 20-21, wherein the processing circuit arrangement further serves to decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on a quantity of available wireless transmission bandwidth.

[0065] Example 23. Video source device according to any of Examples 20-22, further comprising: a display engine means for monitoring a quantity of available wireless transmission bandwidth.

[0066] Example 24. Video source device according to one of Examples 20-23, wherein the encoding means further serves to encode a position of the video subframe within the video full frame into the second bitstream.

[0067] Example 25. Video source device according to one of Examples 20-24, wherein the encoding means further serves to encode an initial video full frame into the second bitstream with the second resolution.

[0068] Example 26. Video source device according to one of claims 20-25, wherein the video source device is a device for virtual reality (VR) or artificial reality (AR).

[0069] Example 27. Wireless display sharing system, comprising: the video source device according to any of Examples 20-26; and a video sink device designed to receive the first bitstream and the second bitstream transmitted wirelessly, to decode the first bitstream, to upscale the decoded first bitstream from the first resolution to the second resolution, to decode the second bitstream, and to produce a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0070] Example 28. Wireless video sink device for display sharing, comprising: a receiver for wirelessly receiving a first bitstream and a second bitstream from a video source device, the second resolution being higher than the first resolution; a decoder for dynamically switching between decoding the first bitstream into a full video frame with a second resolution and the second bitstream into a partial video frame with a second resolution; and a scaling device for upscaling the decoded first bitstream from the first resolution to the second resolution; and a display device for displaying a composite video image based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0071] Example 29. Video sink device according to Example 28, wherein the composite video image is generated with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer.

[0072] Example 30. Video sink device according to one of claims 28-29, wherein the video sink device is a device for virtual reality (VR) or artificial reality (AR).

[0073] Example 31. Wireless display sharing system, comprising: a video source device designed to dynamically switch between encoding a full video frame into a first bitstream of a first resolution and a partial video frame into a second bitstream of a second resolution, the second resolution being higher than the first resolution, and to wirelessly transmit the first bitstream and the second bitstream; and the video sink device according to any of Examples 28-30.

[0074] Example 32. Component of a wireless display sharing video source device, comprising: processing means; and a non-volatile, computer-readable storage medium containing instructions which, when executed by the processing means, cause the processing means to: dynamically switch between encoding a full video frame into a first bitstream of a first resolution and a video subframe into a second bitstream of a second resolution, the second resolution being higher than the second resolution; decide between encoding the full video frame and encoding the video subframe based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, and / or an area-change category of the full video frame;and causing a transmitter to wirelessly transmit the first bitstream and the second bitstream to a video sink device.

[0075] Example 33. Component according to Example 32, wherein the instructions further cause the processing means to: during a period in which the video full frame exhibits a pixel change below a predetermined threshold, encode the video full frame into the second bitstream with the second resolution in order to refresh the video full frame for display at the video sink device with the second resolution.

[0076] Example 34. Component according to one of Examples 32-33, wherein the instructions further cause the processing means to: decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on a quantity of available wireless transmission bandwidth.

[0077] Example 35. Component of one of Examples 32-34, wherein the instructions further cause the processing means to: Encode a position of the video subframe within the video full frame into the second bitstream.

[0078] Example 36. Component according to one of claims 32-35, wherein the instructions further cause the processing means to: encode an initial full video frame into the second bitstream with the second resolution.

[0079] Example 37. Component of a video sink device for wireless display sharing, comprising: processing circuitry; and a non-volatile, computer-readable storage medium comprising instructions which, when executed by the processing means, cause the processing means to: dynamically switch between decoding a first bitstream into a full video frame with a first resolution and a second bitstream into a partial video frame with a second resolution, wherein the first bitstream and the second bitstream were received from a video source device and the second resolution is higher than the first resolution; and upscaling the decoded first bitstream from the first resolution to the second resolution; and generating a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream.

[0080] Example 38. Component according to Example 37, wherein the instructions further cause the processor resources to: generate the composite video image with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer.

[0081] Although the foregoing was described in connection with an exemplary aspect, it is understood that the term "exemplary" is meant merely as an example rather than as the best or optimal. Accordingly, the disclosure is intended to cover alternatives, modifications, and equivalents that may be contained within the scope of protection of the disclosure.

[0082] Although specific aspects have been illustrated and described here, a person skilled in the art will understand that a variety of alternative and / or equivalent implementations can replace the specific aspects shown and described without altering the scope of protection of this disclosure. This application intends to cover all adaptations or variations of the specific aspects discussed herein. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 18 / 193,806

[0001]

Claims

[1] Video source device for wireless display sharing, comprising: an encoder designed to dynamically switch between encoding a full video frame into a first bitstream with a first resolution and a partial video frame into a second bitstream with a second resolution, the second resolution being higher than the first resolution; a processing circuit arrangement designed to decide between encoding the full video frame and encoding the video subframe based on an amount of available wireless transmission bandwidth, a number of pixels in a modified area of ​​the full video frame, spatial complexity of a modified area of ​​the full video frame, temporal complexity of a modified area of ​​the full video frame, or a category of area modification of the full video frame; and a transmitter designed to wirelessly transmit the first bitstream and the second bitstream to a video sink device. [2] Video source device according to claim 1, wherein during a period in which the video full frame has a pixel change below a predetermined threshold, the encoder is further configured to encode the video full frame into the second bitstream with the second resolution in order to refresh the video full frame for display on the video sink device with the second resolution. [3] Video source device according to one or more of claims 1-2, wherein the processing circuit arrangement is further configured as follows: Decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on the amount of available wireless transmission bandwidth. [4] Video source device according to one or more of claims 1-3, further comprising: a display engine designed to monitor the amount of available wireless transmission bandwidth. [5] Video source device according to one or more of claims 1-4, wherein the encoder is further configured to encode a position of the video partial frame within the video full frame into the second bitstream. [6] Video source device according to one or more of claims 1-5, wherein the encoder is further configured to encode an initial full video frame into the second bitstream with the second resolution. [7] Video source device according to one or more of claims 1-6, wherein the video source device is a device for virtual reality (VR) or artificial reality (AR). [8] Wireless display sharing system, comprising: the video source device according to one or more of claims 1-7; and a video sink device designed to receive the first bitstream and the second bitstream transmitted wirelessly, to decode the first bitstream, to upscale the decoded first bitstream from the first resolution to the second resolution, to decode the second bitstream, and to produce a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream. [9] Video sink device for wireless display sharing, comprising: a receiver designed to wirelessly receive a first bitstream and a second bitstream from a video source device, the second resolution being higher than the first resolution; a decoder designed to dynamically switch between decoding the first bitstream into a full video frame with a first resolution and the second bitstream into a partial video frame with a second resolution; and a scaler designed to upscale the decoded first bitstream from the first resolution to the second resolution; and a display designed to show a composite video image based on the decoded and upscaled first bitstream and the decoded second bitstream. [10] Video sink device according to claim 9, wherein the composite video image is generated with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer. [11] Video sink device according to one or more of claims 9-10, wherein the video source device is a device for virtual reality (VR) or artificial reality (AR). [12] Wireless display sharing system comprising: a video source device designed to dynamically switch between encoding a full video frame into a first bitstream with a first resolution and a partial video frame into a second bitstream with a second resolution, the second resolution being higher than the first resolution, and to wirelessly transmit the first bitstream and the second bitstream; and the video sinking device according to claim 9. [13] Component of a video source device for wireless display sharing, comprising: Processing circuit arrangement; and a non-volatile, computer-readable storage medium containing instructions which, when executed by the processing circuitry, cause the processing circuitry to do the following: dynamic switching between encoding a full video frame into a first bitstream with a first resolution and a partial video frame into a second bitstream with a second resolution, where the second resolution is higher than the first resolution; Deciding between encoding the full video frame and encoding the video subframe based on the amount of available wireless transmission bandwidth, the number of pixels in a modified area of ​​the full video frame, the spatial complexity of a modified area of ​​the full video frame, the temporal complexity of a modified area of ​​the full video frame, or an area-change category of the full video frame; and Causing a transmitter to wirelessly send the first bitstream and the second bitstream to a video sink device. [14] Component according to claim 13, wherein the instructions further cause the processing circuit to: During a period in which the video full frame exhibits a pixel change below a predetermined threshold, the video full frame is encoded into the second bitstream with the second resolution in order to refresh the video full frame for display on the video sink device with the second resolution. [15] Component according to one or more of claims 13-14, wherein the instructions further cause the processing circuit to: Decide between encoding the full video frame at the first resolution and encoding the partial video frame at the second resolution based on the amount of available wireless transmission bandwidth. [16] Component according to one or more of claims 13-15, wherein the instructions further cause the processing circuit to: Encoding the position of a video subframe within the video full frame into the second bitstream. [17] Component according to one or more of claims 13-16, wherein the instructions further cause the processing circuit to: Encoding an initial full-frame video into the second bitstream with the second resolution. [18] Component of a video sink device for wireless display sharing, comprising: Processing circuit arrangement; and a non-volatile, computer-readable storage medium containing instructions which, when executed by the processing circuitry, cause the processing circuitry to do the following: dynamic switching between decoding a first bitstream into a full video frame with a first resolution and a second bitstream into a partial video frame with a second resolution, wherein the first and second bitstreams were received from a video source device and the second resolution is higher than the first resolution; and Upscaling the decoded first bitstream from the first resolution to the second resolution; and Generating a composite video image for display based on the decoded and upscaled first bitstream and the decoded second bitstream. [19] Component according to claim 18, wherein the instructions further cause the processing circuit to: Generating the composite video image with the decoded and upscaled first bitstream as the base layer and the decoded second bitstream as a superposition on the base layer.

Images