A display device, multi-application isolated transmission method, and remote receiving device

Abstract

The application discloses a display device, a multi-application isolation transmission method and a remote receiving device. The method creates a virtual screen and an image reader for at least one target application in response to a starting operation of the target application; redirects at least one layer of a current target application to a current virtual screen for each target application, synthesizes at least one layer of the current virtual screen, generates image data, and outputs the image data to the current image reader; in the case that the current image reader triggers a listening callback, converts the image data of the current virtual screen into a structured data stream according to a network device interface protocol, and sends the structured data stream to at least one remote receiving device; in the case that at least one candidate virtual screen matching the subscription demand of any remote receiving device contains the current virtual screen, receives the structured data stream of the current virtual screen through the current remote receiving device, and displays a picture. The method realizes parallel and independent transmission of multiple applications.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device, a multi-application isolation transmission method, and a remote receiving device. Background Technology

[0002] With the increasing prevalence of display devices in smart education, collaborative work, and other scenarios, screen content transmission is being used more and more. Screen content transmission primarily involves transmitting content from a display device to a remote receiving device for display. However, traditional methods often transmit content for the entire screen or a single application, lacking application isolation capabilities and unable to support parallel and independent transmission of multiple applications. Summary of the Invention

[0003] This application provides a display device, a multi-application isolated transmission method, and a remote receiving device to achieve parallel independent transmission of multiple applications.

[0004] In a first aspect, some embodiments provide a display device, including: a display, a communication device, and at least one controller. The display is configured to display content from a broadcast system or network and / or a user interface; the communication device is configured to establish a communication connection with at least one remote receiving device according to a communication protocol, the communication protocol including at least a network device interface protocol; and at least one controller is connected to the display and the communication device and configured to execute instructions to cause the display device to:

[0005] In response to a launch operation of at least one target application, a virtual screen and an image reader are created for each of the at least one target application.

[0006] For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen, generate image data of the current virtual screen, and output the image data to the current image reader corresponding to the current target application;

[0007] When the current image reader triggers a listener callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. If the current virtual screen is included in at least one candidate virtual screen whose subscription requirements match the current virtual screen, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream.

[0008] Secondly, some embodiments also provide a multi-application isolated transmission method applied to the display device provided in the first aspect, the display device including: a display, a communication device, and at least one controller, the method including:

[0009] In response to a launch operation of at least one target application, a virtual screen and an image reader are created for each of the at least one target application.

[0010] For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen, generate image data of the current virtual screen, and output the image data to the current image reader corresponding to the current target application;

[0011] When the current image reader triggers a listener callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. If the current virtual screen is included in at least one candidate virtual screen whose subscription requirements match the current virtual screen, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream.

[0012] Thirdly, some embodiments also provide a remote receiving device, including: a display module, a communication module, and at least one processor. The display module is configured to display content from a broadcast system or network and / or a user interface; the communication module is configured to establish a communication connection with the display device according to a communication protocol, the communication protocol including at least a network device interface protocol; and at least one processor is connected to the display module and the communication module, and is configured to execute instructions to cause the remote receiving device to:

[0013] Acquire subscription requests;

[0014] If at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the system receives the structured data stream of the current virtual screen sent by the display device, and displays the screen on the remote receiving device according to the structured data stream.

[0015] Specifically, the structured data stream of the current virtual screen is obtained by the display device converting the image data of the current virtual screen according to the network device interface protocol when the current image reader corresponding to the current target application triggers a listening callback; the image data of the current virtual screen is obtained by the display device redirecting at least one layer of the current target application to the current virtual screen and compositing at least one layer of the current virtual screen; the image data of the current virtual screen is used to output to the current image reader; the current virtual screen and the current image reader are created by the display device in response to the startup operations of multiple applications for the current target application.

[0016] Fourthly, some embodiments also provide a multi-application isolated transmission method applied to the remote receiving device provided in the third aspect. The remote receiving device includes a display module, a communication module, and at least one processor. The method includes:

[0017] Acquire subscription requests;

[0018] If at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the system receives the structured data stream of the current virtual screen sent by the display device, and displays the screen on the remote receiving device according to the structured data stream.

[0019] Specifically, the structured data stream of the current virtual screen is obtained by the display device converting the image data of the current virtual screen according to the network device interface protocol when the current image reader corresponding to the current target application triggers a listening callback; the image data of the current virtual screen is obtained by the display device redirecting at least one layer of the current target application to the current virtual screen and compositing at least one layer of the current virtual screen; the image data of the current virtual screen is used to output to the current image reader; the current virtual screen and the current image reader are created by the display device in response to the startup operations of multiple applications for the current target application.

[0020] Technical Effects: Some embodiments provide a display device, a multi-application isolated transmission method, and a remote receiving device. In response to the startup operation of at least one target application, a virtual screen and an image reader are created for each launched target application. For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen to generate image data of the current virtual screen. The image data is then output to the current image reader corresponding to the current target application. In this way, each target application has its own dedicated virtual screen and image reader, and the layers of each target application can be redirected to the virtual screen and composited into image data for the virtual screen. The image reader acts as the data output terminal of the virtual screen, thereby capturing image data of at least one virtual screen in parallel, achieving multi-application data isolation. The system provides an independent, crosstalk-free data source for subsequent image data transmission. When the current image reader triggers a listener callback, it converts the image data of the current virtual screen into a structured data stream according to the network device interface protocol and sends it to at least one remote receiving device. This allows the image data of at least one virtual screen to be transmitted to the remote receiving device in parallel and without interference, achieving isolated transmission of multi-application data. If the current virtual screen is included in at least one candidate virtual screen whose subscription request matches that of any remote receiving device, the system receives the structured data stream of the current virtual screen through the current remote receiving device and displays the image on the current remote receiving device based on the structured data stream. In this way, the remote receiving device can subscribe to and determine the structured data stream of at least one candidate virtual screen according to its needs, achieving parallel and directional transmission of multi-application data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 Application scenario diagrams of display devices provided in some embodiments of this application;

[0023] Figure 2 Application architecture diagrams of display devices provided in some embodiments of this application;

[0024] Figure 3 A flowchart illustrating a multi-application isolated transmission method provided in some embodiments of this application;

[0025] Figure 4 Application scenario diagrams of remote receiving devices provided in some embodiments of this application;

[0026] Figure 5 A flowchart illustrating a multi-application isolated transmission method provided in other embodiments of this application;

[0027] Figure 6 A schematic diagram illustrating a preset layout pattern provided in some embodiments of this application;

[0028] Figure 7 A schematic diagram illustrating the application data acquisition process provided in some embodiments of this application;

[0029] Figure 8 A schematic diagram illustrating the application data transmission process provided in some embodiments of this application;

[0030] Figure 9 A flowchart illustrating a multi-application isolated transmission method provided in some embodiments of this application;

[0031] Figure 10 This is a diagram showing the internal structure of a computer device in some embodiments of this application. Detailed Implementation

[0032] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0033] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0034] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0035] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0036] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0037] With the increasing prevalence of display devices in smart education, collaborative work, and other scenarios, screen content transmission is becoming more and more widely used. Screen content transmission primarily involves transmitting content from a display device to a remote receiving device for display. However, traditional methods often transmit content across the entire screen or a single application, lacking application isolation capabilities and unable to support parallel independent transmission across multiple applications. For example, using the Android native MediaProjection API for global screen content capture passively captures all applications displaying the screen content, failing to achieve isolated capture for single or multiple applications, and raising concerns about leaking private content from the main display device. Similarly, single-application mirroring capture methods based on virtual screens only support single-application content capture, unable to achieve parallel independent capture and transmission across multiple applications, and are prone to crosstalk with other application screens during the capture process.

[0038] Based on this, embodiments of this application propose a display device, a multi-application isolated transmission method, and a remote receiving device. The display device includes a display, a communication device, and at least one controller. The display is configured to display content from a broadcast system or network and / or a user interface; the communication device is configured to establish a communication connection with at least one remote receiving device according to a communication protocol, which includes at least a network device interface protocol; and at least one controller is connected to the display and the communication device and configured to execute instructions to cause the display device to perform the multi-application isolated transmission method. The multi-application isolated transmission method includes, in response to the launch operation of at least one target application, creating a virtual screen and an image reader for each launched target application; for each target application, redirecting at least one layer of the current target application to the current virtual screen corresponding to the current target application, compositing at least one layer of the current virtual screen to generate image data of the current virtual screen, and outputting the image data to the current image reader corresponding to the current target application. Thus, each target application has its own dedicated virtual screen and image reader, and each target application's layers can be redirected to the virtual screen and composed. For the image data of the virtual screen, the image reader acts as the data output terminal of the virtual screen, thereby capturing the image data of at least one virtual screen in parallel, realizing isolated acquisition of multi-application data, and providing an independent, crosstalk-free data source for subsequent image data transmission. When the current image reader triggers a listening callback, according to the network device interface protocol, the image data of the current virtual screen is converted into a structured data stream and sent to at least one remote receiving device. Thus, the image data of at least one virtual screen can be transmitted to the remote receiving device in parallel and without interference through the network, realizing isolated transmission of multi-application data. If the current virtual screen is included in at least one candidate virtual screen whose subscription demand matches that of any remote receiving device, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream. In this way, the remote receiving device can subscribe to the structured data stream of at least one candidate virtual screen as needed, realizing parallel and directional transmission of multi-application data.

[0039] In this application embodiment, "display device" generally refers to a device with screen display and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices. Optionally, the display device can be a touch-screen all-in-one machine.

[0040] like Figure 1This is an application scenario diagram of a display device provided in some embodiments of this application. The display device 100 communicates with at least one remote receiving device 200 via a network. The at least one remote receiving device 200 may include remote receiving device 1, remote receiving device 2, remote receiving device 3, etc. The display device 100 includes: a display, a communication device, and at least one controller. The display is configured to display content from a broadcast system or network and / or a user interface; the communication device is configured to establish a communication connection with at least one remote receiving device 200 according to a communication protocol, which includes at least a network device interface protocol; and at least one controller, connected to the display and the communication device, is configured to execute instructions to cause the display device 100 to: in response to the launch operation of at least one target application, create a virtual screen and an image reader for each target application; and for each target application, redirect at least one layer of the current target application to the current virtual screen corresponding to the current target application, so as to... At least one layer of the screen is composited to generate the image data of the current virtual screen, and the image data is output to the current image reader corresponding to the current target application. When the current image reader triggers a listening callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device 200. If the current virtual screen is included in at least one candidate virtual screen whose subscription request matches the current virtual screen of any remote receiving device 200, the structured data stream of the current virtual screen is received by the current remote receiving device 200, and the screen is displayed on the current remote receiving device 200 according to the structured data stream.

[0041] like Figure 2This is an application architecture diagram of a display device provided in some embodiments of this application. The application architecture of the display device 100 includes a system layer, a multi-application isolated virtual screen layer, an application rendering layer, a frame acquisition layer, a parallel scheduling layer, and a structured data stream transmission layer. The system layer includes a system startup module and a permission control module. The system startup module is used to respond to system startup operations, and the permission control module is used to set application whitelists. The multi-application isolated virtual screen layer includes multiple virtual screen modules, such as virtual screen 1, virtual screen 2, virtual screen 3, etc. The multiple virtual screens have independent identifiers (such as IDs), resolutions, and memory. The application rendering layer is used to render multiple authorized applications, such as authorized application 1, authorized application 2, and authorized application 3. The frame acquisition layer includes multiple independent frame acquisition modules. Each frame acquisition module includes an image reader and a frame buffer. For example, frame acquisition module 1 includes an image reader 1 and a frame buffer 1, frame acquisition module 2 includes an image reader 2 and a frame buffer 2, and frame acquisition module 3 includes an image reader 3 and a frame buffer 3. The parallel scheduling layer includes a multi-queue acquisition scheduling module and a thread pool asynchronous scheduling module. The multi-queue acquisition scheduling module is used to acquire image data from multiple frame acquisition modules in parallel, while the thread pool asynchronous scheduling module is used to asynchronously schedule image data from multiple virtual screens using multiple threads. The structured data stream transmission layer is used to transmit the structured data streams corresponding to the image data from multiple virtual screens to the remote receiving device through the Network Device Interface (NDI). The remote receiving device can receive the required structured data streams according to subscription requirements.

[0042] In this way, the system layer responds to the startup operation of at least one target application. The multi-application isolation virtual screen layer creates virtual screens for each of the at least one target application. For each target application, the application rendering layer redirects at least one layer of the current target application to the corresponding current virtual screen to composite at least one layer of the current virtual screen, generating image data for the current virtual screen. The frame acquisition layer creates image readers for each of the at least one target application, outputs the image data of the current virtual screen to the corresponding current image reader, and stores it in the corresponding frame buffer. When the current image reader triggers a listener callback, the parallel scheduling layer converts the image data of the current virtual screen into a structured data stream according to the network device interface protocol. The structured data stream transmission layer sends the structured data stream of the current virtual screen to at least one remote receiving device. If at least one candidate virtual screen matching the subscription request of any remote receiving device contains the current virtual screen, the structured data stream of the current virtual screen is received by the current remote receiving device, and the image is displayed on the current remote receiving device based on the structured data stream.

[0043] In one exemplary embodiment, such as Figure 3 As shown, a multi-application isolated transmission method is provided, which is applied to... Figure 1Taking the display device 100 as an example, the explanation includes the following steps 302 to 306. Wherein:

[0044] Step 302: In response to the launch operation of at least one target application, a virtual screen and an image reader are created for each of the at least one target application.

[0045] The display device contains multiple applications. At least one of these applications is launched, and multi-application isolation transmission is performed on the launched application. For example, the launch operation can be triggered by the target application's icon or automatically initiated through system interaction logic. For instance, an application launch listener is registered at the system layer. When the listener captures the launch signal of any target application, the target application transitions from a non-running state to a running state, or from a background state to a foreground state.

[0046] Optionally, for multiple launch applications, all or some applications can be selected as the target application based on requirements or application permissions. This can be done by selecting launch applications that meet preset requirements or by selecting launch applications that meet the required permissions. For example, the target application could be a whiteboard application, a meeting software, or a document editing software.

[0047] For example, the preset requirements indicate the applications specified by the user that require multi-application isolated transmission. For each launched application, if the currently launched application belongs to the user-specified applications that require multi-application isolated transmission, the currently launched application is selected as the target application. The preset requirements can take the form of a requirement list, requirement instructions, or an application whitelist.

[0048] For example, application permissions indicate whether an application has permission to isolate transport. For each launched application, if the currently launched application has permission to isolate transport, the currently launched application is selected as the target application.

[0049] A virtual display refers to a logical display output simulated by software and without a physical entity. A multi-application isolation virtual display layer creates an independent and dedicated virtual display for each target application. For example, the multi-application isolation virtual display layer calls DisplayManager.createVirtualDisplay() to create a virtual display i for target application i, denoted as VD_i. Each virtual display corresponding to a target application has a unique identifier (such as ID), resolution, and memory allocation.

[0050] Optionally, the resolution of the virtual screen corresponding to each target application is the same as the resolution of the target application, and the memory is greater than the memory size occupied by the target application.

[0051] Optionally, a public display flag (VIRTUAL_DISPLAY_FLAG_PUBLIC) and a secure display flag (VIRTUAL_DISPLAY_FLAG_SECURE) can be configured for each virtual screen, and automatic mirroring (AUTO_MIRROR) can be disabled for each virtual screen.

[0052] An image reader (ImageReader) refers to a processing module that receives and processes image data. The frame acquisition layer creates an independent and dedicated image reader for each target application and binds a frame buffer to each image reader. For example, the frame acquisition layer calls ImageReader.newInstance(width_i,height_i,RGBA_8888,n) to create an image reader i, denoted as IR_i, for target application i.

[0053] Optionally, an image reader is bound to a virtual screen one-to-one. The image reader receives the output of its corresponding virtual screen and stores the received image data in the image reader's frame buffer. For example, IR_i.getSurface() can be set as the output Surface of VD_i.

[0054] Optionally, the image reader operates in continuous capture mode, automatically acquiring each data frame refreshed in the virtual screen.

[0055] At least one target application can be launched simultaneously or not. By creating a virtual screen and image reader for each target application, the content of each target application is transmitted to the corresponding virtual screen and image reader, avoiding the capture of content from other applications and facilitating independent acquisition and transmission of each target application.

[0056] Step 304: For each target application, redirect at least one layer of the current target application to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen, generate image data of the current virtual screen, and output the image data to the current image reader corresponding to the current target application.

[0057] The current target application can be any one of at least one target application. Each target application is displayed on the screen by overlaying at least one layer. Each layer contains different application content, different position coordinates, etc. For example, at least one layer of a target application includes a video layer, a toolbar layer, a text layer, a button layer, etc.

[0058] Application layers are typically sent directly to the compositor on the main screen. This application proposes redirecting each layer of a target application to the virtual screen bound to that target application. All or some layers of each target application can be redirected to the virtual screen bound to that target application. For example, if a target layer containing core application content is identified in at least one layer of the current target application, that target layer is redirected to the current virtual screen bound to that target application. For instance, if the target application contains video playback content, the target layer can be a video layer.

[0059] After redirection, the current virtual screen has at least one layer. The application rendering layer uses a layer compositor to read the layer information of each layer, including the layer stacking order, opacity, position coordinates, etc., and performs stacking calculations based on the layer information to obtain complete image data.

[0060] Optionally, at least one layer of the current virtual screen is composited to generate image data of the current virtual screen, including: determining other layers of the current virtual screen, excluding the target layer, as reference layers; setting rendering disabled and / or animation disabled on the reference layers; setting animation disabled on the target layer; performing layer compositing on at least one of the target layer with animation disabled, the reference layer with rendering disabled, or the reference layer with animation disabled to obtain image data.

[0061] Layer compositing can be performed when any layer is detected, or it can be performed at preset intervals.

[0062] Through layer redirection and layer compositing, the application content of each target application is transmitted to its own dedicated virtual screen. Since each virtual screen is bound to the corresponding image reader of its target application, the image data of each virtual screen is transmitted directionally to the bound image reader. Specifically, once the image data of each frame of the current virtual screen is composited, the current frame image data is sent to the current image reader bound to the current virtual screen.

[0063] Step 306: When the current image reader triggers a listening callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. If the current virtual screen is included in at least one candidate virtual screen whose subscription requirements match the current virtual screen, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream.

[0064] Each image reader registers a listener callback, which can be a listener queue or a callback function. For example, an OnImageAvailableListener callback function can be registered for IR_i and bound to its dedicated capture thread. Using listener callbacks allows for timely responses to new frames, saving system resources.

[0065] When the current image reader triggers a listener callback, it indicates that the image reader has received a new image data frame synthesized from the virtual screen. Using the Network Device Interface (NDI) protocol, the image data of the current virtual screen is encapsulated into a structured data stream, which is then transmitted to the remote receiving device via the NDI network.

[0066] Each frame of the structured data stream contains the identification information of the corresponding virtual screen.

[0067] Optionally, before converting the structured data stream, the method further includes: storing and converting the image data of the current virtual screen to obtain the converted image data; and correspondingly, converting the converted image data of the current virtual screen into a structured data stream according to the network device interface protocol.

[0068] The process of storing the image data for the current virtual screen includes: extracting pixel data (ByteBuffer_i) from the current virtual screen's image data; copying the pixel data (ByteBuffer_i) line by line to the first buffer (BackBuffer_i) according to a preset row step size (removing end-of-line padding); and releasing the buffer slots by calling Image_i.close(). Color conversion is then performed on the pixel data in the first buffer (BackBuffer_i) to obtain the converted image data. For example, a YUV420 to BGRA conversion formula can be used. Alternatively, an RGBA to BGRA channel rearrangement formula (i.e., swapping the R and B channels) can be used. The converted image data can then be written to the second buffer (FrontBuffer_i), and a pointer swap (bufferswap) is performed.

[0069] The remote receiving device and the display device communicate via an NDI network. For example, in a teaching scenario, the display device is the main screen, and at least one remote receiving device is used by the students. In a conference scenario, the display device is the device of the conference facilitator, and at least one remote receiving device is the device of the conference participants.

[0070] Each remote receiving device can receive a structured data stream of all or part of the virtual screens sent by the display device. For example, each remote receiving device can determine at least one candidate virtual screen based on subscription requirements. The subscription requirements indicate the application screen that each remote receiving device needs to display. For instance, the subscription requirements indicate the application name of the subscribed application, and at least one candidate virtual screen matching the application name of the subscribed application is determined based on the binding relationship between the application and the virtual screen.

[0071] In this way, when the current virtual screen is included in at least one candidate virtual screen, the remote receiving device receives the structured data stream of the current virtual screen, and then displays the image in the display module of the remote receiving device according to the structured data stream of the current virtual screen.

[0072] For example, a remote receiving device receives a structured data stream from the current virtual screen, parses the structured data stream to obtain image data and metadata. After encoding, decoding, and rendering the image data, it is sent to the display module for screen display. In this way, the application content of multiple target applications is transmitted in parallel to at least one remote receiving device through the virtual screen and image reader. Each remote receiving device can receive the application content of multiple target applications according to subscription requirements, realizing independent parallel transmission and distribution of multiple applications. This solves the common industry problems of crosstalk and transmission lag caused by multiple applications, and is suitable for scenarios such as smart classrooms and enterprise offices where high transmission accuracy and screen stability are required.

[0073] In the aforementioned multi-application isolated transmission method, in response to the startup operation of at least one target application, a virtual screen and an image reader are created for each launched target application. For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen to generate image data of the current virtual screen. The image data is then output to the current image reader corresponding to the current target application. In this way, each target application has its own dedicated virtual screen and image reader, and the layers of each target application can be redirected to the virtual screen and composited into image data for the virtual screen. The image reader serves as the data output terminal of the virtual screen, thereby capturing image data of at least one virtual screen in parallel, realizing isolated acquisition of multi-application data, and preparing subsequent image data. The transmission provides an independent, crosstalk-free data source. When the current image reader triggers a listener callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. Thus, the image data of at least one virtual screen can be transmitted to the remote receiving device in parallel and without interference, achieving isolated transmission of multi-application data. If the current virtual screen is included in at least one candidate virtual screen whose subscription request matches that of any remote receiving device, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream. In this way, the remote receiving device can subscribe to the structured data stream of at least one candidate virtual screen as needed, achieving parallel directional transmission of multi-application data.

[0074] In an exemplary embodiment, sending a structured data stream to at least one remote receiving device includes: performing block comparison between the image data corresponding to the current virtual screen and a reference image frame earlier than the current time to obtain an image difference rate; determining the transmission frame rate of the structured data stream based on the image difference rate and a preset difference rate range; and sending the structured data stream to at least one remote receiving device based on the transmission frame rate.

[0075] To reduce network data transmission pressure and ensure stable image display, the transmission frame rate can be determined based on the difference between the image data of the current frame and a reference frame earlier than the current time. The structured data stream can then be sent to at least one remote receiving device based on the transmission frame rate.

[0076] The difference between the image data of the current frame and a reference frame earlier than the current time can be represented by the image difference rate. For example, the image difference rate is obtained by comparing the image data corresponding to the current virtual screen with a reference image frame earlier than the current time through block comparison. For example, using a preset block division rule, the image data corresponding to the current virtual screen and the reference image frame are divided into multiple blocks respectively, and the image difference rate is determined based on the differences between corresponding blocks.

[0077] The transmission frame rate of the structured data stream is determined based on the relationship between the image difference rate and a preset difference rate range. For example, if the image difference rate is greater than the maximum value of the preset difference rate range (e.g., 5%), the transmission frame rate is determined to be the first frame rate. For example, if the first frame rate is the current preset frame rate, the structured data stream is transmitted to at least one remote receiving device according to the current preset frame rate.

[0078] If the image difference rate is within a preset difference rate range (e.g., 2%-5%), the transmission frame rate is determined to be a second frame rate lower than the first frame rate. For example, the second frame rate is half the current preset frame rate, and one frame of structured data stream is transmitted to at least one remote receiving device every other frame of structured data stream.

[0079] If the image difference rate is less than the minimum of a preset difference rate range (e.g., 2%), the transmission frame rate is determined to be a third frame rate lower than the second frame rate. For example, if the third frame rate is to transmit one frame every 30 frames, then one frame of structured data stream is transmitted to at least one remote receiving device every 30 frames of structured data stream.

[0080] The preset difference rate range can be flexibly adjusted according to actual needs.

[0081] In this embodiment, the image difference rate is determined by block comparison, and the transmission frame rate is dynamically adjusted. When the image changes little, the transmission frame rate can be automatically reduced to avoid transmitting a large number of duplicate frames and save network bandwidth. Conversely, when the image changes significantly, the transmission frame rate can be automatically increased to ensure stable image display and avoid image latency.

[0082] In an exemplary embodiment, the image data corresponding to the current virtual screen is compared with a reference image frame earlier than the current time to obtain the image difference rate. This includes: dividing the image data corresponding to the current virtual screen into multiple first blocks and dividing the reference image frame into multiple second blocks; the multiple first blocks and the multiple second blocks are in one-to-one correspondence; determining the changed blocks in the multiple first blocks based on the differences between the corresponding first blocks and second blocks; and determining the image difference rate based on the number of changed blocks.

[0083] The segmentation method can employ preset segmentation rules, such as 16×16 or 32×32. The image data corresponding to the current virtual screen and the reference image frame use the same preset segmentation rules to obtain multiple first blocks and multiple second blocks, thus creating a one-to-one correspondence between the multiple first blocks and the multiple second blocks.

[0084] Having a correspondence means being in corresponding positions. That is, if the position of any first block in the image data corresponding to the current virtual screen is consistent with the position of any second block in the reference image frame, then a correspondence is determined between the first block and the second block. Differences between multiple blocks can be processed in parallel.

[0085] The difference between the corresponding first and second blocks can be determined using image similarity calculation or channel absolute difference calculation. For example, if the channel absolute difference exceeds a preset difference, or the image similarity is less than a preset similarity, the corresponding first block is determined to be a changed block.

[0086] Image difference rate can be determined by the number of changed blocks. For example, the ratio between the number of changed blocks and the total number of the first block can be determined, and the image difference rate can be determined based on this ratio. For example, the ratio can be used as the image difference rate, or the ratio can be processed to obtain the image difference rate.

[0087] In this embodiment, by processing the image into blocks, each block can be used as a comparison object for differences. This allows the image difference rate to be updated only when significant changes occur within a given area, ensuring the stability of the transmission frame rate adjustment. The image difference rate is determined by the number of changed blocks, reflecting the actual degree of change in the image content, which aligns with human visual perception and achieves a balance between network bandwidth adjustment and image stability.

[0088] In an exemplary embodiment, a plurality of first pixels in each first block correspond one-to-one with a plurality of second pixels in a corresponding second block; based on the difference between the corresponding first blocks and second blocks, a change block is determined among the plurality of first blocks, including: taking the corresponding first blocks and second blocks as block groups to obtain a plurality of block groups; for each block group, determining the absolute channel difference between the corresponding first pixels and second pixels in the current block group; if the absolute channel difference of the current block group exceeds a preset difference, determining the first block in the current block group as a change block.

[0089] The calculation process for block differences is illustrated using the channel absolute difference calculation method as an example. Each block is grouped, and each block group includes a corresponding first block and a second block. For example, the (i,j)th first block of the image data corresponding to the previous virtual screen and the (i,j)th second block of the reference image frame form the (i,j)th block group. The number of block groups is the same as the number of first blocks.

[0090] The first pixel refers to the pixel in the first block, and the second pixel refers to the pixel in the second block. Since the first and second blocks are divided using the same rules, there is a positional correspondence between each first pixel in the first block and each second pixel in the corresponding second block. By treating the corresponding first and second pixels in each block group as pixel pairs, multiple pixel pairs are obtained.

[0091] Calculating the absolute channel difference for multiple block groups separately refers to calculating the absolute channel difference for each pixel pair in the current block group. For example, for any pixel pair, the absolute difference of the R channel (|R1-R2|), the absolute difference of the G channel (|G1-G2|), and the absolute difference of the B channel (|B1-B2|) are determined separately, resulting in a set of absolute differences. Here, R1, G1, and B1 represent the RGB channels of the first pixel, and R2, G2, and B2 represent the RGB channels of the second pixel.

[0092] In some embodiments, the channel absolute difference of the current block group is determined based on the absolute differences between multiple pixel pairs in the current block group. For example, any one of the mean, maximum, minimum, extreme, mode, median, etc., of the absolute differences between multiple pixel pairs in the current block group can be used as the channel absolute difference of the current block group.

[0093] If the absolute difference of a channel exceeds a preset difference, it indicates a significant difference between the first and second blocks in the corresponding block group. The first block whose absolute channel value exceeds the preset difference is identified as a changed block.

[0094] If the absolute difference of the channels in all block groups does not exceed the preset difference, it is determined that there are no changed blocks among the multiple first blocks.

[0095] In this embodiment, by calculating the absolute channel difference pixel by pixel for each block group, pixel-level differences can be captured; by determining the changed blocks through the absolute channel difference, the accuracy of the identification of changed blocks can be guaranteed.

[0096] In an exemplary embodiment, the image data corresponding to the current virtual screen is compared with a reference image frame earlier than the current time to obtain the image difference rate. This includes: identifying a target dirty region in the image data corresponding to the current virtual screen; the target dirty region is a layer change region marked during the compositing process of at least one layer of the current virtual screen; determining a reference dirty region in the reference image frame that is at the same position as the target dirty region; and comparing the target dirty region with the reference dirty region to obtain the image difference rate.

[0097] In this context, the target dirty region refers to the area of ​​layer change marked during the layer compositing process. When compositing at least one layer in a virtual screen, if content updates occur in all or part of an area of ​​any layer, such as video updates or text input, the layer compositor marks that changed area as the target dirty region.

[0098] Optionally, in the block matching process, the target dirty region in the image data corresponding to the current virtual screen can be identified first. If the target dirty region exists, block matching is performed based on the target dirty region to obtain the image difference rate.

[0099] In the absence of a target dirty region, the image data corresponding to the current virtual screen is compared with the reference image frame earlier than the current time to obtain the image difference rate.

[0100] The process of obtaining the image difference rate by comparing blocks based on the target dirty region includes: identifying a reference dirty region in a reference image frame that is at the same position as the target dirty region; comparing the target dirty region and the reference dirty region by blocks to obtain the image difference rate. Specifically, the target dirty region is divided into multiple first dirty blocks, and the reference image frame is divided into multiple second dirty blocks; the multiple first dirty blocks correspond one-to-one with the multiple second dirty blocks; based on the differences between the corresponding first dirty blocks and second dirty blocks, changing blocks are identified among the multiple first dirty blocks; and the image difference rate is determined based on the number of changing blocks.

[0101] In this embodiment, by identifying the target dirty areas marked during the layer compositing process, which are the areas where the layer content has changed as marked during the layer process, block comparison of the target dirty areas can improve the efficiency of determining the changed blocks and help to quickly and accurately identify the changed blocks.

[0102] In an exemplary embodiment, in response to the launch operation of at least one target application, creating a virtual screen and an image reader for each of the at least one target application includes: loading an application whitelist in response to the launch operation of at least one target application; determining that the current target application's permission verification is passed if the current target application matches any authorized application in the application whitelist; and creating a virtual screen and an image reader for each of the at least one target application whose permission verification is passed.

[0103] Authorized applications refer to pre-specified applications that are allowed to perform multi-application isolated transmission. The identification information of at least one authorized application can be stored in an application whitelist, which can be dynamically added to or deleted from according to configuration requirements.

[0104] At least one target application launched may not be an authorized application. Therefore, when launching at least one target application, an application whitelist is loaded during or after launch, and permissions for at least one target application are verified using the application whitelist. Optionally, during the display device startup phase or the target application initialization phase, a pre-configured application whitelist is read from a secure storage area.

[0105] Optionally, the identifier information of each target application is compared with the identifier information of each authorized application in the application whitelist. If the current target application matches any authorized application in the application whitelist, the current target application's permission verification is deemed successful. In this way, a virtual screen and an image reader are created for each target application that has passed at least one permission verification.

[0106] In this embodiment, a pre-configured application whitelist is used to perform permission verification on at least one target application that is launched, ensuring that the virtual screen and image reader creation process is executed only on the target application that passes the permission verification, thus ensuring the security of multi-application isolated transmission.

[0107] In one exemplary embodiment, such as Figure 4The remote receiving device 200 communicates with the display device 100. The remote receiving device 200 includes a display module, a communication module, and at least one processor. The display module is configured to display content from a broadcast system or network and / or a user interface; the communication module is configured to establish a communication connection with the display device 100 according to a communication protocol, which includes at least a network device interface protocol; and at least one processor, connected to the display module and the communication module, is configured to execute instructions to cause the remote receiving device 200 to: acquire subscription requests; and, if at least one candidate virtual screen matching the subscription request includes the current virtual screen corresponding to the current target application, receive a structured data stream of the current virtual screen sent by the display device 100, and process the structured data stream in the remote receiving device 200. The display process involves several steps: First, the structured data stream of the current virtual screen is obtained by the display device 100 converting the image data of the current virtual screen according to the network device interface protocol when the current image reader corresponding to the current target application triggers a listening callback. Second, the image data of the current virtual screen is obtained by the display device 100 redirecting at least one layer of the current target application to the current virtual screen and compositing at least one layer of the current virtual screen. Third, the image data of the current virtual screen is output to the current image reader. The current virtual screen and the current image reader are created by the display device 100 in response to the startup operations of multiple applications for the current target application. The display device 100 communicating with the remote receiving device 200 can be the same as the display device 100 described in the above embodiments. Therefore, the specific limitations of one or more display device 100 embodiments provided below can be found in the limitations of the display device 100 described above, and will not be repeated here.

[0108] In one exemplary embodiment, such as Figure 5 As shown, a multi-application isolated transmission method is provided, which is applied to... Figure 4 The remote receiving device 200 in the middle, the method includes:

[0109] Step 502: Obtain subscription requirements.

[0110] Step 504: If at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, receive the structured data stream of the current virtual screen sent by the display device, and display the screen on the remote receiving device according to the structured data stream; wherein, the structured data stream of the current virtual screen is obtained by the display device converting the image data of the current virtual screen according to the network device interface protocol when the current image reader corresponding to the current target application triggers the listening callback; the image data of the current virtual screen is obtained by the display device redirecting at least one layer of the current target application to the current virtual screen and compositing at least one layer of the current virtual screen; the image data of the current virtual screen is used to output to the current image reader; the current virtual screen and the current image reader are created by the display device in response to the startup operation of multiple applications for the current target application.

[0111] Some embodiments provide a remote receiving device that, when at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, receives a structured data stream of the current virtual screen sent by a display device, and displays the image on the remote receiving device according to the structured data stream. In this way, the remote receiving device can subscribe to the structured data stream of at least one candidate virtual screen according to demand, realizing parallel directional transmission of multi-application data. Each target application creates a dedicated virtual screen and an image reader. The layers of each target application can be redirected to the virtual screen and synthesized into image data of the virtual screen. The image reader serves as the data output terminal of the virtual screen, thereby capturing the image data of at least one virtual screen in parallel, realizing isolated acquisition of multi-application data, and providing an independent, crosstalk-free data source for image data transmission. The image data of at least one virtual screen can be transmitted to the remote receiving device in parallel and without interference through the network, realizing isolated transmission of multi-application data.

[0112] In an exemplary embodiment, when at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, receiving the structured data stream of the current virtual screen sent by the display device includes: creating at least one receiving thread corresponding to each candidate virtual screen. When at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the structured data stream of the current virtual screen sent by the display device is received through the receiving thread corresponding to the current virtual screen.

[0113] Specifically, a multi-threaded mechanism is used to receive the structured data streams of different candidate virtual screens. The remote receiving device has at least one receiving thread, and each receiving thread corresponds one-to-one with at least one candidate virtual screen. Each receiving thread is used to receive the structured data stream of its corresponding candidate virtual screen.

[0114] Specifically, if at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the structured data stream of the current virtual screen sent by the display device is received through the receiving thread corresponding to the current virtual screen.

[0115] In some embodiments, the remote receiving device broadcasts a query request within a local area network to collect structured data streams from multiple virtual screens sent by the display device. Identification information for all scanned virtual screens is stored in a local list.

[0116] The identification information of each virtual screen stored in the local list is matched with the identification information of at least one candidate virtual screen indicated by the subscription request. If the identification information of any virtual screen is included in the identification information of at least one candidate virtual screen indicated by the subscription request, the current virtual screen is determined to match the subscription request, and the structured data stream of the current virtual screen is received through the receiving thread corresponding to the current virtual screen.

[0117] Optionally, the remote receiving device can communicate with one or more display devices via a network, and the structured data of the multiple virtual screens received by the remote receiving device can come from one or more display devices.

[0118] Optionally, each receiving thread is also used to display the image based on the structured data stream of each candidate virtual screen. The received data can be sent from the receiving thread corresponding to the current virtual screen to the display module, which will then display the image.

[0119] In this embodiment, the remote receiving device can receive the structured data stream of the required virtual screens according to the subscription requirements, without needing to acquire the structured data streams of all scanned virtual screens. Using independent threads to process different data streams ensures that the reception and display of multiple streams do not interfere with each other.

[0120] In one exemplary embodiment, displaying a screen on a remote receiving device based on a structured data stream is configured to: determine the display area corresponding to each of at least one candidate virtual screen based on the number of candidate virtual screens and a preset layout mode; determine the target display area corresponding to the current virtual screen in each display area; and display the structured data stream of the current virtual screen in the target display area.

[0121] The preset layout mode refers to a pre-configured arrangement rule used to indicate the arrangement of the display areas corresponding to at least one virtual screen within the display module. For example, preset layout modes include full-screen mode, side-by-side mode, and grid mode. Figure 6 The diagram shown is a schematic diagram of a preset layout mode in some embodiments.

[0122] In some embodiments, there is a preset mapping relationship between the number of candidate virtual screens and the layout mode. The preset mapping relationship is queried to determine a preset layout mode that matches the number of at least one candidate virtual screen. The area size of the display module is obtained, including the area width and area height. Based on the area size of the display module and the preset layout mode, the display area corresponding to each of the at least one candidate virtual screen is determined within the display module.

[0123] For example, if there is one candidate virtual screen, the preset layout mode can be full-screen mode, which can use the full-screen area of ​​the display module as the display area of ​​the candidate virtual screen.

[0124] For example, if there are two candidate virtual screens, the preset layout mode can be the side-by-side mode, which will horizontally divide the full-screen area of ​​the display module into two side-by-side display areas, which will serve as the display areas for the two candidate virtual screens respectively.

[0125] For example, if there are more than two candidate virtual screens, the preset layout mode can be a grid mode. The optimal target grid row and column number (Cols, Rows) is determined based on the number of candidate virtual screens, K. Here, Cols = ceil(sqrt(K)), and Rows = ceil(K / Cols). For example, if K = 3 or 4, the target grid row and column number can be 2×2 (i.e., 2 rows and 2 columns). The full-screen area of ​​the display device is divided into multiple display areas matching the target grid row and column number, each serving as the display area corresponding to at least one candidate virtual screen.

[0126] Optionally, each display area includes a number, and the remote receiving end has a preset mapping table that records the mapping relationship between the identification information of the virtual screen and the number of the display area. The preset mapping table is queried to determine the target display area corresponding to the current virtual screen.

[0127] Optionally, the structured data stream of the current virtual screen is parsed to obtain image data and metadata. After encoding, decoding, and rendering the image data, it is sent to the display module and displayed in the target display area.

[0128] When a structured data stream of multiple virtual screens is received at the remote receiving end, the images of the multiple virtual screens can be displayed in multiple display areas of the display module.

[0129] Optionally, the structured data stream for each virtual screen is displayed at a proportional scale within its respective display area. The proportional scaling algorithm is as follows: AspectRatio = xres_j / yres_j.

[0130] If the virtual screen is wider, that is, AspectRatio>Region.w / Region.h, then Display_W=Region.w, and Display_H=Region.w / AspectRatio.

[0131] If the virtual screen is taller, that is, AspectRatio<Region.w / Region.h, then Display_H=Region.h, and Display_W=Region.h×AspectRatio.

[0132] If the display area is centered and offset, then Display_X=Region.x+(Region.w-Display_W) / 2.

[0133] Herein, xres_j and yres_j respectively denote the image width and image height corresponding to the structured data stream, AspectRatio denotes the image aspect ratio corresponding to the structured data stream, Region.w and Region.h respectively denote the region width and region height corresponding to the display area, Display_W and Display_H respectively denote the scaled image width and scaled image height, Display_X denotes the offset of the scaled image in the X-axis direction, and Region.x denotes the offset of the position of the display area in the X-axis direction.

[0134] Optionally, performing screen display of the structured data stream of the current virtual screen in the target display area includes: performing bilinear interpolation on the structured data stream of the current virtual screen to obtain an interpolated data stream, and performing screen display of the interpolated data stream in the target display area. Specifically, for each pixel in the structured data stream, weighted processing is performed on the pixel values of a plurality of surrounding pixels of the current pixel, and the weighted result is used as the interpolated pixel value of the current pixel. The interpolated data stream is determined according to the respective interpolated pixel values of the plurality of pixels.

[0135] Optionally, dynamic layout adjustment is performed on the structured data stream of at least one candidate virtual screen. For example, in the case where the reception of the structured data stream of any virtual screen is interrupted or restarted, the number of candidate virtual screens and the preset layout mode are updated, so as to determine the updated display area. For another example, in response to a two-finger zoom operation, Display_W / Display_H is scaled, the ratio between the secondarily scaled image width and the secondarily scaled image height is determined as the updated image aspect ratio, and screen display of the structured data stream is performed according to the updated image aspect ratio. For another example, display parameters are recalculated, including the image width and image height corresponding to the structured data stream, the region width and region height corresponding to the display area, and the like.

[0136] In this embodiment, based on the number of at least one candidate virtual screens and the preset layout mode, the display area corresponding to each of the at least one candidate virtual screens can be automatically determined, thereby achieving automatic adaptation of the display area; the screen of the corresponding virtual screen is displayed in each display area, thereby achieving parallel and independent display of multiple data streams and solving the problem of crosstalk between multiple application screens.

[0137] To illustrate the multi-application isolation transmission method in this solution in detail, the following is a detailed embodiment:

[0138] Some embodiments provide a display device including: a display, a communication device, and at least one controller. The display is configured to display content from a broadcast system or network and / or a user interface; the communication device is configured to establish a communication connection with at least one remote receiving device according to a communication protocol, the communication protocol including at least a network device interface protocol; and at least one controller is connected to the display and the communication device and configured to execute instructions to cause the display device to perform a multi-application isolated transmission method.

[0139] like Figure 7 The diagram shows a flowchart of application data collection in some embodiments. For the i-th target application (i=1,2,...,N) in the application whitelist, the following steps are performed:

[0140] S1, create a virtual screen VD_i and an image reader IR_i. Configure the flags VIRTUAL_DISPLAY_FLAG_PUBLIC + VIRTUAL_DISPLAY_FLAG_SECURE and disable AUTO_MIRROR.

[0141] Set IR_i.getSurface() to the output Surface of VD_i; register OnImageAvailableListener to listen for callbacks in IR_i and bind it to the dedicated capture thread.

[0142] S2, bind the i-th target application to VD_i, SurfaceFlinger performs independent layer composition for VD_i, and outputs it to the Surface of IR_i.

[0143] S3 uses a dedicated rendering optimization processor to perform the following optimization configurations on VD_i:

[0144] (a) Set window_animation_scale, transition_animation_scale, and animator_duration_scale to 0.0 to disable window animation;

[0145] (b) Set FLAG_SHOW_WALLPAPER=false to disable wallpaper rendering;

[0146] (c) The StatusBar and NavigationBar are not loaded on displayID_i, and the virtual screen only renders the core application content of the target application;

[0147] (d) Set the VD_i refresh rate to match the NDI target frame rate (e.g., 30Hz) to avoid wasting frames due to the default 60Hz;

[0148] (e) Configure the Surface usage flag of IR_i to USAGE_GPU_SAMPLED_IMAGE|USAGE_CPU_READ_OFTEN to achieve a GPU direct output zero-copy acquisition path.

[0149] like Figure 8 The diagram illustrates the data transmission process in some embodiments. To send the image data of the i-th virtual screen to the remote receiving device, the following steps are performed:

[0150] When the A1, IR_i callback is triggered, IR_i.acquireLatestImage() is called to obtain Image_i; the processor calls Image_i.getPlanes()[0].getBuffer() to extract pixel data ByteBuffer_i; the processor copies ByteBuffer_i line by line to the first buffer BackBuffer_i according to the preset line step size (removing the end padding); Image_i.close() is called to release the buffer slot.

[0151] A2, the data in BackBuffer_i undergoes color conversion: using the YUV420 to BGRA conversion formula, or RGBA to BGRA channel rearrangement (swapping the R and B channels); the conversion result is written to the second buffer FrontBuffer_i, and pointer swap is performed.

[0152] A3 compares the image data corresponding to the current virtual screen with a reference image frame earlier than the current time. For example, it divides the image data into 16×16 blocks, samples 4 pixels in each block, calculates the absolute difference of each channel, counts the number of changed blocks, and calculates the image difference rate (DiffRate). If there is a target dirty region (from SurfaceFlinger) in the image data corresponding to the current virtual screen, it only compares the blocks within the target dirty region.

[0153] For example, if the image difference rate is greater than the maximum value of a preset difference rate range (e.g., 5%), the transmission frame rate is determined to be the first frame rate. For example, if the first frame rate is the current preset frame rate, the structured data stream is transmitted to at least one remote receiving device according to the current preset frame rate.

[0154] If the image difference rate is within a preset difference rate range (e.g., 2%-5%), the transmission frame rate is determined to be a second frame rate lower than the first frame rate. For example, the second frame rate is half the current preset frame rate, and one frame of structured data stream is transmitted to at least one remote receiving device every other frame of structured data stream.

[0155] If the image difference rate is less than the minimum of a preset difference rate range (e.g., 2%), the transmission frame rate is determined to be a third frame rate lower than the second frame rate. For example, if the third frame rate is to transmit one frame every 30 frames, then one frame of structured data stream is transmitted to at least one remote receiving device every 30 frames of structured data stream.

[0156] The preset difference rate range can be flexibly adjusted according to actual needs.

[0157] A4, according to the NDI protocol, converts the image data of the current virtual screen into a structured data stream and sends it to at least one remote receiving device.

[0158] A5 sends XML metadata via NDIlib_send_send_metadata().

[0159] A6. Each target application maintains three independent threads: Thread_Capture_i for executing A1, Thread_Process_i for executing A2 and A3, and Thread_Send_i for executing A4 and A5. Threads can be connected via a lock-free circular queue (capacity 3 slots), using CAS atomic operations for synchronization. When the queue is full, the oldest frame is overwritten. Thread groups from different applications do not share resources and do not affect each other.

[0160] A7 periodically obtains the network bandwidth estimate BW_est of the remote receiving device through the NDI metadata channel: If the network bandwidth estimate is less than a first preset bandwidth (e.g., 50Mbps), reduce the resolution of VD_i to a first resolution (e.g., 1280×720). If the network bandwidth estimate is greater than or equal to the first preset bandwidth and less than a second preset bandwidth (i.e., 50-150Mbps), maintain the resolution of VD_i at a preset resolution (e.g., 1920×1080). If the network bandwidth estimate is greater than the second preset bandwidth (e.g., 150Mbps), increase the resolution of VD_i to a second resolution (e.g., 2560×1440). For example, VirtualDisplay.resize() can be called to adjust the resolution, rebuild the image reader, and rebind it.

[0161] Some embodiments also provide a remote receiving device, including: a display module, a communication module, and at least one processor. The display module is configured to display content from a broadcast system or network and / or a user interface; the communication module is configured to establish a communication connection with the display device according to a communication protocol, which includes at least a network device interface protocol; and at least one processor is connected to the display module and the communication module, and is configured to execute instructions to cause the remote receiving device to perform a multi-application isolated transmission method. For example... Figure 9 The diagram shown is a flowchart illustrating the application of isolated transmission methods in some embodiments.

[0162] B1 calls NDIlib_find_create_v2() to create a discovery instance, and periodically calls NDIlib_find_get_current_sources() to scan the data stream list, such as the NDI source list; the user selects K sources, and calls NDIlib_recv_create_v3() to create an independent receiving instance for each source.

[0163] B2. Each source creates an independent receiving thread Thread_Recv_j, which calls NDIlib_recv_capture_v3() in a loop to obtain video frames and metadata frames, writes the frame data to RecvBuffer_j, and parses the metadata and stores it in MetaCache_j.

[0164] B3, Display Layout Calculation. Obtain the local screen parameters Screen_W×Screen_H, and calculate the display area {Display_X_j, Display_Y_j, Display_W_j, Display_H_j} for each stream based on the K value (i.e., the number of data streams) and the preset layout mode:

[0165] B4 performs bilinear interpolation scaling on the frame data, uploads the texture via glTexImage2D, the rendering pipeline synthesizes K-way textures to the frame buffer, and calls eglSwapBuffers for display.

[0166] B5, Dynamic Layout Adjustment:

[0167] (a) Layout switching: Recalculate display parameters, take effect in the next frame (≤33ms delay);

[0168] (b) Zoom gesture: Two-finger zoom updates Display_W / H and locks the aspect ratio;

[0169] (c) Flow increase / decrease: Automatically adjust K value and layout when NDI source is disconnected / connected.

[0170] Some embodiments provide a display device, a multi-application isolated transmission method, and a remote receiving device. In response to the launch operation of at least one target application, a virtual screen and an image reader are created for each launched target application. For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application to composite at least one layer of the current virtual screen, generating image data for the current virtual screen. The image data is then output to the current image reader corresponding to the current target application. Thus, each target application has its own dedicated virtual screen and image reader, and each target application's layers can be redirected to the virtual screen and composited into image data for the virtual screen. The image reader serves as the data output terminal for the virtual screen, thereby capturing image data from at least one virtual screen in parallel, achieving isolated acquisition of multi-application data. The system provides an independent, crosstalk-free data source for subsequent image data transmission. When the current image reader triggers a listener callback, it converts the image data of the current virtual screen into a structured data stream according to the network device interface protocol and sends it to at least one remote receiving device. This allows the image data of at least one virtual screen to be transmitted to the remote receiving device in parallel and without interference, achieving isolated transmission of multi-application data. If at least one candidate virtual screen whose subscription request matches that of any remote receiving device includes the current virtual screen, the system receives the structured data stream of the current virtual screen through the current remote receiving device and displays the image on the current remote receiving device based on the structured data stream. In this way, the remote receiving device can subscribe to the structured data stream of at least one candidate virtual screen as needed, achieving parallel and directional transmission of multi-application data.

[0171] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0172] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a multi-application isolated transmission method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0173] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0174] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0175] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0176] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0179] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A display device, characterized in that, include: The display is configured to show content from a broadcast system or network and / or a user interface; A communication device configured to establish a communication connection with at least one remote receiving device according to a communication protocol, the communication protocol including at least a network device interface protocol; and at least one controller, which is connected to the display and the communication device, and is configured to execute instructions to cause the display device to: In response to a launch operation of at least one target application, a virtual screen and an image reader are created for each of the at least one target application. For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen to generate image data of the current virtual screen, and output the image data to the current image reader corresponding to the current target application; When the current image reader triggers a listening callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. If the current virtual screen is included in at least one candidate virtual screen whose subscription requirements match those of any remote receiving device, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream.

2. The display device according to claim 1, characterized in that, The controller is configured to send a structured data stream to at least one remote receiving device. The image data corresponding to the current virtual screen is compared with a reference image frame earlier than the current time to obtain the image difference rate. The transmission frame rate of the structured data stream is determined based on the image difference rate and a preset difference rate range. The structured data stream is sent to at least one remote receiving device according to the transmission frame rate.

3. The display device according to claim 2, characterized in that, The controller performs a block comparison between the image data corresponding to the current virtual screen and a reference image frame earlier than the current time to obtain the image difference rate, and is configured as follows: The image data corresponding to the current virtual screen is divided into multiple first blocks, and the reference image frame is divided into multiple second blocks; There is a one-to-one correspondence between multiple first blocks and multiple second blocks; Based on the differences between the corresponding first and second blocks, the changed blocks are identified among multiple first blocks; The image difference rate is determined based on the number of the changed blocks.

4. The display device according to claim 3, characterized in that, In each first block, multiple first pixels correspond one-to-one with multiple second pixels in the corresponding second block; the controller is configured to determine the changing block among the multiple first blocks based on the difference between the corresponding first and second blocks. By taking the first and second blocks that have a corresponding relationship as block groups, multiple block groups are obtained; For each block group, determine the absolute channel difference between the first and second pixels that have a corresponding relationship in the current block group; If the absolute difference of the channels in the current block group exceeds a preset difference, the first block in the current block group is determined to be a changed block.

5. The display device according to claim 2, characterized in that, The controller performs a block comparison between the image data corresponding to the current virtual screen and a reference image frame earlier than the current time to obtain the image difference rate, and is configured as follows: Identify the target dirty region in the image data corresponding to the current virtual screen; The target dirty area is the layer change area marked during the compositing process of at least one layer of the current virtual screen; In the reference image frame, a reference dirty region is determined that is at the same position as the target dirty region; The target dirty region is compared with the reference dirty region by block comparison to obtain the image difference rate.

6. The display device according to claim 1, characterized in that, The controller executes the launch operation in response to at least one target application, creating a virtual screen and an image reader for each target application, and is configured to: In response to the launch operation of at least one target application, load the application whitelist; For each target application, if the current target application matches any authorized application in the application whitelist, it is determined that the permission verification of the current target application has passed; Create a virtual screen and an image reader for at least one target application that has passed permission verification.

7. A multi-application isolated transmission method, characterized in that, Applied to the display device according to any one of claims 1 to 6, comprising: In response to a launch operation of at least one target application, a virtual screen and an image reader are created for each of the at least one target application. For each target application, at least one layer of the current target application is redirected to the current virtual screen corresponding to the current target application, so as to composite at least one layer of the current virtual screen to generate image data of the current virtual screen, and output the image data to the current image reader corresponding to the current target application; When the current image reader triggers a listening callback, the image data of the current virtual screen is converted into a structured data stream according to the network device interface protocol and sent to at least one remote receiving device. If the current virtual screen is included in at least one candidate virtual screen whose subscription requirements match those of any remote receiving device, the structured data stream of the current virtual screen is received by the current remote receiving device, and the screen is displayed on the current remote receiving device according to the structured data stream.

8. A remote receiving device, characterized in that, include: The display module is configured to display content from a broadcast system or network and / or a user interface; A communication module is configured to establish a communication connection with a display device according to a communication protocol, wherein the communication protocol includes at least a network device interface protocol. And at least one processor, which is connected to the display module and the communication module, and is configured to execute instructions to cause the remote receiving device to: Acquire subscription requests; If at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the structured data stream of the current virtual screen sent by the display device is received, and the screen is displayed on the remote receiving device according to the structured data stream; The structured data stream of the current virtual screen is obtained by the display device converting the image data of the current virtual screen according to the network device interface protocol when the current image reader corresponding to the current target application triggers a listening callback; the image data of the current virtual screen is obtained by the display device redirecting at least one layer of the current target application to the current virtual screen and compositing at least one layer of the current virtual screen; the image data of the current virtual screen is used to output to the current image reader. The current virtual screen and the current image reader are created by the display device in response to the launch operation of multiple applications, for the current target application.

9. The remote receiving device according to claim 8, characterized in that, When the processor executes a process that receives a structured data stream of the current virtual screen sent by the display device, provided that at least one candidate virtual screen matching the subscription request includes the current virtual screen corresponding to the current target application, the processor is configured to: Create at least one receiving thread corresponding to each candidate virtual screen; If at least one candidate virtual screen matching the subscription demand includes the current virtual screen corresponding to the current target application, the structured data stream of the current virtual screen sent by the display device is received through the receiving thread corresponding to the current virtual screen.

10. The remote receiving device according to claim 8, characterized in that, The processor is configured to perform screen display on the remote receiving device according to the structured data stream: Based on the number of at least one candidate virtual screens and the preset layout mode, determine the display area corresponding to each of the at least one candidate virtual screens; Determine the target display area corresponding to the current virtual screen in each display area; The structured data stream of the current virtual screen is displayed in the target display area.

Images