Distributed screen projection method and system based on GPU rendering

Abstract

The invention discloses a distributed screen projection method and system based on GPU rendering, and the method comprises the steps: calling a window management interface and a graphic interface, and creating a virtual screen; setting one or more surface layers on the virtual screen, and drawing image data on the surface layers; and completing the synthesis of the image data on the surface layer by using a graphics processing unit (GPU), encoding the synthesized image data through a multimedia encoding and decoding service, and sending the encoded image data to a receiving end in a distributed manner for screen projection display. According to the method, the virtual screen is created by calling the window management interface and the graphic interface, image data synthesis is completed by utilizing the GPU, efficient coding is performed through the multimedia coding and decoding service, and finally the image data are sent to the receiving end in a distributed manner for screen projection display, so that low-delay and high-resolution compatible distributed screen projection is realized.

Description

Technical Field

[0001] The present invention relates to the field of terminal technology, and in particular to a distributed screen projection method and system based on GPU rendering. Background Art

[0002] In distributed screen projection display technology, distributed screen projection display technology based on CPU rendering is widely used as a common solution. This technology aims to complete image rendering and realize distributed display through the CPU to enhance the user experience. However, this method has obvious limitations: First, since image rendering is completely dependent on the CPU, the picture delay after projection is more than 2 seconds, which poses a significant obstacle to application scenarios with high requirements for real-time and smoothness. Secondly, the existing technology only supports distributed screen projection with the same resolution and lacks support for devices with different resolutions. This means that in actual applications, when faced with display devices with different resolutions, the existing solutions cannot be effectively adapted, which limits its scope of application in diverse display environments.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a distributed screen projection method and system based on GPU rendering are provided, aiming to solve the technical problems in the prior art of too long screen projection delay and inability to support screen projection with different resolutions.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a distributed screen projection method based on GPU rendering, wherein the method comprises:

[0007] Call the window management interface and graphics interface to create a virtual screen;

[0008] Setting one or more surfaces on the virtual screen and drawing image data on the surfaces;

[0009] The graphic processing unit GPU is used to complete the synthesis of the image data on the surface layer, the synthesized image data is encoded through the multimedia encoding and decoding service, and the encoded image data is distributed and sent to the receiving end for screen projection display.

[0010] In one implementation, calling a window management interface and a graphics interface to create a virtual screen includes:

[0011] Call the window management interface to create the initial virtual screen;

[0012] Calling a graphic interface through the window management interface to configure the properties of the initial virtual screen to obtain a basic virtual screen;

[0013] The basic virtual screen is added to a screen group through the window management interface, and a display node for managing the state of the basic virtual screen is created for the basic virtual screen to obtain a virtual screen.

[0014] In one implementation, setting one or more surface layers on the virtual screen and drawing image data on the surface layers includes:

[0015] Calling a multimedia interface to set one or more surfaces on the virtual screen;

[0016] Using a multimedia module to obtain information about the surface layer through a window management interface and a graphic interface;

[0017] Based on the information, image data is rendered on the surface.

[0018] In one implementation, the step of synthesizing the image data on the surface layer by using a graphics processing unit (GPU) includes:

[0019] Traverse each surface layer and calculate synthesis parameters based on the image data on each surface layer;

[0020] Based on the synthesis parameters, a graphics processing unit (GPU) is used to complete the synthesis of the image data on the surface layer.

[0021] In one implementation, traversing each surface layer and calculating synthesis parameters based on image data on each surface layer includes:

[0022] Get the parameter information of the virtual screen;

[0023] Traverse each surface layer, and obtain the size information of each image data based on the image data on each surface layer;

[0024] Calculate the scaling ratio and offset of each image data based on the size information and the parameter information;

[0025] Based on the scaling ratio and the offset, a synthesis parameter is obtained.

[0026] In one implementation, the step of synthesizing the image data on the surface layer based on the synthesis parameters using a graphics processing unit (GPU) includes:

[0027] Create a composite canvas;

[0028] Based on the scaling ratio, scaling processing is performed on each image data;

[0029] Based on the offset, performing offset processing on each image data;

[0030] Based on the scaling ratio and the offset, correspondingly adjusting rendering matrix parameters of each image data;

[0031] Based on the synthesis canvas and the processed image data, a graphics processing unit (GPU) is used to complete synthesis of the image data on the surface layer.

[0032] In one implementation, the step of completing the synthesis of the image data on the surface layer by using the graphics processing unit (GPU) based on the synthesis canvas and the processed image data includes:

[0033] Call the graphics processing unit GPU interface to refresh the rendering;

[0034] Call the nativewindow interface inside the graphics processing unit GPU to apply for the intermediate buffer and add a timestamp;

[0035] Calling the graphics processing unit GPU to render each processed image data into an intermediate buffer;

[0036] The image data in the intermediate buffer is drawn onto the synthesis canvas through the graphics processing unit GPU to complete the synthesis of the image data.

[0037] In a second aspect, an embodiment of the present invention further provides a distributed screen projection system based on GPU rendering, wherein the system includes:

[0038] A virtual screen acquisition module is used to call the window management interface and the graphics interface to create a virtual screen;

[0039] A surface layer setting and image data drawing module, used for setting one or more surface layers on the virtual screen and drawing image data on the surface layers;

[0040] The image data synthesis and screen projection display module is used to use the graphics processing unit GPU to complete the synthesis of the image data on the surface, encode the synthesized image data through the multimedia encoding and decoding service, and distribute the encoded image data to the receiving end for screen projection display.

[0041] In the third aspect, an embodiment of the present invention also provides a terminal, wherein the terminal includes a memory, a processor, and a distributed screen projection program based on GPU rendering stored in the memory and runnable on the processor, and when the processor executes the distributed screen projection program based on GPU rendering, the steps of the distributed screen projection method based on GPU rendering of any one of the above-mentioned schemes are implemented.

[0042] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a distributed screen projection program based on GPU rendering is stored on the computer-readable storage medium, and when the distributed screen projection program based on GPU rendering is executed by a processor, the steps of the distributed screen projection method based on GPU rendering described in any one of the above-mentioned schemes are implemented.

[0043] Beneficial effects: The present invention provides a distributed screen projection method based on GPU rendering. Compared with the prior art, the present invention first calls the window management interface and the graphics interface to create a virtual screen, and constructs an additional display space outside the physical screen, providing an independent and flexible operating environment for subsequent data processing. Next, one or more surfaces are set on the virtual screen, and image data is drawn on the surface to ensure that the image data is organized more orderly and facilitate subsequent processing operations. Then, the graphics processing unit GPU is used to complete the synthesis of the image data on the surface, improve the speed of image processing, and adapt to distributed screen projection of different resolutions; finally, the synthesized image data is encoded through the multimedia encoding and decoding service, and the encoded image data is distributed and sent to the receiving end for screen projection display. The present invention realizes a low-latency and high-resolution compatible screen projection function by integrating virtual screen creation, image data drawing and synthesis, as well as encoding and decoding and distributed transmission technology, improves image quality and transmission efficiency, and ensures that users can obtain a smooth and real-time visual experience when performing remote display. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a specific implementation of a distributed screen projection method based on GPU rendering provided in an embodiment of the present invention.

[0045] Figure 2 A flowchart of obtaining a virtual screen in a distributed screen projection method based on GPU rendering provided in an embodiment of the present invention.

[0046] Figure 3 A flowchart of drawing image data in a distributed screen projection method based on GPU rendering provided in an embodiment of the present invention.

[0047] Figure 4 A flowchart of screen projection display in a distributed screen projection method based on GPU rendering provided in an embodiment of the present invention.

[0048] Figure 5 A functional block diagram of a distributed screen projection system based on GPU rendering provided in an embodiment of the present invention.

[0049] Figure 6 This is a block diagram of the internal structure principle of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] In the field of distributed screen projection display technology, the CPU rendering-based method aims to realize the remote display of screen content through centralized processing of computing resources, so as to simplify the collaborative work between multiple screens and enhance the effects of multimedia presentation and remote collaboration. However, this method has certain limitations, mainly manifested in the long picture delay caused by using the CPU for image rendering. Usually, the delay of the picture after projection will be more than 2 seconds, which is an obvious shortcoming for applications with high real-time requirements. In addition, the current technical solutions can often only support the projection requirements of fixed resolutions, and lack the ability to flexibly adapt between screens with different resolutions, which limits the widespread application of this technology on a variety of terminal devices.

[0052] In order to solve the above problems, this embodiment provides a distributed screen projection method based on GPU rendering. During specific implementation, this embodiment first calls the window management interface and the graphics interface to create a virtual screen, and constructs an additional display space outside the physical screen, providing an independent and flexible operating environment for subsequent data processing. Then, one or more surfaces are set on the virtual screen, and image data is drawn on the surface to ensure that the organization of the image data is more orderly and convenient for subsequent processing operations. Then, the graphics processing unit GPU is used to complete the synthesis of the image data on the surface, improve the speed of image processing, and adapt to distributed screen projection of different resolutions; finally, the synthesized image data is encoded through the multimedia encoding and decoding service, and the encoded image data is distributed and sent to the receiving end for screen projection display. The present invention realizes a low-latency and high-resolution compatible screen projection function by integrating virtual screen creation, image data drawing and synthesis, as well as encoding and decoding and distributed transmission technology, improves image quality and transmission efficiency, and ensures that users can obtain a smooth and real-time visual experience when performing remote display.

[0053] For example, suppose a teacher is teaching online and wants to show the courseware content to remote students. At this time, the teacher can adopt the method proposed by the present invention. First, call the window management interface and the graphics interface to create a virtual screen as an additional display space. The virtual screen can be used to display the PPT or other teaching materials prepared by the teacher, thus avoiding interference with the main screen used by the teacher himself, and also providing more display possibilities for the classroom. Then, one or more surfaces are set on the created virtual screen, and image data that needs to be displayed to students, such as charts and formulas in the course, are drawn on the surface to ensure a clear presentation of the teaching content. Then, when all the image data is ready, the GPU is used to perform synthesis processing of the image data to quickly generate high-quality image output. Regardless of the resolution of the device used by the student, a good display effect can be guaranteed. This process improves the speed of image processing and also ensures compatibility between different devices. Finally, through the multimedia encoding and decoding service, the synthesized image data will be converted into a format suitable for network transmission, and distributedly sent to each student's terminal device through the Internet for screen projection display. Such encoding and decoding and transmission mechanisms ensure that the image data can be transmitted to every corner with low latency and high definition, allowing students to synchronously see the content presented by the teacher, thereby enhancing the interactivity and real-time nature of remote teaching. The method provided by this embodiment can not only improve image quality and transmission efficiency, but also enable teachers and students to obtain a smooth and real-time visual experience, thereby improving the quality and effect of teaching.

[0054] The distributed screen projection method based on GPU rendering provided in this embodiment can be applied to smart terminals, such as Figure 1 As shown in , the specific steps include:

[0055] Step S100: calling the window management interface and the graphics interface to create a virtual screen.

[0056] In this embodiment, first, the window management interface and the graphics interface are called to create a virtual screen, so that an additional display space can be established outside the physical display. The virtual screen can be used to carry customized visual content, such as teaching demonstration materials in an online classroom, thereby providing the necessary basic platform for subsequent image data drawing and distribution.

[0057] Specifically, Figure 2 As shown, step S100 includes the following steps:

[0058] Step S101, calling the window management interface to create an initial virtual screen;

[0059] Step S102, calling a graphic interface through the window management interface to configure the properties of the initial virtual screen to obtain a basic virtual screen;

[0060] Step S103: adding the basic virtual screen to a screen group through the window management interface, and creating a display node for managing the state of the basic virtual screen for the basic virtual screen, to obtain a virtual screen.

[0061] In one implementation, in order to obtain a virtual screen, first, a window management interface is called to create an initial virtual screen. For example, in a Linux environment, a virtual display device (such as Xvfb or Xephyr) can be created through the X Window System or Wayland protocol, which is the initial virtual screen. The initial virtual screen is like an actual screen, but the initial virtual screen is actually a display area simulated in the memory. The purpose of this step is to prepare a display framework for subsequent display operations. Next, the graphics interface is called through the window management interface to configure the properties of the initial virtual screen to obtain a basic virtual screen. The properties include but are not limited to resolution and color depth to meet specific application requirements. For example, the initial virtual screen can be set to a virtual screen with a resolution of 1920x1080 pixels and a color depth of 24 bits, thereby obtaining a basic virtual screen. Such a configuration can ensure high-quality image presentation. This process gives the virtual screen specific characteristics so that it can adapt to different usage scenarios. Then, the basic virtual screen is added to the screen group through the window management interface, so that the basic virtual screen can be managed and controlled. The method of the present invention is not only applicable to a single virtual screen, but can also be extended to multiple virtual screens. By adding the virtual screen to the screen group, space can be reserved for other virtual screens or physical screens that may be added, ensuring the scalability of the system. The screen group can more conveniently configure properties such as resolution, position and size to ensure the consistency and coordination of the displayed content. The management mechanism of the screen group improves the flexibility of the system. Whether it is adjusting the parameters of the current virtual screen or adding a new screen in the future, it can be processed through a unified management interface, avoiding the complexity of managing each screen separately. The design of the screen group also helps to optimize resource allocation. The allocation of GPU resources can be dynamically adjusted according to the number and type of members in the screen group to ensure that each frame of the image can be quickly rendered and transmitted, thereby improving the overall performance. Adding the virtual screen to the screen group can also simplify the code logic, reduce code branches, and improve the readability and maintainability of the code. In addition, a display node for managing the state of the basic virtual screen is created for the basic virtual screen. The display node is responsible for recording and updating the state information of the basic virtual screen, including but not limited to the currently displayed content, position adjustment status, and whether it is currently active, etc., and finally a virtual screen is obtained. This step ensures that the content of the virtual screen can be effectively managed and updated, ensuring the coherence and consistency of the user interface experience. This embodiment achieves the goal of dynamically expanding display capabilities from creating the initial virtual screen to configuring its properties, and then integrating it into the screen group and establishing corresponding display node management, enhancing user experience and application flexibility.

[0062] Step S200: setting one or more surface layers on the virtual screen, and drawing image data on the surface layers.

[0063] In this embodiment, after obtaining a virtual screen, one or more surfaces are first set on the virtual screen, and image data is drawn on the surface to prepare materials for subsequent image synthesis. The image data is stored in a buffer to ensure the continuity and integrity of the data. This step ensures that the image data can be correctly presented on the virtual screen according to the expected layout and attributes, preparing for the subsequent GPU image synthesis and encoding transmission.

[0064] Specifically, Figure 3 As shown, step S200 includes the following steps:

[0065] Step S201, calling a multimedia interface to set one or more surfaces on the virtual screen;

[0066] Step S202: using a multimedia module to obtain information of the surface layer through a window management interface and a graphic interface;

[0067] Step S203: drawing image data on the surface layer based on the information.

[0068] In one implementation, this embodiment first sets one or more surfaces on the previously created virtual screen through the multimedia interface. The surface is a transparent layer for displaying content, which can be moved, scaled, and rotated independently of other layers. The role of this step is to provide a clear target and space for subsequent image data drawing, ensuring that all drawing operations are performed within the specified logical area and will not affect the interface of other applications. For example, on an online education platform, a teacher may want to overlay some annotation text or drawings on top of a teaching video. At this time, one or more such surfaces can be created by calling the API provided by the multimedia interface. For different operating systems, one or more surfaces can be created using DirectX or OpenGL API on Windows systems, or using Vulkan API on Linux systems. Next, the multimedia module is used to obtain information of one or more surfaces through the window management interface and the graphics interface. In this process, it will further communicate with the underlying window management interface (such as User32.dll in Windows or X Window System in Linux) to obtain relevant attribute information of one or more surfaces that have been set, such as position, size, etc. For example, when a user opens one or more video playback windows on a virtual screen, it is necessary to know the exact coordinates and size of these windows in order to correctly adjust the display range of the video screen. These windows are surfaces. Then, based on the information of the one or more surfaces, image data is drawn on the one or more surfaces, and the drawn image data is stored in a buffer. The tools for drawing image data include but are not limited to Skia, OpenGL and DirectX. In this step, the actual image data will be drawn on the corresponding one or more surfaces according to the previously collected information. For example, if the user is watching a high-definition movie, each frame of the video will be loaded and drawn on the corresponding surface. The "drawing" mentioned here actually refers to the process of mapping the image data of each frame to the surface according to the correct pixel coordinates. This process depends on the information of the surface, including the position, size and transparency of the surface. After drawing is completed, the image data will be temporarily saved in one or more buffers (such as double buffering or triple buffering mechanism). The purpose of this is to ensure that all data is latest and complete before the GPU performs image synthesis. At the same time, it can effectively manage memory resources to avoid performance bottlenecks when image data is frequently updated. The buffer also plays a role in smoothing the frame rate. Even in the case of unstable network or processing delay, it can ensure that the image seen by the user will not flicker or freeze.

[0069] Step S300, using a graphics processing unit GPU to complete the synthesis of the image data on the surface, encoding the synthesized image data through a multimedia encoding and decoding service, and sending the encoded image data to a receiving end in a distributed manner for screen projection display.

[0070] In this embodiment, after obtaining the image data, first, the graphics processing unit GPU is used to complete the synthesis of the image data on the surface, ensuring that the data from different sources can be integrated into a complete picture; then, the synthesized image data is encoded through the multimedia encoding and decoding service to reduce the amount of data and optimize the transmission efficiency, and the encoded image data is distributed and sent to the receiving end for screen display, so as to achieve synchronous screen display on different devices, thereby achieving efficient and consistent visual presentation effects. This embodiment uses the powerful parallel computing power of the GPU to quickly complete the synthesis of image data, and then reduces the bandwidth required for data transmission through efficient encoding and decoding technology, thereby achieving distributed sending of the processed image data to multiple receiving ends for synchronous screen display, which not only improves the speed and quality of image processing, but also effectively ensures the consistency and real-time performance of multi-screen display, and enhances the user experience.

[0071] Specifically, Figure 4 As shown, step S300 includes the following steps:

[0072] Step S301, traverse each surface layer, and calculate synthesis parameters based on the image data on each surface layer;

[0073] Step S302: Based on the synthesis parameters, the image data on the surface layer is synthesized using a graphics processing unit (GPU);

[0074] Step S303: encode the synthesized image data through the multimedia encoding and decoding service, and distribute the encoded image data to the receiving end for screen projection display.

[0075] In one implementation, first, based on the created virtual screen, the parameter information of the virtual screen is obtained through the window management interface. The parameter information of the virtual screen includes but is not limited to the width, height, resolution and pixel format of the virtual screen. The parameter information of the virtual screen also includes the target size of the image data on each surface layer, that is, the target width and target height of the image data on each surface layer. The target width and target height of the image data on each surface layer are predetermined and can be set by the user, or automatically calculated by the system according to the layout requirements. Then, traverse each surface layer to obtain the size information of the image data on each surface layer, that is, the original width and original height of the image data on each surface layer, to ensure that the basic characteristics of each image data can be understood, and to lay the foundation for subsequent processing. Next, based on the parameter information and the size information, the scaling ratio and offset are calculated, the purpose of which is to ensure that the image can be correctly scaled to the target size and correctly positioned on the synthetic canvas. Horizontal scaling ratio = target width / original width, vertical scaling ratio = target height / original height. In addition, the appropriate offset needs to be calculated so that the image can be correctly centered in the target area. Horizontal offset = (target width - (original width * horizontal scaling ratio)) / 2, vertical offset = (target height - (original height * vertical scaling ratio)) / 2. For example, if the target width of the image data is 1024 pixels, the target height is 768 pixels, the original width is 1920 pixels, and the original height is 1080 pixels, then the horizontal scaling ratio is (1024 / 1920)≈0.53, which is about 53% of the original size, and the vertical scaling ratio is (768 / 1080)≈0.71, which is about 71% of the original size. In addition, it is necessary to calculate the appropriate offset so that the image can be displayed correctly on the synthesis canvas. The calculation of the offset can be determined by the following formula: the horizontal offset is (1024-1920*0.53) / 2=3.2 pixels, and the vertical offset is (768-1080*0.71) / 2=0.6 pixels. Based on the scaling ratio and offset calculated above, the synthesis parameters are obtained to guide how the surface is placed on the synthesis canvas. With the synthesis parameters, first, create a synthesis canvas for the final image presentation. The size of the synthesis canvas is usually determined by the sum of all image data or a predefined display area size. Next, each image data is scaled using the scaling factor to make the image size fit the target size. Then, the image data is positionally adjusted according to the offset to ensure that the image is correctly displayed on the composite canvas.Next, considering that the painting is done with the help of Skia or other tools, the rendering matrix parameters of each image data are adjusted using the scaling ratio and offset to ensure that the image looks natural and undistorted visually. The specific adjustment method is: the vertical translation value matrix[SkMatrix::kMTransY] in the matrix is ​​updated by the formula matrix[SkMatrix::kMTransY]=matrix[SkMatrix::kMTransY]*vertical scaling ratio+vertical offset. After that, the GPU interface is called to refresh the rendering, which means that the GPU starts processing the image data and is ready to display the image data. In order to effectively manage memory resources, the nativewindow interface inside the GPU is called to apply for the intermediate buffer with a timestamp to ensure that the image data is processed in the correct order, that each frame of the image has enough buffer space when rendering, and that the rendering time of each frame can be tracked, which is very important for applications with high real-time requirements. Next, the GPU is called to render each processed image data into the intermediate buffer. This step actually completes the preliminary rendering of the image. Next, the image data in the intermediate buffer is drawn onto the synthesis canvas through the GPU. This process combines each individual image into a complete picture and completes the synthesis of the image data. After synthesis, the virtual screen displays the synthesized image data. The virtual screen supports adjusting the resolution according to different receiving display devices to ensure that the image can be accurately presented on screens of different sizes and resolutions. Then, the synthesized image data is encoded through the multimedia codec service. This step is to reduce the size of the image data for easy transmission on the network. Finally, the encoded image data is distributed and sent to the receiving end for screen projection display, which means that multiple receiving devices can synchronously receive the same image data stream and display it on their respective screens, thereby achieving the effect of multi-screen synchronous display. This process not only supports the projection requirements of different resolutions, but also effectively shortens the data flow time, making the entire distributed virtual projection process more accurate and smooth, and achieving the technical effect of simplifying complex operations.

[0076] In summary, this embodiment first calls the window management interface and the graphics interface to create a virtual screen, builds an additional display space outside the physical screen, and provides an independent and flexible operating environment for subsequent data processing. Next, one or more surfaces are set on the virtual screen, and image data is drawn on the surface to ensure that the image data is organized more orderly and convenient for subsequent processing operations. Then, the graphics processing unit GPU is used to complete the synthesis of the image data on the surface, improve the speed of image processing, and adapt to distributed screen projection of different resolutions; finally, the synthesized image data is encoded through the multimedia encoding and decoding service, and the encoded image data is distributed and sent to the receiving end for screen projection display. The present invention realizes a low-latency and high-resolution compatible screen projection function by integrating virtual screen creation, image data drawing and synthesis, as well as encoding and decoding and distributed transmission technology, improves image quality and transmission efficiency, and ensures that users can obtain a smooth and real-time visual experience when performing remote display.

[0077] like Figure 5 As shown in , this embodiment also provides a distributed screen projection system based on GPU rendering, which includes: a virtual screen acquisition module 10, a surface setting and image data drawing module 20, and an image data synthesis and screen projection display module 30. Specifically, the virtual screen acquisition module 10 is used to call the window management interface and the graphics interface to create a virtual screen. The surface setting and image data drawing module 20 is used to set one or more surfaces on the virtual screen and draw image data on the surface. The image data synthesis and screen projection display module 30 is used to complete the synthesis of the image data on the surface using a graphics processing unit GPU, encode the synthesized image data through a multimedia encoding and decoding service, and send the encoded image data to the receiving end in a distributed manner for screen projection display.

[0078] In one implementation, the virtual screen acquisition module 10 includes:

[0079] An initial virtual screen acquisition unit, used for calling a window management interface to create an initial virtual screen;

[0080] A basic virtual screen acquisition unit, configured to call a graphic interface through the window management interface to configure the properties of the initial virtual screen to obtain a basic virtual screen;

[0081] The virtual screen acquisition unit is used to add the basic virtual screen to the screen group through the window management interface, and create a display node for the basic virtual screen for managing the state of the basic virtual screen to obtain a virtual screen.

[0082] In one implementation, the surface layer setting and image data drawing module 20 includes:

[0083] A surface layer setting unit, used for calling a multimedia interface to set one or more surface layers on the virtual screen;

[0084] A surface layer information acquisition unit, used to acquire the surface layer information through a window management interface and a graphic interface using a multimedia module;

[0085] The image data drawing and storage unit is used to draw image data on the surface layer based on the information.

[0086] In one implementation, the image data synthesis and screen projection display module 30 includes:

[0087] A synthesis parameter calculation unit, used for traversing each surface layer and calculating synthesis parameters based on image data on each surface layer;

[0088] An image data synthesis unit, configured to complete the synthesis of the image data on the surface layer by using a graphics processing unit (GPU) based on the synthesis parameters;

[0089] The encoding and screen projection display unit is used to encode the synthesized image data through multimedia encoding and decoding services, and distribute the encoded image data to the receiving end for screen projection display.

[0090] In one implementation, the synthesis parameter calculation unit includes:

[0091] A parameter information acquisition subunit, used to acquire parameter information of a virtual screen;

[0092] A size information acquisition subunit, used for traversing each surface layer, and acquiring size information of each image data based on the image data on each surface layer;

[0093] A scaling ratio and offset calculation subunit, used to calculate a scaling ratio and an offset of each image data based on the size information and the parameter information;

[0094] The synthesis parameter acquisition subunit is used to obtain the synthesis parameter based on the scaling ratio and the offset.

[0095] In one implementation, the image data synthesis unit includes:

[0096] A composite canvas creation subunit, used to create a composite canvas;

[0097] A scaling processing subunit, used for performing scaling processing on each image data based on the scaling ratio;

[0098] An offset processing subunit, used for performing offset processing on each image data based on the offset;

[0099] An adjusting subunit, configured to adjust the rendering matrix parameters of each image data accordingly based on the scaling ratio and the offset;

[0100] The synthesis subunit is used to complete the synthesis of the image data on the surface layer by using the graphics processing unit GPU based on the synthesis canvas and the processed image data.

[0101] In one implementation, the synthesis subunit includes:

[0102] A refresh rendering subunit is used to call a graphics processing unit GPU interface to refresh rendering;

[0103] The intermediate buffer application subunit is used to call the nativewindow interface inside the graphics processing unit GPU to apply for the intermediate buffer and add a timestamp;

[0104] An image data rendering subunit, used for calling a graphics processing unit (GPU) to render each processed image data into an intermediate buffer;

[0105] The image data synthesis subunit is used to draw the image data in the intermediate buffer onto the synthesis canvas through the graphics processing unit GPU, so as to complete the synthesis of the image data on the surface layer.

[0106] The working principles of each module in the voice command execution system of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0107] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 6 The terminal may include one or more processors 100 ( Figure 6 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a voice command execution program. When one or more processors 100 execute the computer program 102, the various steps in the voice command execution method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the voice command execution method embodiment can be implemented, which is not limited here.

[0108] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0109] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0110] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed screen projection method based on GPU rendering, characterized in that: The method comprises: Call the window management interface and graphics interface to create a virtual screen; Setting one or more surfaces on the virtual screen and drawing image data on the surfaces; The graphic processing unit GPU is used to complete the synthesis of the image data on the surface layer, the synthesized image data is encoded through the multimedia encoding and decoding service, and the encoded image data is distributed and sent to the receiving end for screen projection display.

2. The distributed screen projection method based on GPU rendering according to claim 1, characterized in that: The calling of the window management interface and the graphics interface to create a virtual screen includes: Call the window management interface to create the initial virtual screen; Calling a graphic interface through the window management interface to configure the properties of the initial virtual screen to obtain a basic virtual screen; The basic virtual screen is added to a screen group through the window management interface, and a display node for managing the state of the basic virtual screen is created for the basic virtual screen to obtain a virtual screen.

3. The distributed screen projection method based on GPU rendering according to claim 2 is characterized in that: The step of setting one or more surface layers on the virtual screen and drawing image data on the surface layers comprises: Calling a multimedia interface to set one or more surfaces on the virtual screen; Using a multimedia module to obtain information about the surface layer through a window management interface and a graphic interface; Based on the information, image data is rendered on the surface.

4. The distributed screen projection method based on GPU rendering according to claim 3 is characterized in that: The step of using a graphics processing unit (GPU) to synthesize the image data on the surface layer includes: Traverse each surface layer and calculate synthesis parameters based on the image data on each surface layer; Based on the synthesis parameters, a graphics processing unit (GPU) is used to complete the synthesis of the image data on the surface layer.

5. The distributed screen projection method based on GPU rendering according to claim 4 is characterized in that: The traversing each surface layer and calculating the synthesis parameters based on the image data on each surface layer includes: Get the parameter information of the virtual screen; Traverse each surface layer, and obtain the size information of each image data based on the image data on each surface layer; Based on the size information and the parameter information, calculating a scaling ratio and an offset of each image; Based on the scaling ratio and the offset, a synthesis parameter is obtained.

6. The distributed screen projection method based on GPU rendering according to claim 5, characterized in that: The step of synthesizing the image data on the surface layer by using a graphics processing unit (GPU) based on the synthesis parameters includes: Create a composite canvas; Based on the scaling ratio, scaling processing is performed on each image data; Based on the offset, performing offset processing on each image data; Based on the scaling ratio and the offset, correspondingly adjusting rendering matrix parameters of each image data; Based on the synthesis canvas and the processed image data, the synthesis of the image data on the surface layer is completed by using the graphics processing unit GPU.

7. The distributed screen projection method based on GPU rendering according to claim 6, characterized in that: The step of completing the synthesis of the image data on the surface layer by using the graphics processing unit (GPU) based on the synthesis canvas and the processed image data includes: Call the graphics processing unit GPU interface to refresh the rendering; Call the nativewindow interface inside the graphics processing unit GPU to apply for the intermediate buffer and add a timestamp; Calling the graphics processing unit GPU to render each processed image data into an intermediate buffer; The image data in the intermediate buffer is drawn onto the synthesis canvas through the graphics processing unit GPU, thereby completing the synthesis of the image data on the surface layer.

8. A distributed screen projection system based on GPU rendering, characterized in that: The system comprises: A virtual screen acquisition module is used to call the window management interface and the graphics interface to create a virtual screen; A surface layer setting and image data drawing module, used for setting one or more surface layers on the virtual screen and drawing image data on the surface layers; The image data synthesis and screen projection display module is used to use the graphics processing unit GPU to complete the synthesis of the image data on the surface, encode the synthesized image data through the multimedia encoding and decoding service, and distribute the encoded image data to the receiving end for screen projection display.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a distributed screen projection program based on GPU rendering stored in the memory and run on the processor. When the processor executes the distributed screen projection program based on GPU rendering, the steps of the distributed screen projection method based on GPU rendering as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a distributed screen projection program based on GPU rendering. When the distributed screen projection program based on GPU rendering is executed by the processor, the steps of the distributed screen projection method based on GPU rendering as described in any one of claims 1-7 are implemented.

Images