Picture display method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a picture display method and device, equipment, a storage medium and a program product, and belongs to the technical field of display. The method comprises the steps that at least one path of image data is acquired, the source of the at least one path of image data comprises at least one of local equipment and source end equipment, and the source end equipment is equipment for picture redirection; based on the at least one path of image data, performing picture rendering in a display window through a plurality of paths of rendering threads to obtain a rendered picture, the same display window supporting display of a plurality of paths of pictures; and displaying the rendered picture through a display screen corresponding to the display window. By adopting the unified multi-path picture display frame provided by the embodiment of the invention, multi-source image data can be rendered in the display window corresponding to the display screen through the unified multi-path rendering thread, and multi-path pictures are supported to be rendered in the same display window, so that support for various display scenes is realized.

Description

[0001] This application is a divisional application of the invention patent with application number 202210443092.5, application date April 25, 2022, and invention name “Screen display method, device, equipment, storage medium and program product”. Technical Field

[0002] The embodiments of the present application relate to the field of display technology, and in particular to a screen display method, device, equipment, storage medium and program product. Background Art

[0003] With the continuous development and update of screen display technology, people have an increasingly urgent need to display multiple images on the same screen, or to display the same image on multiple screens.

[0004] In the related art, multiple screen displays on the same screen or multiple screen displays on the same screen are each implemented on a device, but the same device can only implement one of the functions and cannot uniformly process multiple possible screen display formats. Summary of the invention

[0005] The embodiments of the present application provide a screen display method, device, equipment, storage medium and program product. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a screen display method, the method comprising:

[0007] Acquire at least one channel of image data, where the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection;

[0008] Based on the at least one channel of image data, performing picture rendering in a display window through multiple rendering threads to obtain a rendered picture, wherein the same display window supports displaying multiple channels of pictures;

[0009] The rendering picture is displayed through a display screen corresponding to the display window.

[0010] On the other hand, an embodiment of the present application provides a screen display device, the device comprising:

[0011] A data acquisition module, used for acquiring at least one channel of image data, wherein the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection;

[0012] A picture rendering module, used for performing picture rendering in a display window through multiple rendering threads based on the at least one channel of image data to obtain a rendered picture, wherein the same display window supports displaying multiple channels of pictures;

[0013] A display module is used to display the rendering image through a display screen corresponding to the display window.

[0014] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the screen display method described in the above aspect.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one program code is stored, and the program code is loaded and executed by a processor to implement the screen display method described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the screen display method provided in various optional implementations of the above aspects.

[0017] In an embodiment of the present application, a computer device obtains at least one channel of image data from at least one of a local device and a source device, and renders the image in a display window through a multi-channel rendering thread to obtain a rendered image, and finally displays the rendered image through a display screen corresponding to the display window, wherein the same display window supports the display of multiple channels of images. A unified multi-channel image display framework provided in an embodiment of the present application can be used to render image data from multiple sources (including local devices and redirected devices) in a display window corresponding to the display screen through a unified multi-channel rendering thread, and support the rendering of multiple channels of images in the same display window, thereby realizing support for display scenarios such as multi-screen same display (multiple display screens display the same image), multi-screen different display (multiple display screens display different images), and same-screen multiple display (the same display screen displays multiple different images). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;

[0020] Figure 2A flowchart of a screen display method provided by an exemplary embodiment of the present application is shown;

[0021] Figure 3 is a schematic diagram of a multi-channel video display method shown in an exemplary embodiment of the present application;

[0022] Figure 4 A flowchart of a screen display method provided by another exemplary embodiment of the present application is shown;

[0023] Figure 5 is a diagram of multi-channel rendering data flow and module interaction shown in an exemplary embodiment of the present application;

[0024] Figure 6 is a flowchart of a multi-pass rendering process shown in an exemplary embodiment of the present application;

[0025] Figure 7 is a schematic diagram of an implementation of a display window agent shown in an exemplary embodiment of the present application;

[0026] Figure 8 is a schematic diagram of implementing multi-screen different display shown in an exemplary embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of implementing multi-screen simultaneous display shown in an exemplary embodiment of the present application;

[0028] Fig.10 is a schematic diagram of implementing multiple displays on the same screen according to an exemplary embodiment of the present application;

[0029] Fig.11 A structural block diagram of a picture display device provided by an embodiment of the present application is shown;

[0030] Fig.12 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown. The implementation environment may include: at least one source device 110 and at least one destination device 120.

[0033] The source device 110 is an electronic device with a redirection function, which may be a mobile terminal such as a smart phone, a tablet computer, or a personal computer. Figure 1In the description, the source device 110 includes a tablet computer 111 and a smart phone 112 as an example.

[0034] The object with redirection requirements in the source device 110 is called a redirection object, which can be a screen mirror or a redirection application, and the redirection content of the redirection object can include at least one of video, audio, and control. The redirection application can be a navigation application, a video application, a game application, etc. The embodiment of the present application does not limit the specific type of the screen display application. Optionally, when the redirection object is an application, the application can be in the foreground running state or the background running state during the redirection process.

[0035] In an embodiment of the present application, the same source device can redirect at least two redirection objects at the same time. For example, screen mirroring and video applications can be displayed at the same time, or video applications and navigation applications can be displayed at the same time. Among them, the number of redirection objects that the source device supports to redirect simultaneously has an upper limit, and the upper limit is related to the processing performance of the source device.

[0036] The destination device 120 is an electronic device used to display the redirection results, which may be a smart phone, a tablet computer, a personal computer, a smart TV, or a vehicle-mounted terminal (such as a vehicle computer), etc. Figure 1 In the example, the destination device 120 includes a first vehicle-mounted terminal 121 and a second vehicle-mounted terminal 122, wherein the first vehicle-mounted terminal 121 and the second vehicle-mounted terminal 122 are arranged in the same vehicle, and the first vehicle-mounted terminal 121 and the second vehicle-mounted terminal 122 are respectively controlled by different SoCs (System on Chip). For example, the first vehicle-mounted terminal 121 is an on-board terminal arranged in the front row of the vehicle, and the second vehicle-mounted terminal 122 is an on-board terminal arranged in the back row of the vehicle.

[0037] Optionally, the destination device 120 has at least one screen. Figure 1 As shown, the first vehicle terminal 121 has a first screen 1211 and a second screen 1212 (both controlled by the first SoC 1213 of the first vehicle terminal 121), and the second vehicle terminal 122 has a third screen 1221 and a fourth screen 1222 (both controlled by the second SoC 1223 of the second vehicle terminal 122).

[0038] In some embodiments, when the destination device 120 has at least two screens, at least two screens can display different screen contents at the same time, or the same screen content, and the screen content displayed on each screen can be a single screen object or multiple screen objects (displaying multiple screen objects in split screen).

[0039] It should be noted that, in addition to displaying the screen redirected by the source device, the destination device can also display the application screen of the local application, or local or network video, or image captured by the local camera, that is, the screen source of the destination device can include at least one of the source device and the local device.

[0040] In the embodiment of the present application, the multi-channel image display is embodied in multiple screens with the same display (multiple display screens display the same image), multiple screens with different displays (multiple display screens display different images), and multiple displays on the same screen (the same display screen displays multiple different images). In a possible multi-channel image display scenario, such as Figure 1 As shown, the video application screen in the tablet computer 111 is displayed on the first screen 1211 of the first vehicle-mounted terminal 121; the first game application screen is displayed on the second screen 1212 of the first vehicle-mounted terminal 121. The local camera screen is displayed on the first screen 1211 of the first vehicle-mounted terminal 121, and the local application screen is displayed on the second screen 1212 of the first vehicle-mounted terminal 121, and is displayed in split screen with the first game application. The navigation application in the smart phone 112 is displayed on the first screen 1211 of the first vehicle-mounted terminal 121, and is displayed in split screen with the video application and the local camera screen; the second game application in the smart phone 112 is displayed on the third screen 1221 and the fourth screen 1222 of the second vehicle-mounted terminal 122, that is, the third screen 1221 and the fourth screen 1222 display the same screen content.

[0041] It should be noted that the connection method used between the source device 110 and the destination device 120 may include at least one of WlanAP mode, WiFi-P2P, USB wired connection, and mobile data network, which is not limited in the embodiments of the present application.

[0042] In addition, the above embodiment is only schematically illustrated by taking the mobile terminal displaying images to the vehicle-mounted terminal as an example (i.e., the vehicle-mounted terminal scenario). The solution provided in the embodiment of the present application can be applied to other multi-channel image display scenarios, and this embodiment does not limit the specific application scenario.

[0043] The above embodiment only takes the redirection scenario as an example, and the solution provided in the embodiment of the present application may not include the source device.

[0044] The solution provided in the embodiment of the present application can be used for Figure 1 The destination device in the embodiment can also be used for other devices with multi-channel image display requirements. For the convenience of description, the image display method is used for a computer device as an example in the following embodiments, but it does not constitute a limitation to this.

[0045] Please refer to Figure 2, which shows a flowchart of a screen display method provided by an exemplary embodiment of the present application. The present application embodiment takes the method applied to a computer device as an example for explanation. The method includes:

[0046] Step 201, obtaining at least one channel of image data, wherein the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection.

[0047] In order to realize at least one picture display, the computer device needs to obtain the image data corresponding to each picture. Among them, the computer device can display the picture in the local device or display the picture in the source device through redirection technology.

[0048] In one possible implementation, the computer device displays the screen in the local device, obtains application image data of the local application through the display acquisition module, or obtains camera image data through the local camera, or decodes the local video stream and the network video stream through the video decoding module to obtain video image data output by the video decoder.

[0049] In a possible implementation, the computer device displays the screen in the source device. The computer device, as the destination device, uses the redirection technology and needs to establish video redirectors at the source and destination respectively. After the source device encodes the application image data and the camera image data, the source video redirector transmits the encoding information to the destination video redirector, and the computer device decodes the encoding information through the video decoding module to obtain the image data in the source device.

[0050] Indicatively, Figure 3 As shown, based on the redirection technology, a source video redirector 316 and a destination video redirector 326 are respectively established between the source device 31 and the destination device 32, so that image data is transmitted through the link between the redirectors. In the case of obtaining image data from the source device 31, the first display acquisition module 313 collects data on the screen in the first application 311, and the collected application image data and the camera image data output by the first camera 312 need to be encoded by the video encoding module 314. The encoded data is transmitted to the destination video redirector 326 through the source video redirector 316, and decoded in the destination device 32 via the video decoding module 327, so as to obtain the image data in the source device 31. In the case where multiple applications need to display the screen, the source video processing module 315 is responsible for uniformly processing the image data of each application.

[0051] When the computer device acts as the destination device 32 and obtains image data locally, the computer device directly obtains the camera image data in the second camera 323, obtains the application image data in the second application 322 through the second display acquisition module 325, and decodes the local or network video stream 321 by the single-ended video stream player 324 through the video decoding module 327 to obtain video image data.

[0052] Step 202 , based on at least one channel of image data, image rendering is performed in a display window through multiple rendering threads to obtain a rendered image, wherein the same display window supports displaying multiple channels of images.

[0053] In order to enable each image to be displayed on a corresponding display screen, the computer device creates a corresponding display window for each display screen, and renders the image in the display window through multiple rendering threads to obtain a rendered image.

[0054] In some embodiments, the display windows correspond one-to-one to the display screens.

[0055] In a possible implementation, there are multiple images displayed on at least two display screens, and the computer device creates a corresponding display window for each display screen, wherein the display window is an Activity component, which can be specified in a certain display screen when started. Further, the computer device uniformly renders the images in each display window through multiple rendering threads to obtain a rendered image.

[0056] Indicatively, Figure 3 As shown, the computer device performs image rendering in the first display window 329 and the second display window 3210 respectively through the multi-channel rendering thread 328 according to the acquired image data to obtain a rendered image.

[0057] Step 203: display the rendering image via the display screen corresponding to the display window.

[0058] The computer device displays the rendering image through the display screens corresponding to each display window.

[0059] Indicatively, Figure 3 As shown, the computer device displays the rendering picture in the first display window 329 through the first display screen 331, and displays the rendering picture in the second display window 3210 through the second display screen 332.

[0060] In summary, in the embodiment of the present application, the computer device obtains at least one channel of image data from at least one of the local device and the source device, and renders the image in the display window through a multi-channel rendering thread to obtain a rendered image, and finally displays the rendered image through a display screen corresponding to the display window, wherein the same display window supports the display of multiple channels. A unified multi-channel image display framework provided in the embodiment of the present application can render image data from multiple sources (including local devices and redirection devices) in the display window corresponding to the display screen through a unified multi-channel rendering thread, and support the rendering of multiple channels in the same display window, thereby realizing support for display scenarios such as multi-screen same display (multiple display screens display the same image), multi-screen different display (multiple display screens display different images), and same-screen multiple display (the same display screen displays multiple different images).

[0061] Please refer to Figure 4 , which shows a flowchart of a screen display method provided by another exemplary embodiment of the present application. The present application embodiment is applied to the method Figure 1 Taking the implementation environment shown in FIG. 1 as an example, the method includes:

[0062] Step 401, obtaining at least one channel of image data, wherein the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection.

[0063] The implementation of this step may refer to the above step 201, and this embodiment will not be described in detail here.

[0064] Step 402: Create a rendering module corresponding to the display window through multiple rendering threads, wherein different display windows correspond to different rendering modules.

[0065] Since multiple rendering threads perform image rendering in each display window, a specific rendering module is required to perform specific rendering tasks. Therefore, the computer device creates a rendering module corresponding to each display window through multiple rendering threads, and the rendering module is responsible for rendering each image in the corresponding display window.

[0066] Indicatively, Figure 5 As shown, the Open Graphics Library (OpenGL) multi-path rendering thread 581 creates a first OpenGL rendering module 541 corresponding to the first display window 561 , and creates a second OpenGL rendering module 542 corresponding to the second display window 562 .

[0067] Step 403: Based on at least one channel of image data, a rendering module performs image rendering in a corresponding display window to obtain a rendered image.

[0068] Based on at least one channel of image data, the computer device performs layout rendering in a corresponding display window through a rendering module, thereby obtaining a rendering picture corresponding to the image data and displaying it on a display screen.

[0069] In one possible implementation, the process includes the following steps:

[0070] 1. Create a rendering object corresponding to the image data, which includes vertex coordinates and texture objects.

[0071] The computer device creates a corresponding rendering object based on the acquired image data, which can also be called an OpenGL logical view. The rendering object includes vertex coordinates and texture objects, wherein the vertex coordinates are used to represent the position information of the image data in the display window, and the texture object is used to represent the texture representation of the image data in the display window.

[0072] When the image data corresponds to a single display screen, the computer device generates a single rendering object based on the surface texture corresponding to the image data.

[0073] Since the same image only needs to be displayed on one display screen, the computer device only needs to generate a single rendering object based on the surface texture (SurfaceTexture) corresponding to the image data.

[0074] In a case where the image data corresponds to at least two display screens, the computer device generates at least two rendering objects based on the surface texture corresponding to the image data, wherein different rendering objects correspond to different display windows.

[0075] Since the same picture needs to be displayed on at least two display screens, that is, multiple displays on the same screen, for example, the same picture is projected to two screens of the destination device, or the same application is displayed on two screens, the computer device needs to generate at least two rendering objects based on the surface texture corresponding to the image data, and the rendering objects correspond one-to-one to the display windows corresponding to the display screens.

[0076] Indicative, such as Figure 5 As shown, the image data generated by the first video decoder 511 corresponds to two display screens, so the computer device generates two rendering objects based on the first surface texture 521 corresponding to the image data, namely a first OpenGL rendering object 531 and a second OpenGL rendering object 532.

[0077] The image data generated by the second video decoder 512 and the third video decoder 513 both correspond to a display screen, so the computer device generates a third OpenGL rendering object 533 and a fourth OpenGL rendering object 534 respectively based on the second surface texture 522 and the third surface texture 523 corresponding to the two image data.

[0078] 2. Add the rendering object to the corresponding rendering module.

[0079] When a rendering object is displayed alone on a display screen, the computer device adds the rendering object to a corresponding rendering module; when multiple rendering objects are displayed together on a display screen, the computer device adds the multiple rendering objects together to a corresponding rendering module.

[0080] Indicatively, Figure 5 As shown, the computer device adds the first OpenGL rendering object 531 to the first OpenGL rendering module 541 .

[0081] The computer device adds the second OpenGL rendering object 532 , the third OpenGL rendering object 533 , and the fourth OpenGL rendering object 534 to the second OpenGL rendering module 542 .

[0082] 3. Based on the rendering object, the rendering module performs image rendering in the corresponding display window to obtain a rendered image.

[0083] In a possible implementation, the computer device calls a rendering context environment for the rendering module, performs screen rendering in a corresponding display window, and obtains a rendered screen, wherein the rendering context environment encapsulates a related application programming interface and provides actual rendering capabilities for the rendering module.

[0084] Indicatively, Figure 5 As shown, the OpenGL rendering context environment 592 provides actual rendering capabilities for the first OpenGL rendering module 541 and the second OpenGL rendering module 542 .

[0085] In the case where a rendering module includes a rendering object, the computer device performs image rendering in a corresponding display window through the rendering module according to the rendering object, and displays the rendered composite image using a corresponding surface view control (SurfaceView).

[0086] Indicatively, Figure 5 As shown, the first OpenGL rendering module 541 only includes the first OpenGL rendering object 531 , so the computer device performs image rendering in the corresponding first display window 561 according to the rendering object, and displays the rendered image using the corresponding first surface view control 551 .

[0087] In the case where the rendering module includes at least two rendering objects, the computer device needs to determine layout information of each rendering object.

[0088] When a computer device needs to display multiple images on the same display screen, since the rendering module contains at least two rendering objects and the specific image ratios and sizes that need to be displayed by each rendering object are different, the computer device needs to call the multi-channel layout module to obtain the display coordinates and display size of each rendering object in the display window.

[0089] Among them, the multi-channel layout module dynamically generates the optimal layout style based on the attributes of the image data source corresponding to the rendering object. The computer device generates the highest screen-to-body ratio possible while taking into account the neatness and balance of the layout according to the aspect ratio, horizontal and vertical screen status, and access order of the display screen corresponding to the image data.

[0090] In one possible implementation, the computer device scales each picture according to the original aspect ratio and horizontal or vertical screen state of each picture without deforming the picture, takes into account the attributes of each picture, reasonably adjusts the scaling ratio, and arranges each picture to achieve the highest screen-to-body ratio.

[0091] The computer device performs multi-channel image rendering in a display window through a rendering module based on the layout information of each rendering object to obtain a rendered image.

[0092] In a possible implementation, the computer device renders the pictures corresponding to the respective rendering objects on the corresponding target surface through a rendering module based on the layout information of the respective rendering objects, wherein different rendering modules correspond to different surfaces. Further, the computer device displays the target surface through a surface view control of a display window.

[0093] Different from the related art, in which the computer device renders and displays each image in its own surface view control, in this embodiment, the computer device renders the image corresponding to each rendering object on the corresponding target surface through a rendering module, and displays the rendered and synthesized image through the same surface view control corresponding to the display window.

[0094] Indicatively, Figure 5 As shown, the second OpenGL rendering module 542 includes three rendering objects, namely, a second OpenGL rendering object 532, a third OpenGL rendering object 533 and a fourth OpenGL rendering object 534. The computer device calls the multi-path layout module 591 to reasonably layout the three OpenGL rendering objects according to the properties of the image data source corresponding to each rendering object, and renders the pictures corresponding to each rendering object on the corresponding target surface through the second OpenGL rendering module 542. Furthermore, the target surface is displayed through the second surface view control 552 corresponding to the second display window 562.

[0095] Step 404: display the rendering image via the display screen corresponding to the display window.

[0096] Indicative, such as Figure 5 As shown, the computer device displays the rendering picture in the first display window 561 through the first display screen 571 , and displays the rendering picture in the second display window 562 through the second display screen 572 .

[0097] In an embodiment of the present application, a computer device creates a rendering module corresponding to a display window through multiple rendering threads, and uses a rendering context environment to provide the rendering module with actual rendering capabilities. The rendering module renders the screen according to the image data. In addition, when multiple image data correspond to the same display window, layout information is generated by calling multiple layout modules, and then the rendering module performs unified rendering. The surface view control is used to display the rendered synthesized screen, which helps to improve rendering performance.

[0098] In conjunction with the above embodiments, please refer to Figure 6 , which shows a flowchart of a multi-pass rendering process provided by an exemplary embodiment of the present application.

[0099] Step 601, obtain the rendering thread handle.

[0100] The computer device obtains a rendering thread handle for displaying the window 61, and subsequently calls the OpenGL multi-path rendering thread 62 through the rendering thread handle.

[0101] Step 602, create an OpenGL thread instance.

[0102] In the absence of an OpenGL multi-path rendering thread, the computer device creates an OpenGL multi-path rendering thread instance, which is called by the display form through the rendering thread handle.

[0103] Step 603: Create an instance of the management context environment.

[0104] The computer device creates an instance of a management context environment based on the OpenGL multi-path rendering thread.

[0105] Step 604, return the OpenGL context handle.

[0106] The computer device returns the OpenGL context handle to the OpenGL multi-path rendering thread, and subsequently calls the OpenGL rendering context 63 through the OpenGL context handle.

[0107] Step 605, return the OpenGL thread handle.

[0108] The computer device returns the OpenGL thread handle to the corresponding display window.

[0109] Step 606, initialize the environment.

[0110] The computer device initializes the environment for displaying the window.

[0111] Step 607, create an OpenGL rendering module.

[0112] The computer device creates an OpenGL rendering module for the display window, wherein the display window corresponds to the OpenGL rendering module one by one.

[0113] Step 608, obtain the OpenGL rendering module handle.

[0114] The computer device obtains an OpenGL rendering module handle from the OpenGL rendering module 64, and subsequently calls the OpenGL rendering module 64 through the OpenGL rendering module handle.

[0115] Step 609, start the thread.

[0116] The computer device starts an OpenGL multi-path rendering thread 62 for displaying the window 61 .

[0117] Step 610, create an OpenGL context.

[0118] The computer device creates an OpenGL context by calling the OpenGL rendering context environment 63 .

[0119] Step 611, start creating the current surface of the rendering module.

[0120] The computer device creates a corresponding target surface for the OpenGL rendering module, so that the OpenGL rendering module performs picture rendering on the corresponding target surface.

[0121] Step 612, create an OpenGL surface and switch to the current surface.

[0122] In the case where there are multiple OpenGL rendering modules rendering the picture, the computer device creates an OpenGL surface and switches to the current surface to ensure that the OpenGL rendering module renders the picture on the corresponding target surface.

[0123] Step 613, create an OpenGL rendering object.

[0124] The computer device creates an OpenGL rendering object 65 based on the image data.

[0125] Step 614, obtain the OpenGL rendering object handle.

[0126] The computer device obtains an OpenGL rendering object handle from the OpenGL rendering object, and subsequently calls the OpenGL rendering object 65 through the OpenGL rendering object handle.

[0127] Step 615, add an OpenGL rendering object.

[0128] The computer device adds the OpenGL rendering object to the OpenGL multi-path rendering thread 62, and renders the picture according to the OpenGL rendering object.

[0129] Step 616, layout and drawing.

[0130] The computer device calls the OpenGL rendering module in the OpenGL rendering thread, and performs layout and drawing through the OpenGL rendering module.

[0131] Step 617, draw the OpenGL rendering object.

[0132] Furthermore, the computer device renders the OpenGL rendering object through the OpenGL rendering module 64 .

[0133] When the rendering module includes at least two rendering objects, the rendering object may be changed according to the screen display requirements.

[0134] In a possible implementation, when a rendering object included in a rendering module is changed, in order to reasonably layout the rendering objects according to the attributes of the image data source corresponding to each rendering object, the computer device needs to update the layout information of each rendering object after the change, wherein the change of the rendering object includes adding or deleting.

[0135] Furthermore, the computer device performs screen update rendering in the corresponding display window through a rendering module based on the updated layout information.

[0136] In the above embodiment, when the rendering object included in the rendering module changes, the computer device can dynamically and reasonably arrange the rendering object according to the attributes of the image data source corresponding to the rendering object, thereby improving the rendering performance. In addition, when the rendering object changes, the computer device no longer needs to create redundant surface view controls, but only needs to render and synthesize the changed rendering object through the rendering module, and the synthesized picture is displayed by the same surface view control.

[0137] In order to ensure that the display window corresponding to the display screen can be operated and used according to specific display needs, each display window is managed by a display window agent in the computer device to achieve creation, reuse and destruction of the display window at the right time.

[0138] In a possible implementation, the computer device first determines the display screen corresponding to each channel of image data, and creates a display window for the display screen through a display window agent if there is no display window corresponding to the display screen; if there is a display window corresponding to the display screen, the display window agent selects to reuse the display window corresponding to the display screen. Further, the computer device obtains the display instance contained in the display window through the display window agent, and destroys the display window through the display window agent if the display window does not contain the display instance.

[0139] Indicative, such as Figure 7 As shown, when it is determined that the first image data 711 corresponds to the display screen 731 and there is no display window, the computer device creates a display window 721 through the display window agent 741.

[0140] When one channel of image data corresponding to the display screen 731 is added, the computer device multiplexes the display window 721 through the display window agent 741 to process the second image data 712 .

[0141] When there is no image data corresponding to the display screen 731 , the computer device destroys the display window 721 through the display window agent 741 .

[0142] In the above embodiment, the computer device can manage the display window through the display window agent, and create, reuse and destroy the display window at the right time according to the screen display needs, thereby avoiding the waste of display windows and reducing the storage space occupied during the screen display process.

[0143] According to the solution provided in the embodiment of the present application, the computer device can simultaneously support multiple displays on the same screen, multiple displays on the same screen, and multiple displays on different screens. The following is an example of practical application:

[0144] When multiple display screens are used to display different images, the computer device processes each channel of image data separately, renders the image in a corresponding display window through multiple rendering threads, and finally displays it on a corresponding display screen.

[0145] Indicative, such as Figure 8As shown, the images that the computer device needs to display come from two source devices and a local device, and the four images are displayed on four display screens. In the first source device 81, the first encoder 812 encodes the image data of the first application 811, and the second encoder 814 encodes the image data of the second application 813, and transmits them to the destination device 83 respectively, and is decoded by the corresponding first decoder 831 and second decoder 832 to obtain image data; in the second source device 82, the third encoder 822 encodes the image data of the third application 821, transmits it to the destination device 83, and is decoded by the third decoder 833 to obtain image data; the local video stream 835 is decoded by the fourth decoder 834 to obtain image data. Furthermore, the computer device renders the image in each corresponding display window through a multi-path rendering thread 836, and the display window agent 8311 manages the first display window 837, the second display window 838, the third display window 839 and the fourth display window 8310, and finally displays the image on the first display screen 841, the second display screen 842, the third display screen 843 and the fourth display screen 844 respectively.

[0146] In the case of using multiple display screens to display the same picture, there are two possible ways, namely, multiple display screens of the same destination device display the same picture and display screens of multiple destination devices display the same picture.

[0147] Indicatively, Fig. 9 As shown, the computer device displays the application 911 in the source device 91, the encoder 912 encodes the image data in the application 911, and transmits them to the first destination device 92 and the second destination device 93 respectively, and decodes them respectively by the first decoder 921 and the second decoder 931, and in the first destination device 92, the first multi-channel rendering thread 922 performs image rendering in the first display window 923 and the second display window 924 respectively, and displays the image on the corresponding first display screen 941 and the second display screen 942, wherein the first display window agent 925 manages the first display window 923 and the second display window 924; in the second destination device 93, the second multi-channel rendering thread 932 performs image rendering in the third display window 933, and displays the image on the corresponding third display screen 943, wherein the second display window agent 934 manages the third display window 933.

[0148] When multiple images are displayed on the same display screen, the computer device processes each channel of image data separately, and by calling a multi-channel layout module in a multi-channel rendering thread, each image is reasonably laid out and uniformly rendered, and finally displayed on the same display screen.

[0149] Indicatively, Fig.10 As shown, the images that the computer device needs to display come from two source devices and a local device, and four images are displayed on the same display screen. In the first source device 1001, the first encoder 1012 encodes the image data of the first application 1011, and the second encoder 1014 encodes the image data of the second application 1013, and transmits them to the destination device 1003 respectively, and is decoded by the corresponding first decoder 1031 and second decoder 1032 to obtain image data; in the second source device 1002, the third encoder 1022 encodes the image data of the third application 1021, transmits it to the destination device 1003, and is decoded by the third decoder 1033 to obtain image data; the local video stream 1035 is decoded by the fourth decoder 1034 to obtain image data. Furthermore, the computer device calls the multi-path layout module 1037 through the multi-path rendering thread 1036, performs layout according to the attributes of the image data source, renders the image in the display window 1038, and displays the image on the display screen 1041, wherein the display window agent 1039 manages the display window 1038.

[0150] According to the solution provided in the embodiment of the present application, there is no specific limitation on the number of source devices and destination devices, as well as the number of image data sources and the number of display screens in the source devices and destination devices.

[0151] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0152] Please refer to Fig.11 , which shows a structural block diagram of a screen display device provided by an embodiment of the present application. The device may include:

[0153] The data acquisition module 1101 is used to acquire at least one channel of image data, where the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection;

[0154] A picture rendering module 1102 is used to perform picture rendering in a display window through multiple rendering threads based on the at least one channel of image data to obtain a rendered picture, wherein the same display window supports displaying multiple channels of pictures;

[0155] The display module 1103 is used to display the rendering image through the display screen corresponding to the display window.

[0156] Optionally, the picture rendering module 1102 is used to:

[0157] Creating a rendering module corresponding to a display window through the multi-path rendering threads, wherein different display windows correspond to different rendering modules;

[0158] Based on the at least one channel of image data, the rendering module performs picture rendering in a corresponding display window to obtain the rendered picture.

[0159] Optionally, the picture rendering module 1102 is used to:

[0160] Creating a rendering object corresponding to the image data, the rendering object including vertex coordinates and a texture object;

[0161] Adding the rendering object to the corresponding rendering module;

[0162] Based on the rendering object, the rendering module performs picture rendering in the corresponding display window to obtain the rendered picture.

[0163] Optionally, the picture rendering module 1102 is used to:

[0164] In a case where the image data corresponds to a single display screen, generating a single rendering object based on a surface texture corresponding to the image data;

[0165] In a case where the image data corresponds to at least two display screens, at least two rendering objects are generated based on the surface texture corresponding to the image data, wherein different rendering objects correspond to different display windows.

[0166] Optionally, the picture rendering module 1102 is used to:

[0167] In a case where the rendering module includes at least two rendering objects, determining layout information of each of the rendering objects;

[0168] Based on the layout information of each of the rendering objects, the rendering module performs multi-channel image rendering in the display window to obtain the rendering image.

[0169] Optionally, the picture rendering module 1102 is used to:

[0170] In the case that the rendering module includes at least two rendering objects, the rendering module calls a multi-path layout module to obtain the display coordinates and display size of each rendering object in the display window.

[0171] Optionally, the picture rendering module 1102 is used to:

[0172] Based on the layout information of each rendering object, rendering the picture corresponding to each rendering object on the corresponding target surface through the rendering module, wherein different rendering modules correspond to different surfaces;

[0173] The target surface is displayed through a surface view control of the display window.

[0174] Optionally, the picture rendering module 1102 is used to:

[0175] When a rendering object included in the rendering module is changed, updating the layout information of each rendering object after the change, wherein the rendering object is changed in a manner including adding or deleting;

[0176] Based on the updated layout information, the rendering module performs image update rendering in the corresponding display window.

[0177] Optionally, the display module 1103 is used to:

[0178] Determine the display screen corresponding to each channel of the image data;

[0179] In the case that there is no display window corresponding to the display screen, creating a display window for the display screen through a display window agent;

[0180] In the case that there is a display window corresponding to the display screen, the display window corresponding to the display screen is selected and reused through the display window agent.

[0181] Optionally, the display module 1103 is used to:

[0182] Acquire the display instance contained in the display window through the display window agent;

[0183] In the case that the display window does not contain a display instance, the display window is destroyed by the display window agent.

[0184] Optionally, the data acquisition module 1101 is used to:

[0185] Get application image data of local applications;

[0186] Acquire video image data output by a video decoder, wherein the video decoder is used to decode a local video stream, a network video stream, or a redirected video stream, wherein the redirected video stream is sent by the source device;

[0187] Get the camera image data output by the local camera.

[0188] In summary, in the embodiment of the present application, the computer device obtains at least one channel of image data from at least one of the local device and the source device, and renders the image in the display window through a multi-channel rendering thread to obtain a rendered image, and finally displays the rendered image through a display screen corresponding to the display window, wherein the same display window supports the display of multiple channels. A unified multi-channel image display framework provided in the embodiment of the present application can render image data from multiple sources (including local devices and redirection devices) in the display window corresponding to the display screen through a unified multi-channel rendering thread, and support the rendering of multiple channels in the same display window, thereby realizing support for display scenarios such as multi-screen same display (multiple display screens display the same image), multi-screen different display (multiple display screens display different images), and same-screen multiple display (the same display screen displays multiple different images).

[0189] Please refer to Fig.12 , which shows a block diagram of a computer device provided by an exemplary embodiment of the present application. The computer device 1700 may include one or more of the following components: a processor 1710 , a memory 1720 , and a display screen 1730 .

[0190] The processor 1710 may include one or more processing cores. The processor 1710 uses various interfaces and lines to connect the various parts of the entire computer device 1700, and executes various functions and processes data of the computer device 1700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1720, and calling data stored in the memory 1720. Optionally, the processor 1710 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 1710 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 1710, but may be implemented by a separate chip.

[0191] The memory 1720 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1720 includes a non-transitory computer-readable storage medium. The memory 1720 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data (such as audio data, a phone book), etc., created according to the use of the computer device 1700.

[0192] The display screen 1730 is a component for displaying images. Optionally, at least one display screen is provided in the computer device 1700. For example, the display screen 1730 is a display screen of a smart phone, or the display screen 1730 is a main driver display screen and a co-driver display screen controlled by a vehicle computer, or the display screen 1730 is a folding display screen of a folding screen terminal, etc.

[0193] In addition, those skilled in the art can understand that the structure of the computer device 1700 shown in the above drawings does not constitute a limitation on the computer device, and the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the computer device 1700 also includes a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, a sensor (such as an acceleration sensor, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module and other components, which will not be described in detail here.

[0194] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one program code, and the program code is loaded and executed by a processor to implement the screen display method described in the above embodiments.

[0195] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the screen display method provided in various optional implementations of the above aspects.

[0196] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0197] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A screen display method, characterized in that: The method comprises: Acquire at least one channel of image data, where the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection; Based on the at least one channel of image data, a picture rendering is performed in a display window corresponding to the display screen to obtain the rendered picture corresponding to the display screen, and the picture rendering method includes at least one of a multi-screen same-display scenario, a same-screen multiple-display scenario, and a multi-screen different-display scenario; wherein, in the multi-screen same-display scenario, the rendered picture is obtained by performing picture rendering in a display window corresponding to each display screen based on at least one channel of image data, and the rendered pictures corresponding to different display screens are the same; in the same-screen multiple-display scenario, the rendered picture is obtained by performing multi-channel picture rendering in a display window corresponding to the display screen based on at least two channels of image data; in the multi-screen different-display scenario, the rendered picture is obtained by performing picture rendering in a display window corresponding to each display screen based on the at least one channel of image data, and the rendered pictures corresponding to different display screens are different; The rendering image is displayed through the display screen.

2. The method according to claim 1, characterized in that In the multi-screen co-display scenario, performing picture rendering in a display window corresponding to a display screen based on the at least one channel of image data to obtain the rendered picture corresponding to the display screen includes: Creating a rendering module for the display window corresponding to each display screen through multiple rendering threads, wherein different display windows correspond to different rendering modules; Based on the at least one channel of image data, the rendering module performs image rendering in the display window corresponding to each display screen to obtain the rendered image corresponding to each display screen.

3. The method according to claim 2, characterized in that The step of performing image rendering in the display window corresponding to each display screen by the rendering module based on the at least one channel of image data to obtain the rendered image corresponding to each display screen includes: Creating a rendering object corresponding to the image data, the rendering object including vertex coordinates and texture; Adding the rendering object to the corresponding rendering module; Based on the rendering object, the rendering module performs picture rendering in the display window corresponding to each display screen to obtain the rendering picture corresponding to each display screen.

4. The method according to claim 3, characterized in that The creating a rendering object corresponding to the image data comprises: At least two rendering objects are generated based on the surface texture corresponding to the image data, wherein different rendering objects correspond to the display windows of different display screens.

5. The method according to claim 3, characterized in that: The rendering module performs picture rendering in the display window corresponding to each display screen based on the rendering object to obtain the rendering picture corresponding to each display screen, including: In a case where the rendering module includes at least two rendering objects, determining layout information of each of the rendering objects; Based on the layout information of each of the rendering objects, the rendering module performs multi-channel image rendering in the display window to obtain the rendering image.

6. The method according to claim 5, characterized in that When the rendering module includes at least two rendering objects, determining layout information of each rendering object includes: In the case that the rendering module includes at least two rendering objects, the rendering module calls a multi-path layout module to obtain the display coordinates and display size of each rendering object in the display window.

7. The method according to claim 5, characterized in that The step of performing multi-channel image rendering in the display window by the rendering module based on the layout information of each rendering object to obtain the rendering image includes: Based on the layout information of each rendering object, rendering the picture corresponding to each rendering object on the corresponding target surface through the rendering module, wherein different rendering modules correspond to different surfaces; The target surface is displayed through a surface view control of the display window.

8. The method according to claim 5, characterized in that The method further comprises: When a rendering object included in the rendering module is changed, updating the layout information of each rendering object after the change, wherein the rendering object is changed in a manner including adding or deleting; Based on the updated layout information, the rendering module performs image update rendering in the corresponding display window.

9. The method according to claim 1, characterized in that: In the same-screen multi-display scenario, performing picture rendering in a display window corresponding to a display screen based on the at least one channel of image data to obtain the rendered picture corresponding to the display screen includes: Creating a rendering module for the display window corresponding to the display screen through multiple rendering threads; Based on the at least two channels of image data, the rendering module performs multi-channel image rendering in the display window corresponding to the display screen to obtain the rendered image corresponding to the display screen.

10. The method according to claim 9, characterized in that The method of performing multi-channel image rendering in the display window corresponding to the display screen by the rendering module based on the at least two channels of image data to obtain the rendered image corresponding to the display screen includes: Creating a rendering object corresponding to the image data, the rendering object including vertex coordinates and a texture object; Adding the rendering object to the corresponding rendering module; Based on the rendering object, the rendering module performs multi-channel image rendering in the display window corresponding to the display screen to obtain the rendering image corresponding to the display screen.

11. The method according to claim 10, characterized in that The creating a rendering object corresponding to the image data comprises: A single rendering object is generated based on the surface texture corresponding to the image data.

12. The method according to claim 10, characterized in that The step of performing multi-channel image rendering in the display window corresponding to the display screen by the rendering module based on the rendering object to obtain the rendering image corresponding to the display screen includes: Determining layout information of each of the rendering objects; Based on the layout information of each of the rendering objects, the rendering module performs multi-channel image rendering in the display window to obtain the rendering image.

13. The method according to claim 12, characterized in that The determining layout information of each of the rendering objects includes: The rendering module calls a multi-path layout module to obtain the display coordinates and display size of each of the rendering objects in the display window.

14. The method according to claim 12, characterized in that The step of performing multi-channel image rendering in the display window by the rendering module based on the layout information of each rendering object to obtain the rendering image includes: Based on the layout information of each rendering object, rendering a picture corresponding to each rendering object on a corresponding target surface through the rendering module; The target surface is displayed through a surface view control of the display window.

15. The method according to claim 12, characterized in that The method further comprises: When a rendering object included in the rendering module is changed, updating the layout information of each rendering object after the change, wherein the rendering object is changed in a manner including adding or deleting; Based on the updated layout information, the rendering module performs image update rendering in the corresponding display window.

16. The method according to claim 1, characterized in that In the multi-screen different display scenario, the image rendering is performed in a display window corresponding to the display screen based on the at least one channel of image data to obtain the rendered image corresponding to the display screen, including: Creating a rendering module for the display window corresponding to each display screen through multiple rendering threads, wherein different display windows correspond to different rendering modules; Based on the at least one channel of image data, the rendering module performs image rendering in the display window corresponding to each display screen to obtain the rendered image corresponding to each display screen.

17. The method according to claim 16, characterized in that The step of performing image rendering in the display window corresponding to each display screen by the rendering module based on the at least one channel of image data to obtain the rendered image corresponding to each display screen includes: Creating a rendering object corresponding to the image data, the rendering object including vertex coordinates and a texture object; Adding the rendering object to the corresponding rendering module; Based on the rendering object, the rendering module performs picture rendering in the corresponding display window to obtain the rendered picture.

18. The method according to claim 17, characterized in that The creating a rendering object corresponding to the image data comprises: In a case where the image data corresponds to a single display screen, generating a single rendering object based on a surface texture corresponding to the image data; In a case where the image data corresponds to at least two display screens, at least two rendering objects are generated based on the surface texture corresponding to the image data, wherein different rendering objects correspond to different display windows.

19. The method according to claim 17, characterized in that The step of performing screen rendering in a corresponding display window by means of the rendering module based on the rendering object to obtain the rendered screen includes: In a case where the rendering module includes at least two rendering objects, determining layout information of each of the rendering objects; Based on the layout information of each of the rendering objects, the rendering module performs multi-channel image rendering in the display window to obtain the rendering image.

20. The method according to claim 19, characterized in that When the rendering module includes at least two rendering objects, determining layout information of each rendering object includes: In the case that the rendering module includes at least two rendering objects, the rendering module calls a multi-path layout module to obtain the display coordinates and display size of each rendering object in the display window.

21. The method according to claim 19, characterized in that The step of performing multi-channel image rendering in the display window by the rendering module based on the layout information of each rendering object to obtain the rendering image includes: Based on the layout information of each rendering object, rendering the picture corresponding to each rendering object on the corresponding target surface through the rendering module, wherein different rendering modules correspond to different surfaces; The target surface is displayed through a surface view control of the display window.

22. The method according to claim 19, characterized in that The method further comprises: When a rendering object included in the rendering module is changed, updating the layout information of each rendering object after the change, wherein the rendering object is changed in a manner including adding or deleting; Based on the updated layout information, the rendering module performs image update rendering in the corresponding display window.

23. The method according to any one of claims 1 to 22, characterized in that: The method further comprises: Determine the display screen corresponding to each channel of the image data; In the case that there is no display window corresponding to the display screen, creating a display window for the display screen through a display window agent; In the case that there is a display window corresponding to the display screen, the display window corresponding to the display screen is selected and reused through the display window agent.

24. The method according to claim 23, characterized in that The method further comprises: Acquire the display instance contained in the display window through the display window agent; In the case that the display window does not contain a display instance, the display window is destroyed by the display window agent.

25. The method according to any one of claims 1 to 22, characterized in that: The obtaining of at least one channel of image data includes at least one of the following methods: Get application image data of local applications; Acquire video image data output by a video decoder, wherein the video decoder is used to decode a local video stream, a network video stream, or a redirected video stream, wherein the redirected video stream is sent by the source device; Get the camera image data output by the local camera.

26. A screen display device, characterized in that: The device comprises: A data acquisition module, used for acquiring at least one channel of image data, wherein the source of the at least one channel of image data includes at least one of a local device and a source device, and the source device is a device for performing image redirection; A picture rendering module, used for performing picture rendering in a display window corresponding to a display screen based on the at least one channel of image data to obtain the rendered picture corresponding to the display screen, wherein the picture rendering method includes at least one of a multi-screen same-display scenario, a same-screen multiple-display scenario, and a multi-screen different-display scenario; wherein, in the multi-screen same-display scenario, the rendered picture is obtained by performing picture rendering in a display window corresponding to each display screen based on at least one channel of image data, and the rendered pictures corresponding to different display screens are the same; in the same-screen multiple-display scenario, the rendered picture is obtained by performing multi-channel picture rendering in a display window corresponding to the display screen based on at least two channels of image data; in the multi-screen different-display scenario, the rendered picture is obtained by performing picture rendering in a display window corresponding to each display screen based on the at least one channel of image data, and the rendered pictures corresponding to different display screens are different; A display module is used to display the rendering image through the display screen.

27. A computer device, characterized in that: The computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the screen display method as described in any one of claims 1 to 25.

28. A computer-readable storage medium, characterized in that: At least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by the processor to implement the screen display method as described in any one of claims 1 to 25.

29. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the screen display method as described in any one of claims 1 to 25.

Images