Image rendering method, device and electronic device

By creating the host player and slave player in video editing technology, and not parsing the audio of slave media data in the terminal device, it solves the problem that the terminal device takes too much computing resources when rendering multiple media data in a single window. The problem improves the user experience.

CN115529492BActive Publication Date: 2024-11-15HISENSE VISUAL TECH CO LTD
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202211007931.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-22
Publication Date
2024-11-15
Estimated Expiration
2042-08-22

AI Technical Summary

Technical Problem

In the field of video editing technology, when the terminal device renders multiple media data in a single window, it is easy to occupy a large amount of computing resources, resulting in flash problems and affecting the user experience.

Method used

By creating the host player and the slave player, the host player is responsible for rendering the main media data, while the slave player does not parse the audio data from the slave data, reduces the computing resource usage, and loads it in the host player and the slave player. The data starts broadcasting and performs the target function to determine the target texture image for rendering.

Benefits of technology

It effectively reduces the computing resource usage of terminal devices when rendering multiple media data in a single window, reduces flash frequency and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115529492B_ABST
    Figure CN115529492B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image rendering method, device and electronic device, which relates to the field of video editing technology and is used to solve the problem of how to reduce the computing resources occupied by a terminal device when rendering multiple media data in a single window. The method includes: in response to a selection operation of a target function in a target application, obtaining a main media data, at least one slave media data, and the first frame rate information of the main media data; creating a main player for rendering the main media data, and at least one slave player for rendering the slave media data; when the main player has preloaded the main media data and the slave player has preloaded the slave media data, starting the main player and the slave player; performing a target operation corresponding to the target function on the target frame data to determine a target texture image; rendering the target texture image to obtain a display image of the current frame.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of video editing technology, and in particular to an image rendering method, device and electronic device. Background Art

[0002] At present, in the field of video editing technology, when a terminal device renders a piece of media data (such as a video) in a single window, the terminal device can render the video well. When a terminal device renders multiple pieces of media data in a single window, a large amount of computing resources will be occupied. When the configuration parameters of the terminal device (such as memory, processor, etc.) are low, flickering problems are likely to occur, resulting in a low user experience.

[0003] Therefore, how to reduce the computing resources occupied by a terminal device when rendering multiple media asset data in a single window has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an image rendering method, device and electronic device.

[0005] The technical solution of the present disclosure is as follows:

[0006] In a first aspect, the present disclosure provides an image rendering method, comprising: in response to a selection operation of a target function in a target application, obtaining a main media asset data, at least one slave media asset data, and first frame rate information of the main media asset data; creating a main player for rendering the main media asset data, and at least one slave player for rendering the slave media asset data; wherein, a slave player renders a slave media asset, and the slave player does not parse the audio data in the slave media asset data; when the main player has preloaded the main media asset data and the slave player has preloaded the slave media asset data, starting the main player and the slave player; performing a target operation corresponding to the target function on the target frame data to determine a target texture image; wherein the target frame data includes frame data of the main player in the current frame, or the target frame data includes frame data of the main player in the current frame and frame data of the slave player in the current frame; rendering the target texture image to obtain a display image of the current frame.

[0007] In some feasible examples, before performing a target operation corresponding to a target function on target frame data and determining a target texture image, the image rendering method provided by the present disclosure further includes: based on first frame rate information, when determining that the first frame rate information is a fixed frame rate, performing frame synchronization on the slave media data of the slave player and the main media data of the main player according to the media clock of the main media data; and performing the following operations on each slave player: eliminating frame data in the slave media data to obtain the eliminated slave media data.

[0008] In some implementable examples, the target frame data includes frame data of the main player in the current frame; a target operation corresponding to a target function is performed on the target frame data to determine a target texture image, including: when there is no frame data in the current frame of the slave player, a target operation corresponding to the target function is performed on both the frame data of the slave player in the previous frame and the frame data of the main player in the current frame to determine the target texture image.

[0009] In some implementable examples, the target frame data includes frame data of the main player in the current frame and frame data of the slave player in the current frame; a target operation corresponding to a target function is performed on the target frame data to determine a target texture image, including: in the case where there is frame data of the slave player in the current frame, a target operation corresponding to the target function is performed on both the frame data of the slave player in the current frame and the frame data of the main player in the current frame to determine the target texture image.

[0010] In some feasible examples, the image rendering method provided by the present disclosure also includes: when capturing a frame image from the media data after being eliminated, if there is no frame image in the current frame, determining that the slave player corresponding to the eliminated media data does not have frame data in the current frame; when capturing a frame image from the media data after being eliminated, if there is a frame image in the current frame, determining that the slave player corresponding to the eliminated media data does have frame data in the current frame.

[0011] In some feasible examples, frame data in the slave media asset data are eliminated to obtain the eliminated slave media asset data, including: obtaining a first total duration of the rendering duration corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering duration corresponding to each frame of the slave media asset data before the current frame; according to the first total duration and the second total duration, the frame data in the slave media asset data are eliminated to obtain the eliminated slave media asset data.

[0012] In some feasible examples, according to the first total duration and the second total duration, frame data in the secondary media asset data is removed to obtain the removed secondary media asset data, including:

[0013] When the absolute value of the difference between the first total duration and the second total duration is greater than or equal to the product of the preset threshold and the rendering duration corresponding to each frame of the main media asset data before the current frame, the frame data corresponding to the current frame in the slave media asset data is discarded; when the absolute value of the difference between the first total duration and the second total duration is less than the preset threshold and the product of the rendering duration corresponding to each frame of the main media asset data before the current frame, the frame data corresponding to the current frame in the slave media asset data is retained.

[0014] In some feasible examples, the image rendering method provided by the present disclosure also includes: based on the first frame rate information, when it is determined that the first frame rate information is not a fixed frame rate, frame synchronization of the slave media data of the slave player and the master media data of the master player according to the system clock.

[0015] In a second aspect, the present disclosure provides an image rendering device, comprising: a processing unit, for controlling an acquisition unit to acquire a main media asset data, at least one slave media asset data, and a first frame rate information of the main media asset data in response to a selection operation of a target function in a target application; the processing unit, further for creating a main player for rendering the main media asset data acquired by the acquisition unit, and at least one slave player for rendering the slave media asset data acquired by the acquisition unit; wherein a slave player renders a slave media asset, and the slave player does not parse the audio data in the slave media data; the processing unit, further for starting the main player and the slave player when the main player preloads the main media asset data acquired by the acquisition unit and the slave player preloads the slave media asset data acquired by the acquisition unit; the processing unit, further for performing a target operation corresponding to the target function on the target frame data to determine a target texture image; wherein the target frame data includes frame data of the main player in the current frame, or the target frame data includes frame data of the main player in the current frame and frame data of the slave player in the current frame; the processing unit, further for rendering the target texture image, and controlling the display unit to display the display image of the current frame.

[0016] In some feasible examples, the processing unit is further used to perform frame synchronization on the slave media data of the slave player and the main media data of the master player according to the media clock of the main media data when the first frame rate information obtained by the acquisition unit is determined to be a fixed frame rate; the processing unit is further used to perform the following operation on each slave player: remove frame data from the slave media data obtained by the acquisition unit to obtain the removed slave media data.

[0017] In some implementable examples, the target frame data includes frame data of the main player in the current frame; the processing unit is specifically used to perform a target operation corresponding to the target function on the frame data of the slave player in the previous frame and the frame data of the main player in the current frame when there is no frame data in the slave player in the current frame, so as to determine the target texture image.

[0018] In some implementable examples, the target frame data includes frame data of the main player in the current frame and frame data of the slave player in the current frame; the processing unit is specifically used to perform a target operation corresponding to the target function on the frame data of the slave player in the current frame and the frame data of the main player in the current frame when there is frame data of the slave player in the current frame, so as to determine the target texture image.

[0019] In some feasible examples, the processing unit is further used to, when capturing a frame image from the media asset data after being removed, if there is no frame image in the current frame, determine that the slave player corresponding to the removed media asset data does not have frame data in the current frame; the processing unit is further used to, when capturing a frame image from the media asset data after being removed, if there is a frame image in the current frame, determine that the slave player corresponding to the removed media asset data does have frame data in the current frame.

[0020] In some feasible examples, the acquisition unit is specifically used to obtain a first total duration of the rendering time corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering time corresponding to each frame of the slave media asset data before the current frame; the processing unit is specifically used to eliminate frame data in the slave media asset data according to the first total duration obtained by the acquisition unit and the second total duration obtained by the acquisition unit to obtain the eliminated slave media asset data.

[0021] In some feasible examples, the processing unit is specifically used to discard the frame data corresponding to the current frame in the slave media asset data obtained by the acquisition unit when the absolute value of the difference between the first total duration obtained by the acquisition unit and the second total duration obtained by the acquisition unit is greater than or equal to the product of a preset threshold and the rendering duration corresponding to each frame of the main media asset data before the current frame; the processing unit is specifically used to retain the frame data corresponding to the current frame in the slave media asset data obtained by the acquisition unit when the absolute value of the difference between the first total duration obtained by the acquisition unit and the second total duration obtained by the acquisition unit is less than the preset threshold and the product of the rendering duration corresponding to each frame of the main media asset data before the current frame.

[0022] In some feasible examples, the processing unit is further used to perform frame synchronization on the slave media data of the slave player and the master media data of the master player according to the system clock when determining that the first frame rate information is not a fixed frame rate based on the first frame rate information obtained by the acquisition unit.

[0023] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory being used to store a computer program; the processor being used to enable the electronic device to implement any one of the image rendering methods provided in the first aspect when executing the computer program.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor as any image rendering method provided in the first aspect.

[0025] In a fifth aspect, the present invention provides a computer program product. When the computer program product is run on a computer, the computer executes any one of the image rendering methods provided in the first aspect.

[0026] It should be noted that the above computer instructions may be stored in whole or in part on a first computer-readable storage medium, wherein the first computer-readable storage medium may be packaged together with the processor of the image rendering device, or may be packaged separately from the processor of the image rendering device, which is not limited in the present disclosure.

[0027] The description of the second, third, fourth and fifth aspects in the present disclosure can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0028] In the present disclosure, the name of the above-mentioned image rendering device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of the present disclosure, they belong to the scope of the claims of the present disclosure and their equivalent technologies.

[0029] These and other aspects of the present disclosure will become more apparent from the following description.

[0030] Compared with the prior art, the technical solution provided by the present invention has the following advantages:

[0031] When the user needs to render multiple media asset data, the target function of the target application can be selected. The target is used to create a main player and at least one slave player after receiving the selection operation of the target function. When there are multiple slave players, the slave player does not parse the audio data in the slave media asset data, thereby reducing the occupation of computing resources. Furthermore, after the main player and the slave player have loaded the media asset data, the main player and the slave player are started. Then, the target operation corresponding to the target function is performed on the target frame data to determine the target texture image. After that, the target texture image is rendered to obtain the display image of the current frame.

[0032] In addition, when the image rendering method provided by the present disclosure is applied in a scenario where multiple media data are rendered in a single window, the target is used to create a slave player. Since there is no processing unit in the slave player for parsing the corresponding audio data, the computing resources occupied by multiple slave players when parsing the audio data can be reduced. Since the computing resources occupied by the slave player are reduced when the target application renders multiple media data in a single window, the computing resources occupied by the target application when rendering multiple media data in a single window can be reduced, which solves the problem of how to reduce the computing resources occupied by the terminal device when rendering multiple media data in a single window in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1 One of the flowcharts of the image rendering method provided in the embodiment of the present application;

[0036] Figure 2 One of the structural schematic diagrams of the display device in the image rendering method provided in the embodiment of the present application;

[0037] Figure 3 The second structural schematic diagram of the display device in the image rendering method provided in the embodiment of the present application;

[0038] Figure 4 One of the flowcharts of the image rendering method provided in the embodiment of the present application;

[0039] Figure 5 The second flowchart of the image rendering method provided in the embodiment of the present application;

[0040] Figure 6 The third flowchart of the image rendering method provided in the embodiment of the present application;

[0041] Figure 7 A fourth flowchart of the image rendering method provided in an embodiment of the present application;

[0042] Figure 8 A fifth flowchart of the image rendering method provided in an embodiment of the present application;

[0043] Fig. 9 The sixth flowchart of the image rendering method provided in the embodiment of the present application;

[0044] Fig.10 A schematic diagram of the structure of a display device 200 provided in an embodiment of the present application;

[0045] Fig.11 A schematic diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0048] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0049] The Surface Texture in the disclosed embodiment refers to a new class added from Android 3.0 (Application Program Interface (API) 11). Surface Texture is used to capture image frames in a video stream, which can be a camera preview or video decoding data.

[0050] The Surface in the embodiments of the present disclosure refers to the surface layer, which is mainly used for screen display in the Android system.

[0051] OpenGL ES (OpenGL for Embedded Systems) in the disclosed embodiments is a subset of the Open Graphics Library (OpenGL) 3D graphics API, and is designed for embedded devices such as mobile phones, personal digital assistants (PDAs) and game consoles.

[0052] The EGL in the embodiments of the present disclosure refers to the first layer between the rendering plane and OpenGL ES, which is responsible for connecting the rendering plane and managing the OpenGL ES context.

[0053] The YUV in the disclosed embodiments is a color encoding method. "Y" represents brightness (Luminance or Luma), that is, the grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0054] The BufferQueue in the disclosed embodiment is the core of all graphics processing operations in Android. Its function is very simple: to connect the party that generates the graphics data buffer (the producer) to the party that receives the data for display or further processing (the consumer).

[0055] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application, such as Figure 1 As shown, the user can operate the display device 200 through the mobile terminal 300 and the control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user instructions through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices can also be used to control the display device 200.

[0056] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and realize connection communication through the network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 also communicates data with the display device 200 through various communication methods. The display device 200 can be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The display device 200 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide an intelligent network TV function that provides a computer support function.

[0057] In some embodiments, the electronic device provided by the embodiment of the present application may be the above-mentioned display device 200. Among them, the display device 200 responds to the selection operation of the target function in the target application (such as: the target application is an application for video editing, and the target function is a picture-in-picture function or a multi-grid function), obtains a main media asset data, at least one slave media asset data, and the first frame rate information of the main media asset data, such as: after the user selects the picture-in-picture function, the user can find at least two required videos in the media asset data stored in the display device 200, and determine the main media asset data and the slave media asset data in the selected video. In this way, the display device 200 can perform picture-in-picture rendering on the main media asset data and the slave media asset data, such as: creating a main player for rendering the main media asset data, and at least one slave player for rendering the slave media asset data. Since the slave player does not parse the audio data in the slave media asset data, the occupation of the computing resources of the display device 200 can be reduced. At the same time, when the main player preloads the main media asset data and the slave player preloads the slave media asset data, the main player and the slave player are started. Afterwards, the target operation corresponding to the target function is performed on the target frame data to determine the target texture image. Finally, the display image of the current frame is obtained by rendering the target texture image, so that the user can check the display image corresponding to the current frame, thereby improving the user experience.

[0058] Figure 2 FIG. 2 shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment. Figure 2 The display device 200 shown includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be a display with a touch function, such as a touch display. The tuner 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0059] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0060] In some examples, the display device 200 of one or more embodiments is a TV set 1, and the operating system of the TV set 1 is an Android system. Figure 3 As shown, the television 1 can be logically divided into an application layer (Applications) (referred to as "application layer") 21, an application framework layer (Application Framework) (referred to as "framework layer") 22, an Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer") 23, and a kernel layer 24.

[0061] Among them, the application layer 21 includes one or more applications. The application can be a system application or a third-party application. For example, the application layer 21 includes a target application, and the target application can provide a video editing function. The framework layer 22 provides an application programming interface (API) and a programming framework for the application of the application layer 21. The system runtime layer 23 provides support for the upper layer, namely the framework layer 22. When the framework layer 22 is used, the Android operating system will run the C / C++ library contained in the system runtime layer 23 to implement the functions to be implemented by the framework layer 22. The kernel layer 24 serves as a software middleware between the hardware layer and the application layer 21, and is used to manage and control hardware and software resources.

[0062] In some examples, the kernel layer 24 includes a first driver and a second driver, the first driver is used to send the user operation collected by the detector 230 to the target application, and the second driver is used to control the display 260 to display the display image sent by the display unit 212.

[0063] The target application in the TV set 1 is started. Afterwards, the first driver is used to send the user operation collected by the detector 230 to the target application for identification. Afterwards, the processing unit 211 of the target application responds to the target operation received by the acquisition unit 210 (such as: selecting the picture-in-picture function in the target application), and at this time, the processing unit 211 creates a main player for rendering the main media asset data, and N slave players for rendering the slave media asset data. N is a constant greater than or equal to 1.

[0064] The main player at least includes: a file download module, a decapsulation module, a video decoding module, an audio decoding module, an audio and video synchronization module, a video rendering module and an audio rendering module; the slave player at least includes: a file download module, a decapsulation module, a video decoding module, a multi-video synchronization module and a video rendering module;

[0065] The file download module is used to read the main media data selected by the user, or the slave media data from the memory of the display device 200. The decapsulation module is used to decapsulate the main media data or the slave media data read by the file download module according to a preset protocol, thereby obtaining video data and audio data. The video decoding module is used to parse the video data generated by the decapsulation module to obtain video parsing data. The audio decoding module is used to parse the audio data generated by the decapsulation module to obtain audio parsing data. The audio and video synchronization module is used to synchronize the video parsing data generated by the video decoding module and the audio parsing data generated by the audio decoding module, thereby ensuring the consistency of display. The video rendering module is used to render the video data synchronized by the audio and video synchronization module to obtain video data of the display image (such as: data in YUV format, or data in red (RED, R) green (GREEN, G) blue (BLUE, B) format). The audio rendering module is used to render the audio data synchronized by the audio and video synchronization module to obtain audio data of the display image. The multi-video synchronization module is used to synchronize the video parsing data parsed by the video decoding module in the player with the video parsing data parsed by the video decoding module in the anchor, so as to ensure the consistency of display.

[0066] When rendering the frame data of the master player and the slave player, only when the frame data of the master player arrives, a request is made to render the frame data of the master player and the slave player.

[0067] Afterwards, the processing unit 211 starts playing the main player and the slave player when the main player has preloaded the main media asset data obtained by the acquisition unit 210, and the slave player has preloaded the slave media asset data obtained by the acquisition unit 210. The processing unit 211 determines that the first frame rate information of the main media asset data obtained by the acquisition unit 210 is a fixed frame rate, and then performs frame synchronization on the slave media asset data of the slave player and the main media asset data of the main player according to the media clock of the main media asset data; the processing unit 211 performs the following operations on each slave player: removes the frame data in the slave media asset data to obtain the removed slave media asset data. Alternatively, when it is determined that the first frame rate information is not a fixed frame rate, the slave media asset data of the slave player and the main media asset data of the main player are frame synchronized according to the system clock. The processing unit 211 performs the following operations on each slave player: retains the frame data in the slave media asset data. Afterwards, when the OpenGL ES in the processing unit 211 determines that the frame data of the main player has arrived, the data acquisition unit is controlled to acquire the frame data of the current frame from each slave player (such as: the frame data from the media asset data after being eliminated, or the frame data from the media asset data). Afterwards, the video editing unit of the OpenGL ES in the processing unit 211 performs the target operation corresponding to the target function on the target frame data to determine the target texture image; the video editing unit of the OpenGL ES in the processing unit 211 renders the target texture image, and controls the display unit 212 to display the display image of the current frame, such as: after the video editing module in the processing unit 211 renders the target texture image, the rendered texture image is stored in the frame buffer object (FBO), and then EGL reads the rendered texture image corresponding to the current frame from the FBO, and controls the display unit 212 to display the rendered texture image corresponding to the current frame.

[0068] Specifically, the storage unit 213 of the target application can be used to store the program code of the TV 1, and can also be used to store data generated by the TV 1 during operation, such as data in a write request, media asset data saved by a user, and the like.

[0069] In some examples, the memory in the TV 1 stores media data. After the user opens the target application on the TV 1 and selects the target function of the target application, the target application displays the media data currently stored in the TV 1. The user can then select the required media data, so that the target application can obtain a main media data and at least one slave media data.

[0070] In some other examples, no media data is stored in the memory of TV 1. After the user opens the target application on TV 1 and selects the target function of the target application, the target application requests media data from server 400 (e.g., determining the media data based on the user portrait corresponding to the user account currently logged in to TV 1). Then, the target application displays the media data sent by server 400, and the user can select the required media data, so that the target application can obtain a main media data and at least one slave media data.

[0071] In other examples, the memory in the TV set 1 does not store media data. After the TV set 1 establishes a communication connection with the user device, the user starts the target application on the TV set 1, and selects the target function of the target application. At this time, the TV set 1 requests the user device to read the media data. After receiving the request to read the media data sent by the TV set 1, the user device sends the media data stored in the memory of the user device to the TV set 1. After the TV set 1 receives the media data sent by the user device, the target application displays the media data sent by the user device, and the user can select the required media data, so that the target application can obtain a main media data and at least one slave media data.

[0072] In some examples, the main media data and the secondary media data can be determined according to the order in which the user selects the media data, such as determining that the first selected media data is the main media data, and the media data selected thereafter is the secondary media data. Alternatively, the main media data and the secondary media data can be determined according to the user's settings for the media data, such as when the user selects the media data, the user can be prompted whether the current media data is set as the main media data or the secondary media data. When the user chooses to set it as the main media data, there is no need to prompt whether the current media data is set as the main media data or the secondary media data in the subsequent process of selecting the media data. In this way, the main media data and the secondary media data can be determined.

[0073] Specifically, the electronic device provided in the embodiment of the present disclosure may be the above-mentioned display device 200, which is not limited here.

[0074] The primary media asset data and the secondary media asset data involved in this application may be data authorized by the user or fully authorized by all parties.

[0075] In the following embodiments, the method of the embodiment of the present application is described by taking the above-mentioned display device 200 as an example in which the execution subject of the image rendering method provided by the embodiment of the present disclosure is executed.

[0076] The present application embodiment provides an image rendering method, such as Figure 4 As shown, the image rendering method may include S11-S15.

[0077] S11. In response to a selection operation of a target function in a target application, a main media asset data, at least one slave media asset data, and first frame rate information of the main media asset data are obtained.

[0078] S12: Create a main player for rendering the main media asset data and at least one slave player for rendering the slave media asset data, wherein one slave player renders one slave media asset data, and the slave player does not parse the audio data in the slave media asset data.

[0079] Specifically, when the slave player does not parse the audio data in the slave media asset data, the video rendering of the slave player will not be affected by the original audio and video synchronization module, which can reduce the usage of computing resources while ensuring the user experience.

[0080] S13: When the main player has preloaded the main media asset data, and the slave player has preloaded the slave media asset data, start playing the main player and the slave player.

[0081] S14, performing a target operation corresponding to a target function on the target frame data to determine a target texture image, wherein the target frame data includes frame data of the master player in the current frame, or the target frame data includes frame data of the master player in the current frame and frame data of the slave player in the current frame.

[0082] In some examples, the target function may be a picture-in-picture function, and the target operation is to perform picture-in-picture rendering. Here, the target function may be the picture-in-picture function as an example for explanation. In some other examples, the target function may be other filter functions, and the target operation is to perform other filter rendering, which is not limited here.

[0083] S15. Render the target texture image to obtain a display image of the current frame.

[0084] In some examples, when creating the master player and the slave player, the processing module in the target application also needs to dynamically generate the texture id corresponding to each player through OpenGL, and generate the Surface Texture corresponding to each player based on the texture id of each player, and then generate the Surface (surface layer) based on the Surface Texture and pass it to the player. Among them, SurfaceTexture contains a BufferQueue. When the player queues new data, it will provide a callback notification of "onFrameAvailable" (i.e. "frame data available") to the target application. At this time, the application can request rendering. When calling "updateTexImage" (i.e. "update texture image"), the previously occupied buffer will be released, and a new buffer will be obtained from the BufferQueue for EGL calls for editing and rendering. Figure 3 The data acquisition unit in uses updateTexImage to obtain frame data.

[0085] In this way, the target application can determine the number of available frame data of each slave player through the number of "onFrameAvailable" (ie, "frame data available") callbacks, and the value is initialized to 0.

[0086] Secondly, edit each frame of data of the master player and the slave player in a loop. Editing is performed only when the "frame data available event" of the master player is received, and when the "frame data available" event of the slave player is received, only the "number of available frame data" value of the slave player is increased by 1. When editing, the external texture data of the master media asset data is first output to FBO. Then the "number of available frame data" of each slave player of the slave media asset data is judged in turn. If its value is greater than 0, the data in the FBO output by the rendering pipeline of the master media asset data corresponding to the master media asset data is used as the input of the rendering pipeline of the slave media asset data corresponding to the current slave media asset data, and video editing is performed. For example: picture-in-picture overlay synthesis processing, and the texture synthesized by the slave player pipeline corresponding to the slave media asset data is output to FBO, and the number of available frame data of the slave player is reduced by 1. If the "number of available frame data" is less than 1, the texture of the current slave player rendering pipeline uses the external slave player frame data texture retained in the previous frame, without updating the external texture object. After that, the "number of available frame data" of the remaining slave players is determined until all slave players have finished processing. Then the data in the last slave player pipeline FBO is output to the screen rendering pipeline, and finally the edited frame data is rendered to the screen or virtual screen. Finally, when all videos are processed, the synchronous rendering of the entire single-window multi-channel video ends.

[0087] It can be seen that for scenes where multiple videos need to be rendered in the same window, rendering is requested only when the frame data of the primary media data arrives. For the secondary media data, the rendering pipeline of the secondary media data is selected on demand based on whether there is available frame data of the secondary media data to update the external texture object, so as to avoid flickering in the display area of ​​the secondary media data.

[0088] From the above, it can be seen that when the image rendering method provided by the present disclosure is applied in a scenario where multiple media data are rendered in a single window, the target is used to create a slave player. Since there is no processing unit in the slave player for parsing the corresponding audio data, the computing resources occupied by multiple slave players when parsing the audio data can be reduced. Since the computing resources occupied by the slave player are reduced when the target application renders multiple media data in a single window, the computing resources occupied by the target application when rendering multiple media data in a single window can be reduced, which solves the problem of how to reduce the computing resources occupied by the terminal device when rendering multiple media data in a single window in the prior art.

[0089] In some possible implementation examples, combined with Figure 4 ,like Figure 5 As shown, the image rendering method provided by the embodiment of the present disclosure also includes: S16 and S17.

[0090] S16. When it is determined according to the first frame rate information that the first frame rate information is a fixed frame rate, frame synchronization is performed on the slave media asset data of the slave player and the master media asset data of the master player according to the media asset clock of the master media asset data.

[0091] In some examples, the first frame rate information corresponds to one or more frame rates, and when the first frame rate information corresponds to only one frame rate, it indicates that the first frame rate information is a fixed frame rate. When the first frame rate information includes multiple frame rates, it indicates that the first frame rate information is not a fixed frame rate.

[0092] Specifically, the system clock is a real clock (such as the clock used by electronic devices), and the media clock is a concept, which is only used to synchronize the playback progress of the current master player to the slave player. For example, when the master player performs internal audio and video synchronization, the master player returns the current playback progress to the target application, and the target application then sets the playback progress of the master player to the slave player, that is, the media clock.

[0093] S17. For each slave player, perform the following operations: remove frame data from the slave media asset data to obtain the removed slave media asset data.

[0094] In some examples, the second frame rate information of the slave media data may differ from the first frame rate information of the main media data, such as: the frame rate corresponding to the second frame rate information is different from the frame rate corresponding to the first frame rate information. In one case, the frame rate corresponding to the second frame rate information is less than or equal to the frame rate corresponding to the first frame rate information. In this case, there is no need to discard the frame data in the slave media data. In another case, the frame rate corresponding to the second frame rate information is greater than the frame rate corresponding to the first frame rate information. In this case, in the same time, since the total number of frame data (such as frame images) in the slave media data is greater than the total number of frame data in the main media data, the frame data in the slave media data needs to be eliminated, so that the total number of frame data in the slave media data in the same time is less than or equal to the total number of frame data in the main media data, to ensure that flickering does not occur.

[0095] In some feasible examples, the target frame data includes the frame data of the main player in the current frame; Figure 5 ,like Figure 6 As shown, the above S14 can be specifically implemented through the following S140.

[0096] S140. When there is no frame data in the current frame of the slave player, perform a target operation corresponding to the target function on both the frame data of the previous frame of the slave player and the frame data of the current frame of the master player to determine a target texture image.

[0097] In some embodiments, the situation where the slave player does not have frame data in the current frame includes the following scenarios:

[0098] 1. The slave player loses a frame in the current frame, resulting in no frame data in the slave player in the current frame.

[0099] Second, in conjunction with the example given in S17 above, since the total number of frame data (such as frame images) in the slave media asset data is greater than the total number of frame data in the master media asset data, the frame data in the slave media asset data needs to be removed so that the total number of frame data in the slave media asset data in the same time period is less than or equal to the total number of frame data in the master media asset data. At this time, it may happen that the slave player does not have frame data in the current frame due to the removal of the frame data of the current frame.

[0100] In some examples, since there is no frame data in the current frame from the player, the frame data (such as frame image) of the host in the current frame can be converted into a first texture image, and then the target operation corresponding to the target function is performed on the first texture image to determine the target texture image. For example, if the target function is a picture-in-picture function and the target operation is picture-in-picture rendering, the target texture image is the second texture image obtained after picture-in-picture rendering is performed on the first texture image.

[0101] In some feasible examples, the target frame data includes the frame data of the master player in the current frame and the frame data of the slave player in the current frame; Figure 5 ,like Figure 6 As shown, the above S14 can be specifically implemented through the following S141.

[0102] S141. When frame data exists in the current frame of the slave player, a target operation corresponding to the target function is performed on both the frame data of the slave player in the current frame and the frame data of the master player in the current frame to determine a target texture image.

[0103] In some examples, when there is frame data in the current frame of the slave player, it is necessary to perform a target operation corresponding to the target function on both the frame data of the slave player in the current frame and the frame data of the master player in the current frame. Prior to this, the frame data (such as a frame image) of the slave player in the current frame can be converted into a third texture image, and the frame data of the master player in the current frame can be converted into a fourth texture image. Afterwards, the third texture image and the fourth texture image are superimposed to obtain a fifth texture image. The target operation corresponding to the target function is performed on the fifth texture image to obtain a sixth texture image, and the target texture image is determined to be the sixth texture image.

[0104] In some possible implementation examples, combined with Figure 6 ,like Figure 7 As shown, the image rendering method provided by the embodiment of the present disclosure further includes: S18 and S19.

[0105] S18. When capturing a frame image for the removed slave media asset data, if there is no frame image in the current frame, it is determined that the slave player corresponding to the removed slave media asset data does not have frame data in the current frame.

[0106] In some examples, the frame image can be captured by Surface Texture from the removed slave media data. In combination with the example given in S140 above, when Surface Texture captures the frame image from the removed slave media data, due to frame loss from the slave player or the removal of the frame data of the current frame, the frame image cannot be captured. In this case, it can be determined that the slave player corresponding to the removed slave media data does not have frame data in the current frame.

[0107] S19: when capturing a frame image for the removed slave media asset data, if a frame image exists in the current frame, it is determined that a slave player corresponding to the removed slave media asset data has frame data in the current frame.

[0108] In some possible implementation examples, combined with Figure 5 ,like Figure 8 As shown, the above S17 can be specifically implemented through the following S170.

[0109] S170. Perform the following operations for each slave player: obtain a first total duration of the rendering duration corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering duration corresponding to each frame of the slave media asset data before the current frame; and remove frame data from the slave media asset data according to the first total duration and the second total duration to obtain the removed slave media asset data.

[0110] In some possible implementation examples, combined with Figure 8 ,like Fig. 9 As shown, the above S170 can be specifically implemented through the following S1700.

[0111] S1700. Perform the following operations for each slave player: obtain a first total duration of the rendering duration corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering duration corresponding to each frame of the slave media asset data before the current frame; if the absolute value of the difference between the first total duration and the second total duration is greater than or equal to the product of a preset threshold and the rendering duration corresponding to each frame of the main media asset data before the current frame, discard the frame data corresponding to the current frame in the slave media asset data; if the absolute value of the difference between the first total duration and the second total duration is less than the product of the preset threshold and the rendering duration corresponding to each frame of the main media asset data before the current frame, retain the frame data corresponding to the current frame in the slave media asset data.

[0112] In some examples, after starting the main player, it is also necessary to monitor events such as frame loss, buffering, and error reporting of the main player. When events such as frame loss, buffering, and error reporting occur in the main player, it is necessary to record the number of frames where the event occurs, so that the first total duration can be determined based on the number of frames. For example, after synchronizing the main media asset data and the slave media asset data, the duration of each frame of the main media asset data is accumulated to obtain the first total duration, such as: the frame rate of the video corresponding to the main media asset data is 25fps, then the rendering duration of each frame of the main media asset data is 1000 / 25=40ms. The duration of each frame of the slave media asset data is accumulated to obtain the second total duration, such as: the frame rate of the video corresponding to the slave media asset data is 50fps, then the rendering duration of each frame of the main media asset data is 1000 / 50=20ms.

[0113] Assume that the preset threshold is After synchronizing the master media data with the slave media data, the current frame is the first frame. At this time, the first total duration is 40ms, and the second total duration is equal to 20ms. Since |40ms-20ms|=20ms, 20ms is equal to 20ms. Therefore, the frame data of the first frame of the media asset data is discarded, and the second frame of the media asset data is taken. At this time, the second total duration is 20ms+20ms=40ms. Since |40ms-40ms|=0ms, 0ms is less than 20ms. The frame data of the second frame of the media asset data is retained.

[0114] The current main media data is the second frame, and the third frame of the secondary media data is taken. At this time, the first total duration is 40ms+40ms=80ms, and the second total duration is 40ms+20ms=60ms. Since |80ms-60ms|=20ms, 20ms is equal to 20ms. Therefore, the frame data of the third frame of the media asset data is discarded, and the fourth frame of the media asset data is taken. At this time, the second total duration is 60ms+20ms=80ms. Since |80ms-80ms|=0ms, 0ms is less than 20ms. Then the fourth frame of the slave media asset data is retained. In this way, the above operation is performed on each slave player, and the removed slave media asset data corresponding to each slave player is obtained. Specifically, the initial values ​​of the first total duration and the second total duration are both 0.

[0115] It is necessary to explain that in order to ensure that the slave media asset data and the main media asset data can have frame data in each frame, when the main media asset data and the slave media asset data are synchronized, for the same frame in the main media asset data, the total number of frame data selected from the slave media asset data meets the following conditions: the absolute value of the difference between the first total duration and the second total duration is less than the sum of the products of the preset threshold and the rendering duration corresponding to each frame of the main media asset data before the current frame (rounded down if it is not an integer), such as: the rendering duration of each frame of the main media asset data is 40ms, and the rendering duration of each frame of the slave media asset data is 30ms. Assuming the preset threshold is After synchronizing the master media data with the slave media data, the current frame is the first frame. At this time, the first total duration is 40ms, and the second total duration is equal to 30ms. Since |40ms-30ms|=10ms, 10ms is less than 20ms. The frame data from the first frame of the media asset data is retained.

[0116] The current main media data is the second frame, and the second frame of the secondary media data is continued. At this time, the first total duration is 40ms+40ms=80ms, and the second total duration is 30ms+30ms=60ms. Since |40ms-60ms|=20ms, 20ms is equal to 20ms. Therefore, the frame data of the second frame of the media data is discarded, and the third frame of the media data is taken. At this time, if there is no frame data in the third frame of the media data (such as frame loss, etc.), the second total duration is equal to 30ms+30ms+30ms=90ms. Since |80ms-90ms| is less than Therefore, the fourth frame of the media asset data is taken. At this time, the second total duration is equal to 30ms+30ms+30ms+30ms=120ms. Since |80ms-120ms| is greater than Therefore, no processing is performed on the fourth frame of the slave media asset data, and the master media asset data is determined to be a non-existent data frame of the slave media asset data corresponding to the second frame.

[0117] After that, the current main media asset data is the third frame, and the fourth frame of the slave media asset data is obtained. In this way, the above operation is performed on each slave player, and the removed slave media asset data corresponding to each slave player is obtained.

[0118] It can be seen that for the main media data with a fixed frame rate, the slave player calculates the rendering time of each frame of the main media data based on the first frame rate information to drop frames or wait for the slave media data, such as: the slave player drops frames in proportion in the multi-video synchronization module or adopts the double-speed playback logic, or the slave player chooses whether to call the rendering pipeline of the slave media data to release the frame buffer of the slave media data based on whether there is frame data in the current frame, so as to avoid the problem of small window flashing. For non-fixed frame rate sources, the slave player performs synchronous rendering based on the system clock.

[0119] In some possible implementation examples, combined with Figure 4 ,like Figure 5 As shown, the image rendering method provided by the embodiment of the present disclosure also includes: S20.

[0120] S20. When it is determined based on the first frame rate information that the first frame rate information is not a fixed frame rate, frame synchronization is performed on the slave media asset data of the slave player and the master media asset data of the master player according to the system clock.

[0121] In some examples, if it is determined based on the first frame rate information that the first frame rate information is not a fixed frame rate, the slave player is set to use the system clock for synchronization. When the system clock is used for synchronization, if the frame data of the slave player is not taken away for a long time, the newly generated frame data will be discarded by the slave player.

[0122] When the user performs operations such as pause or jump on the main player or the slave player, the target rendering cumulative duration and the current rendering cumulative duration of the slave player are reset. In this way, by monitoring the status of the main player, when the status of the main player changes, the target application can synchronously adjust the status of the slave player. For example, when the main player pauses playback due to buffering, the application pauses the playback of the slave player.

[0123] When the target application receives a frame loss event of the main player, the first total duration needs to be increased by the duration of each frame of the main media asset data multiplied by the number of lost frames.

[0124] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0125] The embodiment of the present application can divide the functional modules of the image rendering device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0126] like Fig.10 As shown, an embodiment of the present application provides a structural diagram of a display device 200 , which includes a processor 101 , a communicator 102 , and a display 103 .

[0127] The processor 101 is used to control the communicator 102 to obtain a main media asset data, at least one slave media asset data, and first frame rate information of the main media asset data in response to a selection operation of a target function in a target application; the processor 101 is also used to create a main player for rendering the main media asset data obtained by the communicator 102, and at least one slave player for rendering the slave media asset data obtained by the communicator 102; wherein a slave player renders a slave media asset, and the slave player does not parse the audio data in the slave media asset data; the processor 101 is also used to pre-load the main player After the main media asset data acquired by the communicator 102 is loaded, and the slave player is preloaded with the slave media asset data acquired by the communicator 102, the main player and the slave player are started; the processor 101 is also used to perform a target operation corresponding to a target function on the target frame data, and determine a target texture image; wherein the target frame data includes the frame data of the main player in the current frame, or the target frame data includes the frame data of the main player in the current frame and the frame data of the slave player in the current frame; the processor 101 is also used to render the target texture image, and control the display 103 to display the display image of the current frame.

[0128] In some feasible examples, the processor 101 is further used to perform frame synchronization on the slave media data of the slave player and the main media data of the master player according to the media clock of the main media data when the first frame rate information obtained by the communicator 102 is determined to be a fixed frame rate; the processor 101 is further used to perform the following operation on each slave player: remove the frame data in the slave media data obtained by the communicator 102 to obtain the removed slave media data.

[0129] In some implementable examples, the target frame data includes frame data of the main player in the current frame; the processor 101 is specifically used to perform a target operation corresponding to the target function on the frame data of the slave player in the previous frame and the frame data of the main player in the current frame when there is no frame data of the slave player in the current frame, so as to determine the target texture image.

[0130] In some implementable examples, the target frame data includes frame data of the main player in the current frame and frame data of the slave player in the current frame; the processor 101 is specifically used to perform a target operation corresponding to the target function on the frame data of the slave player in the current frame and the frame data of the main player in the current frame when there is frame data of the slave player in the current frame, so as to determine the target texture image.

[0131] In some feasible examples, the processor 101 is further used to, when capturing a frame image from the media asset data after being removed, if there is no frame image in the current frame, determine that the slave player corresponding to the removed media asset data does not have frame data in the current frame; the processor 101 is further used to, when capturing a frame image from the media asset data after being removed, if there is a frame image in the current frame, determine that the slave player corresponding to the removed media asset data does have frame data in the current frame.

[0132] In some feasible examples, the communicator 102 is specifically used to obtain a first total duration of the rendering duration corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering duration corresponding to each frame of the slave media asset data before the current frame; the processor 101 is specifically used to remove frame data from the slave media asset data according to the first total duration obtained by the communicator 102 and the second total duration obtained by the communicator 102 to obtain the removed slave media asset data.

[0133] In some feasible examples, the processor 101 is specifically used to discard the frame data corresponding to the current frame in the slave media data obtained by the communicator 102 when the absolute value of the difference between the first total duration obtained by the communicator 102 and the second total duration obtained by the communicator 102 is greater than or equal to the product of a preset threshold and the rendering duration corresponding to each frame of the main media data before the current frame; the processor 101 is specifically used to retain the frame data corresponding to the current frame in the slave media data obtained by the communicator 102 when the absolute value of the difference between the first total duration obtained by the communicator 102 and the second total duration obtained by the communicator 102 is less than the preset threshold and the product of the rendering duration corresponding to each frame of the main media data before the current frame.

[0134] In some feasible examples, the processor 101 is further used to perform frame synchronization on the slave media data of the slave player and the master media data of the master player according to the system clock when determining that the first frame rate information is not a fixed frame rate based on the first frame rate information obtained by the communicator 102.

[0135] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and its role will not be repeated here.

[0136] Of course, the display device 200 provided in the embodiment of the present application includes but is not limited to the above modules, for example, the display device 200 may also include a memory 104. The memory 104 may be used to store the program code of the display device 200, and may also be used to store data generated by the display device 200 during operation, such as data in a write request.

[0137] As an example, combining Figure 3 The function implemented by the acquisition unit 210 in the display device 200 is the same as the function of the communicator 101, the function implemented by the processing unit 211 is the same as the function of the processor 102, the function implemented by the display unit 212 is the same as the function of the display 103, and the function implemented by the storage module 213 is the same as the function of the memory 104.

[0138] like Fig.11As shown, the embodiment of the present application also provides a chip system, which can be applied to the display device 200 in the aforementioned embodiment. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 can be the processor in the above-mentioned display device 200. The processor 1501 and the interface circuit 1502 can be interconnected through a line. The processor 1501 can receive and execute computer instructions from the memory of the above-mentioned display device 200 through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the display device 200 can execute the various steps executed by the display device 200 in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0139] The embodiment of the present application also provides a computer-readable storage medium for storing computer instructions executed by the above-mentioned display device 200.

[0140] The embodiment of the present application also provides a computer program product, including computer instructions executed by the above-mentioned display device 200.

[0141] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image rendering method, characterized in that: include: In response to a selection operation of a target function in a target application, acquiring a main media asset data, at least one sub-media asset data, and first frame rate information of the main media asset data; Creating a main player for rendering the main media asset data and at least one slave player for rendering the slave media asset data; wherein one slave player renders one slave media asset data, and the slave player does not parse the audio data in the slave media asset data; When the master player has preloaded the master media asset data, and the slave player has preloaded the slave media asset data, starting the master player and the slave player; Performing a target operation corresponding to the target function on the target frame data to determine a target texture image; wherein the target frame data includes frame data of the master player in the current frame, or the target frame data includes frame data of the master player in the current frame and frame data of the slave player in the current frame; Rendering the target texture image to obtain a display image of the current frame; wherein the rendering includes: When receiving a "frame data available event" from a slave player, add 1 to the "number of available frame data" value of the slave player; When receiving the "frame data available event" of the main player, the external texture data of the main media asset data is output to the frame buffer object FBO; and then the "number of available frame data" of each slave player of the slave media asset data is judged in turn; If the available number of frame data is greater than 0, the data in the FBO output by the rendering pipeline of the master media asset data corresponding to the master media asset data is used as the input of the rendering pipeline of the slave media asset data corresponding to the current slave media asset data, and video editing processing is performed; If the available number of frame data is less than 1, the texture of the current slave player rendering pipeline adopts the external slave player frame data texture retained in the previous frame; the remaining slave players' "available number of frame data" is judged until all slave players are processed; the data in the last slave player pipeline FBO is output to the screen rendering pipeline, and finally the edited and processed frame data is rendered to the screen or virtual screen.

2. The image rendering method according to claim 1, characterized in that: Before performing the target operation corresponding to the target function on the target frame data and determining the target texture image, the method further includes: When it is determined according to the first frame rate information that the first frame rate information is a fixed frame rate, frame synchronization is performed on the slave media asset data of the slave player and the master media asset data of the master player according to the media asset clock of the master media asset data; The following operation is performed for each of the slave players: the frame data in the slave media asset data is removed to obtain the removed slave media asset data.

3. The image rendering method according to claim 2, characterized in that: The target frame data includes the frame data of the main player in the current frame; Performing a target operation corresponding to the target function on the target frame data to determine a target texture image includes: When there is no frame data in the current frame of the slave player, the target operation corresponding to the target function is performed on both the frame data of the slave player in the previous frame and the frame data of the master player in the current frame to determine the target texture image.

4. The image rendering method according to claim 2, characterized in that: The target frame data includes the frame data of the master player in the current frame and the frame data of the slave player in the current frame; Performing a target operation corresponding to the target function on the target frame data to determine a target texture image includes: In the case that the slave player has frame data in the current frame, the target operation corresponding to the target function is performed on both the frame data of the slave player in the current frame and the frame data of the master player in the current frame to determine the target texture image.

5. The image rendering method according to claim 2, characterized in that: The removing frame data from the secondary media asset data to obtain the secondary media asset data after the removal includes: Obtaining a first total duration of the rendering duration corresponding to each frame of the main media asset data before the current frame, and a second total duration of the rendering duration corresponding to each frame of the secondary media asset data before the current frame; According to the first total duration and the second total duration, frame data in the secondary media asset data is removed to obtain the removed secondary media asset data.

6. The image rendering method according to claim 5, characterized in that: The removing frame data from the secondary media asset data according to the first total duration and the second total duration to obtain the secondary media asset data after the removal includes: When the absolute value of the difference between the first total duration and the second total duration is greater than or equal to the product of a preset threshold and a rendering duration corresponding to each frame of the primary media asset data before the current frame, discard the frame data corresponding to the current frame in the secondary media asset data; When the absolute value of the difference between the first total duration and the second total duration is smaller than the product of a preset threshold and the rendering duration corresponding to each frame of the main media data before the current frame, the frame data corresponding to the current frame in the slave media data is retained.

7. The image rendering method according to claim 2, characterized in that: The method further comprises: When it is determined according to the first frame rate information that the first frame rate information is not a fixed frame rate, frame synchronization is performed on the slave media asset data of the slave player and the master media asset data of the master player according to the system clock.

8. An image rendering device, characterized in that: include: A processing unit, configured to control the acquisition unit to acquire a main media asset data, at least one sub-media asset data, and first frame rate information of the main media asset data in response to a selection operation of a target function in a target application; The processing unit is further used to create a main player for rendering the main media asset data acquired by the acquisition unit, and at least one slave player for rendering the slave media asset data acquired by the acquisition unit; wherein one slave player renders one slave media asset data, and the slave player does not parse the audio data in the slave media asset data; The processing unit is further configured to start playing the main player and the slave player when the main player has preloaded the main media asset data acquired by the acquisition unit and the slave player has preloaded the slave media asset data acquired by the acquisition unit; The processing unit is further used to perform a target operation corresponding to the target function on the target frame data to determine a target texture image; wherein the target frame data includes frame data of the master player in the current frame, or the target frame data includes frame data of the master player in the current frame and frame data of the slave player in the current frame; The processing unit is further configured to render the target texture image to obtain a display image of the current frame; wherein the rendering includes: When receiving a "frame data available event" from a slave player, add 1 to the "number of available frame data" value of the slave player; When receiving the "frame data available event" of the main player, the external texture data of the main media asset data is output to the frame buffer object FBO; and then the "number of available frame data" of each slave player of the slave media asset data is judged in turn; If the available number of frame data is greater than 0, the data in the FBO output by the rendering pipeline of the master media asset data corresponding to the master media asset data is used as the input of the rendering pipeline of the slave media asset data corresponding to the current slave media asset data, and video editing processing is performed; If the available number of frame data is less than 1, the texture of the current slave player rendering pipeline adopts the external slave player frame data texture retained in the previous frame; the remaining slave players' "available number of frame data" is judged until all slave players are processed; the data in the last slave player pipeline FBO is output to the screen rendering pipeline, and finally the edited and processed frame data is rendered to the screen or virtual screen.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image rendering method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image rendering method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Method and system for utilizing GPU (Graphic Processing Unit) to realize video playing

    CN102413375A

  • Distributed playing scene picture splicing method for large screen

    CN104125494A

  • Terminal-based real-time beautifying and stream pushing method for video in webcast

    CN106230841A

  • Distributed picture rendering and picture playing control method, apparatus and device

    CN109164997A

  • Video playing method and device, electronic equipment and readable storage medium

    CN111954022A