Video processing method, device, computer equipment, and medium

Abstract

The present application proposes a video processing method, apparatus, computer equipment, medium and product. Applied to a video client, the video client is configured with a first graphics card and a second graphics card; the method includes: obtaining a video frame to be processed, and decoding the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data; storing the decoded frame data in a decoding output buffer, wherein the decoding output buffer is a data storage structure based on a bidirectional linked list to store decoded frame data, allowing the decoding thread and any one of the rendering threads based on the second graphics card to operate the decoding output buffer; obtaining the decoded frame data to be rendered from the bidirectional linked list, and rendering the decoded frame data to be rendered through a rendering thread to obtain a rendered video frame picture. The present application can meet the video playback requirements in special scenarios, improve video processing efficiency and reduce latency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of video processing, and more particularly to a video processing method, a video processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of computer technology, various videos are ubiquitous in all aspects of daily life. In the video field, the complete video processing process on the client side usually includes: decoding the video frame, rendering the decoded result, and outputting and playing the rendered video image.

[0003] Currently, conventional video processing on the client side usually relies on a single graphics card to complete decoding and rendering processes, resulting in low processing efficiency. Summary of the Invention

[0004] The embodiments of the present application propose a video processing method, apparatus, computer equipment, medium and product, which support the decoding process and rendering process to be executed separately by two graphics cards, thereby improving video processing efficiency and reducing the delay of video frame processing.

[0005] In one aspect, an embodiment of the present application provides a video processing method, which is applied to a video client, wherein the video client is configured with a first graphics card and a second graphics card; the method includes:

[0006] Obtaining a video frame to be processed, and decoding the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data;

[0007] Storing the decoded frame data in a decode output buffer, wherein the decode output buffer stores the decoded frame data in a data storage structure based on a bidirectional linked list, and allowing either a decoding thread or a rendering thread based on the second graphics card to operate the decode output buffer;

[0008] The decoded frame data to be rendered is obtained from the bidirectional linked list, and the decoded frame data to be rendered is rendered by the rendering thread to obtain a rendered video frame picture.

[0009] In one aspect, an embodiment of the present application provides a video processing device, characterized in that it is applied to a video client, the video client being configured with a first graphics card and a second graphics card; the device includes:

[0010] An acquisition unit, configured to acquire a video frame to be processed;

[0011] A processing unit, configured to decode the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data;

[0012] a storage unit, configured to store decoded frame data in a decoded output buffer, wherein the decoded frame data is stored in the decoded output buffer using a data storage structure based on a bidirectional linked list, and wherein either a decoding thread or a rendering thread based on the second graphics card is allowed to operate the decoded output buffer;

[0013] The acquisition unit is further used to obtain the decoded frame data to be rendered from the bidirectional linked list;

[0014] The processing unit is used to perform rendering processing on the decoded frame data to be rendered through the rendering thread to obtain a rendered video frame picture.

[0015] In a possible implementation, the processing unit is further configured to perform the following operations:

[0016] Create a linked list head and a linked list tail. The linked list head is used to store the pointer of the first node stored in the decoded output buffer, and the linked list tail is used to store the pointer of the last node stored in the decoded output buffer.

[0017] Generate a doubly linked list in the initial state based on the created linked list head and linked list tail;

[0018] Among them, in the initial state, the node pointers stored corresponding to the head of the linked list and the tail of the linked list in the bidirectional linked list are both empty.

[0019] In a possible implementation, the processing unit decodes the video frame to be processed based on the decoding thread started by the first graphics card, and after obtaining the decoded frame data, is further configured to perform the following operations:

[0020] Obtain write permission to the decoding output cache through thread locking of the decoding thread;

[0021] Obtaining a target node according to the decoded frame data obtained by the decoding process, and adding the target node to the bidirectional linked list in the initial state;

[0022] After determining that the target node is successfully added to the doubly linked list, the decoding thread is called to release the lock.

[0023] In one possible implementation, the target node is used to store: decoded frame data, a front pointer for pointing to a previous node, and a back pointer for pointing to a next node; the previous node refers to a node that is located before the target node in the decoding output buffer, and the next node refers to a node that is located after the target node in the decoding output buffer.

[0024] In one possible implementation, the processing unit adds the target node to the doubly linked list in the initial state to perform the following operations:

[0025] The forward pointer used by the target node to point to the previous node is pointed to the head of the doubly linked list, and the backward pointer used by the target node to point to the next node is pointed to the first node pointed to by the head of the linked list;

[0026] Point the linked list head to the target node to add the target node to the doubly linked list;

[0027] If the node pointed to by the tail of the doubly linked list is empty, point the tail of the linked list to the target node.

[0028] In one possible implementation, the processing unit obtains decoded frame data to be rendered from a bidirectional linked list, and performs the following operations:

[0029] The rendering thread queries whether there is an available node in the doubly linked list. If so, the rendering thread obtains the read permission of the decoded output cache by thread locking.

[0030] Remove the available nodes obtained from the query from the doubly linked list;

[0031] If it is determined that the available node has been successfully removed from the doubly linked list, the lock release operation is performed through the rendering thread.

[0032] In one possible implementation, if the node pointed to by the tail of the doubly linked list is not empty, it is determined that there is an available node in the doubly linked list, and the available node is the last node in the doubly linked list; the processing unit removes the available node obtained by the query from the doubly linked list, and performs the following operations:

[0033] If there are at least two nodes in the doubly linked list, point the tail pointer of the doubly linked list to the node before the last node in the doubly linked list to remove the last node from the doubly linked list; or,

[0034] If the available node is the only node in the doubly linked list, the node pointed to by the pointer at the end of the doubly linked list is set to empty to remove the last node from the doubly linked list.

[0035] In a possible implementation, the processing unit is further configured to perform the following operations:

[0036] Detect whether the video client is configured with an integrated graphics card. If so, use the integrated graphics card as the first graphics card and the independent graphics card as the second graphics card.

[0037] If not, the working graphics card is used as the second graphics card, and the graphics card other than the working graphics card is used as the first graphics card.

[0038] In a possible implementation, the processing unit is further configured to perform the following operations:

[0039] Determine whether the second graphics card meets the idle condition. If so, call the rendering thread of the second graphics card to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation;

[0040] If not, the decoding thread of the first graphics card is called to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation.

[0041] On the one hand, an embodiment of the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned video processing method.

[0042] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned video processing method.

[0043] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described video processing method.

[0044] In an embodiment of the present application, a video client is configured with a first graphics card and a second graphics card. First, the first graphics card can obtain the video frame to be processed, and decode the video frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data; then, the decoded frame data can be stored in the decoding output buffer, which is a data storage structure based on a bidirectional linked list to store the decoded frame data, allowing the decoding thread and any thread based on the rendering thread of the second graphics card to operate the decoding output buffer; finally, the decoded frame data to be rendered is obtained from the bidirectional linked list, and the rendering thread performs rendering processing on the decoded frame data to be rendered to obtain a rendered video frame picture. It can be seen that the present application divides the decoding and rendering tasks of the video frame into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the working mode of the dual graphics card can effectively alleviate the problem that the video client's video playback frame rate is lower than the video stream reception frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technical objects in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a schematic diagram of the principle of a video processing solution provided in an embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of the architecture of a video processing system provided by an embodiment of the present application;

[0048] Figure 3 This is a flow chart of a video processing method provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the structure of a bidirectional linked list provided in an embodiment of the present application;

[0050] Figure 5 This is a flow chart of another video processing method provided in an embodiment of the present application;

[0051] Figure 6 This is a schematic diagram of a decoding thread workflow provided by an embodiment of the present application;

[0052] Figure 7 This is a schematic diagram of a process for inserting a node into a bidirectional linked list provided by an embodiment of the present application;

[0053] Figure 8 This is a workflow diagram of a rendering thread provided in an embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of a process for removing a node from a bidirectional linked list provided in an embodiment of the present application;

[0055] Figure 10 This is a scene diagram of a game screen provided in an embodiment of the present application;

[0056] Figure 11 This is a scene diagram of an image processing method for cloud gaming provided by an embodiment of the present application;

[0057] Figure 12 is a structural diagram of a video processing device provided in an embodiment of the present application;

[0058] Figure 13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0060] 1. Principle of video processing solution:

[0061] This application provides a video processing solution that supports assigning decoding tasks to the first graphics card and rendering tasks to the second graphics card when the video client is equipped with two graphics cards (i.e., display interface cards, also known as display adapters, which are hardware used to convert display information required by the computer system and provide the converted display information to the monitor to control the display of the corresponding information). This solution can be applied to real-time video streaming playback scenarios such as cloud gaming, video calls, and game live streaming, and can improve video processing efficiency and reduce playback latency. See [Note: The following sentences appear to be unrelated and should be omitted.] Figure 1 , Figure 1 This is a schematic diagram of the principle of a video processing solution provided by the embodiment of the present application. Figure 1 The principle of the image processing solution provided by this application is roughly explained:

[0062] like Figure 1 As shown, the video client is configured with two graphics cards (integrated graphics card and independent graphics card; or, independent graphics card and independent graphics card). It should be noted that this application will give priority to specifying the CPU (Central Processing Unit) integrated graphics card to handle decoding tasks, and the independent graphics card to handle rendering tasks. If the video client is not configured with an integrated graphics card, the rendering tasks will be handled based on the main graphics card (default working graphics card), and the other graphics card will handle decoding tasks. Since the rendering capability of the integrated graphics card is relatively weak, this application gives priority to using the integrated graphics card to perform decoding tasks to improve the efficiency of video processing. Figure 1As shown, graphics card A is responsible for decoding tasks (hereinafter referred to as the first graphics card), and graphics card B is responsible for rendering tasks (hereinafter referred to as the second graphics card). ① When receiving video stream data (which may include one or more video frames), graphics card A obtains the video frame to be processed from the video stream data, and decodes the video frame to be processed based on the decoding thread started by graphics card A to obtain decoded frame data; ② Graphics card A stores the decoded frame data in the decoding output cache, which is implemented based on a data storage structure of a bidirectional linked list. The bidirectional linked list is used to indicate that at the same time, any one of the decoding thread and the rendering thread started by the second graphics card is allowed to operate the decoding output cache; ③ Graphics card B obtains the decoded frame data to be rendered from the decoding output cache (bidirectional linked list), and renders the decoded frame data to be rendered based on the rendering thread to obtain the rendered video frame picture. Subsequently, the rendered video frame picture can be output in the video client.

[0063] It can be seen that the present application divides the decoding and rendering tasks of video frames into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the dual graphics card working mode can effectively alleviate the problem of the video client having a video playback frame rate lower than the video stream receiving frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing.

[0064] Next, we will introduce in detail the key technical terms involved in the above-mentioned video processing solutions:

[0065] 2. Related technical terms involved in video processing solutions:

[0066] This application mainly provides cloud technology (cloud gaming and cloud computing) and blockchain technology. In order to enable a clearer understanding of the technical solutions provided by the embodiments of this application, the technical terms involved in the embodiments of this application are introduced below.

[0067] (1) Cloud technology:

[0068] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.

[0069] The video processing solution of this application mainly relates to cloud gaming and cloud computing technologies in the field of cloud technology. That is, the video processing solution of this application can be applied to application scenarios of real-time video streaming, such as cloud gaming and game live streaming. In the application scenarios of real-time video streaming, a large number of cloud computing technologies are involved. Therefore, the following is a detailed introduction to cloud gaming and cloud computing technologies respectively:

[0070] ① Cloud gaming:

[0071] The video processing solution provided in this application can be applied to cloud gaming scenarios. So-called cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud gaming scenario, the game is not played on the player's game terminal, but on a cloud server, which renders the game scene / game screen into a video and audio stream (audio and video stream) and transmits it to the game client over the network. The game client does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain the operation instructions entered by the game player and send them to the cloud server.

[0072] Specifically, in a cloud gaming scenario, when the cloud server pushes the game video stream of the target cloud game to the video client, the first graphics card in the video client can obtain the video frame to be processed from the game video stream, and decode the video frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data, and store the decoded frame data in the decoding output cache. Next, the second graphics card obtains the decoded frame data to be rendered from the bidirectional linked list, and renders the decoded frame data to be rendered through the rendering thread to obtain the rendered game screen. Subsequently, the rendered game screen can be output in the video client. Due to the use of dual graphics card video processing, network freezes, delays and other problems in cloud gaming scenarios can be guaranteed, thereby improving the gaming experience.

[0073] ② Cloud computing:

[0074] In this application, the decoding process of the video frame (decoding the video frame to be processed through the decoding thread of the first graphics card) and the rendering process (rendering the decoded frame data to be rendered through the rendering thread) both involve a large amount of data calculation and data storage services. The above processes require a lot of computer operating costs. Therefore, this application can implement corresponding video decoding and picture rendering processes based on cloud computing technology.

[0075] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides resources is called the "cloud." To users, the resources in the "cloud" appear to be infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis. As a provider of the basic capabilities for cloud computing, a cloud computing resource pool (referred to as a cloud platform), generally referred to as an IaaS (Infrastructure as a Service) platform, is established. Various types of virtual resources are deployed in the cloud computing resource pool for external customers to choose and use.

[0076] (2) Blockchain technology:

[0077] Blockchain is a new application model for computer technologies such as distributed data storage, peer-to-peer (P2P) transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database consisting of a series of data blocks (also called blocks) linked together using cryptographic methods. Each block contains information about a batch of online transactions, which is used to verify the validity of that information (to prevent counterfeiting) and generate the next block. Blockchain uses cryptography to ensure that data cannot be tampered with or forged.

[0078] In this application, the decoded frame data, decoded output buffer, and rendered video frame data involved in the video processing process can be sent to the blockchain for storage. Based on the blockchain's tamper-proof and traceable characteristics, the security of the video processing process can be improved and the leakage of video data can be avoided.

[0079] It is particularly important to note that in the subsequent specific implementation methods of this application, relevant data such as object information is involved. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0080] Combined with the above description of the video processing solution provided by this application and the related technical terms involved, next, combined with Figure 2 The architecture of the video processing system provided in the embodiments of the present application is described in detail.

[0081] See Figure 2 , Figure 2 : This is a schematic diagram of the architecture of a video processing system provided by an embodiment of the present application. The architecture diagram of the image processing system may include at least: a terminal device cluster and a cloud server 204. Among them, the terminal device cluster includes: terminal device 201, terminal device 202, and terminal device 203, etc.; a communication connection can be established between the cloud server 204 and any terminal device in the terminal device cluster (such as terminal device 201) based on a wired or wireless method. For example, data (such as video stream data, which may include multiple video frames to be processed) can be transmitted between the cloud server 204 and the terminal device 201 based on a real-time communication protocol. It should be understood that a video client is running in any terminal device in the terminal device cluster, and the video client is used to decode, render, and play the video stream data transmitted by the cloud server 204 to output the decoded and rendered video frame images in the video client. Among them, the cloud server 204 can be a video database that stores various video source data, such as game videos, live videos, etc. According to different business scenarios, the terminal device can download or request response video stream data from the cloud server 204. It should be noted that the number of terminal devices in the terminal device cluster is for example only, and the embodiments of the present application do not limit the number of terminal devices.

[0082] The terminal device 201, terminal device 202, and terminal device 203 mentioned in this application may include but are not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile Internet devices (MIDs), intelligent voice interaction devices, vehicle-mounted terminals, roadside devices, aircraft, wearable devices, smart home appliances, or wearable devices with video processing functions such as smart watches, smart bracelets, pedometers, etc.

[0083] The cloud server 204 mentioned in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms, etc.

[0084] Next, taking any terminal device (e.g., terminal device 201) in the image processing system as an example, the data interaction process between the terminal device 201 and the cloud server 204 is described accordingly:

[0085] Taking the game live broadcast scenario as an example, after receiving the operation instruction input by the target game player, the terminal device 201 can send the operation instruction to the cloud server 204. After receiving the operation instruction for the target cloud game, the cloud server 204 can obtain the game video stream of the target cloud game and push the game video stream to the terminal device 201. In this application, the terminal device 201 needs to be configured with a first graphics card and a second graphics card. The terminal device 201 can obtain the video frame to be processed from the game video stream, and decode the video frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data; then, the terminal device 201 can store the decoded frame data in the decoding output buffer. The decoding output buffer is a data storage structure based on a bidirectional linked list to store decoded frame data, allowing the decoding thread and any one of the rendering threads based on the second graphics card to operate the decoding output buffer; finally, the terminal device 201 obtains the decoded frame data to be rendered from the bidirectional linked list, and renders the decoded frame data to be rendered through the rendering thread to obtain the rendered video frame picture (game picture).

[0086] In one possible implementation, the video processing system provided by the embodiments of this application can be deployed on a blockchain. For example, each terminal device included in a terminal device cluster can be treated as a node device of the blockchain, together forming a blockchain network. Therefore, the video processing process in this application can be executed on a blockchain, which can not only ensure the fairness and impartiality of the video processing process, but also make the video processing process traceable, thereby improving the security of the video processing process.

[0087] It can be understood that the video processing system described in the embodiment of the present application is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0088] Based on the foregoing description of the video processing solution and the video processing system of the present application, the specific embodiments involved in the video processing solution will be described in detail with reference to the accompanying drawings.

[0089] See Figure 3 , Figure 3 This is a flow chart of a video processing method provided by an embodiment of the present application. Figure 2 The terminal device in the video processing system shown in the figure has a video client running in the terminal device, and the video client is configured with a first graphics card and a second graphics card. The video processing method mainly includes but is not limited to the following steps S301 to S303:

[0090] S301: Acquire a video frame to be processed, and decode the video frame to be processed by a decoding thread of a first graphics card to obtain decoded frame data.

[0091] In the embodiment of the present application, the video frame to be processed can be obtained from a segment of video stream data. Specifically, the video frame to be processed can be any frame in a segment of video stream data, or the video frame to be processed can be the first frame in a segment of video stream data, or the video frame to be processed can be a specified frame in a segment of video stream data. The embodiment of the present application does not specifically limit this. In addition, the video frame can include but is not limited to: game screen frames in a game scene, live screen frames in a live broadcast scene, video screen frames in a video scene, etc.

[0092] In one possible implementation, it is detected whether the video client is configured with an integrated graphics card. If so, the integrated graphics card is used as the first graphics card, and the independent graphics card is used as the second graphics card. If not, the working graphics card is used as the second graphics card, and the other graphics card other than the working graphics card is used as the first graphics card. Normally, when the video client is configured with dual graphics cards (i.e., two graphics cards), the two graphics cards can be an integrated graphics card-independent graphics card, and an independent graphics card-independent graphics card, respectively. Since the rendering capability of the integrated graphics card is relatively weak, the embodiment of the present application will give priority to determining whether the video client is configured with an integrated graphics card. If so, the integrated graphics card is used as the first graphics card, and the other independent graphics card is used as the second graphics card; when the video client is not configured with an integrated graphics card, the independent graphics card of the main graphics card (working graphics card) can be used as the second graphics card, and the remaining independent graphics card can be used as the first graphics card. Based on this method, it is possible to ensure that each graphics card can give full play to its respective capabilities and improve the efficiency of video processing.

[0093] The first graphics card is used to start a decoding thread to perform decoding tasks, and the second graphics card is used to start a rendering thread to perform rendering tasks. Specifically, the decoding thread of the first graphics card can decode the video frames to be processed to obtain decoded frame data. The decoding process may include: converting the video frame format and interpreting the video information.

[0094] S302: Storing the decoded frame data in a decoded output buffer. The decoded output buffer stores the decoded frame data in a data storage structure based on a bidirectional linked list, allowing either a decoding thread or a rendering thread based on the second graphics card to operate the decoded output buffer.

[0095] Specifically, the decoding frame data is stored in the decoding output buffer by the decoding thread of the first graphics card. Wherein, the bidirectional linked list means that any node included in it carries a bidirectional pointer, for example, see Figure 4 , Figure 4 This is a schematic diagram of the structure of a bidirectional linked list provided in an embodiment of the present application. Figure 4 As shown, any node in the doubly linked list (e.g., node b) has a previous pointer and a next pointer. The previous pointer points to the previous node (node ​​a), and the next pointer points to the next node (node ​​c). Similarly, the same is true for nodes a, c, and d, forming a doubly linked list.

[0096] In one possible implementation, a bidirectional linked list in an initial state can be generated. The specific process includes: creating a linked list header and a linked list tail, the linked list header is used to store the pointer of the first node stored in the decoded output buffer, and the linked list tail is used to store the pointer of the last node stored in the decoded output buffer; based on the created linked list header and linked list tail, a bidirectional linked list in an initial state is generated; wherein, in the bidirectional linked list in the initial state, the node pointers stored in the linked list header and the linked list tail are both empty. Subsequently, after adding the node to the bidirectional linked list in the initial state, a complete bidirectional linked list including multiple nodes can be generated. wherein, any node in the bidirectional linked list stores: the video frame decoding data corresponding to the node, the previous item pointer of the node, and the next item pointer of the node.

[0097] In one possible implementation, after the decoding thread started by the first graphics card decodes the video frame to be processed and obtains the decoded frame data, it also includes: first, obtaining the write permission of the decoding output cache through the decoding thread based on thread locking; then, obtaining the target node according to the decoded frame data obtained by the decoding process, and adding the target node to the bidirectional linked list in the initial state; finally, after determining that the target node is successfully added to the bidirectional linked list, calling the decoding thread to execute the lock release operation.

[0098] The target node is used to store: decoded frame data, a forward pointer to the previous node, and a backward pointer to the next node; the forward node refers to the node before the target node in the decoded output buffer, and the next node refers to the node after the target node in the decoded output buffer. Figure 4 As shown, for example, if the target node is node b, then the previous node of the target node is node a, and the next node of the target node is node c; for another example, if the target node is node c, then the previous node of the target node is node b, and the next node of the target node is node d.

[0099] In addition, in the embodiment of the present application, allowing any one of the decoding thread and the rendering thread based on the second graphics card to operate the decoding output cache means: at the same time, allowing any one of the decoding thread or the rendering thread to operate the decoding output cache, that is, the decoding thread and the rendering thread cannot perform operations on the decoding output cache at the same time. Specifically, the present application can allow any one of the decoding thread and the rendering thread based on the second graphics card to operate the decoding output cache by thread locking. The so-called thread locking may include: a locking position, which represents that once one of the threads (such as the decoding thread) acquires the lock, the other threads (rendering threads) will be blocked to this position. When the locked thread (decoding thread) completes execution and releases the lock, the other threads (rendering threads) will continue to execute backward according to the position at the time of blocking. For example, the decoding thread locks LOCK, and the decoding thread releases the lock release. In addition, the decoding thread operates the decoding output cache mainly including writing the decoded frame data from the decoding output cache, and the decoded frame data is packaged in the node, so the decoding thread is used to insert the node into the decoding output cache (bidirectional linked list); the rendering thread operates the decoding output cache mainly including reading the decoded frame data from the decoding output cache to complete the rendering operation, that is, the rendering thread is used to read the decoded frame data in the node from the decoding output cache (bidirectional linked list) and perform rendering operations on the read decoded frame data.

[0100] S303: Obtain the decoded frame data to be rendered from the bidirectional linked list, and perform rendering processing on the decoded frame data to be rendered by the rendering thread to obtain a rendered video frame picture.

[0101] It should be understood that the picture content of the video frame in the embodiment of the present application may vary according to different business scenarios. Specifically, if the video frame to be processed is a game frame in a game scene, the rendered video frame picture may include the game picture; if the video frame to be processed is a live frame in a live scene, the rendered video frame picture may include the live picture; if the video frame to be processed is an image frame in a video scene, the rendered video frame picture may include the video picture.

[0102] In one possible implementation, the decoded frame data to be rendered is obtained from a doubly linked list, specifically including: first, querying whether there is an available node in the doubly linked list through the rendering thread; if so, obtaining read permission for the decoded output cache based on thread locking; then, removing the available node obtained from the query from the doubly linked list; finally, if it is determined that the available node has been successfully removed from the doubly linked list, executing the lock release operation through the rendering thread. The decoded frame data stored in the available node is the decoded frame data to be rendered obtained from the doubly linked list. Subsequently, the decoded frame data included in the available node is rendered by the rendering thread to obtain a rendered video frame.

[0103] In one possible implementation, a determination is made as to whether the second graphics card meets an idle condition. If so, the rendering thread of the second graphics card is called upon to retrieve the decoded frame data to be rendered from a doubly linked list and perform the rendering operation. If not, the first graphics card is called upon to retrieve the decoded frame data to be rendered from the doubly linked list and perform the rendering operation. The idle condition may include whether the rendering thread of the second graphics card is idle. If so, the second graphics card is determined to meet the idle condition. In this manner, the second graphics card can be kept operational, and if rendering processing is required, temporary rendering processing can be performed based on the first graphics card to ensure efficient video rendering.

[0104] In an embodiment of the present application, a video client is configured with a first graphics card and a second graphics card. First, the first graphics card can obtain the video frame to be processed, and decode the video frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data; then, the decoded frame data can be stored in the decoding output buffer, which is a data storage structure based on a bidirectional linked list to store the decoded frame data, allowing the decoding thread and any thread based on the rendering thread of the second graphics card to operate the decoding output buffer; finally, the decoded frame data to be rendered is obtained from the bidirectional linked list, and the rendering thread performs rendering processing on the decoded frame data to be rendered to obtain a rendered video frame picture. It can be seen that the present application divides the decoding and rendering tasks of the video frame into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the working mode of the dual graphics card can effectively alleviate the problem that the video client's video playback frame rate is lower than the video stream reception frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing.

[0105] See Figure 5 , Figure 5 This is a flow chart of another video processing method provided by the embodiment of the present application. Figure 2 The terminal device in the video processing system shown in the figure has a video client running in the terminal device, and the video client is configured with a first graphics card and a second graphics card. The video processing method mainly includes but is not limited to the following steps S501 to S505:

[0106] S501: The decoding thread obtains a video frame to be processed.

[0107] S502: Decoding the video frame to be processed by a decoding thread of the first graphics card to obtain decoded frame data.

[0108] S503: The decoding thread stores the decoded frame data into the decoding output buffer.

[0109] In an embodiment of the present application, after completing the decoding of a video frame, the decoding thread will create a node for the video frame. This node contains the decoded data of the video frame, a front pointer for pointing to the previous node, and a back pointer for pointing to the next node. Before placing the node into the decoding output cache, the decoding thread will obtain the operation permission of the decoding output cache by locking. After the locking is successful, the front pointer pointing to the previous node of the node to be inserted points to the head of the linked list, and the back pointer pointing to the next node points to the first node pointed to by the head of the linked list. The decoding thread finally points the head of the linked list to the node to be inserted. In this way, the node to be inserted becomes the first node of the linked list. If the node pointed to by the tail of the linked list is empty at this time, it means that this is the first node of the linked list. At this time, the tail of the linked list will also be set to point to this node. After the node is inserted, the decoding thread releases the lock.

[0110] Specifically, see Figure 6 , Figure 6 This is a schematic diagram of a decoding thread workflow provided by an embodiment of the present application. Figure 6 As shown, the workflow of the decoding thread mainly includes the following steps:

[0111] ①Get the video data stream.

[0112] Specifically, the decoding thread can receive a video data stream from a cloud server, where the video data stream can include, for example, a game video stream in a game scenario, a live video stream in a live broadcast scenario, an audio and video stream in a video scenario, etc. The embodiments of this application do not specifically limit the specific scenarios involved in the video data stream. It is understood that the video data stream can be understood as including at least one video frame streaming media data.

[0113] ② Determine whether a video frame is received.

[0114] Specifically, the decoding thread can obtain a video frame from the video data stream as the video frame to be processed. For example, any video frame in the video data stream can be used as the video frame to be processed; another example is the first frame in the video data stream; another example is the specified frame in the video data stream. If it is determined that the video frame to be processed has been received, step 3 is triggered; if the video frame to be processed has not been received, step 1 is triggered.

[0115] ③ The decoding thread decodes the video frames to be processed.

[0116] ④ Create a target node containing the decoded frame data.

[0117] The target node is used to store: decoded frame data, a front pointer pointing to a previous node, and a back pointer pointing to a next node; the previous node refers to the node before the target node in the decoded output buffer, and the next node refers to the node after the target node in the decoded output buffer.

[0118] ⑤Lock and obtain write permission.

[0119] Specifically, thread locking can be used to obtain operation permissions for the decoded output cache, where the decoding thread is primarily used to write decoded data (decoded frame data) to the decoded output cache. Thread locking can include: a lock position, which represents the position at which other threads (rendering threads) will be blocked once a thread (e.g., a decoding thread) acquires the lock. When the locked thread (decoding thread) completes execution and releases the lock, other threads (rendering threads) will continue to execute backward based on the position at which they were blocked. For example, the decoding thread locking method can specifically include thread locking LOCK.

[0120] ⑥Add the target node to the doubly linked list.

[0121] In one possible implementation, the decoding thread adds the target node to the doubly linked list, which may include the following steps: first, pointing the front pointer of the target node used to point to the previous node to the head of the doubly linked list, and pointing the back pointer of the target node used to point to the next node to the first node pointed to by the head of the linked list; then, pointing the head of the linked list to the target node to add the target node to the doubly linked list; finally, if the node pointed to by the tail of the doubly linked list is empty, pointing the tail of the linked list to the target node.

[0122] See Figure 7 , Figure 7 This is a flow chart of inserting a node into a bidirectional linked list provided by an embodiment of the present application. Figure 7 As shown, if the node to be inserted is node e, the front pointer of node e can be pointed to the head of the linked list, and the back pointer of node e can be pointed to the first node (node ​​a) pointed to by the head of the linked list. Finally, the head of the linked list is pointed to node e. In this way, node e can be added to the doubly linked list, and node e becomes the first node in the doubly linked list.

[0123] ⑦Release the lock.

[0124] Specifically, after determining that the target node has been successfully added to the bidirectional linked list, the decoding thread may execute an operation of releasing the lock.

[0125] S504: The rendering thread obtains decoded frame data to be rendered from the bidirectional linked list.

[0126] S505: Rendering is performed on the decoded frame data to be rendered by a rendering thread to obtain a rendered video frame.

[0127] In this embodiment of the present application, the decoded output buffer uses a doubly linked list-based data storage structure to store decoded frame data, allowing either the decoding thread or the rendering thread based on the second graphics card to access the decoded output buffer. During idle time, the rendering thread continuously queries the decoded output buffer for decoded frame data to be rendered by checking whether the pointer at the end of the linked list is null. If it is not null, indicating that there is decoded frame data to be rendered in the decoded output buffer, the rendering thread attempts to lock the buffer. Upon successful locking, the rendering thread removes the node from the linked list. Specifically, this is accomplished by simply setting the end of the linked list to the node immediately preceding the last node. In a special case, if the node is the only node in the linked list, the pointer to the previous node points to the beginning of the linked list. In this case, the rendering thread sets the node pointed to by the end of the linked list to null. After the removal operation is complete, the rendering thread releases the lock and begins rendering the video data in the removed node. After rendering is complete, the rendering thread destroys the node and begins a new round of cache query operations.

[0128] Specifically, see Figure 8 , Figure 8 This is a workflow diagram of a rendering thread provided by an embodiment of the present application. Figure 8 As shown, the workflow of the rendering thread mainly includes the following steps:

[0129] (1) The rendering thread starts.

[0130] (2) Determine whether there is available node data.

[0131] Specifically, the rendering thread continuously queries the decoding output cache for video data to be rendered when it is idle. The idle state mentioned here is to ensure that the second graphics card is in working state at any time. That is, as long as the rendering thread is in idle state, the rendering thread is triggered to query the decoding output cache for data to be rendered. If so, the rendering of the video data is executed. After rendering, it goes to query the decoding output cache again, and so on. In this way, the rendering thread can be ensured to be in a high-intensity working state, thereby improving the efficiency of video processing.

[0132] In a specific implementation, the rendering thread determines whether there is available node data, which may include: if the node pointed to by the tail of the doubly linked list is not empty, determining that there is an available node in the doubly linked list, and the available node is the last node in the doubly linked list, then triggering the execution of step (3); if there is no available node in the doubly linked list, then triggering the execution of step (1).

[0133] (3) Lock and obtain read permission.

[0134] Specifically, thread locking can be used to obtain operation permissions for the decoded output cache, where the rendering thread is mainly used to read decoded data (decoded frame data) from the decoded output cache. Thread locking can include: a lock position, which represents the position at which other threads (decoding threads) will be blocked once a thread (such as a rendering thread) acquires the lock. When the locked thread (rendering thread) completes execution and releases the lock, other threads (decoding threads) will continue to execute backward based on the position at the time of blocking. For example, the rendering thread locking method can specifically include thread locking LOCK.

[0135] (4) Remove the last node from the doubly linked list.

[0136] In specific implementation, removing the last node from the doubly linked list through the rendering thread may specifically include: if there are at least two nodes in the doubly linked list, pointing the pointer at the end of the doubly linked list to the node before the last node in the doubly linked list, so as to remove the last node from the doubly linked list; or, if the available node is the only node in the doubly linked list, setting the node pointed to by the pointer at the end of the doubly linked list to empty, so as to remove the last node from the doubly linked list.

[0137] See Figure 9 , Figure 9 This is a flow chart of removing a node from a bidirectional linked list provided by an embodiment of the present application. Figure 9 As shown, if the node to be removed is node d, the pointer at the end of the doubly linked list is pointed to the node before the last node in the doubly linked list (node ​​c). In this way, node d can be removed from the doubly linked list, and the removed node d is an available node. The decoded frame data stored in the available node is the decoded frame data to be rendered. Optionally, if the doubly linked list only includes node d, that is, node d is the only node in the doubly linked list, then after removing node d, the node pointed to by the end of the linked list is set to empty.

[0138] (5) Release the lock.

[0139] Specifically, when it is determined that the available node is successfully removed from the bidirectional linked list, the rendering thread executes an operation of releasing the lock.

[0140] (6) Decoded frame data in the rendering node.

[0141] In this application, if the video frame is a game frame in a game scene, the specific process of decoding the frame data in the rendering thread rendering node may include: obtaining the game texture data included in the decoded frame data, rendering the game screen based on the game texture data, and obtaining the rendered game screen. Among them, the game texture data refers to the data used to render the game screen, that is, the data used to render the game content in the target cloud game to obtain the game screen. The data format of the game texture data may include a two-dimensional array, a three-dimensional array, etc., which is used to store the color, position, background environment and other parameters of the corresponding game screen. Among them, the game content may include scene content and user interface content. The scene content refers to the content in the game such as trees, roads, buildings, etc., and the user interface content refers to the content that can be operated by the user in the game, such as the virtual keyboard and virtual controls in the game. Please refer to Figure 10 , Figure 10 This is a scene diagram of a game screen provided by an embodiment of the present application. Figure 10 In the game screen shown, street lights (401), traffic signs (402), grass (403), buildings (404), etc. are scene contents, and virtual controls such as "Start Game" (405) and "Exit Game" (406) are user interface contents. In addition, the game screen refers to the screen that can be displayed in the game client (for example, Figure 10 The game screen is usually obtained by superimposing the game scene screen and the user interface screen. The game scene screen is rendered by the scene content, and the user interface screen is rendered by the user interface content.

[0142] (7) Destroy the node.

[0143] Specifically, after the rendering thread completes rendering of the decoded frame data in a node, it can then query the next node from the decoded output cache and execute the data rendering process of the next node, and so on, until the rendering of the decoded frame data to be rendered in the decoded output cache is completed.

[0144] Based on the above description, it should be understood that the video processing solution provided by this application can be applied to various types of cloud gaming scenarios. Figure 11 , Figure 11 This is a scene diagram of an image processing method for cloud gaming provided by an embodiment of the present application. Figure 11As shown, in a cloud gaming scenario, ① the user can enter game operation instructions for the target cloud game in the video client (game client). For example, game operation instructions may include: rotation instructions, jump instructions, movement instructions, etc. After the game client responds to the game operation instruction, it sends the game operation instruction to the cloud server. ② After receiving the game operation instruction for the target cloud game, the cloud server can query the game video stream that matches the target cloud game and return the game video stream (video data stream) to the game client. ③ The game client is configured with a first graphics card and a second graphics card. After receiving the game video stream, the game client can obtain the game frame to be processed from the game video stream, and decode the game frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data, and store the decoded frame data in a decoding output cache whose data storage structure is a doubly linked list. ④ The rendering thread of the second graphics card continuously queries the doubly linked list to see whether there is game data to be rendered during idle time. If so, the rendering thread obtains the game data to be rendered (decoded frame data) from the doubly linked list, and renders the decoded frame data to be rendered through the rendering thread to obtain the rendered game screen. Subsequently, the corresponding game screen is output in the game client. In this way, for any frame of the game, the decoding, rendering and other processing logic can be completely performed based on the above method, and finally the complete game screen can be presented in the game client.

[0145] In an embodiment of the present application, in application scenarios of real-time video streaming playback such as cloud gaming, live video broadcasting, and video conferencing, it is often required that the delay between the video frame content being sent from the server to the local (video client, game client, etc.) playback be lower than the user-perceivable delay, which is closely related to the user experience of the application. When the processing frame rate of the video client is lower than the video stream frame rate, the client will accumulate a large number of video frames that cannot be processed in a timely manner, making the application unable to work normally. In an embodiment of the present application, based on the feature that the video client is configured with dual graphics cards (a first graphics card and a second graphics card), making full use of the dual graphics card resources of the video client helps to improve the processing speed of the video client for the video data stream, thereby greatly improving the user experience. Specifically, the present application divides the decoding and rendering tasks of video frames into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the dual graphics card working mode can effectively alleviate the problem of the video client having a video playback frame rate lower than the video stream receiving frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing.

[0146] The above details the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below. Next, combined with the video processing scheme provided by the above embodiment of the present application, the relevant devices of the embodiment of the present application are introduced accordingly.

[0147] See Figure 12 , Figure 12 1 is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. The video processing device 1200 can be applied to the application server in the aforementioned embodiment. The video processing device 1200 can be a computer program (including program code) running on a computer device, for example, the video processing device 1200 is an application software; the video processing device 1200 can be used to execute the corresponding steps of the video processing method provided in an embodiment of the present application. The video processing device 1200 includes:

[0148] An acquisition unit 1201 is configured to acquire a video frame to be processed;

[0149] The processing unit 1202 is configured to decode the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data;

[0150] A storage unit 1203 is configured to store the decoded frame data in a decoded output buffer. The decoded output buffer stores the decoded frame data in a data storage structure based on a bidirectional linked list, and allows either a decoding thread or a rendering thread based on the second graphics card to operate the decoded output buffer.

[0151] The acquisition unit 1201 is further configured to acquire decoded frame data to be rendered from the bidirectional linked list;

[0152] The processing unit 1202 is configured to perform rendering processing on the decoded frame data to be rendered through a rendering thread to obtain a rendered video frame.

[0153] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0154] Create a linked list head and a linked list tail. The linked list head is used to store the pointer of the first node stored in the decoded output buffer, and the linked list tail is used to store the pointer of the last node stored in the decoded output buffer.

[0155] Generate a doubly linked list in the initial state based on the created linked list head and linked list tail;

[0156] Among them, in the initial state, the node pointers stored corresponding to the head of the linked list and the tail of the linked list in the bidirectional linked list are both empty.

[0157] In a possible implementation, the processing unit 1202 decodes the video frame to be processed based on the decoding thread started by the first graphics card, and after obtaining the decoded frame data, is further configured to perform the following operations:

[0158] Obtain write permission to the decoding output cache through thread locking of the decoding thread;

[0159] Obtaining a target node according to the decoded frame data obtained by the decoding process, and adding the target node to the bidirectional linked list in the initial state;

[0160] After determining that the target node is successfully added to the doubly linked list, the decoding thread is called to release the lock.

[0161] In one possible implementation, the target node is used to store: decoded frame data, a front pointer for pointing to a previous node, and a back pointer for pointing to a next node; the previous node refers to a node that is located before the target node in the decoding output buffer, and the next node refers to a node that is located after the target node in the decoding output buffer.

[0162] In one possible implementation, the processing unit 1202 adds the target node to the doubly linked list in the initial state to perform the following operations:

[0163] The forward pointer used by the target node to point to the previous node points to the head of the doubly linked list, and the backward pointer used by the target node to point to the next node points to the first node pointed to by the head of the linked list;

[0164] Point the linked list head to the target node to add the target node to the doubly linked list;

[0165] If the node pointed to by the tail of the doubly linked list is empty, point the tail of the linked list to the target node.

[0166] In a possible implementation, the processing unit 1202 obtains the decoded frame data to be rendered from the bidirectional linked list, and performs the following operations:

[0167] The rendering thread queries whether there is an available node in the doubly linked list. If so, the rendering thread obtains the read permission of the decoded output cache by thread locking.

[0168] Remove the available nodes obtained from the query from the doubly linked list;

[0169] If it is determined that the available node has been successfully removed from the doubly linked list, the lock release operation is performed through the rendering thread.

[0170] In one possible implementation, if the node pointed to by the tail of the doubly linked list is not empty, it is determined that there is an available node in the doubly linked list, and the available node is the last node in the doubly linked list; the processing unit 1202 removes the available node obtained by the query from the doubly linked list, and performs the following operations:

[0171] If there are at least two nodes in the doubly linked list, point the tail pointer of the doubly linked list to the node before the last node in the doubly linked list to remove the last node from the doubly linked list; or,

[0172] If the available node is the only node in the doubly linked list, the node pointed to by the pointer at the end of the doubly linked list is set to empty to remove the last node from the doubly linked list.

[0173] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0174] Detect whether the video client is configured with an integrated graphics card. If so, use the integrated graphics card as the first graphics card and the independent graphics card as the second graphics card.

[0175] If not, the working graphics card is used as the second graphics card, and the graphics card other than the working graphics card is used as the first graphics card.

[0176] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0177] Determine whether the second graphics card meets the idle condition. If so, call the rendering thread of the second graphics card to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation;

[0178] If not, the decoding thread of the first graphics card is called to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation.

[0179] In an embodiment of the present application, a video client is configured with a first graphics card and a second graphics card. First, the first graphics card can obtain the video frame to be processed, and decode the video frame to be processed through the decoding thread of the first graphics card to obtain decoded frame data; then, the decoded frame data can be stored in the decoding output buffer, which is a data storage structure based on a bidirectional linked list to store the decoded frame data, allowing the decoding thread and any thread based on the rendering thread of the second graphics card to operate the decoding output buffer; finally, the decoded frame data to be rendered is obtained from the bidirectional linked list, and the rendering thread performs rendering processing on the decoded frame data to be rendered to obtain a rendered video frame picture. It can be seen that the present application divides the decoding and rendering tasks of the video frame into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the working mode of the dual graphics card can effectively alleviate the problem that the video client's video playback frame rate is lower than the video stream reception frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing.

[0180] See Figure 13 , Figure 13 1300 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 1300 is used to execute the steps performed by the video client in the aforementioned method embodiment. The computer device 1300 includes: one or more processors 1301; one or more input devices 1302; one or more output devices 1303; and a memory 1304. The processors 1301, input devices 1302, output devices 1303, and memory 1304 are connected via a bus 1305. The memory 1304 is used to store a computer program, which includes program instructions. The processor 1301 is used to call the program instructions stored in the memory 1304 to perform the following operations:

[0181] Obtaining a video frame to be processed, and decoding the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data;

[0182] Storing the decoded frame data in a decode output buffer, wherein the decode output buffer stores the decoded frame data in a data storage structure based on a bidirectional linked list, and allowing either a decoding thread or a rendering thread based on the second graphics card to operate the decode output buffer;

[0183] The decoded frame data to be rendered is obtained from the bidirectional linked list, and the decoded frame data to be rendered is rendered by the rendering thread to obtain a rendered video frame picture.

[0184] In a possible implementation, the processor 1301 is further configured to perform the following operations:

[0185] Create a linked list head and a linked list tail. The linked list head is used to store the pointer of the first node stored in the decoded output buffer, and the linked list tail is used to store the pointer of the last node stored in the decoded output buffer.

[0186] Generate a doubly linked list in the initial state based on the created linked list head and linked list tail;

[0187] Among them, in the initial state, the node pointers stored corresponding to the head of the linked list and the tail of the linked list in the bidirectional linked list are both empty.

[0188] In a possible implementation, the processor 1301 decodes the video frame to be processed based on the decoding thread started by the first graphics card, and after obtaining the decoded frame data, is further configured to perform the following operations:

[0189] Obtain write permission to the decoding output cache through thread locking of the decoding thread;

[0190] Obtaining a target node according to the decoded frame data obtained by the decoding process, and adding the target node to the bidirectional linked list in the initial state;

[0191] After determining that the target node is successfully added to the doubly linked list, the decoding thread is called to release the lock.

[0192] In one possible implementation, the target node is used to store: decoded frame data, a front pointer for pointing to a previous node, and a back pointer for pointing to a next node; the previous node refers to a node that is located before the target node in the decoding output buffer, and the next node refers to a node that is located after the target node in the decoding output buffer.

[0193] In one possible implementation, the processor 1301 adds the target node to the doubly linked list in the initial state to perform the following operations:

[0194] The forward pointer used by the target node to point to the previous node is pointed to the head of the doubly linked list, and the backward pointer used by the target node to point to the next node is pointed to the first node pointed to by the head of the linked list;

[0195] Point the linked list head to the target node to add the target node to the doubly linked list;

[0196] If the node pointed to by the tail of the doubly linked list is empty, point the tail of the linked list to the target node.

[0197] In one possible implementation, the processor 1301 obtains decoded frame data to be rendered from the bidirectional linked list, and performs the following operations:

[0198] The rendering thread queries whether there is an available node in the doubly linked list. If so, the rendering thread obtains the read permission of the decoded output cache by thread locking.

[0199] Remove the available nodes obtained from the query from the doubly linked list;

[0200] If it is determined that the available node has been successfully removed from the doubly linked list, the lock release operation is performed through the rendering thread.

[0201] In one possible implementation, if the node pointed to by the tail of the doubly linked list is not empty, it is determined that there is an available node in the doubly linked list, and the available node is the last node in the doubly linked list; the processor 1301 removes the available node obtained by the query from the doubly linked list, and performs the following operations:

[0202] If there are at least two nodes in the doubly linked list, point the tail pointer of the doubly linked list to the node before the last node in the doubly linked list to remove the last node from the doubly linked list; or,

[0203] If the available node is the only node in the doubly linked list, the node pointed to by the pointer at the end of the doubly linked list is set to empty to remove the last node from the doubly linked list.

[0204] In a possible implementation, the processor 1301 is further configured to perform the following operations:

[0205] Detect whether the video client is configured with an integrated graphics card. If so, use the integrated graphics card as the first graphics card and the independent graphics card as the second graphics card.

[0206] If not, the working graphics card is used as the second graphics card, and the graphics card other than the working graphics card is used as the first graphics card.

[0207] In a possible implementation, the processor 1301 is further configured to perform the following operations:

[0208] Determine whether the second graphics card meets the idle condition. If so, call the rendering thread of the second graphics card to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation;

[0209] If not, the decoding thread of the first graphics card is called to obtain the decoded frame data to be rendered from the bidirectional linked list and perform the rendering operation.

[0210] In the embodiment of the present application, based on the characteristic that the video client is equipped with dual graphics cards (a first graphics card and a second graphics card), the dual graphics card resources of the video client are fully utilized, which helps to improve the processing speed of the video client for the video data stream, thereby greatly improving the user experience. Specifically, the present application divides the decoding and rendering tasks of the video frame into two graphics cards for processing, which can reduce the load of a single graphics card. Compared with the traditional decoding and rendering processing based on a single graphics card, it avoids the performance degradation of a single graphics card due to excessive load; in addition, the working mode of the dual graphics card can effectively alleviate the problem of the video client's video playback frame rate being lower than the video stream receiving frame rate (i.e., playback delay) due to insufficient performance of a single graphics card or network fluctuations, thereby improving the efficiency of video processing and reducing the delay of video frame processing.

[0211] In addition, it should be noted here that: the embodiment of the present application also provides a computer storage medium, and a computer program is stored in the computer storage medium, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the method in the corresponding embodiment above, so it will not be described in detail here. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one place, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0212] According to one aspect of the present application, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, so that the computer device can perform the method in the corresponding embodiment described above. Therefore, a detailed description thereof will not be given here.

[0213] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0214] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video processing method, characterized in that: Applied to a video client, the video client is configured with a first graphics card and a second graphics card; the method includes: Receiving video stream data in a video stream playback application scenario, obtaining a to-be-processed video frame from the video stream data, and decoding the to-be-processed video frame through a decoding thread of the first graphics card to obtain decoded frame data; The decoded frame data is stored in a decoded output buffer, wherein the decoded frame data is stored in the decoded output buffer using a data storage structure based on a doubly linked list, wherein the doubly linked list is used to indicate that at the same time, any one of the decoding thread and the rendering thread based on the second graphics card is allowed to operate the decoded output buffer based on thread locking; wherein the thread locking indication has a locking position, wherein the locking position is used to indicate that after the target thread in the decoding thread and the rendering thread performs a locking operation, another thread other than the target thread that performs the locking operation is blocked at the locking position until the target thread that performs the locking operation performs a release operation, and the other thread continues to execute backward according to the locking position at the time of blocking; the target thread is any one of the decoding thread and the rendering thread; wherein the doubly linked list includes a plurality of nodes, and any node stores: the video frame decoding data corresponding to the node, the previous item pointer of the node, and the next item pointer of the node; If the rendering thread is in an idle state and obtains the read permission of the decoding output cache, the decoded frame data to be rendered is obtained from the bidirectional linked list, and the rendering thread renders the decoded frame data to be rendered to obtain a rendered video frame picture.

2. The method according to claim 1, wherein The method further comprises: Create a linked list head and a linked list tail, wherein the linked list head is used to store a pointer to the first node stored in the decoding output buffer, and the linked list tail is used to store a pointer to the last node stored in the decoding output buffer; Generate a doubly linked list in the initial state based on the created linked list head and linked list tail; Wherein, in the initial state, the node pointers stored corresponding to the head of the linked list and the tail of the linked list in the bidirectional linked list are both empty.

3. The method according to claim 2, wherein After decoding the to-be-processed video frame by the decoding thread of the first graphics card to obtain decoded frame data, the method further includes: Obtaining write permission for the decoding output cache through a decoding thread based on thread locking; Obtaining a target node according to the decoded frame data obtained by decoding, and adding the target node to the bidirectional linked list in the initial state; After determining that the target node is successfully added to the bidirectional linked list, the decoding thread is called to execute a lock release operation.

4. The method according to claim 3, wherein The target node is used to store: the decoded frame data, a front pointer pointing to a previous node, and a back pointer pointing to a next node; the previous node refers to a node located before the target node in the decoded output buffer, and the next node refers to a node located after the target node in the decoded output buffer.

5. The method according to claim 4, wherein Adding the target node to the bidirectional linked list in the initial state includes: The forward pointer of the target node pointing to the preceding node is pointed to the head of the doubly linked list, and the backward pointer of the target node pointing to the succeeding node is pointed to the first node pointed to by the head of the linked list; Point the linked list head to the target node to add the target node to the doubly linked list; If the node pointed to by the tail of the bidirectional linked list is empty, the tail of the linked list is pointed to the target node.

6. The method according to claim 1, wherein The step of obtaining the decoded frame data to be rendered from the bidirectional linked list and performing rendering processing on the decoded frame data to be rendered by the rendering thread to obtain a rendered video frame picture includes: querying, by a rendering thread, whether there is an available node in the bidirectional linked list; if so, obtaining read permission for the decoded output buffer based on thread locking, rendering the decoded frame data included in the available node to display the video frame corresponding to the decoded frame data included in the available node; Remove the available node obtained from the query from the bidirectional linked list; If it is determined that the available node has been successfully removed from the bidirectional linked list, the rendering thread performs a lock release operation.

7. The method according to claim 6, wherein If the node pointed to by the tail of the doubly linked list is not empty, it is determined that there is an available node in the doubly linked list, and the available node is the last node in the doubly linked list; The removing the available node obtained by querying from the bidirectional linked list includes: If there are at least two nodes in the doubly linked list, point the pointer of the tail of the doubly linked list to the node before the last node in the doubly linked list, so as to remove the last node from the doubly linked list; or, If the available node is the only node in the doubly linked list, the node pointed to by the pointer at the end of the doubly linked list is set to empty, so as to remove the last node from the doubly linked list.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: Detecting whether the video client is configured with an integrated graphics card, and if so, using the integrated graphics card as the first graphics card and the independent graphics card as the second graphics card; If not, the working graphics card is used as the second graphics card, and the graphics cards other than the working graphics card are used as the first graphics card.

9. The method according to any one of claims 1 to 7, wherein: The method further comprises: Determine whether the second graphics card meets the idle condition, and if so, call the rendering thread of the second graphics card to obtain the decoded frame data to be rendered from the bidirectional linked list and perform a rendering operation; If not, the first graphics card is called to obtain the decoded frame data to be rendered from the bidirectional linked list and perform a rendering operation.

10. A video processing device, characterized in that: Applied to a video client, the video client is configured with a first graphics card and a second graphics card; the device comprises: An acquisition unit, configured to receive video stream data in a video stream playback application scenario and acquire a video frame to be processed from the video stream data; a processing unit, configured to decode the video frame to be processed through a decoding thread of the first graphics card to obtain decoded frame data; A storage unit, configured to store the decoded frame data in a decoded output buffer, wherein the decoded frame data is stored in the decoded output buffer using a data storage structure based on a doubly linked list, wherein the doubly linked list is configured to indicate that at the same time, any one of the decoding thread and the rendering thread based on the second graphics card is allowed to operate the decoded output buffer based on thread locking; wherein the thread locking indication includes a locking position, wherein the locking position is configured to indicate that after the target thread in the decoding thread and the rendering thread performs a locking operation, another thread other than the target thread that performs the locking operation is blocked at the locking position until the target thread that performs the locking operation performs a release operation, and the other thread continues to execute backward according to the locking position at the time of blocking; the target thread is either the decoding thread or the rendering thread; wherein the doubly linked list includes a plurality of nodes, and any node stores: the video frame decoding data corresponding to the node, a previous item pointer of the node, and a next item pointer of the node; The acquisition unit is further configured to acquire the decoded frame data to be rendered from the bidirectional linked list if the rendering thread is in an idle state and obtains the read permission of the decoding output buffer; The processing unit is used to perform rendering processing on the decoded frame data to be rendered through the rendering thread to obtain a rendered video frame picture.

11. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the video processing method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the video processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the video processing method according to any one of claims 1 to 9.

Images