Image decoding method, device, readable medium and electronic device based on buffer pool

Abstract

The present application belongs to the field of cloud computing technology, and relates to a buffer pool-based image decoding method, device, computer-readable medium, and electronic device. The method is applied to the client, and includes: obtaining a coded code stream, constructing a frame object corresponding to a single-frame image based on the coded code stream; applying for storage space in the buffer pool based on the Frame object, and storing the Frame packet corresponding to the single-frame image in the buffer pool; inputting the Frame packet to the decoding end at the maximum frame sending speed, wherein the maximum frame sending speed is obtained by adjusting the dynamic capability of the decoding end based on a test code stream with the same parameters as the single-frame image after performing a dynamic capability detection on the decoding end. The present application can cache data packets through the buffer pool, and input the Frame packets in the buffer pool to the decoding end at the maximum frame sending speed, thereby avoiding the performance degradation of the decoding end, alleviating network congestion, and achieving the effect of stable flow control, thereby improving the decoding performance of the decoding end.

Description

Technical Field

[0001] The present application belongs to the field of cloud computing technology, and specifically relates to a buffer pool-based image decoding method, a buffer pool-based image decoding device, a computer-readable medium, and an electronic device. Background Art

[0002] With the rapid development of network technology, end users have higher demands on the performance of terminal devices. For example, when watching videos or playing games, they hope that the pictures are clear, smooth, and without any interruptions. This also places extremely high requirements on the video encoding and decoding delay. For example, when the video frame rate is 60fps (frames per second), the encoding and decoding delay needs to be less than 16ms. When the video frame rate is 120fps, the encoding and decoding delay needs to be less than 8ms, and so on.

[0003] At present, when performing video encoding and decoding, the data transmitted from the network is generally not managed. That is, no matter how much data is transmitted from the network, it will be immediately transmitted to the chip for encoding and decoding. However, if a large number of data packets are transmitted from the network at the same time, and the upper-level framework does not manage the data and transmits all data packets directly to the chip, it will cause the chip to be overloaded and overheated, reducing the chip's work efficiency, increasing the delay of encoding and decoding, and ultimately reducing the video viewing experience. The video playback screen will be severely stuck and delayed. Summary of the Invention

[0004] The purpose of this application is to provide a buffer pool-based image decoding method, a buffer pool-based image decoding device, a computer-readable medium and an electronic device, which can overcome the problems existing in the related art that a large number of data packets enter the chip for decoding at the same time, resulting in chip performance degradation, serious decoding delay, and video image freeze.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a buffer pool-based image decoding method is provided, the method comprising: obtaining an encoded code stream, constructing a frame object corresponding to a single-frame image based on the code stream; applying for storage space in a buffer pool based on the Frame object, and storing a Frame packet corresponding to the single-frame image in the buffer pool; inputting the Frame packet to a decoding end at a maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing a dynamic capability detection on the decoding end based on a test code stream having the same parameters as the single-frame image.

[0007] According to one aspect of an embodiment of the present application, a buffer pool-based image decoding device is provided, the device comprising: an object construction module, configured to obtain an encoded code stream and construct a frame object corresponding to a single-frame image based on the encoded code stream; a space application module, configured to apply for storage space in a buffer pool based on the Frame object and store a Frame packet corresponding to the single-frame image in the buffer pool; and a decoding module, configured to input the Frame packet to a decoding end at a maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing a dynamic capability detection on the decoding end based on a test code stream having the same parameters as the single-frame image.

[0008] In some embodiments of the present application, the object construction module is configured to: parse the encoded code stream to obtain frame information corresponding to the single-frame image, the frame information including frame type, timestamp, frame size and serial number; and construct the Frame object according to the frame information and the memory address corresponding to the single-frame image.

[0009] In some embodiments of the present application, the buffer pool includes: unapplied unused space, applied unused space and used space; the Frame object includes frame information and a memory address corresponding to the single-frame image; the space application module is configured to: send a storage space application instruction to the buffer pool based on the memory address, and determine the size of the Frame packet according to the frame size in the frame information and the size of the frame information; when the size of the Frame packet is less than or equal to the size of the unapplied unused space, obtain space in the unapplied unused space as applied space according to the size of the Frame packet; and, when the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame packet, store the Frame packet in the applied unused space.

[0010] In some embodiments of the present application, the space application module is further configured to: when the size of the Frame package is larger than the unapplied unused space, wait for the used space to be released until the size of the unapplied unused space is larger than or equal to the size of the Frame package; and, when the size of the remaining space in the applied unused space is smaller than the size of the Frame package, wait for the used space to be released until the size of the remaining space in the applied unused space is larger than or equal to the size of the Frame package.

[0011] In some embodiments of the present application, the space application module is further configured to: store the Frame packets in the buffer pool in sequence according to the timestamp and / or index number in the Frame object.

[0012] In some embodiments of the present application, the parameter includes a set frame rate; the decoding module includes: a first input unit, used to input the test code stream at the set frame rate to the decoding end for decoding, so as to obtain a benchmark output frame rate and a benchmark single-frame average decoding delay; a first comparison unit, used to determine a first frame rate threshold according to the set frame rate and a first coefficient, and compare the benchmark output frame rate with the first frame rate threshold; a first determination unit, used to use the set frame rate as the maximum frame sending speed when the benchmark output frame rate is less than the first frame rate threshold; a second input unit, used to increase the set frame rate to obtain a frame sending speed when the benchmark output frame rate is greater than or equal to the first frame rate threshold, and input the test code stream at the frame sending speed to the decoding end for decoding, so as to obtain an output frame rate and a single-frame average decoding delay; a first calculation unit, used to determine the maximum frame sending speed based on the frame sending speed, the benchmark single-frame average decoding delay, the output frame rate, and the single-frame average decoding delay.

[0013] In some embodiments of the present application, the first calculation unit includes: a second comparison unit, used to determine a second frame rate threshold based on the frame sending speed and a second coefficient, and compare the output frame rate with the second frame rate threshold; a second determination unit, used to use the frame sending speed as the maximum frame sending speed when the output frame rate is less than the second frame rate threshold; a second calculation unit, used to determine the maximum frame sending speed based on the frame sending speed, the benchmark single-frame average decoding delay and the single-frame average decoding delay when the output frame rate is greater than or equal to the second frame rate threshold.

[0014] In some embodiments of the present application, the second calculation unit includes: a third comparison unit, used to compare the benchmark single-frame average decoding delay and the single-frame average decoding delay; a third determination unit, used to use the frame delivery speed as the maximum frame delivery speed when the single-frame average decoding delay is greater than the benchmark single-frame average decoding delay; and a fourth calculation unit, used to determine the maximum frame delivery speed according to the frame delivery speed when the single-frame average decoding delay is less than or equal to the benchmark single-frame average decoding delay.

[0015] In some embodiments of the present application, the third determination unit includes: a fourth comparison unit, used to compare the frame sending speed with the frame rate threshold; a fourth determination unit, used to use the frame sending speed as the maximum frame sending speed when the frame sending speed is greater than the frame rate threshold; an update unit, used to update the frame sending speed when the frame sending speed is less than or equal to the frame rate threshold, and input the test code stream to the decoding end for decoding at the updated frame sending speed to obtain an updated output frame rate and an updated single-frame average decoding delay; a fifth calculation unit, used to determine the maximum frame sending speed based on the updated frame sending speed, the benchmark single-frame average decoding delay, the updated output frame rate and the updated single-frame average decoding delay.

[0016] In some embodiments of the present application, the buffer pool-based image decoding device also includes: an acquisition module, used to obtain the network congestion duration and the number of image frames waiting to apply for buffer pool storage space; an emptying module, used to detect whether there is an immediately refreshed image IDR frame in the Frame packet stored in the buffer pool when the network congestion duration is greater than a preset duration and the number of image frames is greater than a preset threshold, and to clear the buffer pool according to the detection result.

[0017] In some embodiments of the present application, the clearing module is configured to: when the IDR frame exists in the buffer pool, obtain the target index number corresponding to the IDR frame; discard the Frame packet in the buffer pool whose index number is less than the target index number, and at the same time send the Frame packet corresponding to the IDR frame from the buffer pool to the decoding end.

[0018] In some embodiments of the present application, the clearing module is configured to: when the IDR frame does not exist in the buffer pool, send an IDR frame acquisition request to the server; receive the IDR frame sent by the server in response to the IDR frame acquisition request, clear the buffer pool, and send the Frame packet containing the IDR frame from the buffer pool to the decoding end.

[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the buffer pool-based image decoding method in the above technical solution is implemented.

[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the buffer pool-based image decoding method in the above technical solution by executing the executable instructions.

[0021] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the buffer pool-based image decoding method in the above technical solution.

[0022] The buffer pool-based image decoding method provided by the embodiment of the present application, after obtaining the encoded code stream, constructs a frame object corresponding to a single frame image based on the encoded code stream, and sends an instruction to the buffer pool based on the Frame object to apply for storage space in the buffer pool. After successfully applying for storage space in the buffer pool, the Frame packet corresponding to the single frame image is stored in the buffer pool, and then the Frame packet stored in the buffer pool is input to the decoding end for decoding at the maximum frame sending rate. The maximum frame sending rate is obtained by adjusting the dynamic capacity of the decoding end after performing dynamic capacity detection on the decoding end based on a test code stream with the same parameters as the single frame image. On the one hand, the present application can cache the data packets incoming from the network in the buffer pool when the network is congested, thereby preventing the decoding end from being impacted by a large amount of data and causing the decoding end to degrade in performance. On the other hand, it can obtain the maximum frame sending rate that maximizes the decoding performance of the decoding end by dynamically detecting the load capacity of the decoding end. When the Frame packets in the buffer pool are input to the decoding end at this maximum frame sending rate, it can not only alleviate network congestion but also achieve the effect of stable flow control, thereby improving the decoding performance of the decoding end.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0025] Figure 1 The structural diagram of the system architecture of the buffer pool-based image decoding method in the embodiment of the present application is schematically shown.

[0026] Figure 2 The following schematically illustrates a flow chart of steps of a buffer pool-based image decoding method in an embodiment of the present application.

[0027] Figure 3 The schematic diagram of the structure of the buffer pool in the embodiment of the present application is shown schematically.

[0028] Figure 4The flowchart of applying for storage space in the buffer pool to cache Frame packets in an embodiment of the present application is schematically shown.

[0029] Figure 5 The flowchart of obtaining the maximum frame sending speed in an embodiment of the present application is schematically shown.

[0030] Figure 6 The flowchart of determining the maximum frame sending speed according to the output frame rate and the frame sending speed in an embodiment of the present application is schematically shown.

[0031] Figure 7 The flowchart of determining the maximum frame sending speed according to the benchmark single-frame average decoding delay and the single-frame average decoding delay in an embodiment of the present application is schematically shown.

[0032] Figure 8 The flowchart of determining the maximum frame sending speed according to the frame sending speed in an embodiment of the present application is schematically shown.

[0033] Figure 9 The structural block diagram of the image decoding device based on the buffer pool in an embodiment of the present application is schematically shown.

[0034] Figure 10 The following schematically shows a block diagram of a computer system structure of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0036] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0039] In the related technologies of this application, with the development of audio and video technology, the requirements for codec delay in audio and video frameworks are getting higher and higher, especially in scenarios such as cloud games and video calls, which have extremely high requirements for codec delay. Taking Android terminals as an example, Android terminals currently generally use the Mediacodec codec interface developed by Google to call the chip's codec capabilities to encode and decode the code stream, but if you want to obtain low-latency codecs on Android terminals, you need to rely on the chip's hardware codec capabilities. When a large number of data packets are transmitted to the network at the same time, and the upper-level framework does not manage the data and directly inputs the data packets into the chip, it will cause the chip workload to be too high and overheat, ultimately reducing the chip's work efficiency and increasing the codec delay. Users will feel that the picture is severely stuck and delayed, which reduces the user experience.

[0040] Whether it is an Android terminal or a terminal equipped with other operating systems, encoding and decoding are performed through the encoder and decoder in the chip. Therefore, if the network is congested and the upper-level framework does not manage the large number of incoming data packets, there will be problems such as reduced chip efficiency, serious encoding and decoding delays, and poor user experience.

[0041] In response to related technologies in the field, embodiments of the present application propose a buffer pool-based image decoding method. This buffer pool-based image decoding method can be applied to any video-related scenario, such as gaming, video calling, live streaming, and so on. Before describing the buffer pool-based image decoding method in embodiments of the present application in detail, the technical terms involved in this application are first explained.

[0042] 1. Encoding: The process of converting information from one form or format to another, also known as the code of a computer programming language.

[0043] 2. Decoding: The reverse process of encoding, which is the process of restoring information from its encoded form to its original state before encoding.

[0044] 3. Frame rate: The speed at which the audio and video frames are sent to the codec module in the chip. For example, 60fps means that 60 frames of data are sent to the chip per second.

[0045] 4. Set frame rate: The game screen frame rate set by the user during cloud gaming. The server will encode at this set frame rate, and the client also needs to decode at this set frame rate.

[0046] 5.Frame package: The type name of the data cached in the buffer pool, which includes the code stream data, frame type, timestamp, frame size, index number, and resolution corresponding to a single frame image.

[0047] 6. Average single-frame decoding delay: The decoding delay is calculated by taking the average decoding time of each frame during the decoding process.

[0048] 7. Frame rate: The frequency at which a bitmap image, in units of frames, appears continuously on the display.

[0049] 8. Cloud gaming: Based on cloud computing technology, games run on remote servers. End customers do not need to download or install, nor do they need to consider terminal configuration. As long as there is a network, they can play games with very high computing power.

[0050] Next, an exemplary system architecture applying the technical solution of this application is described.

[0051] Figure 1 The following schematically shows an exemplary system architecture block diagram applying the technical solution of the present application.

[0052] like Figure 1 As shown, the system architecture 100 may include a client 101, a server 102, and a network 103. The client 101 includes various electronic devices with display screens, such as smart phones, tablet computers, laptops, desktop computers, smart TVs, and smart car terminals. In different scenarios, the number of clients 101 is different. For example, in a cloud gaming scenario, there may be only one client 101, and users participate in cloud gaming through the client 101. In a video call scenario, there may be multiple clients 101, and users of each client 101 may make video calls through the client 101, and so on. The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The network 103 may be a communication medium of various connection types that can provide a communication link between the client 101 and the server 102, such as a wired communication link or a wireless communication link.

[0053] Depending on the implementation requirements, the system architecture in the embodiment of the present application can have any number of clients, networks, and servers. For example, the server can be a server group composed of multiple server devices. In addition, the technical solutions provided in the embodiment of the present application can be applied to the client 101.

[0054] In one embodiment of the present application, client 101 is equipped with a chip having a codec module and a buffer pool. The buffer pool caches one or more Frame packets, which are generated by processing data packets containing image frames sent from the network to client 101. After dynamically detecting the chip's load capacity and adjusting the maximum frame delivery rate of the buffer pool corresponding to the highest chip decoding performance based on the dynamic capacity, the Frame packets in the buffer pool can be delivered to the decoding end for decoding based on the maximum frame delivery rate to obtain image frames for rendering and display.

[0055] In one embodiment of the present application, when dynamically detecting the load capacity of a chip to adjust the maximum frame sending rate of a buffer pool corresponding to the highest decoding performance of the chip based on the dynamic capacity, first, data such as a benchmark output frame rate and a benchmark single-frame average decoding delay determined by the client at a set frame rate in the current scenario and inputting a test stream into a decoding end for decoding are obtained. Then, the frame sending rate is re-determined based on the set frame rate, and the test stream is input into the decoding end at the re-determined frame sending rate to obtain data such as the decoding output frame rate and the average single-frame decoding delay at the re-determined frame sending rate. Finally, the maximum frame sending rate is determined based on the re-determined frame sending rate, the obtained decoding output frame rate and the average single-frame decoding delay, and the benchmark decoding output frame rate and the benchmark single-frame average decoding delay corresponding to the set frame rate.

[0056] In one embodiment of the present application, the buffer pool-based image decoding method in the present application can be used in cloud gaming, video calls, live broadcasts and other scenarios. Accordingly, the server 102 can be a cloud server that provides cloud computing services. That is, the present application involves cloud storage and cloud computing technology.

[0057] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0058] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0059] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0060] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.

[0061] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0062] Based on logical functional divisions, the PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging tools. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0063] The following describes in detail the technical solutions provided by the present application, such as the buffer pool-based image decoding method, buffer pool-based image decoding device, computer-readable medium, and electronic device, in conjunction with specific implementation methods.

[0064] Figure 2 The schematic diagram shows a flow chart of the steps of a buffer pool-based image decoding method in one embodiment of the present application. The buffer pool-based image decoding method is executed by a client, which can be Figure 1 Client 101 in Figure 2 As shown, the buffer pool-based image decoding method in the embodiment of the present application may mainly include the following steps S210 to S230.

[0065] Step S210: obtaining a coded code stream, and constructing a frame object corresponding to a single frame image based on the coded code stream;

[0066] Step S220: applying for storage space in a buffer pool based on the Frame object, and storing the Frame package corresponding to the single-frame image in the buffer pool;

[0067] Step S230: Input the Frame packet to the decoding end at the maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing dynamic capability detection on the decoding end based on a test code stream having the same parameters as the single frame image.

[0068] In the buffer pool-based image decoding method provided in an embodiment of the present application, after obtaining a coded bitstream, a frame object corresponding to a single-frame image is constructed based on the bitstream, and a command is sent to the buffer pool based on the Frame object to apply for storage space in the buffer pool. After successfully applying for storage space in the buffer pool, the Frame packet corresponding to the single-frame image is stored in the buffer pool, and then the Frame packet stored in the buffer pool is input to the decoding end at a maximum frame sending rate. The maximum frame sending rate is obtained by adjusting the dynamic capacity of the decoding end after dynamically detecting the capacity of the decoding end based on a test bitstream with the same parameters as the single-frame image. On the one hand, the present application can cache data packets incoming from the network in the buffer pool when the network is congested, thereby preventing a large amount of data from impacting the decoding end and causing a performance degradation of the decoding end. On the other hand, the maximum frame sending rate that maximizes the chip decoding performance can be obtained by dynamically detecting the load capacity of the decoding end. When the Frame packets in the buffer pool are input to the decoding end at this maximum frame sending rate, not only can network congestion be alleviated, but also the effect of stable flow control can be achieved, thereby improving the decoding performance of the decoding end.

[0069] The specific implementation of each method step of the buffer pool-based image decoding method is described in detail below.

[0070] In step S210, a coded code stream is obtained, and a frame object corresponding to a single frame image is constructed based on the code stream.

[0071] In one embodiment of the present application, in the field of video encoding and decoding, each frame of the video is usually encoded separately, and then the encoded data packets are sent to the client in sequence. The client parses each data packet to obtain the image frame therein, and finally renders and plays it. Accordingly, in an embodiment of the present application, when a large number of data packets generated during network congestion are cached by a buffer pool, these data packets can be split into multiple Frame packets corresponding to a single frame image, and then all the Frame packets are arranged in order and sequentially transmitted to the decoding end for decoding, wherein the decoding end can specifically be a chip, and the decoding is implemented by a decoding module provided in the chip.

[0072] In one embodiment of the present application, before applying for storage space from the buffer pool and caching the Frame package in the buffer pool, it is necessary to obtain the encoded code stream and construct a Frame object and Frame package corresponding to a single frame image according to the encoded code stream.

[0073] Since a frame of image can be divided into multiple slices, each slice is divided into multiple macroblocks, and each macroblock is divided into multiple sub-blocks, a large image can be decomposed into small blocks, which can be easily encoded in space. When transmitting multiple image frames that constitute a video, they are transmitted in a binary stream, which is composed of multiple NALU units (Network Abstract Layer Units). The NALU unit includes a start code, a NALU header, and a NALU load (Payload). The start code is used to identify the start of a NALU unit. The NALU header includes a type parameter NAL Type. The NALU load is specifically RBSP (Raw Byte Sequence Payload) data, which contains the encoded data of the video. Therefore, after the client receives a data packet containing a single-frame image information transmitted from the network, it can first extract the NALU header from it to obtain the NALU type, and then splice the RBSP data in the data packet to form a code stream corresponding to the single-frame image. Finally, the RBSP data is parsed according to the obtained NALU type to obtain the frame information corresponding to the single-frame image. The frame information specifically includes the frame type, timestamp, frame size and sequence number, and further, it can also include information such as resolution. Among them, the frame type is specifically divided into intra-frame coded frames (I frames), forward predictive coded frames (P frames), and bidirectional predictive interpolated coded frames (B frames). The I frame also includes the instant decoding refresh IDR frame, which is the first image frame in an image sequence.

[0074] After obtaining the frame information corresponding to a single-frame image, a Frame object can be constructed based on the frame information and the memory address corresponding to the single-frame image. Based on the Frame object, storage space in the buffer pool can be requested. If storage space in the buffer pool is requested, the Frame packet corresponding to the single-frame image can be cached in the buffer pool, waiting to be transmitted to the decoding end for decoding. If storage space in the buffer pool is not requested, the process continues to wait until cache space is requested. The Frame packet includes the encoding stream and frame information corresponding to the single-frame image. In other words, the Frame packet includes the encoding stream, frame type, timestamp, frame size, and sequence number corresponding to the single-frame image. Furthermore, it can also include information such as resolution.

[0075] In step S220, storage space in a buffer pool is requested based on the Frame object, and the Frame packet including the code stream and the frame information is stored in the buffer pool.

[0076] In one embodiment of the present application, a Frame object includes frame information and a memory address corresponding to a single-frame image. A buffer pool is provided in the client and corresponds to the client's memory. Therefore, a storage space request instruction can be sent to the buffer pool based on the memory address to request storage space. This storage space is the memory space. When requesting storage space, it is possible to determine whether there is sufficient storage space in the buffer pool for allocation based on the frame size in the frame information and the size of the frame information. The frame size corresponds to the size of the encoded code stream corresponding to the single-frame image. Since the Frame packet consists of the encoded code stream and frame information, the size of the encoded code stream corresponds to the frame size. Therefore, the size of the Frame packet can be obtained based on the frame size and the size of the frame information, and storage space can then be requested based on the size of the Frame packet.

[0077] In one embodiment of the present application, a buffer pool object is also stored in the client, and the buffer pool object stores the cached Frame packets and the status information of the entire buffer pool. The storage space of the buffer pool is divided into three parts, such as Figure 3 As shown, the buffer pool 300 includes a used space 301, an applied unused space 302, and an unapplied unused space 303, wherein the used space 301 is used to store Frame packets that have been input to the decoding end but have not been decoded. If a decoding error occurs, the corresponding Frame packet can be retrieved from the used space 301 according to the index number for decoding. If the decoding is completed, the Frame packet is deleted from the used space 301; the applied unused space 302 is used to store Frame packets for which storage space has been applied but have not been input to the decoding end. When the Frame packet is input to the decoding end, the Frame packet is transferred from the applied unused space 302 to the used space 301; the unapplied unused space 303 is a space waiting to be applied. In this embodiment of the present application, multiple Frame packets can be sent to the decoding end for decoding at the same time. Accordingly, multiple Frame packets can be stored simultaneously in the used space 301 and the applied unused space 302, such as Figure 3 As shown, three Frame packets are stored in the used space 301 , and two Frame packets are stored in the applied unused space 302 .

[0078] Correspondingly, the status information of the buffer pool includes: total space size, size of used space, size of unapplied unused space, size of applied unused space, number of stored frames, number of IDR frames, etc., wherein the total space size is equal to the sum of the sizes of used space, unapplied unused space and applied unused space, and the total space size depends on the resolution of the code stream in the current scene, and the unit is Frame. For example, when the resolution of the code stream in the current scene is 720P, the total space size can be set to 100 Frames, and the used space can be set to a maximum of 50 Frames, that is, the buffer pool can store a total of 100 Frame packages, of which the used space can store a maximum of 50 Frame packages. If the resolution of the code stream in the current scene is 1080P, the total space size can be set to 70 Frames, and the used space can be set to a maximum of 35 Frames, that is, the buffer pool can store a total of 70 Frame packages, of which the used space can store a maximum of 35 Frame packages. Furthermore, the size of the available space can be determined based on the size of the total space and the size of the used space, that is, the sum of the sizes of the unapplied unused space and the applied unused space. For example, when the resolution of the code stream is 720P, the size of the available space is 50Frames. When the resolution of the code stream is 1080P, the size of the available space is 65Frames. At the same time, the size of the unapplied unused space and the size of the applied unused space can be allocated based on the size of the available space according to actual needs. For example, when the size of the available space is 50Frames, the unapplied unused space and the applied unused space are both set to 25Frames, and so on. This application does not make specific limitations on this.

[0079] In one embodiment of the present application, when applying for storage space from the buffer pool, the application is made in units of Frame. Since the size of each Frame packet is different, the size of the storage space applied for is also different. Figure 4 The schematic diagram shows the process of applying for storage space in the buffer pool to cache the Frame package. Figure 4As shown, in step S401, it is determined whether the size of the Frame package is less than or equal to the size of the unapplied unused space; if the determination result is yes, step S402 is executed, and if the determination result is no, step S403 is executed; in step S402, the corresponding applied space is obtained from the unapplied unused space according to the size of the Frame package; in step S403, the used space is waited for to be released until the size of the unapplied unused space is greater than or equal to the size of the Frame package; in step S404, it is determined whether the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame package; if the determination result is yes, step S405 is executed, and if the determination result is no, step S406 is executed; in step S405, the Frame package is placed in the applied unused space; in step S406, the used space is waited for to be released until the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame package.

[0080] Among them, the available space in the applied unused space and the unapplied unused space changes dynamically. When a new Frame package is transmitted, space is first applied for from the unapplied unused space. If the size of the Frame package is less than or equal to the size of the unapplied unused space, part or all of the space in the unapplied unused space is allocated to the Frame package as the applied space. Then, it is determined whether the size of the remaining space in the applied unused space is greater than or equal to the Frame package. If so, the Frame package is stored in the applied unused space and the applied space is released. If not, wait for the used space to be released, and the Frame package cached in the applied unused space is transferred to the used space until the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame package, so that the Frame package in the applied space can be transferred to the applied unused space and the applied space is released at the same time. If the size of a Frame packet is larger than the size of the unapplied, unused space, the system waits for the used space to be released, and then moves the Frame packets in the applied, unused space to the used space. Simultaneously, the Frame packets in the applied space are moved to the applied, unused space, freeing up the applied space. This way, if the size of the unapplied, unused space is larger than or equal to the size of the Frame packet, some or all of it can be allocated as applied space to the Frame packet. It's worth noting that the size of the unapplied, unused space in this application is larger than or equal to the theoretical maximum size of the Frame packet. This ensures that when the unapplied, unused space is not claimed by a Frame packet, the corresponding applied space can be allocated to the new Frame packet.

[0081] In one embodiment of the present application, when storing Frame packets in a buffer pool, the Frame packets can be sorted according to the timestamps and / or index numbers contained in the Frame objects in the Frame packets and stored in order in the buffer pool. Then, when the Frame packets are sent to the decoding end for decoding, they are also sent in order, so that the video image displayed by the client is coherent. It is worth noting that when the source end (server) that sends the data packets containing the image frames does not fail, the timestamps and index numbers in the Frame packets usually correspond to each other, and the client receives the data packets in chronological order. Therefore, the Frame packets can be sorted only according to the index number or timestamp. However, when the source end (server) fails, the transmission order may be disrupted. In this case, it is necessary to sort according to both the timestamp and the index number to ensure that the Frame packets are sorted in order and sent to the decoding end for parsing.

[0082] In step S230, the Frame packet is input to the decoding end at the maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing dynamic capability detection on the decoding end based on a test code stream having the same parameters as the single frame image.

[0083] In one embodiment of the present application, while caching Frame packets, the buffer pool also sends the cached Frame packets to the decoding end for decoding. However, in order to achieve the effect of stable traffic control, improve the decoding performance of the decoding end, and at the same time improve the transmission efficiency and decoding efficiency of the Frame packets, it is necessary to send the Frame packets to the decoding end at a higher and stable frame sending speed. In the embodiment of the present application, in order to determine the maximum frame sending speed, the load capacity of the decoding end, that is, the chip, can be dynamically detected and adjusted based on the dynamic capacity to obtain the speed.

[0084] Generally speaking, as a decoding end, the chip will increase its operating frequency as the frame sending speed increases, thereby speeding up the chip decoding speed. However, once the frame sending speed reaches a threshold, the chip decoding speed will drop significantly. Therefore, it is necessary to detect the frame sending speed threshold that causes the chip decoding speed to drop.

[0085] In one embodiment of the present application, when performing dynamic capability detection, it is first necessary to obtain the parameter configuration corresponding to the image frame in the current scene, then obtain a test code stream with the same parameter configuration based on the parameter configuration, and finally perform detection based on the test code stream.

[0086] Figure 5 The schematic diagram of the process of obtaining the maximum frame sending speed is shown as follows: Figure 5As shown, in step S501, parameters corresponding to image frames in the current scene are obtained, and the parameters specifically include parameters such as a set frame rate. In step S502, a test stream having the same parameters is obtained based on the parameters. In step S503, the test stream is input to a decoder at the set frame rate to obtain a benchmark output frame rate and a benchmark single-frame average decoding delay. In step S504, a first frame rate threshold is determined based on the set frame rate and a first coefficient, and the benchmark output frame rate is compared with the first frame rate threshold. In step S505, when the benchmark output frame rate is less than the first frame rate threshold, the set frame rate is used as the maximum frame sending rate. In step S506, when the benchmark output frame rate is greater than or equal to the first frame rate threshold, the set frequency is increased to obtain the frame sending rate. In step S507, the test stream is input to a decoder at the frame sending rate to obtain the output frame rate and the single-frame average decoding delay. In step S508, the maximum frame sending rate is determined based on the frame sending rate, the benchmark single-frame average decoding delay, the output frame rate, and the single-frame average decoding delay.

[0087] The test bitstream in step S502 may be an encoded bitstream transmitted in real time by the server and having the same resolution, set frame rate and other parameters as the current video.

[0088] In step S504, the first coefficient can be any value in the interval [0.85, 0.95], for example, it can be 0.9. That is, when the reference output frame rate is less than 90% of the set frame rate, the current frame sending speed is considered to be the maximum frame sending speed. This is because under normal circumstances, the output frame rate of the decoding end is close to the set frame rate, but in the case of network congestion, the performance of the decoding end will decline, resulting in a decrease in the output frame rate of the decoding end. If the output frame rate drops below a certain proportion of the frame sending speed, it means that the decoding performance of the decoding end has begun to weaken. Therefore, the frame sending speed under this condition can be used as the maximum frame sending speed, and the Frame packets in the buffer pool can be sent to the decoding end for decoding. Under the premise of ensuring the decoding performance of the decoding end, decoding is performed with higher efficiency to alleviate network congestion.

[0089] The benchmark output frame rate, benchmark single-frame average decoding delay, output frame rate, and single-frame average decoding delay obtained in steps S503 and S507 can be obtained by counting decoding results of a preset duration, for example, by decoding results of 2s of the encoded bitstream. Of course, they can also be obtained by counting decoding results of other decoding durations of the encoded bitstream. This embodiment of the present application does not specifically limit this.

[0090] In step S506, when determining the frame sending speed, the set frame rate can be increased according to a fixed amplitude or a random amplitude. For example, when the set frame rate is 60fps, the set frame rate can be increased with a fixed amplitude such as 10fps, 20fps, 30fps, etc., and the set frame rate can also be increased with different amplitudes.

[0091] In one embodiment of the present application, when the benchmark output frame rate is greater than or equal to the first frame rate threshold, the maximum frame sending speed can be determined based on the increased frame sending speed, the benchmark single-frame average decoding delay, the output frame rate corresponding to the increased frame sending speed, and the single-frame average decoding delay. Specifically, the maximum frame sending speed can be obtained by judgment from three levels, wherein the first level is to determine the maximum frame sending speed based on the output frame rate and the increased frame sending speed, the second level is to determine the maximum frame sending speed based on the benchmark single-frame average decoding delay and the single-frame average decoding delay, and the third level is to determine the maximum frame sending speed based on the increased frame sending speed.

[0092] In one embodiment of the present application, when determining the maximum frame feed speed based on the above three levels, the processing logic of the three levels is interrelated. For example, the maximum frame feed speed can be determined first from the first level. If the maximum frame feed speed cannot be determined based on the first level, the maximum frame feed speed is determined from the second level. Correspondingly, if the maximum frame feed speed cannot be determined based on the second level, the maximum frame feed speed is determined from the third level. Furthermore, when the maximum frame feed speed cannot be determined based on the third level, the current frame feed speed is updated. That is, the current frame feed speed is increased based on the current frame feed speed according to a fixed or random increment to obtain an updated frame feed speed. The maximum frame feed speed is then determined based on the updated frame feed speed. The above process is repeated until the maximum frame feed speed is obtained.

[0093] It is worth noting that, although the above-described embodiment sequentially determines the maximum frame feed speed based on the judgment processes at the first, second, and third levels, the embodiments of the present application include but are not limited to the above-described judgment processes. That is, in the embodiments of the present application, the maximum frame feed speed can be determined in any order based on the processing logic at the first, second, and third levels. For example, the determination of whether to use the processing logic at the first level to determine the maximum frame feed speed can be first made based on the judgment result at the second level, and then the determination of whether to use the processing logic at the third level to determine the maximum frame feed speed can be made based on the judgment result at the first level, and so on. Accordingly, if the maximum frame feed speed still cannot be determined based on the last judgment result, the frame feed speed is re-updated, and the maximum frame feed speed is determined based on the updated frame feed speed and corresponding parameters until the maximum frame feed speed is obtained.

[0094] Next, taking the example of executing the processing logic of the first level, the second level and the third level in sequence, the process of determining the maximum frame sending speed according to the three different levels in the embodiment of the present application is described in detail.

[0095] Figure 6 The following schematically shows a flow chart of determining the maximum frame sending speed according to the output frame rate and the frame sending speed. Figure 6 As shown, in step S601, the second frame rate threshold is determined according to the set frame rate and the second coefficient, and the output frame rate is compared with the second frame rate threshold; in step S602, when the output frame rate is less than the second frame rate threshold, the frame sending speed is used as the maximum frame sending speed; in step S603, when the output frame rate is greater than or equal to the first frame rate threshold, the maximum frame sending speed is determined according to the frame sending speed, the benchmark single-frame average decoding delay and the single-frame average decoding delay.

[0096] The second coefficient is also any value in the interval [0.85, 0.95] and may be the same as or different from the first coefficient. This embodiment of the present application does not impose any specific limitation on this.

[0097] Figure 7 The flowchart of determining the maximum frame sending speed according to the benchmark single-frame average decoding delay and the single-frame average decoding delay is shown schematically. Figure 7 As shown, in step S701, the benchmark single-frame average decoding delay and the single-frame average decoding delay are compared; in step S702, when the single-frame average decoding delay is greater than the benchmark single-frame average decoding delay, the frame sending speed is used as the maximum frame sending speed; in step S703, when the single-frame average decoding delay is less than or equal to the benchmark single-frame average decoding delay, the maximum frame sending speed is determined according to the frame sending speed.

[0098] The benchmark single-frame average decoding delay is obtained by inputting the test code stream into the decoding end for decoding at the set frame rate. When the current frame sending speed is obtained by increasing the frame rate on the basis of the set frame rate, and the single-frame average decoding delay obtained by inputting the test code stream into the decoding end for decoding at the current frame sending speed is greater than the benchmark single-frame average decoding delay, it indicates that the decoding end still has good decoding performance corresponding to the current frame sending speed. Therefore, the current frame sending speed can be used as the maximum frame sending speed.

[0099] Figure 8 The following schematically shows a flow chart of determining the maximum frame sending speed according to the frame sending speed. Figure 8As shown, in step S801, the frame sending speed is compared with the frame rate threshold; in step S802, when the frame sending speed is greater than the frame rate threshold, the frame sending speed is used as the maximum frame sending speed; in step S803, when the frame sending speed is less than or equal to the frame rate threshold, the frame sending speed is updated, and the test code stream is input to the decoding end for decoding according to the updated frame sending speed to obtain an updated output frame rate and an updated single-frame average decoding delay; in step S804, the maximum frame sending speed is determined according to the updated frame sending speed, the benchmark single-frame average decoding delay, the updated output frame rate, and the updated single-frame average decoding delay.

[0100] The frame rate threshold can be the maximum output frame rate corresponding to the decoder. For example, at a resolution of 1080P, the maximum output frame rate of the decoder is 240fps. When the frame rate is increased to above 240fps, it can be directly determined as the maximum frame rate. The maximum output frame rate of the decoder varies with the resolution. Therefore, when determining the frame rate threshold, the maximum output frame rate corresponding to the decoder can be determined as the frame rate threshold based on the resolution of the video in the current scene. The frame rate can then be determined based on the determined frame rate threshold.

[0101] In one embodiment of the present application, after determining the maximum frame delivery rate, this maximum frame delivery rate can be set as the maximum release rate of the buffer pool. When the network data packet transmission rate exceeds the maximum frame delivery rate of the buffer pool, the buffer pool will deliver the cached frame packets to the decoder at the maximum release rate until network congestion disappears. When the buffer pool transmits the cached frame packets to the decoder, it sends the frame packets in sequence according to their index numbers, ensuring that the decoder can decode the code streams corresponding to the frame packets in order to obtain the image frames for rendering and display.

[0102] In one embodiment of the present application, if the network congestion is serious and all spaces in the buffer pool are occupied, the network congestion can be alleviated by clearing the buffer pool.

[0103] In one embodiment of the present application, the degree of network congestion can be determined from two aspects: the duration of network congestion and the number of image frames waiting for buffer pool storage space. When the network congestion duration exceeds a preset duration and the number of image frames waiting for buffer pool storage space is greater than a preset threshold, the buffer pool is cleared to alleviate network congestion. The preset duration and preset threshold can be set according to actual needs. For example, the preset duration can be set to 5 seconds and the preset threshold can be set to 30 frames. That is, when the network congestion duration exceeds 5 seconds and the number of image frames waiting for buffer pool storage space is greater than 30 frames, the network congestion can be alleviated by clearing the buffer pool.

[0104] In one embodiment of the present application, when clearing the buffer pool, different clearing methods can be used depending on whether there are IDR frames in the buffer pool. Specifically, it can be detected whether there is an IDR frame for immediately refreshing the image in the Frame packet stored in the buffer pool, and the buffer pool can be cleared according to the detection result. When there is an IDR frame in the Frame packet stored in the buffer pool, the target index number corresponding to the IDR frame is obtained, and all Frame packets with index numbers less than the target index number cached in the buffer pool are discarded, and the Frame packet corresponding to the IDR frame is directly sent to the decoding end; when there is no IDR frame in the Frame packet stored in the buffer pool, the client sends an IDR frame acquisition request to the server, and when receiving the IDR frame sent by the server in response to the IDR frame acquisition request, all Frame packets in the buffer pool are cleared, and the Frame packet corresponding to the received IDR frame is sent to the decoding end.

[0105] By clearing the Frame packets in the buffer pool, the available space in the buffer pool can be increased, and more Frame packets can be cached, alleviating network congestion. Although clearing the Frame packets in the buffer pool has some impact on the video display effect, compared with the lag caused by network congestion, the impact of clearing the buffer pool is negligible.

[0106] In one embodiment of the present application, by alleviating network congestion, a smooth network can be ensured, which can ensure that the decoding end can smoothly digest the Frame packets sent by the buffer pool, and the buffer pool will not hoard Frame packets. When the client receives the Frame packets transmitted by the network, it can be directly sent to the decoding end. At this time, the buffer pool is equivalent to a transparent intermediate layer, and no processing of the Frame packets is required.

[0107] The buffer pool-based image decoding method in the embodiments of the present application can be applied to any scenario involving image frame processing, such as cloud gaming, video calls, live broadcasts, video playback, and other scenarios. Since cloud gaming, video calls, live broadcasts, video playback, and other scenarios have high requirements for image smoothness, network congestion can cause image freezes, seriously affecting the experience of players, callers, and video viewers. Therefore, the buffer pool-based image decoding method in the embodiments of the present application can be used to cache and decode received data packets to ensure the image quality and smoothness of games and videos.

[0108] Next, taking the cloud gaming scenario as an example, the buffer pool-based image decoding method in the embodiment of the present application is specifically described.

[0109] Cloud gaming is a game that runs on a remote server. Clients do not need to download or install the game, nor do they need to consider terminal configuration. As long as there is a network, they can play highly computationally intensive games. From a data transmission perspective, the remote server encodes the image information that constitutes each image frame of the cloud game to form a data packet corresponding to each image frame, and then sends the data packet to the client via the network. After receiving the data packet, the client splits it into frame packets and caches them in a buffer pool. The client then sends them to the decoder at a certain frame rate for decoding to obtain the corresponding image information. Finally, the obtained image information is rendered, and a clear and smooth picture can be seen on the client's display interface.

[0110] When the client receives a data packet, splits it into Frame packets and caches them in the buffer pool, it can specifically obtain the NALU type by extracting the NALU header in the data packet, and then splicing the RBSP data contained in multiple NALU units in the data packet to form a code stream corresponding to the image frame. Finally, the code stream is parsed by NALU according to the NALU type to obtain the frame information corresponding to the image frame. The frame information may include the frame type, timestamp, frame size and index number Index, and further, it may also include information such as resolution; then, a Frame object can be constructed based on the frame information and the memory address of the frame, and storage space can be applied to the buffer pool based on the Frame object. The buffer pool is divided into three parts: applied unused space, unapplied unused space, and used space. When the size of the unapplied unused space is greater than or equal to the image When the size of the Frame packet corresponding to the frame is determined, part or all of the unused space in the unused space is allocated as the requested space to the Frame packet according to the size of the Frame packet. Then, the remaining space in the requested unused space is checked to see if it is greater than or equal to the size of the Frame packet. If the remaining space is greater than or equal to the size of the Frame packet, the Frame packet is stored in the requested unused space and the unused space is released. If the remaining space is less than the size of the Frame packet, the used space is waited for to be released until the remaining space in the requested unused space is greater than or equal to the size of the Frame packet. If the unused space is less than the size of the Frame packet, the used space is waited for to be released until the remaining space in the requested unused space is greater than or equal to the size of the Frame packet. The Frame packet is constructed based on the code stream and frame information corresponding to the image frame.

[0111] When the network is congested, multiple Frame packets are cached in the buffer pool. In order to alleviate network congestion and improve the decoding performance of the decoding end, the buffer pool can send the cached Frame packets to the decoding end for decoding at the maximum frame sending rate. The maximum frame sending rate is the maximum release rate of the buffer pool, which can be obtained by dynamically detecting the load capacity of the decoding end and adjusting it based on the dynamic capacity. Specifically, the configuration parameters such as the resolution and set frame rate corresponding to the current cloud game can be obtained; then a test stream with the same configuration parameters can be obtained from the server in real time, and the test stream is input to the decoding end at the set frame rate for decoding. The output frame rate and the average decoding delay of a single frame of the test stream after decoding for 2s are calculated as the benchmark output frame rate and the benchmark average decoding delay of a single frame; then the benchmark output frame rate is compared with the first frame rate threshold. The first frame rate threshold is the result of the set frame rate and the first coefficient. The first coefficient is any value in the interval [0.85, 0.95]. When the benchmark output frame rate is less than the first frame rate threshold, When the set frame rate is used as the maximum frame rate, when the benchmark output frame rate is greater than or equal to the first frame rate threshold, the set frame rate is increased to obtain the frame rate, and the test stream is input to the decoder for decoding at the frame rate to calculate the output frame rate and average single-frame decoding delay of the test stream after 2s decoding. The set frame rate can be increased by a fixed or random increment. Finally, the maximum frame rate is determined based on the comparison results by comparing the output frame rate with the frame rate after applying the second coefficient, comparing the average single-frame decoding delay with the benchmark single-frame average decoding delay, and comparing the frame rate with the frame rate threshold. The second coefficient is also any value in the interval [0.85, 0.95] and can be the same as or different from the first coefficient.

[0112] After determining the maximum frame sending speed, the Frame packets in the buffer pool are sent to the decoding end for decoding at the maximum frame sending speed, and rendered according to the decoded image data to form a cloud game interface. Since the buffer pool can cache some Frame packets when the network is congested and send the Frame packets to the decoding end at a stable maximum frame sending speed, it can slow down the work efficiency of the decoding end caused by excessive workload due to network congestion, reduce encoding delay, and improve the gaming experience of cloud games.

[0113] Furthermore, when the network congestion duration is greater than a preset duration and the number of image frames waiting to apply for buffer pool storage space is greater than a preset threshold, the buffer pool can be cleared to reduce the decoding pressure caused by network congestion. When the buffer pool is cleared, if the Frame packets stored in the buffer pool contain IDR frames, the Frame packets with index numbers less than the target index number corresponding to the IDR frame are discarded, and the Frame packets containing the IDR frames are directly sent to the decoding end for decoding. If the Frame packets stored in the buffer pool do not contain IDR frames, a new IDR frame is requested from the server. When the Frame packet containing the IDR frame is received, the buffer pool is cleared, and the Frame packet is sent to the decoding end for decoding.

[0114] The buffer pool-based image decoding method in the present application, after obtaining the encoded code stream, constructs a frame object corresponding to a single frame image based on the encoded code stream, and sends an instruction to the buffer pool based on the Frame object to apply for storage space in the buffer pool. After successfully applying for storage space in the buffer pool, the Frame packet containing the code stream and frame information is stored in the buffer pool, and then the Frame packet stored in the buffer pool is input to the decoding end at the maximum frame sending rate. The maximum frame sending rate is obtained by adjusting the dynamic capacity of the decoding end after performing a dynamic capacity detection on the decoding end based on a test code stream with the same parameters as the single frame image. On the one hand, the present application can cache the data packets incoming from the network in the buffer pool when the network is congested, thereby avoiding the impact of a large amount of data on the decoding end, resulting in a decrease in the performance of the decoding end; on the other hand, it can obtain the maximum frame sending rate that maximizes the decoding performance of the decoding end by dynamically detecting the load capacity of the decoding end. When the Frame packet in the buffer pool is input to the decoding end at the maximum frame sending rate, it can not only alleviate network congestion, but also achieve the effect of stable flow control, thereby improving the decoding performance of the decoding end.

[0115] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0116] The following introduces an apparatus embodiment of the present application, which can be used to execute the buffer pool-based image decoding method in the above-mentioned embodiment of the present application. Figure 9 The structure block diagram of the image decoding device based on the buffer pool provided by the embodiment of the present application is schematically shown. Figure 9 As shown, the buffer pool-based image decoding device 900 includes: an object construction module 910, a space application module 920 and a decoding module 930. Specifically:

[0117] An object construction module 910 is configured to obtain a coded code stream and construct a Frame object corresponding to a single-frame image based on the coded code stream. A space application module 920 is configured to apply for storage space in a buffer pool based on the Frame object and store the Frame packet corresponding to the single-frame image in the buffer pool. A decoding module 930 is configured to input the Frame packet to a decoding end at a maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing a dynamic capability detection on the decoding end based on a test code stream having the same parameters as the single-frame image.

[0118] In some embodiments of the present application, based on the above technical solution, the object construction module 910 is configured to: parse the encoded code stream to obtain frame information corresponding to the single-frame image, the frame information including frame type, timestamp, frame size and serial number; construct the Frame object according to the frame information and the memory address corresponding to the single-frame image.

[0119] In some embodiments of the present application, the buffer pool includes: unapplied unused space, applied unused space and used space; the Frame object includes frame information and a memory address corresponding to the single-frame image; based on the above technical solution, the space application module 920 is configured to: send a storage space application instruction to the buffer pool based on the memory address, and determine the size of the Frame package according to the frame size in the frame information and the size of the frame information; when the size of the Frame package is less than or equal to the size of the unapplied unused space, obtain the space in the unapplied unused space as the applied space according to the size of the Frame package; and when the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame package, store the Frame package in the applied unused space.

[0120] In some embodiments of the present application, based on the above technical solution, the space application module 920 is further configured to: when the size of the Frame package is larger than the unapplied unused space, wait for the used space to be released until the size of the unapplied unused space is larger than or equal to the size of the Frame package; and, when the size of the remaining space in the applied unused space is smaller than the size of the Frame package, wait for the used space to be released until the size of the remaining space in the applied unused space is larger than or equal to the size of the Frame package.

[0121] In some embodiments of the present application, based on the above technical solution, the space application module 920 is further configured to: store the Frame packets in the buffer pool in sequence according to the timestamp and / or index number in the Frame object.

[0122] In some embodiments of the present application, the parameter includes a set frame rate. Based on the above technical solution, the decoding module 930 includes: a first input unit, configured to input the test code stream to the decoding end for decoding at the set frame rate, so as to obtain a benchmark output frame rate and a benchmark single-frame average decoding delay; a first comparison unit, configured to determine a first frame rate threshold according to the set frame rate and a first coefficient, and compare the benchmark output frame rate with the first frame rate threshold; a first determination unit, configured to use the set frame rate as the maximum frame sending speed when the benchmark output frame rate is less than the first frame rate threshold; a second input unit, configured to increase the set frame rate to obtain a frame sending speed when the benchmark output frame rate is greater than or equal to the first frame rate threshold, and input the test code stream to the decoding end for decoding at the frame sending speed, so as to obtain an output frame rate and a single-frame average decoding delay; and a first calculation unit, configured to determine the maximum frame sending speed based on the frame sending speed, the benchmark single-frame average decoding delay, the output frame rate, and the single-frame average decoding delay.

[0123] In some embodiments of the present application, based on the above technical solution, the first calculation unit includes: a second comparison unit, used to determine a second frame rate threshold based on the frame sending speed and the second coefficient, and compare the output frame rate with the second frame rate threshold; a second determination unit, used to use the frame sending speed as the maximum frame sending speed when the output frame rate is less than the second frame rate threshold; a second calculation unit, used to determine the maximum frame sending speed based on the frame sending speed, the benchmark single-frame average decoding delay and the single-frame average decoding delay when the output frame rate is greater than or equal to the second frame rate threshold.

[0124] In some embodiments of the present application, based on the above technical solution, the second calculation unit includes: a third comparison unit, used to compare the benchmark single-frame average decoding delay and the single-frame average decoding delay; a third determination unit, used to use the frame delivery speed as the maximum frame delivery speed when the single-frame average decoding delay is greater than the benchmark single-frame average decoding delay; a fourth calculation unit, used to determine the maximum frame delivery speed according to the frame delivery speed when the single-frame average decoding delay is less than or equal to the benchmark single-frame average decoding delay.

[0125] In some embodiments of the present application, based on the above technical solution, the third determination unit includes: a fourth comparison unit, used to compare the frame sending speed with the frame rate threshold; a fourth determination unit, used to use the frame sending speed as the maximum frame sending speed when the frame sending speed is greater than the frame rate threshold; an update unit, used to update the frame sending speed when the frame sending speed is less than or equal to the frame rate threshold, and input the test code stream to the decoding end for decoding at the updated frame sending speed to obtain an updated output frame rate and an updated single-frame average decoding delay; a fifth calculation unit, used to determine the maximum frame sending speed based on the updated frame sending speed, the benchmark single-frame average decoding delay, the updated output frame rate and the updated single-frame average decoding delay.

[0126] In some embodiments of the present application, based on the above technical solution, the buffer pool-based image decoding device 900 also includes: an acquisition module, used to obtain the network congestion duration and the number of image frames waiting to apply for buffer pool storage space; an emptying module, used to detect whether there is an immediately refreshed image IDR frame in the Frame packet stored in the buffer pool when the network congestion duration is greater than a preset duration and the number of image frames is greater than a preset threshold, and to clear the buffer pool according to the detection result.

[0127] In some embodiments of the present application, based on the above technical solution, the clearing module is configured as follows: when the IDR frame exists in the buffer pool, obtaining the target index number corresponding to the IDR frame; discarding the Frame packet in the buffer pool whose index number is less than the target index number, and at the same time sending the Frame packet corresponding to the IDR frame from the buffer pool to the decoding end.

[0128] In some embodiments of the present application, based on the above technical solution, the clearing module is configured to: when the IDR frame does not exist in the buffer pool, send an IDR frame acquisition request to the server; receive the IDR frame sent by the server in response to the IDR frame acquisition request, clear the buffer pool, and send the Frame packet containing the IDR frame from the buffer pool to the decoding end.

[0129] The specific details of the buffer pool-based image decoding device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0130] Figure 10 The computer system structure block diagram of the electronic device for implementing the embodiment of the present application is schematically shown. The electronic device may be as follows: Figure 1 The client 101 and the server 102 are shown in FIG.

[0131] It should be noted that Figure 10The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0132] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 (ROM) or the program loaded from the storage part 1008 into the random access memory 1003 (RAM). Various programs and data required for system operation are also stored in the random access memory 1003. The CPU 1001, the read-only memory 1002, and the random access memory 1003 are connected to each other via a bus 1004. An input / output interface 1005 (i.e., an I / O interface) is also connected to the bus 1004.

[0133] In some embodiments, the following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a local area network card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 1010 as needed, so that a computer program read therefrom can be installed into the storage section 1008 as needed.

[0134] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1009 and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit 1001, the various functions defined in the system of the present application are performed.

[0135] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0138] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable an electronic device to execute the method according to the embodiments of the present application.

[0139] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0140] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A buffer pool-based image decoding method, characterized in that: Applied to the client, including: Obtain a coded code stream, and construct a frame object corresponding to a single frame image based on the coded code stream; Applying for storage space in a buffer pool based on the Frame object, and storing the Frame package corresponding to the single-frame image in the buffer pool; Inputting the Frame packet to the decoding end at a maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing dynamic capability detection on the decoding end based on a test bitstream having the same parameters as the single frame image, and the maximum frame sending rate is used to indicate the frame sending rate of the buffer pool corresponding to the highest decoding performance of the decoding end; The parameters include setting a frame rate, and after performing dynamic capability detection on the decoding end according to a test bitstream having the same parameters as the single-frame image, adjusting the maximum frame sending rate based on the dynamic capability includes: Inputting the test code stream to the decoding end at the set frame rate for decoding to obtain a benchmark output frame rate and a benchmark single-frame average decoding delay; determining a first frame rate threshold according to the set frame rate and a first coefficient, and comparing the reference output frame rate with the first frame rate threshold; When the reference output frame rate is less than the first frame rate threshold, the set frame rate is used as the maximum frame sending speed.

2. The method according to claim 1, characterized in that The constructing a frame object corresponding to a single frame image based on the encoded code stream includes: Parsing the encoded code stream to obtain frame information corresponding to the single-frame image, the frame information including frame type, timestamp, frame size, and sequence number; The Frame object is constructed according to the frame information and the memory address corresponding to the single-frame image.

3. The method according to claim 1, characterized in that The buffer pool includes: unused space that has not been applied for, unused space that has been applied for, and used space; the Frame object includes frame information and memory address corresponding to the single frame image; The applying for storage space in a buffer pool based on the Frame object and storing the Frame package corresponding to the single-frame image in the buffer pool includes: Sending a storage space request instruction to the buffer pool based on the memory address, and determining the size of the Frame packet according to the frame size in the frame information and the size of the frame information; When the size of the Frame packet is less than or equal to the size of the unapplied and unused space, obtaining space in the unapplied and unused space as the applied space according to the size of the Frame packet; and When the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame packet, the Frame packet is stored in the applied unused space.

4. The method according to claim 3, characterized in that The method further comprises: When the size of the Frame packet is larger than the unapplied unused space, waiting for the used space to be released until the size of the unapplied unused space is larger than or equal to the size of the Frame packet; and When the size of the remaining space in the applied unused space is smaller than the size of the Frame packet, wait for the used space to be released until the size of the remaining space in the applied unused space is greater than or equal to the size of the Frame packet.

5. The method according to claim 4, characterized in that The method further comprises: The received Frame packets are stored in the buffer pool in sequence according to the timestamp and / or index number in the Frame object.

6. The method according to claim 1, characterized in that After detecting the dynamic capability of the decoding end according to the test bitstream having the same parameters as the single-frame image, adjusting the maximum frame sending rate based on the dynamic capability includes: When the benchmark output frame rate is greater than or equal to the first frame rate threshold, increasing the set frame rate to obtain a frame sending speed, and inputting the test stream to the decoding end for decoding at the frame sending speed to obtain an output frame rate and a single-frame average decoding delay; The maximum frame sending speed is determined according to the frame sending speed, the benchmark single-frame average decoding delay, the output frame rate and the single-frame average decoding delay.

7. The method according to claim 6, characterized in that The determining the maximum frame sending speed according to the frame sending speed, the benchmark single-frame average decoding delay, the output frame rate, and the single-frame average decoding delay includes: determining a second frame rate threshold according to the frame sending speed and the second coefficient, and comparing the output frame rate with the second frame rate threshold; When the output frame rate is less than the second frame rate threshold, using the frame sending speed as the maximum frame sending speed; When the output frame rate is greater than or equal to the second frame rate threshold, a maximum frame sending rate is determined according to the frame sending rate, the benchmark single-frame average decoding delay, and the single-frame average decoding delay.

8. The method according to claim 7, characterized in that The determining the maximum frame sending speed according to the frame sending speed, the benchmark single-frame average decoding delay, and the single-frame average decoding delay includes: Comparing the benchmark single-frame average decoding delay with the single-frame average decoding delay; When the benchmark single-frame average decoding delay is greater than the single-frame average decoding delay, the frame sending speed is used as the maximum frame sending speed; When the benchmark single-frame average decoding delay is less than or equal to the single-frame average decoding delay, the maximum frame sending speed is determined according to the frame sending speed.

9. The method according to claim 8, characterized in that Determining the maximum frame sending speed according to the frame sending speed includes: Comparing the frame sending speed with a frame rate threshold; When the frame sending speed is greater than the frame rate threshold, the frame sending speed is used as the maximum frame sending speed; When the frame sending speed is less than or equal to the frame rate threshold, updating the frame sending speed, and inputting the test code stream to the decoding end for decoding at the updated frame sending speed to obtain an updated output frame rate and an updated single-frame average decoding delay; The maximum frame sending speed is determined according to the updated frame sending speed, the reference single-frame average decoding delay, the updated output frame rate and the updated single-frame average decoding delay.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Obtain the network congestion duration and the number of image frames waiting to apply for buffer pool storage space; When the network congestion duration is greater than a preset duration and the number of image frames is greater than a preset threshold, it is detected whether there is an immediate refresh image IDR frame in the Frame packets stored in the buffer pool, and the buffer pool is cleared according to the detection result.

11. The method according to claim 10, characterized in that The detecting whether there is an immediate refresh image IDR frame in the Frame packets stored in the buffer pool, and clearing the buffer pool according to the detection result, includes: When the IDR frame exists in the buffer pool, obtaining a target index number corresponding to the IDR frame; The Frame packets in the buffer pool whose index numbers are smaller than the target index number are discarded, and the Frame packets corresponding to the IDR frame are sent from the buffer pool to the decoding end.

12. The method according to claim 10, characterized in that The detecting whether there is an immediate refresh image IDR frame in the Frame packets stored in the buffer pool, and clearing the buffer pool according to the detection result, includes: When the IDR frame does not exist in the buffer pool, sending an IDR frame acquisition request to the server; The IDR frame sent by the server in response to the IDR frame acquisition request is received, the buffer pool is cleared, and the Frame packet containing the IDR frame is sent from the buffer pool to the decoding end.

13. An image decoding device based on a buffer pool, characterized in that: Configuration on the client, including: An object construction module is used to obtain a coding stream and construct a frame object corresponding to a single frame image based on the coding stream; a space application module, configured to apply for storage space in a buffer pool based on the Frame object, and store the Frame package corresponding to the single-frame image in the buffer pool; a decoding module, configured to input the Frame packet to a decoding end at a maximum frame sending rate, wherein the maximum frame sending rate is obtained by adjusting the dynamic capability of the decoding end after performing dynamic capability detection on the decoding end based on a test bitstream having the same parameters as the single-frame image, and the maximum frame sending rate is used to indicate the frame sending rate of the buffer pool corresponding to the highest decoding performance of the decoding end; The parameters include setting a frame rate, and after performing dynamic capability detection on the decoding end according to a test bitstream having the same parameters as the single-frame image, adjusting the maximum frame sending rate based on the dynamic capability includes: Inputting the test code stream to the decoding end at the set frame rate for decoding to obtain a benchmark output frame rate and a benchmark single-frame average decoding delay; determining a first frame rate threshold according to the set frame rate and a first coefficient, and comparing the reference output frame rate with the first frame rate threshold; When the reference output frame rate is less than the first frame rate threshold, the set frame rate is used as the maximum frame sending speed.

14. A computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the buffer pool-based image decoding method according to any one of claims 1 to 12 is implemented.

15. An electronic device, characterized in that: include: processor; as well as a memory for storing instructions; The processor executes the instructions stored in the memory to implement the buffer pool-based image decoding method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The method comprises computer instructions, which, when executed on a computer, enable the computer to execute the buffer pool-based image decoding method according to any one of claims 1 to 12.

Images