Video playing method, device, equipment and program product

Abstract

The invention provides a video playing method and device, equipment and a program product. The method comprises the following steps: acquiring the number of jamming times of a video decoder in a historical period; determining a video decoder based on a size relationship between the number of jamming times and a jamming threshold value of the current period; decoding the video stream data based on the video decoder in the current period; and when decoding fails, switching the video decoder in the current period from the hard decoder to the soft decoder, and re-decoding the video stream data based on the soft decoder to obtain video data corresponding to the video stream data. According to the invention, the decoding performance and the playing experience can be considered in the video playing process.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and in particular, to a video playing method, apparatus, device, and program product. Background Art

[0002] In current online video players, due to the diversity of systems, there are deficiencies in the compatibility of system players. Generally, professional adaptive players need to use self-developed players. In order to provide users with high-definition picture quality and smooth playing experience at the video playing end, this leads to an increasing demand for the performance of video decoders on mobile devices, and the audio and video decoding capabilities of underlying hardware need to be adopted. However, due to the diversity of system devices, compatibility problems such as the decoder getting stuck and unable to output pictures, and insufficient decoding performance resulting in unsmooth pictures often occur.

[0003] In related technologies, there is a lack of an effective solution that takes into account both decoding performance and playing experience during video playback. Summary of the Invention

[0004] Embodiments of this application provide a video playing method, apparatus, device, computer-readable storage medium, and computer program product, which can take into account both decoding performance and playing experience.

[0005] The technical solution of the embodiments of this application is implemented as follows:

[0006] Embodiments of this application provide a video playing method, the method includes:

[0007] Obtain the number of times the video decoder gets stuck within a historical period;

[0008] When the number of times of getting stuck is greater than or equal to the stuck threshold of the current period, determine the video decoder within the current period as a software decoder; when the number of times of getting stuck is less than the stuck threshold of the current period, determine the video decoder within the current period as a hardware decoder;

[0009] Decode the video stream data based on the video decoder within the current period;

[0010] When the decoding fails, switch the video decoder within the current period from the hardware decoder to the software decoder, and re-decode the video stream data based on the software decoder to obtain the video data corresponding to the video stream data.

[0011] Embodiments of this application provide a video playing apparatus, including:

[0012] A data acquisition module, configured to obtain the number of times the video decoder gets stuck within a historical period;

[0013] A decoding strategy module, configured to determine the video decoder in the current cycle as a soft decoder when the stuck count is greater than or equal to the stuck threshold of the current cycle, and determine the video decoder in the current cycle as a hard decoder when the stuck count is less than the stuck threshold of the current cycle;

[0014] A video decoding module, configured to decode video stream data based on the video decoder in the current cycle;

[0015] A decoding stuck recognition module, configured to switch the video decoder in the current cycle from the hard decoder to the soft decoder when the decoding fails, and re-decode the video stream data based on the soft decoder to obtain video data corresponding to the video stream data;

[0016] In the above solution, the video decoding module is further configured to construct a first thread corresponding to the video decoder; initialize the video decoder in the current cycle based on the first thread to obtain the initialized video decoder; and decode the video stream data based on the initialized video decoder;

[0017] In the above solution, the decoding stuck recognition module is further configured to determine the playback state of the video decoder in the current cycle as a stuck state; and increase the stuck count of the video decoder in the historical cycle of the next cycle when the playback state is the stuck state;

[0018] In the above solution, the decoding stuck recognition module is further configured to send a callback message to the playback control layer, where the callback message is used to prompt the playback control layer that the video decoder in the current cycle fails to decode and the rendering attributes in the current cycle need to be updated;

[0019] In the above solution, a player module is configured to update the rendering attributes in the current cycle in response to receiving an update instruction for the rendering attributes in the current cycle sent by the playback control layer, where the updated rendering attributes are used to successfully render the video data to the terminal, and the terminal is used to play the video data;

[0020] In the above solution, the video decoding module is further configured to construct a second thread corresponding to the soft decoder; initialize the soft decoder based on the second thread to obtain the initialized soft decoder; and decode the video stream data based on the initialized soft decoder;

[0021] In the above solution, the video decoding module is further configured to construct a third thread; and release the hard decoder in the current cycle and the first thread corresponding to the hard decoder in the current cycle based on the third thread;

[0022] In the above solution, a protocol parsing module is configured to obtain a playback address of a video and obtain streaming media data based on the playback address; parse the streaming media data based on a streaming media protocol to obtain the video stream data;

[0023] In the above solution, the video decoding module is further configured to obtain a stuck threshold of the previous cycle; when hard decoders in the previous N consecutive cycles of the current cycle all fail to decode, reduce the stuck threshold of the previous cycle to obtain the stuck threshold of the current cycle, where N is a positive integer greater than 1;

[0024] In the above solution, the video decoding module is further configured to obtain a stuck threshold of the previous cycle; when the video decoder in the previous cycle is a software decoder, determine a target cycle, where the target cycle is the cycle closest to the current cycle among multiple cycles in the historical cycle in which the video decoder is switched from the hard decoder to the software decoder; when the number of cycles between the target cycle and the current cycle is greater than a number threshold, increase the stuck threshold of the previous cycle to obtain the stuck threshold of the current cycle;

[0025] In the above solution, the decoding strategy module is further configured to obtain multiple hard decoders from a set of video decoders, where the performances of the multiple hard decoders are different; divide the stuck threshold of the current cycle based on the performances of the multiple hard decoders to obtain multiple stuck threshold intervals, where the stuck threshold intervals and the hard decoders are in one-to-one correspondence, and the stuck threshold corresponding to the stuck threshold interval is negatively correlated with the performance of the hard decoder; when the number of stuck times is within a target stuck threshold interval, determine the video decoder in the current cycle as the hard decoder corresponding to the target stuck threshold interval, where the target stuck threshold interval is one of the multiple stuck threshold intervals;

[0026] In the above solution, the video decoding module is further configured to perform a prediction process by invoking a neural network model based on the number of stuck times in the previous cycle to obtain the stuck threshold of the current cycle, where the neural network model is trained based on the number of stuck times in the previous sample cycle and the label of the stuck threshold.

[0027] An embodiment of the present application provides an electronic device, including:

[0028] A memory for storing computer-executable instructions;

[0029] A processor, when executing the computer-executable instructions stored in the memory, implements the video playback method provided by the embodiment of the present application.

[0030] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement the video playback method provided by the embodiment of the present application when being executed by a processor.

[0031] An embodiment of the present application provides a computer program product including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the video playback method provided by the embodiment of the present application is implemented.

[0032] The embodiment of the present application has the following beneficial effects:

[0033] Based on the magnitude relationship between the number of freezes and the freeze threshold of the current period, different video decoders are determined for the current period, avoiding the video playback continuously entering the frozen state for multiple times, taking into account both the decoding performance of the decoder and the playback experience. Then, based on the video decoder within the current period, the video stream data is decoded. When the decoding fails, the video decoder within the current period is switched from a hard decoder to a soft decoder, and the video stream data is re-decoded based on the soft decoder to obtain the video data for playback, avoiding the user being unable to obtain the video data for playback within the current period, thereby improving the user's playback experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic structural diagram of the video playback system architecture provided by the embodiment of the present application;

[0035] Figure 2 is a schematic structural diagram of the electronic device provided by the embodiment of the present application;

[0036] Figure 3A is a first flowchart of the video playback method provided by the embodiment of the present application;

[0037] Figure 3B is a second flowchart of the video playback method provided by the embodiment of the present application;

[0038] Figure 3C is a third flowchart of the video playback method provided by the embodiment of the present application;

[0039] Figure 3D is a fourth flowchart of the video playback method provided by the embodiment of the present application;

[0040] Figure 3E is a fifth flowchart of the video playback method provided by the embodiment of the present application;

[0041] Figure 3F is a sixth flowchart of the video playback method provided by the embodiment of the present application;

[0042] Figure 3GIt is the seventh process schematic diagram of the video playback method provided by the embodiments of the present application;

[0043] Figure 3H It is the eighth process schematic diagram of the video playback method provided by the embodiments of the present application;

[0044] Figure 3I It is the ninth process schematic diagram of the video playback method provided by the embodiments of the present application;

[0045] Figure 4 It is the schematic diagram of the video playback principle provided by the embodiments of the present application. Detailed implementation manners

[0046] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0047] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0048] In the following description, the terms "first / second / third" are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0050] In the embodiments of the present application, when collecting and processing relevant data in practical applications, the informed consent or separate consent of the personal information subject should be obtained strictly in accordance with the requirements of relevant national laws and regulations, and subsequent data use and processing behaviors should be carried out within the scope authorized by laws and regulations and the personal information subject.

[0051] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described, and the nouns and terms involved in the embodiments of this application are applicable to the following explanations.

[0053] 1) Cycle. In a video playback scenario, when the application completes initialization, one video playback of the application is equivalent to one cycle. When the current video is closed and the next video is opened, the next cycle is entered.

[0054] 2) Hard decoder. A decoder that calls the decoding ability of the hardware to decode video stream data. The hard decoder has high playback fluency and low occupancy rate of the Central Processing Unit (CPU).

[0055] 3) Soft decoder. A decoder that calls the decoding ability of the software to perform simulated decoding of video stream data. The computing power of the soft decoder comes from the CPU, and the utilization rate of the hardware is relatively low.

[0056] 4) Decoder set. The decoder set includes at least one hard decoder corresponding to the graphics processor (i.e., hardware), and the performance of the hard decoder corresponds one-to-one with the performance of the graphics processor.

[0057] 5) Rendering attribute. A drawing buffer area (Surface) on the memory for rendering processing. The drawing buffer area corresponds to the video decoder and is used to render video data into the terminal.

[0058] In the related art, when the server recognizes that the decoder freeze problem occurs a certain number of times, the next playback will directly downgrade to software decoding playback, but the current playback still shows a black screen, which still affects the user's playback experience. When the video playback downgrades from hardware decoding playback to software decoding playback, software decoding is always used, and the decoding performance is poor, which is not suitable for video playback.

[0059] Based on the above analysis, the applicant found that the decoding downgrading method in the related art cannot balance the decoding performance and playback experience of video playback. To address the above problems, the embodiments of this application provide a video playback method, device, equipment, computer-readable storage medium, and computer program product that can balance decoding performance and playback experience.

[0060] See Figure 1 , Figure 1It is a schematic diagram of the video playback system architecture provided by the embodiments of the present application. To support a video playback application, a terminal (exemplarily showing terminal 400) is connected to a server 200 through a network 300. The network 300 can be a wide area network, a local area network, or a combination of the two.

[0061] The terminal 400 is used to send the number of times the video decoder has jammed during the historical period to the server 200 through the network 300. The server 200 is used to determine the video decoder for decoding based on the magnitude relationship between the number of jams and the jam threshold of the current period, decode the video stream data based on the video decoder in the current period. When the decoding fails, the video decoder in the current period is switched from a hard decoder to a soft decoder, and the video stream data is re-decoded based on the soft decoder to obtain the video data corresponding to the video stream data, and the obtained video data is returned to the terminal 400. The terminal 400 displays the obtained video data through a graphical interface 410.

[0062] The following illustrates an example of the terminal 400 for video playback.

[0063] In some embodiments, the terminal 400 can independently complete the video playback task. For example, the terminal 400 is used to obtain the number of times the video decoder has jammed during the historical period, and determine the video decoder for decoding based on the magnitude relationship between the number of jams and the jam threshold of the current period relying on the computing resources of the terminal 400 itself, decode the video stream data based on the video decoder in the current period. When the decoding fails, the video decoder in the current period is switched from a hard decoder to a soft decoder, and the video stream data is re-decoded based on the soft decoder to obtain the video data corresponding to the video stream data, and the obtained video data is displayed through the graphical interface 410.

[0064] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0065] The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present application.

[0066] Embodiments of this application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, be used as needed, and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the high development and application of the Internet industry, and the promotion of demands such as search services, social networks, mobile commerce, and open collaboration, in the future, each item may have its own hash code identification mark and needs to be transmitted to the background system for logical processing. Data at different levels will be processed separately, and various types of industry data require the support of a powerful system background, which can only be achieved through cloud computing.

[0067] Embodiments of this application can be implemented through artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0068] Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or basic model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] See Figure 2 , Figure 2 is a schematic structural diagram of an electronic device provided by an embodiment of this application for implementing a video playback method. Figure 2 The electronic device 500 shown can be Figure 1Among the terminal 400 or the server 200, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. Each component in the server 200 is coupled together through a bus system 540. It can be understood that the bus system 440 is used to implement the connection and communication between these components. The bus system 540 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 540.

[0070] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, and other input buttons and controls;

[0072] In some embodiments, when the embodiment independently completes the video playback task by the terminal 400, the server 200 provided by the embodiments of the present application does not include the user interface 530.

[0073] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disc drives, etc. The memory 550 optionally includes one or more storage devices that are physically located away from the processor 510.

[0074] The memory 550 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), and the volatile memory can be a random access memory (RAM, Random Access Memory). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, the memory 550 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are illustrated below.

[0076] An operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0077] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), etc.;

[0078] A presentation module 553, for enabling the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with a user interface 530 (e.g., a display screen, a speaker, etc.);

[0079] In some embodiments, when the terminal 400 independently completes a video playback task, the server 200 provided by the embodiments of the present application may not include the presentation module 553;

[0080] An input processing module 554, for detecting and translating one or more user inputs or interactions from one of one or more input devices 532; in some embodiments, when the terminal 400 independently completes a video playback task, the server 200 provided by the embodiments of the present application may not include the presentation module 553.

[0081] In some embodiments, the device provided by the embodiments of the present application may be implemented in software. Figure 2 A video playback device 555 stored in the memory 550 is shown, which may be software in the form of programs and plugins, etc., including the following software modules: a data acquisition module 5551, a decoding strategy module 5552, a video decoding module 5553, a decoding freeze identification module 5554, a player module 5555, and a protocol parsing module 5556. These modules are logical, so they can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0082] In some other embodiments, the device provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video playing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0083] It should be noted that in the examples of video playing below, those skilled in the art can apply the video playing method provided by the embodiments of the present application to the processing of video playing according to the understanding of the following text.

[0084] See Figure 3A , Figure 3A is the first flowchart of the video playing method provided by the embodiments of the present application, which will be described in combination with the steps shown in Figure 3A The video playing method provided by the embodiments of the present application can be implemented independently by a server or a terminal, or jointly implemented by a server and a terminal. The following will take the joint implementation of a server and a terminal as an example for description.

[0085] In step 101, obtain the number of times the video decoder gets stuck within a historical period.

[0086] In some embodiments, the number of stuck times is the number of times the video decoder is in a stuck state within a historical period, that is, the number of times the video decoder switches from a hard decoder to a soft decoder within a historical period. Among them, when the player is in the initial startup stage, that is, the current period is the initial period, the number of times the video decoder gets stuck within the historical period is set to 0.

[0087] In step 102, when the number of stuck times is greater than or equal to the stuck threshold of the current period, determine the video decoder in the current period as a soft decoder; when the number of stuck times is less than the stuck threshold of the current period, determine the video decoder in the current period as a hard decoder.

[0088] It should be noted that the stuck threshold for each period can be fixed or dynamically changed.

[0089] Taking the dynamically changing jamming threshold as an example, when the current cycle is the initial cycle, the jamming threshold is the preset jamming threshold, and when the current cycle is not the initial cycle, the jamming threshold can be dynamically configured.

[0090] Exemplarily, when the number of jamming times with a value of 3 is greater than or equal to the jamming threshold of the current cycle with a value of 3, the video decoder in the current cycle is determined to be a soft decoder, and when the number of jamming times with a value of 2 is less than the jamming threshold of the current cycle with a value of 3, the video decoder in the current cycle is determined to be a hard decoder.

[0091] In some embodiments, referring to Figure 3B , Figure 3B is the second process schematic diagram of the video playing method provided by the embodiments of the present application. For Figure 3A in step 102 of Figure 3B determining the video decoder in the current cycle as a hard decoder, it can be implemented through

[0092] In step 1021, obtain multiple hard decoders from the video decoder set.

[0093] Among them, the performances of multiple (i.e., at least two) hard decoders are different.

[0094] In some embodiments, the video decoder set contains multiple hard decoders with different performances, and the order of the multiple hard decoders is sorted from high to low according to the performance of the hard decoders.

[0095] Exemplarily, the video decoder set contains a first hard decoder and a second hard decoder. Among them, the performance of the first hard decoder is higher than that of the second hard decoder, and the order of the hard decoders in the video decoder set is the first hard decoder, the second hard decoder.

[0096] In step 1022, based on the performances of the multiple hard decoders, divide the jamming threshold of the current cycle to obtain multiple jamming threshold intervals.

[0097] Among them, the jamming threshold intervals and the hard decoders are in one-to-one correspondence, and the jamming threshold corresponding to the jamming threshold interval is negatively correlated with the performance of the hard decoder.

[0098] In some embodiments, the jamming threshold intervals can be evenly divided or unevenly divided. Through different division methods, the hard decoders are divided into different jamming threshold intervals. The higher the performance of the hard decoder, the smaller the value of the jamming threshold of the corresponding jamming threshold interval, and the lower the performance of the hard decoder, the larger the value of the jamming threshold of the corresponding jamming threshold interval.

[0099] For example, the stuck threshold for the current cycle is evenly divided. When the stuck threshold is 4 and the set of video decoders includes 2 hard decoders, the stuck threshold for the current cycle is divided into a first interval with a stuck threshold of [1, 2] and a second interval with a stuck threshold of [3, 4]. The hard decoders are sorted from highest to lowest performance to obtain a first decoder and a second decoder respectively. Since there is a one-to-one correspondence between the stuck threshold intervals and the hard decoders, when the number of stuck times is in the first interval, the first decoder is used as the video decoder for the current cycle; when the number of stuck times is in the second interval, the second decoder is used as the video decoder for the current cycle.

[0100] For example, the stuck threshold for the current cycle is unevenly divided. When the stuck threshold is 4 and the set of video decoders includes 2 hard decoders, the stuck threshold for the current cycle is divided into a first interval with a stuck threshold of [1, 3] and a second interval with a stuck threshold of [4]. The hard decoders are sorted from highest to lowest performance to obtain a first decoder and a second decoder respectively. Since there is a one-to-one correspondence between the stuck threshold intervals and the hard decoders, when the number of stuck times is in the first interval, the first decoder is used as the video decoder for the current cycle; when the number of stuck times is in the second interval, the second decoder is used as the video decoder for the current cycle.

[0101] In step 1023, when the number of stuck times is within the target stuck threshold interval, the video decoder for the current cycle is determined as the hard decoder corresponding to the target stuck threshold interval.

[0102] Among them, the target stuck threshold interval is one of the multiple stuck threshold intervals.

[0103] In some embodiments, the stuck threshold interval includes the lower bound and the upper bound corresponding to the stuck threshold interval, and the upper bound is greater than or equal to the lower bound. When the number of stuck times is greater than or equal to the lower bound of a certain stuck threshold interval and less than the upper bound of the current stuck threshold interval, this stuck threshold interval is used as the target stuck threshold interval.

[0104] For example, the lower bound of a certain stuck threshold interval is 2 and the upper bound is 4. Since the number of stuck times of 3 satisfies being less than the upper bound of this stuck threshold interval and greater than the lower bound of this stuck threshold interval, this stuck threshold interval is used as the target stuck threshold interval, and the video decoder for the current cycle is determined as the hard decoder corresponding to the target stuck threshold interval.

[0105] Through the embodiments of the present application, the hard decoder for the current cycle is determined based on the number of stuck times in the historical cycle, and by matching hard decoders with different performances, it is ensured that users preferentially use hard decoders for video playback, taking into account both the decoding performance of the decoder and the playback experience.

[0106] Through the embodiments of the present application, different video decoders are determined based on the relationship between the number of stuck times and the stuck threshold of the current cycle, so as to avoid the video playback entering the stuck state continuously for multiple times and improve the user's playback experience.

[0107] Continue to refer to Figure 3A , in step 103, the video decoder in the current cycle is used to decode the video stream data.

[0108] In some embodiments, the video stream data is obtained by pre-encoding the video data and can be decoded by the video decoder in the current cycle. When the decoding is successful, the video data corresponding to the video stream data is obtained.

[0109] In some embodiments, refer to Figure 3C , Figure 3C is the third process schematic diagram of the video playback method provided by the embodiments of the present application. Figure 3A The steps shown in 103 can be implemented through Figure 3C steps 1031 to 1033, which will be specifically described below.

[0110] In step 1031, a first thread corresponding to the video decoder is constructed.

[0111] In some embodiments, a fourth thread for creating memory resources is constructed, and a first thread corresponding to the video decoder is constructed based on the fourth thread. The first thread corresponding to the video decoder is used to execute the initialization process of the video decoder, and the fourth thread is used to execute the construction of the first thread corresponding to the video decoder.

[0112] In step 1032, the video decoder in the current cycle is initialized based on the first thread to obtain the initialized video decoder.

[0113] In some embodiments, the video decoder in the current cycle is initialized based on the first thread. During the initialization process, the parameters corresponding to the video decoder are obtained, and the parameters are assigned to the video decoder to obtain the initialized video decoder.

[0114] In step 1033, the video stream data is decoded based on the initialized video decoder.

[0115] In some embodiments, the video stream data is a video vector obtained by encoding the original video data by an encoder, and the video vector is decoded in the video decoder to obtain the video data corresponding to the video stream data.

[0116] Continue to refer to Figure 3A, in step 104, when decoding fails, switch the video decoder in the current cycle from a hard decoder to a soft decoder, and re-decode the video stream data based on the soft decoder to obtain the video data corresponding to the video stream data.

[0117] In some embodiments, when decoding fails, the video stream data is re-decoded using a soft decoder, so that video data can still be obtained when the current cycle is stuck.

[0118] Here, when decoding fails, a third thread for releasing memory resources is constructed, and based on the third thread, the hard decoder in the current cycle and the first thread corresponding to the hard decoder in the current cycle are released.

[0119] Exemplarily, a third thread different from the first thread corresponding to the hard decoder in the current cycle is constructed, and instructions for releasing the hard decoder in the current cycle and the first thread corresponding to the hard decoder in the current cycle are executed through the third thread to release the hard decoder in the current cycle and the first thread corresponding to the hard decoder in the current cycle. By executing the release of the first thread corresponding to the hard decoder through the third thread, the newly constructed thread can release the first thread corresponding to the stuck hard decoder, thus avoiding the problem that the video cannot continue to play due to the hard decoder being stuck and improving the smoothness of video playback.

[0120] In some embodiments, refer to Figure 3D , Figure 3D is the fourth process schematic diagram of the video playback method provided by the embodiments of the present application. For Figure 3A the re-decoding of the video stream data based on the soft decoder in step 104, it can be implemented through Figure 3D steps 1041 to 1043 of

[0121] In step 1041, a second thread corresponding to the soft decoder is constructed.

[0122] In some embodiments, a fourth thread for creating memory resources is constructed, and a second thread corresponding to the soft decoder is constructed based on the fourth thread. Among them, the second thread corresponding to the soft decoder is used to execute the initialization process of the soft decoder, and the fourth thread is used to execute the construction of the second thread corresponding to the soft decoder.

[0123] In step 1042, the soft decoder is initialized based on the second thread to obtain an initialized soft decoder.

[0124] In some embodiments, the soft decoder in the current cycle is initialized based on the first thread. During the initialization process, the parameters corresponding to the soft decoder are obtained and assigned to the soft decoder to obtain an initialized soft decoder.

[0125] In step 1043, the video stream data is decoded based on the initialized soft decoder.

[0126] In some embodiments, the video stream data is a video vector obtained by encoding the original video data by an encoder, and the video vector is decoded in the soft decoder to obtain the video data corresponding to the video stream data.

[0127] Through the embodiments of the present application, when the current cycle is in a stuck state, the video data obtained by re-decoding the soft decoder can be rendered to the terminal, improving the user's playback experience.

[0128] In some embodiments, refer to Figure 3E , Figure 3E is the fifth process schematic diagram of the video playback method provided by the embodiments of the present application. Before switching the video decoder in the current cycle from the hard decoder to the soft decoder in step 104, steps 201 to 202 of Figure 3E are executed, which will be specifically described below.

[0129] In step 201, the playback state of the video decoder in the current cycle is determined to be a stuck state.

[0130] In some embodiments, the initial state of the playback state of the video decoder in each cycle is a normal state.

[0131] In step 202, when the playback state is a stuck state, the number of stuck times of the video decoder in the historical cycle of the next cycle is increased.

[0132] In some embodiments, the number of stuck times of the video decoder in the historical cycle of the next cycle is increased to accurately record the number of stuck times of the video decoder, so as to determine whether the video decoder is a hard decoder or a soft decoder based on the number of stuck times and the stuck threshold in the next cycle.

[0133] For example, when the value of the number of stuck times in the current cycle is 2, when it is detected that the playback state in the current cycle is a stuck state, the value of the number of stuck times in the current cycle is incremented by 1 to obtain the value of the number of stuck times in the current cycle.

[0134] Continuing with the above example, the number of stuck times in the current cycle with a value of 3 is obtained, that is, the value of the number of stuck times in the historical cycle of the next cycle is 3, and the stuck threshold in the current cycle is 3. Then, in the next cycle, since the number of stuck times in the historical cycle of the next cycle is equal to the stuck threshold in the next cycle, the video decoder in the next cycle is determined to be a soft decoder.

[0135] Continuing with the above example, the number of freezes in the current cycle with a value of 3 is obtained. That is, the number of freezes in the historical cycle of the next cycle has a value of 3, and the freeze threshold in the current cycle is 4. Then, in the next cycle, since the number of freezes in the historical cycle of the next cycle is less than the freeze threshold of the next cycle, the video decoder in the next cycle is determined to be the hard decoder.

[0136] Through the embodiments of the present application, when decoding fails, the number of freezes of the video decoder is accurately recorded to determine the number of freezes in the historical cycle of the next cycle, so as to determine the video decoder based on the number of freezes. When the number of freezes in the historical cycle of the next cycle is greater than or equal to the freeze threshold, the hard decoder is no longer used for decoding, avoiding the situation of decoding failure caused by using the hard decoder for decoding, and improving the playback efficiency of video playback.

[0137] In some embodiments, referring to Figure 3F , Figure 3F is the sixth flowchart of the video playback method provided by the embodiments of the present application. Before switching the video decoder in the current cycle from the hard decoder to the soft decoder in step 104, steps 301 to 302 of Figure 3F are executed, which will be specifically described below.

[0138] In step 301, a callback message is sent to the playback control layer.

[0139] Among them, the callback message is used to prompt the playback control layer that the video decoder in the current cycle has decoding failure and the rendering attributes in the current cycle need to be updated.

[0140] In some embodiments, when the video decoder decodes fails, the video decoder needs to be decoded again to obtain video data. Since there is a corresponding relationship between the current video decoder and the current rendering attributes, the video data obtained by re - decoding cannot be rendered to the terminal through the current rendering attributes. In order to enable the terminal to render the video data obtained after re - decoding, a callback message is sent to the playback control layer, and the playback control layer is used to instruct the player to update the rendering attributes.

[0141] In step 302, in response to receiving the update instruction for the rendering attributes in the current cycle sent by the playback control layer, the rendering attributes in the current cycle are updated.

[0142] Among them, the updated rendering attributes are used to successfully render the video data to the terminal, and the terminal is used to play the video data.

[0143] In some embodiments, after receiving the callback message, the playback control layer sends an update instruction for the rendering attributes in the current cycle.

[0144] In some embodiments, there is a corresponding relationship between the updated rendering attributes and the software decoder, so that the video data obtained by decoding the video stream data by the software decoder can be successfully rendered to the terminal.

[0145] In some embodiments, refer to Figure 3G , Figure 3G which is the seventh process schematic diagram of the video playback method provided by the embodiments of the present application. Before obtaining the number of freezes of the video decoder in the historical period in step 101, execute Figure 3G steps 401 to 402 of

[0146] In step 401, obtain the playback address of the video, and obtain the streaming media data based on the playback address.

[0147] In some embodiments, the streaming media data includes video stream data encoded based on the original video data and audio stream data encoded based on the original audio data.

[0148] In step 402, parse the streaming media data based on the streaming media protocol to obtain the video stream data.

[0149] In some embodiments, the Streaming Protocol is a protocol used to transmit streaming media data over a network. The streaming media protocol is used to decompose the streaming media data into video stream data and audio stream data.

[0150] In some embodiments, refer to Figure 3H , Figure 3H which is the eighth process schematic diagram of the video playback method provided by the embodiments of the present application. After obtaining the number of freezes of the video decoder in the historical period in step 101, execute Figure 3H steps 501 to 502 of

[0151] In step 501, obtain the freeze threshold of the previous cycle.

[0152] In some embodiments, the freeze threshold of the current cycle can be dynamically updated according to the freeze threshold of the previous cycle.

[0153] In step 502, when the hard decoder fails to decode in the first N consecutive cycles of the current cycle, reduce the freeze threshold of the previous cycle to obtain the freeze threshold of the current cycle, where N is a positive integer greater than 1.

[0154] For example, when N is set to 3, when the hard decoder fails to decode in the first 3 consecutive cycles of the current cycle, and the freeze threshold of the previous cycle is 8, subtract 1 from the freeze threshold of the previous cycle to obtain the freeze threshold of the current cycle with a value of 7.

[0155] According to the embodiments of the present application, when the video decoder continuously fails to decode, the stuck threshold is reduced, thereby reducing the maximum number of stuck times that can directly use the soft decoder for degradation. That is, when a certain number of stuck times is reached, the situation of using the hard decoder for decoding and resulting in decoding failure can be avoided. Therefore, the playback efficiency of video playback is improved.

[0156] In some embodiments, refer to Figure 3I , Figure 3I is the ninth process schematic diagram of the video playback method provided by the embodiments of the present application. After obtaining the number of stuck times of the video decoder in the historical period in step 101, execute Figure 3I steps 601 to 603 below, which will be specifically described.

[0157] In step 601, obtain the stuck threshold of the previous period.

[0158] In some embodiments, the stuck threshold of the current period can be dynamically updated according to the stuck threshold of the previous period.

[0159] In step 602, when the video decoder in the previous period is a soft decoder, determine the target period.

[0160] Among them, the target period is the period closest to the current period among the multiple periods in the historical period when the video decoder is switched from a hard decoder to a soft decoder.

[0161] In some embodiments, the target period is used to indicate that the video decoder in all periods between the target period and the current period is a soft decoder.

[0162] In step 603, when the number of periods between the target period and the current period is greater than the number threshold, increase the stuck threshold of the previous period to obtain the stuck threshold of the current period.

[0163] In some embodiments, in order to improve the decoding quality of video playback, after continuously using the soft decoder for decoding multiple times, the stuck threshold can be increased so that the number of stuck times in the historical period is greater than the stuck threshold, and the video decoder is determined to be a hard decoder, improving the playback experience and decoding performance of video playback.

[0164] In some embodiments, the number threshold is used to represent the maximum number of consecutive periods in which the video decoder uses only the soft decoder for decoding.

[0165] For example, the value of the stuck threshold of the previous period is 3, the value of the number of periods between the target period and the current period is 5, the number threshold is 4, and adding 1 to the stuck threshold of the previous period gives the stuck threshold of the current period with a value of 4.

[0166] Through the embodiments of the present application, when there are consecutive video decoder cycles that only use the software decoder for decoding, the stuck threshold is dynamically adjusted to improve the decoding performance of video playback in continuous software decoding scenarios, taking into account both decoding performance and playback experience.

[0167] In some embodiments, after obtaining the number of stuck times of the video decoder in the historical cycle in step 101, a neural network model is called based on the number of stuck times in the previous cycle for prediction processing to obtain the stuck threshold for the current cycle.

[0168] Among them, the neural network model is trained based on the number of stuck times and the label of the stuck threshold in the previous sample cycle.

[0169] In some embodiments, based on the number of stuck times and the label of the stuck threshold in the previous sample cycle, a neural network model is called for training to obtain the predicted value of the stuck threshold. A loss function is constructed based on the predicted value of the stuck threshold and the label of the stuck threshold, and the parameters of the neural network model are modified through the backpropagation algorithm. When the loss function converges, the parameters of the neural network model are saved.

[0170] Through the embodiments of the present application, the neural network model is used to predict the stuck threshold for the current cycle by itself, improving the configuration efficiency of the stuck threshold, and thus further improving the playback efficiency of video playback.

[0171] In some embodiments, decoding failure includes at least one of the following: the initialization time of the video decoder in the current cycle is greater than the initialization time threshold, and the decoding time of the video decoder in the current cycle is greater than the decoding time threshold.

[0172] In some embodiments, the decoding process includes initializing the video decoder and decoding the video stream data based on the initialized video decoder to obtain the video data corresponding to the video stream data. Decoding failure is caused by any reason that causes the video stream data to be unable to obtain the video data during the decoding process. The embodiments of the present application do not limit the reasons for decoding failure.

[0173] In some embodiments, when the initialization time of the video decoder in the current cycle is greater than the initialization time threshold, the video decoder decoding fails.

[0174] For example, when the initialization time of the video decoder is 1 second and the time threshold is 0.8 seconds, since the initialization time is greater than the time threshold, the video decoder decoding fails.

[0175] In some embodiments, when the decoding time of the initialized video decoder in the current cycle is greater than the time threshold, the video decoder decoding fails.

[0176] Exemplarily, when the decoding time of the video decoder after initialization is 1 second and the time threshold is 0.8 seconds, since the decoding time is greater than the time threshold, the hard decoder decoding fails.

[0177] Next, an exemplary application of the video playback method provided in the embodiments of the present application in an actual application scenario will be described.

[0178] In video playback, the currently generally used decoding strategy is to give priority to hard decoding and use soft decoding as a fallback. However, MediaCodec is a decoder for the underlying hardware encapsulated by the system (i.e., the hard decoder). Due to device diversity, there are many compatibility issues, especially the problem of MediaCodec getting stuck, which can cause the entire decoding thread to get stuck and unable to decode and output the picture (i.e., video data), and there is no error message, so it cannot be perceived by the system. Even if the video playback is exited, the current decoding thread cannot exit normally, and it also affects the next MediaCodec initialization. Once a problem occurs, it will keep getting stuck, resulting in the user always experiencing a black screen during playback and affecting the user's viewing experience.

[0179] At the same time, because MediaCodec is bound to Surface (i.e., the rendering property), once MediaCodec gets stuck, Surface cannot be unbound from it, which will cause the picture still not to be output even after falling back to soft decoding. The general compatibility solution is to directly fall back to soft decoding playback the next time when it is recognized that the decoder gets stuck a certain number of times, but the current playback still shows a black screen, which still affects the user's playback experience. At the same time, if soft decoding is always used, the decoding performance may not be sufficient, resulting in a problem of unsmooth pictures.

[0180] Therefore, in the embodiments of the present application, based on the number of times the video decoder gets stuck within the historical period, the video decoder for the current period is dynamically configured. In this way, the efficiency of video playback is improved. Then, when decoding failure is recognized, the upper layer of the player (i.e., the playback control layer) is notified to update Surface, and then it is dynamically and imperceptibly downgraded to soft decoding playback, which can not only solve the black screen problem of the current playback and improve the user's playback experience, but also dynamically adjust the downgrading strategy, taking into account both decoding performance and playback experience.

[0181] Through the video playback method provided in the embodiments of the present application, obtain the number of times of switching from the hard decoder getting stuck to the soft decoder (i.e., the stuck threshold), obtain the number of times the mediacodec hard decoder gets stuck (i.e., the number of times the video decoder gets stuck), judge the size relationship between the number of times of getting stuck and the stuck threshold within the current life cycle (i.e., the current period) based on the number of times of getting stuck and the stuck threshold, determine the video decoder based on the size relationship, when the video decoder decoding fails, update the Surface (i.e., the rendering property) corresponding to the current video decoder, re-decode using the soft decoder, and render the decoded video data to the terminal.

[0182] Taking video playback as an example, refer to Figure 4 , Figure 4 which is the schematic diagram of video playback provided by the embodiments of the present application.

[0183] In Figure 4 , Figure 4 it contains multiple modules for video playback in the embodiments of the present application, such as background configuration module 101, player module 102, upper-layer player module 103, protocol parsing module 104, video decoding module 105, decoding freeze recognition module 106, decoding strategy module 107, image post-processing module 108, image rendering module 109, audio decoding module 110, sound post-processing module 111, and sound rendering module 112, etc.

[0184] The background configuration module 101 provides a decoding strategy for the server to dynamically configure that after the video decoder freezes N times, it can be downgraded to software decoding, and after continuously attempting software decoding multiple times, it can attempt hardware decoding again, which is convenient for dynamic configuration according to different model devices (i.e., application scenarios), so as to maximize the balance between decoding performance and playback experience. When the APP (i.e., application program) is initialized, the background configuration module 101 can be pulled, and then the decoding strategy provided by the background configuration module 101 is configured into the decoding strategy module 107 through the player module 102, so that the decoding strategy provided by the background configuration module 101 can be used in the decoding strategy module 107.

[0185] The player module 102 is responsible for creating a player, inputting the playback address of the video source (i.e., video) for playback, and implementing playback control (such as playback control methods such as play, pause, dragging the time point of the playback progress bar (seek), and obtaining the playback time point, etc.). Among them, the application system of the player can be one of multiple application systems such as the Android system and the IOS system. The player module 102 also includes an interface for callback messages of decoder freeze and an interface for dynamically updating the Surface, so as to facilitate dynamically notifying the upper-layer player to update the Surface in a timely manner when the decoding freeze is recognized, and setting the updated Surface into the video decoding module 105, which is convenient for using the new Surface when downgrading to software decoding, thus solving the problem of black screen during this playback.

[0186] The upper-layer player module 103, in addition to basic playback control methods, when receiving the callback message of decoder freeze, creates a new Surface and dynamically updates it to the player.

[0187] The protocol parsing module 104 parses the streaming media file (i.e., streaming media data) obtained from network transmission according to the corresponding streaming media protocol, and separates it into a video bitstream (i.e., video stream data) and an audio bitstream (i.e., audio stream data).

[0188] The video decoding module 105 creates a decoding thread and the corresponding hardware decoder or software decoder according to the encoding format and decoding strategy of the parsed video bitstream, and decodes the obtained video bitstream into the original image data (i.e., video data), and its form can be the original image data in the form of YUV, RGB, etc. In order to improve the video decoding performance, the general decoding strategy is to first judge whether hardware decoding is supported. As long as hardware decoding is supported, hardware decoding is preferentially used. If hardware decoding is not supported, software decoding is used as a fallback.

[0189] The decoding stuck recognition module 106 provides the timeout detection logic for the main methods of the decoder. For example, initialize the timeout detection. Before the decoder is initialized, create a timeout detection thread and set a certain initialization time-consuming threshold. When the initialization has not been completed after exceeding this threshold, it is considered that the initialization is stuck, and a callback message indicating that the decoder is stuck is notified to the upper-layer module 103 of the player, so that the upper-layer module 103 of the player can update the Surface in time after receiving this callback message and set the updated Surface to the video decoding module 105.

[0190] The decoding strategy module 107 provides two functions. One is to provide a decoding strategy to judge whether the number of times the hardware decoder is stuck within the historical life cycle exceeds the threshold dynamically configured in the background (i.e., the stuck threshold). If it does not exceed, hardware decoding is preferentially used. If hardware decoding fails or gets stuck, it falls back to software decoding. If it exceeds, software decoding is directly used, and then try hardware decoding again after the next life cycle or after trying software decoding N times according to the background configuration, where N is an integer greater than 1, maximizing the balance between decoding performance and playback experience.

[0191] The second is that after receiving the callback message indicating that the decoder is stuck in the decoding stuck recognition module 106, directly use a new thread to release the decoder and the decoding thread, and at the same time create a new decoding thread and a software decoder, and use the new Surface for rendering to solve the black screen problem of this playback.

[0192] The image post-processing module 108 generally implements a series of image algorithms according to needs, such as super-resolution reconstruction, image enhancement and other image algorithms to improve the visual subjective quality of the image.

[0193] The image rendering module 109 displays the original image data through a peripheral device (i.e., the terminal). In the Android system, MediaCodec and Surface are used together to render the original image data. The Surface is set to MediaCodec, and MediaCodec performs the rendering.

[0194] The audio decoding module 110 creates a decoding thread and the corresponding audio hard decoder or soft decoder according to the encoding format and decoding strategy of the parsed audio bitstream, and decodes the obtained audio bitstream into raw audio data, generally in the form of PCM data. The decoding strategy can be selected according to the actual situation and is not limited in the embodiments of the present application.

[0195] The audio post-processing module 111 generally implements audio algorithms according to needs, such as audio algorithms like pitch change, noise separation, virtual surround sound effects, etc., to improve the subjective auditory quality of the sound.

[0196] The sound rendering module 112 displays the raw audio data through a peripheral device, and generally renders it through AudioTrack on the Android system.

[0197] The following explains the process of the video playback method provided by the embodiments of the present application in combination with the above modules.

[0198] First, when the player starts playing, it first pulls the background configuration module 101 and sets the number of times of switching from hard decoder to soft decoder (i.e., the stuck threshold) of the background configuration module 101 to the video decoding module 105. According to the problems that occur in the online devices, background dynamic configuration can be performed to achieve an optimal experience.

[0199] Second, a player is created in the player module 102 and the playback address of the video source is passed in, and the streaming media file obtained from the playback address is parsed to obtain a video bitstream and an audio bitstream.

[0200] Third, a decoder is created and initialized in the video decoding module 105. When creating the decoder, the video decoding module 105 obtains a decoder list from the decoding strategy module 107, and then selects the first decoder for initialization. On the basis of the decoding strategy, a decision factor index for the stuck of the Mediacodec hard decoder is added to determine whether the number of times the Mediacodec hard decoder is stuck during the current life cycle exceeds the threshold configured in the background. If it exceeds, there is only software decoding in the decoder list obtained by the decoding strategy module, and no more attempts are made for hardware decoding.

[0201] Fourth, when the video decoding module 105 creates a decoder, it starts the decoding stuck recognition module 106 to detect whether the decoder initialization has timed out or the decoding time has timed out. If it times out (i.e., decoding fails), the number of times of decoder initialization timeout is recorded for convenient use in the decoding strategy module 107 next time. At the same time, a message to update the Surface is notified to the upper-layer player (i.e., the playback controller) in the player upper-layer module 103, and the decoding strategy module 107 is notified to release the decoding thread and switch the decoder.

[0202] Fifth, after receiving the updated Surface message, the player re-creates the Surface and sets it to the video decoding module 105.

[0203] Sixth, after receiving the callback message that the decoder is stuck, the decoding policy module 107 uses a new thread to release the decoding thread, and at the same time creates a new decoding thread and a software decoder, thus effectively solving the problem of black screen during playback.

[0204] Seventh, in the video decoding module 105, the video stream is decoded into raw image data (i.e., video data) based on the software decoder. In the image post-processing module 108, image processing is performed on the raw image data. In the image rendering module 109, the processed raw image data is displayed through a peripheral device.

[0205] Eighth, in the audio decoding module 110, the audio stream is decoded into raw sound data based on a hardware decoder or a software decoder. In the sound post-processing module 111, sound processing is performed on the raw sound data. In the sound rendering module 112, the processed raw sound data is displayed through a peripheral device.

[0206] In summary, the video playback method provided by the embodiments of the present application can effectively solve the problem of black screen during playback caused by the hardware decoder being stuck and improve the user's viewing experience through dynamic decoding stuck detection and dynamic Surface update; by dynamically configuring the downgrading policy, it solves the problem that the software decoding is always used after the hardware decoding is stuck, which may lead to the problem of unsmooth picture due to insufficient decoding performance. Through dynamic configuration according to different models, highly flexible policy iteration is realized, without relying on the APP version update, and the decoding performance and playback experience are maximally balanced.

[0207] Next, continue to describe the exemplary structure of the software module implementation of the video playback device 555 provided by the embodiments of the present application. In some embodiments, as Figure 2 shown, the software modules in the video playback device 555 stored in the memory 550 may include:

[0208] The data acquisition module 5551 is used to obtain the number of times the video decoder is stuck within a historical period.

[0209] The decoding policy module 5552 is used to determine the video decoder in the current period as a software decoder when the number of stuck times is greater than or equal to the stuck threshold of the current period, and determine the video decoder in the current period as a hardware decoder when the number of stuck times is less than the stuck threshold of the current period.

[0210] The video decoding module 5553 is used to decode the video stream data based on the video decoder in the current period.

[0211] The decoding freeze module 5554 is used to switch the video decoder from the hardware decoder to the software decoder within the current cycle when decoding fails, and re-decode the video stream data based on the software decoder to obtain the video data corresponding to the video stream data.

[0212] In some embodiments, the video decoding module 5553 is further used to construct a first thread corresponding to the video decoder; initialize the video decoder within the current cycle based on the first thread to obtain the initialized video decoder; and decode the video stream data based on the initialized video decoder.

[0213] In some embodiments, the decoding freeze module 5554 is further used to determine the playback state of the video decoder within the current cycle as the freeze state; when the playback state is the freeze state, increase the freeze count of the video decoder within the historical cycle of the next cycle.

[0214] In some embodiments, the decoding freeze module 5554 is further used to send a callback message to the playback control layer, where the callback message is used to prompt the playback control layer that the video decoder within the current cycle has decoding failure and the rendering attributes within the current cycle need to be updated.

[0215] In some embodiments, the player module 5555 is used to update the rendering attributes within the current cycle in response to receiving an update instruction for the rendering attributes within the current cycle sent by the playback control layer, where the updated rendering attributes are used to successfully render the video data to the terminal, and the terminal is used to play the video data.

[0216] In some embodiments, the video decoding module 5553 is further used to construct a second thread corresponding to the software decoder; initialize the software decoder based on the second thread to obtain the initialized software decoder; and decode the video stream data based on the initialized software decoder.

[0217] In some embodiments, the video decoding module 5553 is further used to construct a third thread; and release the hardware decoder within the current cycle and the first thread corresponding to the hardware decoder within the current cycle based on the third thread.

[0218] In some embodiments, the protocol parsing module 5556 is used to obtain the playback address of the video and obtain the streaming media data based on the playback address; and parse the streaming media data based on the streaming media protocol to obtain the video stream data.

[0219] In some embodiments, the video decoding module 5553 is further used to obtain the freeze threshold of the previous cycle; when the hardware decoders within the first N consecutive cycles of the current cycle all have decoding failures, reduce the freeze threshold of the previous cycle to obtain the freeze threshold of the current cycle, where N is a positive integer greater than 1.

[0220] In some embodiments, the video decoding module 5553 is further configured to obtain the stuck threshold of the previous cycle; when the video decoder in the previous cycle is a software decoder, determine the target cycle, where the target cycle is the cycle closest to the current cycle among multiple cycles in the historical cycles when the video decoder is switched from a hardware decoder to a software decoder; when the number of cycles between the target cycle and the current cycle is greater than the number threshold, increase the stuck threshold of the previous cycle to obtain the stuck threshold of the current cycle.

[0221] In some embodiments, the decoding policy module 5552 is further configured to obtain multiple hardware decoders from the set of video decoders, where the performances of the multiple hardware decoders are different; based on the performances of the multiple hardware decoders, divide the stuck threshold of the current cycle to obtain multiple stuck threshold intervals, where each stuck threshold interval corresponds to a hardware decoder one-to-one, and the stuck threshold corresponding to the stuck threshold interval is negatively correlated with the performance of the hardware decoder; when the number of stuck times is within the target stuck threshold interval, determine the video decoder in the current cycle as the hardware decoder corresponding to the target stuck threshold interval, where the target stuck threshold interval is one of the multiple stuck threshold intervals.

[0222] In some embodiments, the video decoding module 5553 is further configured to perform a prediction process by invoking a neural network model based on the number of stuck times in the previous cycle to obtain the stuck threshold of the current cycle, where the neural network model is trained based on the number of stuck times and the label of the stuck threshold in the previous sample cycle.

[0223] An embodiment of the present application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the video playing method described above in the embodiments of the present application.

[0224] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, where computer-executable instructions or a computer program are stored. When the computer-executable instructions or the computer program are executed by a processor, the processor will be caused to execute the video playing method provided in the embodiments of the present application. For example, Figures 3A to 3I as shown in the video playing method.

[0225] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0226] In some embodiments, the computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a stand-alone program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0227] As an example, the computer-executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code).

[0228] As an example, the computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected via a communication network.

[0229] In summary, through the embodiments of the present application, by means of the relationship between the number of freezes of the video decoder in the historical period and the current period freeze threshold, different video decoders are determined to avoid the video playback continuously entering the frozen state, taking into account both the decoding performance of the decoder and the playback experience. Then, based on the video decoder in the current period, the video stream data is decoded. When the decoding fails, the video decoder in the current period is switched from the hard decoder to the soft decoder, and the video stream data is re-decoded based on the soft decoder to obtain the video data corresponding to the video stream data. When the current period is in the frozen state, the video data re-decoded by the soft decoder can be rendered to the terminal to avoid the playback black screen problem caused by the hard decoder freeze, improving the user's playback experience.

[0230] The above is only the embodiments of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video playback method, characterized in that, The method includes: Obtaining the number of freezes of the video decoder within a historical period; When the number of freezes is greater than or equal to the freeze threshold of the current period, determining the video decoder within the current period as a software decoder, and when the number of freezes is less than the freeze threshold of the current period, determining the video decoder within the current period as a hardware decoder; Decoding video stream data based on the video decoder within the current period; When the decoding fails, switching the video decoder within the current period from the hardware decoder to the software decoder, and re-decoding the video stream data based on the software decoder to obtain video data corresponding to the video stream data.

2. The method according to claim 1, characterized in that, The decoding of the video stream data based on the video decoder within the current period includes: Constructing a first thread corresponding to the video decoder; Initializing the video decoder within the current period based on the first thread to obtain the initialized video decoder; Decoding the video stream data based on the initialized video decoder.

3. The method according to claim 1, characterized in that, Before switching the video decoder within the current period from the hardware decoder to the software decoder, the method further includes: Determining the playback state of the video decoder within the current period as a frozen state; When the playback state is the frozen state, increasing the number of freezes of the video decoder within the historical period of the next period.

4. The method according to claim 1, wherein Before switching the video decoder within the current period from the hardware decoder to the software decoder, the method further includes: Sending a callback message to the playback control layer, where the callback message is used to prompt the playback control layer that the video decoder within the current period has decoding failure and the rendering attributes within the current period need to be updated; In response to receiving an update instruction for the rendering attributes within the current period sent by the playback control layer, updating the rendering attributes within the current period, where the updated rendering attributes are used to successfully render the video data to the terminal, and the terminal is used to play the video data.

5. The method according to claim 1, wherein The re-decoding of the video stream data based on the software decoder includes: Constructing a second thread corresponding to the software decoder; Initializing the software decoder based on the second thread to obtain the initialized software decoder; Decoding the video stream data based on the initialized software decoder.

6. The method according to any one of claims 1 to 5, characterized in that, Before the re-decoding of the video stream data based on the software decoder, the method further includes: Constructing a third thread; Releasing the hardware decoder within the current period and the first thread corresponding to the hardware decoder within the current period based on the third thread.

7. The method according to claim 1, wherein Before obtaining the number of freezes of the video decoder within the historical period, the method further includes: Obtaining the playback address of the video and obtaining streaming media data based on the playback address; Parsing the streaming media data based on the streaming media protocol to obtain the video stream data.

8. The method according to claim 1, characterized in that, After obtaining the number of freezes of the video decoder within the historical period, the method further includes: Obtaining the freeze threshold of the previous period; When the hard decoders in the first N consecutive cycles of the current cycle all fail to decode, reduce the stuck threshold of the previous cycle to obtain the stuck threshold of the current cycle, where N is a positive integer greater than 1.

9. The method according to claim 1, wherein After obtaining the number of times the video decoder is stuck in the historical cycle, the method further includes: Obtain the stuck threshold of the previous cycle; When the video decoder in the previous cycle is a soft decoder, determine the target cycle, where the target cycle is the cycle closest to the current cycle among the multiple cycles in the historical cycle that switch the video decoder from the hard decoder to the soft decoder; When the number of cycles between the target cycle and the current cycle is greater than the number threshold, increase the stuck threshold of the previous cycle to obtain the stuck threshold of the current cycle.

10. The method according to claim 1, characterized in that, The determining the video decoder in the current cycle as a hard decoder includes: Obtain multiple hard decoders from the video decoder set, where the performances of the multiple hard decoders are different; Based on the performances of the multiple hard decoders, divide the stuck threshold of the current cycle to obtain multiple stuck threshold intervals, where each stuck threshold interval corresponds to a hard decoder one by one, and the stuck threshold corresponding to the stuck threshold interval is negatively correlated with the performance of the hard decoder; When the number of stuck times is within the target stuck threshold interval, determine the video decoder in the current cycle as the hard decoder corresponding to the target stuck threshold interval, where the target stuck threshold interval is one of the multiple stuck threshold intervals.

11. The method according to claim 1, wherein After obtaining the number of times the video decoder is stuck in the historical cycle, the method further includes: Call a neural network model for prediction processing based on the number of stuck times in the previous cycle to obtain the stuck threshold of the current cycle, where the neural network model is trained based on the number of stuck times and the label of the stuck threshold in the previous sample cycle.

12. The method according to claim 1, wherein The decoding failure includes at least one of the following: the initialization time of the video decoder in the current cycle is greater than the initialization time threshold, and the decoding time of the video decoder in the current cycle is greater than the decoding time threshold.

13. A video playback device, characterized in that, The device includes: A data acquisition module for obtaining the number of times the video decoder is stuck in the historical cycle; A decoding strategy module for determining the video decoder in the current cycle as a soft decoder when the number of stuck times is greater than or equal to the stuck threshold of the current cycle, and determining the video decoder in the current cycle as a hard decoder when the number of stuck times is less than the stuck threshold of the current cycle; A video decoding module for decoding the video stream data based on the video decoder in the current cycle; A decoding stuck recognition module for, when the decoding fails, switching the video decoder in the current cycle from the hard decoder to the soft decoder and re-decoding the video stream data based on the soft decoder to obtain the video data corresponding to the video stream data.

14. An electronic device, characterized in that, The electronic device includes: A memory for storing computer-executable instructions; A processor, when executing computer-executable instructions or a computer program stored in the memory, implements the video playback method according to any one of claims 1 to 12.

15. A computer program product, comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by the processor, the video playback method according to any one of claims 1 to 12 is implemented.

Images