Video playing processing method and device, electronic equipment and storage medium

Abstract

The application provides a video playing processing method and device, electronic equipment and storage medium; the method comprises: acquiring a video file; decoding the video file and playing the decoded video frame data in a third-party player interface; in response to a first switching instruction indicating that the third-party player is switched from the foreground to the background, hiding the third-party player interface and sending the video frame data to a native player in the terminal device, so that the native player continues the playing progress of the third-party player and continues to play the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the display screen of the terminal device. Through the application, the continuity of video playing can be ensured when the third-party player is switched from the foreground to the background, thereby realizing the seamless user experience of watching.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a video playback processing method, apparatus, electronic device and storage medium. Background Technology

[0002] For various mobile operating systems (including iOS and Android), due to operating system limitations, third-party media players will be suspended by the operating system and unable to continue playing videos after being moved to the background for a period of time. Additionally, if the third-party media player continues to render videos in the background after being moved there, crashes are likely to occur.

[0003] If you use the operating system's native player (such as AV-Player in iOS) to avoid the aforementioned issues with third-party players, the limitations of the native player's hardware decoding capabilities mean that it can only guarantee normal playback of videos on third-party players. However, if you switch to the native player (such as using AV-Player), playback will fail, resulting in a poor user experience.

[0004] In summary, there is currently no effective solution to ensure the continuity of video playback when a third-party player switches from the foreground to the background. Summary of the Invention

[0005] This application provides a video playback processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can ensure the continuity of video playback when a third-party player switches from the foreground to the background, thereby achieving a seamless viewing user experience.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a video playback processing method applied to a third-party player in a terminal device, including:

[0008] Get the video file;

[0009] The video file is decoded, and the decoded video frame data is played in a third-party player interface;

[0010] In response to a first switching command instructing the third-party player to switch from the foreground to the background, the third-party player interface is hidden, and the video frame data is sent to the native player in the terminal device, so that...

[0011] The native player continues the playback progress of the third-party player and continues to play the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the terminal device's display screen.

[0012] This application provides a video playback processing apparatus for use with a third-party player in a terminal device, including:

[0013] The acquisition module is used to acquire video files;

[0014] A decoding module is used to decode the video file;

[0015] The playback module is used to play decoded video frame data in a third-party player interface;

[0016] The hidden module is used to hide the interface of the third-party player in response to a first switching command that instructs the third-party player to switch from the foreground to the background.

[0017] The sending module is used to send the video frame data to the native player in the terminal device, so that the native player can continue the playback progress of the third-party player and continue playing the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the display screen of the terminal device.

[0018] This application provides an electronic device, including:

[0019] Memory, used to store executable instructions;

[0020] The processor, when executing executable instructions stored in the memory, implements the video playback processing method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video playback processing method provided in this application.

[0022] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the video playback processing method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] When a third-party player is switched from the foreground to the background, its decoding capabilities are reused to decode the video file, and the decoded video frame data is sent to the native player. In this way, the native player does not need to decode the video file; it can simply continue playing the video frame data sent by the third-party player within the native player interface. This avoids playback failures when switching from a third-party player to the native player, thus achieving seamless video playback when the third-party player switches from the foreground to the background, improving the user's viewing experience. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the video playback processing system 100 provided in this application embodiment;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application;

[0029] Figure 5 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application;

[0030] Figure 6 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram illustrating an application scenario of the video playback processing method provided in the embodiments of this application;

[0032] Figure 8 This is a schematic diagram of the overall architecture of the video playback processing method provided in the embodiments of this application;

[0033] Figure 9 This is a schematic diagram illustrating the principle of playing video frame data provided in the embodiments of this application;

[0034] Figure 10 This is a schematic diagram illustrating the principle of playing audio frame data provided in the embodiments of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0038] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0041] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0042] 2) Native media player, which is the media player pre-installed in the operating system of the terminal device, usually used as the default media player.

[0043] 3) Third-party media players: These are media players that users download and install from app stores. They are different from native media players and are used to improve the shortcomings of the operating system and realize personalized functions, providing users with a richer user experience.

[0044] 4) Picture-in-Picture (PiP): This is a video content presentation method that refers to the simultaneous playback of another video in a small area of ​​the screen while a full-screen video is playing.

[0045] 5) Software Development Kit (SDK): Also known as a software development toolkit, it is a collection of development tools used to create application software for specific software packages, software frameworks, hardware platforms, operating systems, etc. In a broader sense, a software development kit refers to a collection of related documents, examples, and tools that assist in the development of a particular type of software.

[0046] 6) Background Task: This refers to a process provided by the operating system that can run in the background. Even if the application has been suspended or is no longer running, the background task belonging to that application can still silently perform related operations. It needs to be used in conjunction with background mode; for example, for the iOS operating system, it needs to be configured in the Project Settings of the iOS development IDE (such as Xcode).

[0047] 7) Background mode: A process execution mode that does not occupy display or input devices and cannot interact with the user. In order for an application to continue running after being switched from the foreground to the background, it is necessary to set the conditions for the application to run in the background (such as adding the application to a whitelist that will not be suspended by the operating system). These conditions are called background mode.

[0048] 8) FFmpeg: An open-source computer program used to record, convert, and stream digital audio and video. Licensed under the LGPL or GPL, it provides a complete solution for recording, converting, and streaming audio and video. It includes the highly advanced audio and video codec library libavcodec. To ensure high portability and encoding / decoding quality, much of the code in libavcodec was developed from scratch.

[0049] 9) Open Graphics Library (OpenGL): It is a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics. This interface consists of nearly 350 different function calls, used to draw everything from simple graphics bits to complex three-dimensional scenes.

[0050] 10) Metal: is a low-level rendering application programming interface that provides the minimum level required by software, ensuring that the software can run on different graphics chips.

[0051] 11) Moving Picture Experts Group (MPEG): This is a video format. The MPEG file format is an international standard for motion picture compression algorithms. It uses lossy compression methods to reduce redundant information in moving pictures while ensuring a dynamic image refresh rate of 30 frames per second.

[0052] 12) Group of Picture (GOP): A group of consecutive frames consisting of one I-frame and several B / P frames. It is the basic unit accessed by the video image encoder and decoder. Its order will be repeated until the end of the video. Among them, the I-frame is the internally coded frame (also known as the key frame), the P-frame is the forward prediction frame, and the B-frame is the bidirectional interpolation frame.

[0053] 13) HTTP Live Streaming (HLS) is an adaptive bitrate streaming media transmission protocol. HLS describes a set of tools and programs for providing audio and video services over the Internet. A video file can be divided into multiple video slices (TSs). The delivery location and order of these slices are stored in a set of XML files called playlists, which end with the file extension m3u8. For example, assuming a video slice is 10 seconds long, a one-hour movie can be cut into 360 10-second video slices, and then a file called a playlist is created, which contains the video name, location, and slice playback sequence (along with metadata describing the codec, resolution, and bitrate, etc.). The process of creating these video slices is called video segmentation.

[0054] This application provides a video playback processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can ensure the continuity of video playback when a third-party player switches from the foreground to the background, thereby achieving a seamless viewing experience for the user. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as a terminal device, or it can be implemented collaboratively by a terminal device and a server.

[0055] The following description uses the video playback processing method provided in the embodiments of this application, which is implemented collaboratively by a terminal device and a server, as an example.

[0056] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video playback processing system 100 provided in this application embodiment. It aims to support an application that can maintain video playback continuity when a third-party player switches from the foreground to the background. Figure 1As shown, the video playback processing system 100 includes: a server 200, a network 300, and a terminal device 400. The terminal device 400 runs a third-party player 410 and a native player 420 that comes with the operating system. The network 300 can be a wide area network or a local area network, or a combination of both.

[0057] In some embodiments, taking a network playback scenario as an example, a third-party player 410 can receive video files sent by a server 200 via a network 300. For example, after receiving a download request from the third-party player 410, the server 200 can send a playlist (e.g., an m3u8 file, where m3u8 is a text file in UTF-8 encoding format) to the third-party player 410, so that the third-party player 410 can sequentially download multiple segments of the video file from the server 200 according to the playlist (TS, TS is a container format, full name MPEG-TS, the file is divided into three layers: TS layer, PES layer, and ES layer, where the ES layer is the audio and video data, and the PES layer adds time to the audio and video data). The TS layer adds necessary information for data stream identification and transmission to the PES layer (such as stamps and descriptive information for data frames). Each segment carries the information required for decoding and can be decoded independently (i.e., without depending on other segments). The third-party player 410 decodes multiple segments sequentially according to the receiving order and plays the decoded video frame data in the third-party player interface. Subsequently, in response to the first switching instruction instructing the third-party player to switch from the foreground to the background (e.g., receiving a user's click operation on the "Home" button used to return to the home screen), the third-party player 410 hides the third-party player interface (i.e., switches from the foreground to the background) and sends the video frame data to the native player 420 in the terminal device 400. After receiving the video frame data sent by the third-party player 410, the native player 420 continues the playback progress of the third-party player 410 and continues playing the video frame data in the native player interface in a picture-in-picture display mode. In this way, seamless switching of video frame data playback is achieved, improving the user's viewing experience.

[0058] In other embodiments, taking a local playback scenario as an example, the video playback processing method provided in this application embodiment can also be... Figure 1The terminal device 400 shown in the diagram executes independently. For example, a third-party player 410 responds to a playback request triggered by the user, reads a video file from the local storage location of the terminal device 400, then calls the built-in SDK to decode the video file and plays the decoded audio frame data in the third-party player interface. Subsequently, when the third-party player 410 receives the first switching instruction sent by the operating system of the terminal device 400, it hides the third-party player interface (i.e., switches from the foreground to the background) and sends video frame data to the native player 420 in the terminal device 400, so that the native player 420 continues the playback progress of the third-party player 410 (for example, assuming that when the third-party player 410 receives the first switching instruction sent by the operating system of the terminal device 400, the video frame data corresponding to the 70th second of the video file is being played in the third-party player interface, then the native player 420 continues to play the video frame data corresponding to the subsequent time period starting from the 70th second in the native player interface), and continues to play the video frame data in the native player interface in a picture-in-picture display mode. In this way, the continuity of video playback can be guaranteed when the third-party player switches from the foreground to the background, improving the user's viewing experience.

[0059] In other embodiments, the embodiments of this application can also be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0060] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0061] Example, Figure 1 The server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal device 400 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0062] The structure of the electronic device provided in the embodiments of this application will be further described below. Taking the electronic device as a terminal device as an example, see... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0063] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0065] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0066] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0067] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0068] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0069] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0070] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0071] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0072] In some embodiments, the video playback processing apparatus provided in this application can be implemented in software. Figure 2 A video playback processing device 555 stored in memory 550 is shown. This device can be software in the form of programs and plugins, including the following software modules: acquisition module 5551, decoding module 5552, playback module 5553, hiding module 5554, sending module 5555, creation module 5556, transmission module 5557, stop module 5558, and decapsulation module 5559. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions they implement. It should be noted that... Figure 2 For ease of explanation, all the above modules are shown at once, but this should not be construed as excluding the implementation of the video playback processing device 555, which may only include the acquisition module 5551, decoding module 5552, playback module 5553, hiding module 5554, and sending module 5555. The functions of each module will be described below.

[0073] The video playback processing method provided in this application will be specifically described below with reference to exemplary applications and implementations of the terminal devices provided in the embodiments of this application.

[0074] It should be noted that the terminal device provided in this application embodiment runs a third-party player and the native player that comes with the operating system. The operating system installed in the terminal device includes, but is not limited to, iOS, Android, Symbian, Blackberry, and Windows. The video playback processing method provided in this application embodiment will be specifically described from the perspective of the interaction between the third-party player and the native player running on the terminal device.

[0075] See Figure 3 , Figure 3 This is a flowchart illustrating the video playback processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0076] In step 101, the third-party player obtains the video file.

[0077] In some embodiments, taking a local playback scenario as an example, a third-party player responds to a video playback request by reading the video file to be played from the local storage location of the terminal device. For example, the third-party player interface displays the names of at least one video file to be played. When it receives a user's click operation on the name of video file A, the third-party player reads video file A from the local storage location of the terminal device. That is to say, the video file stored locally on the terminal device can be in a non-streaming media format, meaning that the video file needs to be completely downloaded before it can be decoded and played, such as Windows Media Video (WMV) files, MKV file format video files, etc.

[0078] In other embodiments, taking a network playback scenario as an example, a third-party player, in response to a user-triggered video playback operation (e.g., receiving a click from the user on the "Start Play" button displayed on the third-party player's interface), sends a video playback request to the server. The server, in response to the video playback request from the third-party player, sends a playlist of the requested video file to the third-party player. Subsequently, the third-party player sequentially downloads multiple video segments of the video file from the server according to the playlist. Each video segment carries the information required for decoding, thus allowing it to be decoded independently, i.e., without relying on other video segments. In other words, for video files in a network playback scenario, the video file's encapsulation format can be a streaming media format, meaning the video file does not need to be completely downloaded or undergo additional transcoding to be decoded and played, such as FLV (Flash Video) format video files.

[0079] In some embodiments, a third-party player can also obtain video files by: obtaining a multimedia file, wherein the multimedia file is obtained by encapsulating a video file, an audio file, and a subtitle file; and decapsulating the multimedia file to obtain a video file. For example, the server can pre-encapsulate an RMVB format video file, an MP3 format audio file, and an SRT format subtitle file into an MP4 or MKV format multimedia file. After obtaining the multimedia file, the third-party player can first decapsulate the multimedia file, for example, by calling a decapsulator to decompose the synthesized multimedia file into a video file, an audio file, and a subtitle file. The subtitles in the subtitle file are used to play synchronously with the video frame data in the third-party player interface. For example, assuming that the video frame data corresponding to the 90th second of the video file is currently playing in the third-party player interface, the subtitle corresponding to the 90th second of the subtitle file can be played synchronously.

[0080] It should be noted that when a third-party player switches from the foreground to the background, it can send a subtitle file to the native player so that the native player can continue playing the subtitles from the third-party player's playback progress within the native player's interface.

[0081] In step 102, the third-party player decodes the video file and plays the decoded video frame data in the third-party player interface.

[0082] In some embodiments, a software development kit (SDK) can be pre-built into the third-party player. This SDK integrates various types of decoders (e.g., FFMPEG decoder, Video ToolBox decoder, MediaCodec decoder, Dolby proprietary decoder, etc.). The third-party player can then decode the acquired video file by calling the built-in SDK to perform the following processes: determining the encoding format of the video file; and calling a decoder matching the encoding format to decode the video file. For example, assuming the third-party player's built-in SDK determines that the video file to be played is encoded using MPEG, it can call the FFMPEG decoder to decode the video file and obtain the corresponding video frame data. After obtaining the video frame data, the third-party player can render the video frame data decoded by the SDK using tools such as OpenGL or Metal, thereby displaying the video image rendered based on the video frame data in the third-party player's interface.

[0083] In other embodiments, while playing the decoded video frame data in the third-party player interface, the third-party player can also play the decoded audio frame data. This is done after the execution... Figure 3 After step 102 shown, you can also execute... Figure 4 The illustrated step 106 will combine Figure 4 The steps shown are explained.

[0084] In step 106, the third-party player decodes the audio file and plays audio frame data synchronized with the currently playing video frame data according to the currently playing video frame data in the third-party player interface.

[0085] In some embodiments, after decomposing a multimedia file to obtain an audio file, the third-party player can call its built-in SDK to decode the audio file. For example, the SDK first determines the encoding format of the decomposed audio file, and then calls a decoder that matches the encoding format to decode the audio file. For instance, assuming the audio file's encoding format is Dolby audio, the SDK can call a Dolby audio decoder to decode the audio file. After the SDK decodes the audio file to obtain audio frame data, the third-party player can play audio frame data that is synchronized with the currently playing video frame data in terms of playback time, based on the currently playing video frame data in the third-party player's interface. For instance, assuming the video frame data corresponding to the 70th second of the video file is currently playing in the third-party player's interface, the third-party player can synchronously play the audio frame data corresponding to the 70th second.

[0086] For example, a third-party player can play synchronized audio frame data in the following way: The decoded audio frame data is filled into at least one buffer (also called a buffer, which is part of the memory space; that is, a certain amount of storage space is reserved in the memory space to buffer input or output data; this reserved space is called a buffer) in the terminal device's memory space, and at least one buffer is stored in the audio queue; the audio output device (such as the terminal device's built-in speaker or external headphones, such as headphones with a wired or wireless connection to the terminal device) is invoked to read the audio frame data filled in at least one buffer from the audio queue, so that the audio output device outputs the corresponding audio signal. For example, taking the iOS operating system as an example, the AV-Player player can first read the audio frame data decoded by the SDK from the hard drive into the Audio Queue buffer (for example, assuming there are 3 buffers in the Audio Queue, namely buffer 1, buffer 2 and buffer 3, and each buffer can store 10 seconds of audio frame data, then the audio frame data from the 1st to the 10th second can be stored in buffer 1, the audio frame data from the 11th to the 20th second can be stored in buffer 2, and the audio frame data from the 21st to the 30th second can be stored in buffer 3), and then send the buffer to the audio queue; then the AV-Player player can use the Audio... The interface provided by the Queue tells the external speaker (i.e., audio output device), such as the built-in speaker of the terminal device, or headphones connected to the terminal device via USB or Bluetooth, that there is audio frame data in the buffer and it can be played (for example, the external speaker first reads the audio frame data stored in buffer 1 of the audio queue for playback, then reads the audio frame data stored in buffer 2 of the audio queue for playback, and so on). When the audio frame data in a buffer has finished playing, the Audio Queue tells the AV-Player that there is an empty buffer that can be used to fill audio frame data (for example, when the audio frame data from the 1st to the 10th second stored in buffer 1 has finished playing, the audio frame data from the 31st to the 40th second can continue to be filled into buffer 1), and repeat the above process until all audio frame data has finished playing.

[0087] See also Figure 3 In step 103, the third-party player responds to the first switching instruction that instructs the third-party player to switch from the foreground to the background by hiding the third-party player interface.

[0088] In some embodiments, the first switching instruction may be generated by the operating system of the terminal device and sent to the third-party player, and is generated when at least one of the following occurs: receiving a trigger operation instructing the third-party player to switch from the foreground to the background (e.g., receiving a user's click operation on the "Home" button for returning to the home screen, or receiving a user-triggered swipe operation to exit the third-party player); receiving a voice call request (e.g., when the third-party player is playing video frame data in full-screen mode in the third-party player interface, the operating system of the terminal device receives a voice call request, and the operating system can generate a first switching instruction instructing the third-party player to switch from the foreground to the background, and send the first switching instruction to the third-party player, so that the third-party player switches from the foreground to the background, thereby allowing the user to continue watching video frame data while processing the voice call request); receiving a video call request (e.g., when the third-party player is playing decoded video frames in full-screen mode in the third-party player interface). When the terminal device's operating system receives a video call request, it can generate a first switching instruction instructing the third-party player to switch from the foreground to the background. This instruction is then sent to the third-party player, allowing the user to continue watching video frame data while processing the video call request. If the currently playing video frame data includes recommendation information (such as advertisements), and the playback duration of this recommendation information exceeds a time threshold (e.g., when playing decoded video frame data in a third-party player interface), the terminal device's operating system can use a machine learning model to identify the video frame data. If the operating system detects that an advertisement is currently playing in the third-party player interface and the advertisement's playback duration exceeds a time threshold, it can generate and send a first switching instruction to the third-party player, reducing the interference of recommendation information and improving the user's viewing experience.

[0089] It should be noted that after receiving the first switching instruction sent by the terminal device's operating system, the third-party player can hide its interface (i.e., switch from the foreground to the background) and continue running in the background (for example, the third-party player can be added to the whitelist of players allowed to run in the background so that it will not be suspended by the operating system when switching from the foreground to the background, and can continue to run in the background). For example, it can continue to call the built-in SDK to decode video files or continue to play the decoded audio frame data.

[0090] In other embodiments, when a third-party player responds to a first switching instruction instructing the third-party player to switch from the foreground to the background, it may also perform the following processing: based on the progress of the video frame data that the native player continues to play in the native player interface (e.g., it can be represented by the timestamp of the currently playing video frame), it plays audio frame data that is synchronized with the video frame data played in the native player interface (in terms of playback time), thereby ensuring that the timestamp of the currently playing audio frame is synchronized with the timestamp of the video frame.

[0091] For example, when a third-party player receives the first switching instruction sent by the terminal device's operating system and switches from the foreground to the background, it can continue playing the audio frame data decoded by the SDK in the background mode. For instance, the third-party player can continue playing the audio frame data synchronized with the video frame data played in the native player interface (for example, assuming that the native player interface is currently playing the video frame data corresponding to the 90th second). That is, the third-party player continues to play the audio frame data corresponding to the 90th second in the background mode.

[0092] In step 104, the third-party player sends video frame data to the native player in the terminal device.

[0093] In some embodiments, the decoding of video files by the third-party player can be pre-decoding, wherein the range of pre-decoding can be: a predetermined duration (e.g., 10 seconds) after the current playback time point or a predetermined number of decoding units (i.e., the smallest decoding unit, such as a GOP in MPEG, a TS in HLS, etc.). Then, the third-party player can send video frame data to the native player in the terminal device in the following way: send the following data from the video frame data of the predetermined duration to the native player: video frame data whose playback time is after the first moment, wherein the first moment is the moment when the third-party player switches from the foreground to the background.

[0094] For example, taking an MPEG video file as an example, a third-party player can call its built-in SDK to pre-decode subsequent time segments. Taking MP4 as an example, the decoding unit is GOP (for example, after the SDK decodes the first GOP, it enters a waiting state. After the video frame data decoded for the first GOP has finished playing, it decodes the second GOP, and so on). Suppose that when the third-party player switches from the foreground to the background, the built-in SDK has already completed decoding for the 7th GOP (assuming that the video frame data decoded for the 7th GOP corresponds to the playback time segment from the 70th to the 90th second). At the same time, suppose that the video frame data corresponding to the 80th second of the video file is currently playing in the third-party player interface, then the third-party player can send the video frame data decoded for the 7th GOP but not yet played (i.e., the video frame data corresponding to the 80th to the 90th second) to the native player. At the same time, the third-party player can also send the video frame data decoded for subsequent GOPs to the native player. In this way, the amount of data sent can be reduced.

[0095] It should be noted that the aforementioned predetermined duration can be a fixed value (e.g., 10 seconds) or a dynamic value that is adaptively adjusted according to the hardware configuration of the terminal device. For example, when the hardware configuration of the terminal device is higher than the configuration threshold, it indicates that the terminal device has strong computing power, and the corresponding predetermined duration can be longer (e.g., 20 seconds); when the hardware configuration of the terminal device is lower than the configuration threshold, it indicates that the terminal device has weak computing power, and the corresponding predetermined duration can be shorter (e.g., 5 seconds). This application embodiment does not specifically limit this.

[0096] In other embodiments, the decoding of the video file by the third-party player can be pre-decoding, wherein the scope of pre-decoding can be the entire playback time period of the video file. Then, the third-party player can send video frame data to the native player in the terminal device in the following way: send video frame data corresponding to the subsequent time period to the native player, wherein the subsequent time period is the time period after the first moment in the playback timeline of the video file, and the first moment is the moment when the third-party player switches from the foreground to the background.

[0097] For example, for video files with a playback duration less than a duration threshold (e.g., 5 minutes) or a resolution lower than a resolution threshold, a third-party player can call its built-in SDK to decode the video file all at once, obtaining the video frame data corresponding to the entire playback time period (assuming the video file duration is 5 minutes, then the entire playback time period is from second 1 to second 300). Assuming that when the third-party player switches from the foreground to the background, it is playing the video frame data corresponding to second 60 of the video file in the third-party player interface, the third-party player can send the video frame data corresponding to the subsequent time period (i.e., second 60 to second 300) to the native player, so that the native player can continue playing the video frame data in the native player interface from second 60 (i.e., the native player can continue playing the video frame data corresponding to the subsequent time period from second 60 in the native player interface).

[0098] It's worth noting that when a third-party player sends video frame data decoded by the SDK to the native player, it can also store the video frame data in the memory corresponding to the instance running the third-party player. For example, it can store video frame data that is less than a duration threshold from the current playback time in memory and delete it after the storage duration expires. For instance, assuming the current playback time is the 90th second of the video file, the third-party player can store video frame data in memory while sending the corresponding subsequent time period video frame data to the native player. For example, it can store the video frame data corresponding to the 90th to 100th seconds decoded by the SDK. Then, when the video file plays to the 100th second in the native player, the video frame data corresponding to the 90th to 100th seconds stored in memory can be deleted. In this way, valuable memory space can be saved while ensuring normal playback of the video file.

[0099] In other embodiments, when a third-party player sends video frame data to the native player in a terminal device, it may also perform the following processing: send audio frame data decoded by the SDK to the native player so that the native player can continue playing the audio frame data, following the playback progress of the third-party player.

[0100] For example, a third-party player can send audio frame data to the native player in the following ways: Perform one of the following processes: Send the following data from the audio frame data of a predetermined duration to the native player: audio frame data whose playback time is after the first moment (corresponding to the case where the SDK pre-decodes the range of the predetermined duration after the current playback time, where the predetermined duration can be the playback time segment corresponding to the video frame data decoded by the SDK for 1 GOP in the video file, such as 10 seconds); Send the audio frame data from the complete playback timeline of the video file after the first moment to the native player (corresponding to the case where the SDK pre-decodes the range of the entire playback time segment of the audio file, such as for audio files whose playback duration is less than the duration threshold, such as audio files with a total duration of only 5 minutes).

[0101] It should be noted that the method for sending audio frame data is similar to the method for sending video frame data described above, and can be implemented by referring to the method for sending video frame data described above. The embodiments of this application will not be described again here.

[0102] In step 105, the native player continues the playback progress of the third-party player and continues to play video frame data in the native player interface.

[0103] Here, the size of the native player interface is smaller than the screen size of the terminal device. In other words, the native player continues the playback progress of the third-party player in a picture-in-picture display mode, and continues to play video frame data in the native player interface.

[0104] In some embodiments, after receiving video frame data sent by a third-party player, the native player can resume playback from the third-party player and continue playing the video frame data in the native player interface. For example, suppose that when the third-party player receives a first switching instruction sent by the terminal device's operating system and switches from the foreground to the background, the video frame data corresponding to the 70th second of the video file is being played in the third-party player interface. Then, the native player can resume playback from the third-party player and continue playing the video frame data for the subsequent time period starting from the 70th second of the video file in the native player interface. In this way, the continuity of video playback can be maintained when the third-party player switches from the foreground to the background, achieving a seamless viewing experience.

[0105] In other embodiments, when the native player continues the playback progress of the third-party player and continues to play video data frames in the native player interface, the third-party player can also perform the following processing: send video frame data whose playback time is before the first moment to the native player so that the native player responds to the instruction to start playback from the second moment. In the native player interface, based on the received video frame data whose playback time is before the first moment, playback starts from the video frame data corresponding to the second moment, wherein the second moment is earlier than the first moment.

[0106] For example, after a third-party player switches from the foreground to the background, it can send video frame data whose playback time is after the first moment to the native player, as well as video frame data whose playback time is before the first moment. In this way, when the native player receives an instruction to start playback from the second moment (i.e., the user wants to watch video frame data corresponding to an earlier moment in the native player), the native player can start playing from the video frame data corresponding to the second moment (e.g., the playback time is the 90th second of the video file) based on the received video frame data whose playback time is before the first moment (e.g., the playback time is the video frame data corresponding to the playback time from the 1st second to the 90th second) in the native player interface.

[0107] In other embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application, as shown below. Figure 5 As shown, the native player executes... Figure 3 Before step 105 shown, the following steps can also be performed: Figure 5 Steps 107 and 108 shown will be combined Figure 5 The steps shown are explained.

[0108] In step 107, the native player creates a picture-in-picture controller object.

[0109] In step 108, the native player passes a reference to the native player interface to the picture-in-picture controller object, so that the native player interface is bound to the picture-in-picture controller object.

[0110] Here, the picture-in-picture controller object is used to control the native player interface to use a display mode smaller than the screen size.

[0111] For example, taking the native player as AVPlayer as an example, AVPlayer is a video playback player based on the iOS operating system. AVPlayer cannot display the video content on its own; it needs to add graphics to the layer that needs to be displayed using the AVPlayer Layer (i.e., the native player interface) to display the video content. Therefore, before AVPlayer continues to play video frame data in the AVPlayer Layer, it can also perform the following processing: create a new AVPictureInPictureController object and pass a reference to the AVPlayer Layer used to display the video content to the AVPictureInPictureController object. That is, a strong reference to the AVPictureInPictureController object is required to achieve the picture-in-picture function. In other words, after binding the AVPlayer Layer to the AVPictureInPictureController object, the AVPlayer Layer can be managed through the AVPictureInPictureController object. For example, it can control the AVPlayer Layer to use a display mode smaller than the screen size of the terminal device (i.e., control the AVPlayer Layer to be displayed in picture-in-picture mode).

[0112] In other embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the video playback processing method provided in an embodiment of this application, as shown below. Figure 6 As shown, after execution Figure 3 After step 105 shown, you can also execute... Figure 6 Steps 109 and 110 shown will be combined Figure 6 The steps shown are explained.

[0113] In step 109, the third-party player responds to a second switching instruction that instructs it to switch from the background back to the foreground and displays the third-party player interface.

[0114] In some embodiments, the second switching instruction may be generated by the operating system of the terminal device and sent to the third-party player, and is generated when at least one of the following occurs: receiving a trigger operation instructing the third-party player to switch back to the background (e.g., receiving a user's click operation on the icon of the third-party player displayed on the desktop); the voice call ends (e.g., when the operating system of the terminal device detects that the voice call has ended, it can generate a second switching instruction instructing the third-party player to switch back to the foreground and send the second switching instruction to the third-party player to make the third-party player switch back to the foreground); the video call ends (e.g., when the operating system of the terminal device detects that the video call has ended, it can generate a second switching instruction instructing the third-party player to switch back to the foreground and send the second switching instruction to the third-party player to make the third-party player switch back to the foreground); the currently playing video frame data does not include recommendation information (e.g., when the operating system of the terminal device detects that the advertisement has finished playing, it can generate a second switching instruction instructing the third-party player to switch back to the foreground and send the second switching instruction to the third-party player to make the third-party player switch back to the foreground).

[0115] In other embodiments, after receiving a second switching instruction from the operating system of the terminal device, the third-party player can display the third-party player interface (i.e., switch from the background to the foreground) and send a hiding notification to the native player so that the native player hides the native player interface. The methods for hiding the native player interface include: closing the native player (i.e., closing the instance running the native player and reclaiming the memory corresponding to the instance); or keeping the instance running the native player running and only closing the native player interface.

[0116] For example, continuing from the previous text, when the native player interface is hidden (e.g., the instance running the native player is closed and the memory corresponding to the instance is reclaimed, in which case the possibility of switching from the third-party player to the native player later is relatively small), the third-party player can also perform the following processing: stop sending video frame data to the native player.

[0117] For example, continuing from the previous point, when the native player interface is hidden (e.g., keeping the native player instance running but only closing the native player interface; in this case, the likelihood of switching back from a third-party player to the native player later is high, and the native player will automatically delete older video frame data, such as video frames older than a predetermined duration from the current playback time), if the transmission of video frame data to the native player has not yet been completed, the third-party player can still perform the following processing: continue sending video frame data whose playback time is after the third moment to the native player, where the third moment is the moment the third-party player switches back to the foreground from the background. For example, suppose that when the third-party player switches back to the foreground, the native player interface is playing the 100th second of the video file. Then, by keeping the native player instance running but only closing the native player interface, the third-party player can continue sending the video frame data after the 100th second, obtained from SDK decoding, to the native player. In this way, when switching back from the third-party player to the native player later, the native player can directly play the video based on the received video frame data in the native player interface, further improving the switching speed.

[0118] In step 110, the third-party player continues the playback progress of the native player and continues to play video frame data in the third-party player interface.

[0119] In some embodiments, when a third-party player receives a second switching instruction from the operating system of the terminal device and switches back to the foreground from the background, the third-party player interface can be redisplayed, and the playback progress of the native player can be resumed. Video frame data can continue to be played in the third-party player interface. For example, when the third-party player switches back to the foreground, the video frame data corresponding to the 120th second of the video file is being played in the native player interface. Then, the third-party player can continue to play the video frame data corresponding to the subsequent time period starting from the 120th second in the third-party player interface.

[0120] It should be noted that, regarding the situation where the third-party player continues to play the audio frame data decoded by the SDK through the native player when the third-party player switches from the foreground to the background, the third-party player can stop sending the decoded audio frame data to the native player when the third-party player switches back from the background to the foreground. Of course, considering that the player may switch again in the future, the third-party player can also continue to send the decoded audio frame data to the native player. This application embodiment does not specifically limit this.

[0121] The video playback processing method provided in this application embodiment continues to reuse the decoding capabilities of the third-party player to decode the video file when the third-party player is switched from the foreground to the background, and sends the decoded video frame data to the native player. In this way, the native player does not need to decode the video file, but only needs to continue playing the video frame data sent by the third-party player in the native player interface, avoiding the situation where playback fails when switching from the third-party player to the native player. This achieves the function of seamless video playback when the third-party player switches from the foreground to the background, improving the user's viewing experience.

[0122] The following uses the iOS operating system as an example to illustrate an exemplary application of the embodiments of this application in a real-world application scenario.

[0123] With the development of internet technology, video playback applications (corresponding to the aforementioned third-party players) have become widely used in people's daily lives and have gradually become an indispensable part of their lives. These video playback applications all use either the operating system's built-in player (hereinafter referred to as the system player) or implement their own "picture-in-picture" functionality to achieve video playback. When a user exits the background, they invariably pause video playback by setting the player's state machine (e.g., switching from playing to pause). However, users desire a seamless video viewing experience. The solution provided by relevant technologies relies on the picture-in-picture capability offered by the iOS operating system. However, this solution has a drawback: the hardware decoding capabilities of the iOS operating system are limited. This can easily lead to videos playing normally within the application but failing when switched to the system player, resulting in a very poor user experience.

[0124] In view of this, this application provides a video playback processing method. By embedding a video playback component (i.e., SDK) in a video playback application, seamless switching of video playback based on the iOS operating system can be achieved. The solution provided by this application can achieve seamless video playback when switching from the foreground to the background without affecting the user experience and effect, and greatly improves the user experience when using video playback applications.

[0125] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating an application scenario of the video playback processing method provided in the embodiments of this application, such as... Figure 7As shown, a third-party player's playback interface 701 is displayed on the screen 700 of a terminal device (such as a mobile phone). When the third-party player is switched from the foreground to the background, the playback interface 701 of the third-party player is hidden on the screen 700, and the AV-Player player (i.e., the system player) built into the iOS operating system is enabled. The video frame data continues to be output in the playback interface 702 of the AV-Player player in picture-in-picture display mode. In this way, seamless video playback can be achieved when switching from the foreground to the background.

[0126] The video playback processing method provided in the embodiments of this application will be described in detail below.

[0127] The technical solution provided in this application is based on the playback framework of the iOS operating system and relies on the picture-in-picture capability of the system player. That is, when a third-party player switches from the foreground to the background, it can enable the system player to continue playing the video.

[0128] The following will combine Figure 8 The specific implementation process of the video playback processing method provided in the embodiments of this application will be described. For example... Figure 8As shown, the playback components (SDK) built into the third-party player are used first. These components include the Demuxer API, Decoder API, and Renderer API. The Demuxer API includes FFMPEG Demuxer and a self-developed Demuxer. For example, the Demuxer thread can call the Demuxer API to depackage multimedia files into video and audio files. The Decoder API includes FFMPEG software decoding, MediaCodec, VideoToolBox, AudioToolBox, and Dolby audio proprietary decoder. For example, the Decoder thread can call the Decoder API to decode the depackaged video and audio files to obtain video and audio frame data. The Renderer API includes OpenGL, Metal, Audio Track, OpenSL, and AudioQueue. For example, the Renderer thread can call the Renderer API to decode the depackaged video and audio files to obtain video and audio frame data. The API (using OpenGL or Metal to display video frame data) decodes the video file, outputting video and audio frame data. Then, it processes the video and audio frame data obtained from the SDK decoding. The video frame data processing is as follows: it is provided to the iOS operating system's built-in AV-Player for output → bound to the picture-in-picture controller (AVPictureInPictureController) → the picture-in-picture display mode is enabled to continue outputting video frame data. The audio frame data processing uses a low-overhead method based on Audio Queue, suitable for recording and playing music on iOS and Mac OS X. The general process is: first, the audio frame data is read into the AudioQueue buffer, and the buffer is sent to the audio queue. Then, the external speaker is informed that there is audio frame data in the buffer and it can be played. Finally, after the external speaker finishes playing, the buffer is cleared, and the above steps are repeated until all audio frame data has been played.

[0129] In other words, Figure 8 The third-party player's built-in SDK on the left is responsible for decoding video and audio files and outputting the decoded video and audio frame data. For the video frame data, when the third-party player switches from the foreground to the background, it can enable the AV-Player built into the iOS operating system and continue playing the video frame data decoded by the SDK through the AV-Player. For the audio frame data, it can continue to call the Audio Queue for playback output.

[0130] The following will combine Figure 9 The video portion will be explained in detail.

[0131] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of playing video frame data provided in the embodiments of this application, such as... Figure 9 As shown, after decapsulating the multimedia file to obtain the video file, the video file can be decoded to obtain video frame data. This video frame data can then be displayed using OpenGL or Metal. Furthermore, when a third-party player switches from the foreground to the background, the built-in AVPlayer player of the iOS operating system can be used to continue displaying the video frame data, achieving seamless playback switching. This ensures that the video frame data played by AVPlayer is still obtained by reusing the SDK's own decoding capabilities, which is far superior to relying on the hardware decoding capabilities of the iOS operating system.

[0132] The following will continue to combine Figure 10 The audio portion will be explained in detail.

[0133] For example, see Figure 10 , Figure 10 This is a schematic diagram illustrating the principle of playing audio frame data provided in the embodiments of this application, such as... Figure 10As shown, for the audio part, the audio playback function can be implemented based on the Audio Queue framework (Audio Queue is an audio processing framework encapsulated in the iOS operating system, which can be used to play or record audio and supports platform-level audio format encoding and decoding. Audio Queue is a software object used to play or record audio in the iOS or Mac OS X operating system. It is an AudioQueueRef data type defined in AudioQueue.h, which includes multiple Buffers, each of which is a buffer of some audio frame data, a Buffer Queue, which is an ordered queue composed of multiple Buffers, and a Callback used to notify the hard disk that the audio frame data in the Buffer has finished playing and that new audio frame data can be filled into the Buffer). The main steps include: 1. From the hard disk (Hard 1. Read audio frame data from the hard drive (e.g., disk); 2. Store the audio frame data into multiple buffers (e.g., three buffers, buffer 1, buffer 2, and buffer 3); 3. Send the three buffers filled with audio frame data into the audio queue; 4. Notify the external speaker to play the audio frame data in the audio queue; 5. After playback, clean up related resources. The operation flow for the buffer is as follows: First, call the corresponding method (e.g., Callback) to read the audio frame data from the hard drive into the buffer of the Audio Queue, and send the buffer filled with audio frame data into the audio queue; then, through the interface provided by the Audio Queue, notify the external speaker (e.g., the microphone of the terminal device) that the buffer has stored audio frame data and can be played; when the audio frame data in one buffer has finished playing, the Audio Queue can use Callback to notify that there is currently an empty buffer (e.g., Figure 10 The buffer 1 shown can be used to fill new audio frame data. Repeat the above steps until all audio frame data has been played.

[0134] The video playback processing method provided in this application embodiment can enable a third-party player to seamlessly play videos when switching from the foreground to the background. It not only effectively utilizes the hardware decoding capabilities improved by the operating system itself, but also utilizes the software decoding and custom decoding capabilities provided by the SDK built into the third-party player, so that the normal output of audio and video is not affected when switching between the foreground and the background, thereby effectively improving the user's viewing experience.

[0135] The following continues to describe the exemplary structure of the video playback processing device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the video playback processing device 555 in the memory 550 may include: an acquisition module 5551, a decoding module 5552, a playback module 5553, a hiding module 5554, and a sending module 5555.

[0136] The acquisition module 5551 is used to acquire video files; the decoding module 5552 is used to decode video files; the playback module 5553 is used to play the decoded video frame data in the third-party player interface; the hiding module 5554 is used to hide the third-party player interface in response to a first switching command instructing the third-party player to switch from the foreground to the background; and the sending module 5555 is used to send video frame data to the native player in the terminal device so that the native player can continue the playback progress of the third-party player and continue playing video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the terminal device's display screen.

[0137] In some embodiments, the third-party player has a built-in software development kit (SDK) that integrates various types of decoders; the decoding module 5552 is also used to call the SSD to perform the following processes: determine the encoding format of the video file; and call a decoder that matches the encoding format to decode the video file.

[0138] In some embodiments, the sending module 5555 is further configured to send the following data from video frame data of a predetermined duration to the native player: video frame data whose playback time is after a first moment, wherein the first moment is the moment when the third-party player switches from the foreground to the background.

[0139] In some embodiments, the sending module 5555 is further configured to send video frame data corresponding to a subsequent time period to the native player, wherein the subsequent time period is the time period after the first moment in the playback timeline of the video file, and the first moment is the moment when the third-party player switches from the foreground to the background.

[0140] In some embodiments, when the native player continues the playback progress of the third-party player and continues to play video frame data in the native player interface, the sending module 5555 is further configured to send video frame data whose playback time is before the first moment to the native player, so that the native player responds to the instruction to start playback from the second moment. In the native player interface, based on the received video frame data whose playback time is before the first moment, playback starts from the video frame data corresponding to the second moment, wherein the second moment is earlier than the first moment.

[0141] In some embodiments, the video playback processing apparatus 555 further includes a creation module 5556 and a transmission module 5557, wherein the creation module 5556 is used to create a picture-in-picture controller object before continuing to play video frame data in the native player interface; the transmission module 5557 is used to pass a reference to the native player interface to the picture-in-picture controller object so that the native player interface is bound to the picture-in-picture controller object, wherein the picture-in-picture controller object is used to control the native player interface to adopt a display mode smaller than the screen size.

[0142] In some embodiments, the first switching instruction is generated by the operating system of the terminal device and sent to the third-party player, and is generated when at least one of the following occurs: a trigger operation instructing the third-party player to switch from the foreground to the background is received; a voice call request is received; a video call request is received; or the currently playing video frame data includes recommendation information.

[0143] In some embodiments, the hiding module 5554 is further configured to hide the native player interface in response to a second switching instruction instructing the third-party player to switch back to the foreground from the background; the playback module 5553 is further configured to display the third-party player interface and continue the playback progress of the native player, continuing to play video frame data in the third-party player interface.

[0144] In some embodiments, the video playback processing apparatus 555 further includes a stop module 5558, which is used to stop sending video frame data to the native player when the native player interface is hidden.

[0145] In some embodiments, the sending module 5555 is further configured to, when the native player interface is hidden, if the video frame data has not yet been transmitted to the native player, continue to send video frame data whose playback time is after the third moment, wherein the third moment is the moment when the third player switches back to the foreground from the background.

[0146] In some embodiments, the acquisition module 5551 is further configured to acquire a multimedia file, which is obtained by encapsulating a video file, an audio file, and a subtitle file; the video playback processing device 555 further includes a decapsulation module 5559, configured to decapsulate the multimedia file to obtain a video file; the decapsulation module 5559 is further configured to decapsulate the multimedia file to obtain an audio file and a subtitle file, wherein the subtitles in the subtitle file are used to be played synchronously with the video frame data in a third-party player interface.

[0147] In some embodiments, the decoding module 5552 is further configured to decode the audio file; the playback module 5553 is further configured to play audio frame data synchronized with the currently playing video frame data according to the currently playing video frame data in the third-party player interface.

[0148] In some embodiments, the playback module 5553 is further configured to fill audio frame data into at least one buffer and store at least one buffer into an audio queue; and call an audio output device to read the audio frame data filled in at least one buffer from the audio queue so that the audio output device outputs the corresponding audio signal.

[0149] In some embodiments, the playback module 5553 is further configured to, in response to a first switching instruction instructing a third-party player to switch from the foreground to the background, play audio frame data synchronized with the video frame data played in the native player interface, based on the progress of the video frame data that the native player continues to play in the native player interface.

[0150] In some embodiments, the sending module 5555 is further configured to send audio frame data to the native player when sending video frame data to the native player in the terminal device, so that the native player can continue playing the audio frame data following the playback progress of the third-party player.

[0151] In some embodiments, the sending module 5555 is further configured to perform one of the following processes: sending the following data from audio frame data of a predetermined duration to the native player: audio frame data whose playback time is after a first moment; sending audio frame data from the complete playback timeline of the video file after the first moment to the native player, wherein the first moment is the moment when the third-party player switches from the foreground to the background.

[0152] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore, it will not be repeated. For any technical details not covered in the video playback processing apparatus provided in this application embodiment, please refer to... Figures 3 to 6 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0153] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the video playback processing method described in this application.

[0154] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the video playback processing method provided in this application. For example, ... Figures 3 to 6 The video playback processing method shown in any of the attached figures.

[0155] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0156] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0157] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0158] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video playback processing method, characterized in that, The method, which is applied to a third-party media player in a terminal device, includes: Get the video file; The video file is decoded, and the decoded video frame data is played in a third-party player interface; In response to a first switching command instructing the third-party player to switch from the foreground to the background, the third-party player interface is hidden, and the video frame data is sent to the native player in the terminal device, so that... The native player continues the playback progress of the third-party player and continues to play the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the terminal device's display screen.

2. The method according to claim 1, characterized in that, The third-party player has a built-in software development kit, which integrates various types of decoders; Decoding the video file includes: The software development kit is invoked to perform the following processing: Determine the encoding format of the video file; The video file is decoded by calling the decoder that matches the encoding format.

3. The method according to claim 1, characterized in that, Sending the video frame data to the native player in the terminal device includes: Send the following data from video frame data of a predetermined duration to the native player: video frame data whose playback time is after a first moment, wherein the first moment is the moment when the third-party player switches from the foreground to the background.

4. The method according to claim 1, characterized in that, Sending the video frame data to the native player in the terminal device includes: The video frame data corresponding to the subsequent time period is sent to the native player, wherein the subsequent time period is the time period in the playback timeline of the video file that is after the first moment, and the first moment is the moment when the third-party player switches from the foreground to the background.

5. The method according to claim 3 or 4, characterized in that, When the native player continues playback of the video frame data in the native player interface, following the playback progress of the third-party player, the method further includes: Send video frame data whose playback time is before the first moment to the native player, so that In response to the instruction to start playback from the second moment, the native player, in the native player interface, starts playback from the video frame data corresponding to the second moment based on the received video frame data whose playback time is earlier than the first moment, wherein the second moment is earlier than the first moment.

6. The method according to claim 1, characterized in that, Before continuing to play the video frame data in the native player interface, the method further includes: Create a picture-in-picture controller object; The native player interface is passed a reference to the picture-in-picture controller object to bind the native player interface to the picture-in-picture controller object, wherein the picture-in-picture controller object is used to control the native player interface to adopt a display mode smaller than the screen size.

7. The method according to claim 1, characterized in that, The first switching instruction is generated by the operating system of the terminal device and sent to the third-party player, and is generated when at least one of the following conditions is met: Received a trigger operation instructing the third-party player to switch from the foreground to the background; A voice call request has been received. A video call request has been received. The currently playing video frame data includes recommendation information.

8. The method according to claim 1, characterized in that, The method further includes: In response to a second switching command instructing the third-party player to switch from the background back to the foreground, the native player interface is hidden, and the third-party player interface is displayed. Continuing the playback progress of the native player, the video frame data is played again in the third-party player interface.

9. The method according to claim 8, characterized in that, When the native player interface is hidden, the method further includes: Stop sending the video frame data to the native player.

10. The method according to claim 8, characterized in that, When the native player interface is hidden, the method further includes: In response to the fact that the video frame data has not yet been fully transmitted to the native player, the system continues to send video frame data with a playback time after the third moment to the native player, wherein the third moment is the moment when the third-party player switches back to the foreground from the background.

11. The method according to claim 1, characterized in that, The acquisition of the video file includes: Obtain a multimedia file, wherein the multimedia file is obtained by encapsulating the video file, audio file, and subtitle file; The multimedia file is decapsulated to obtain the video file; The method further includes: The multimedia file is decapsulated to obtain the audio file and the subtitle file, wherein the subtitles in the subtitle file are used to play synchronously with the video frame data in the third-party player interface.

12. The method according to claim 11, characterized in that, When decoding the video file and playing the decoded video frame data in a third-party player interface, the method further includes: The audio file is decoded, and audio frame data synchronized with the currently playing video frame data is played according to the currently playing video frame data in the third-party player interface.

13. The method according to claim 12, characterized in that, The audio frame data that is synchronized with the currently playing video frame data includes: The audio frame data is filled into at least one buffer, and the at least one buffer is stored in an audio queue; The audio output device is invoked to read the audio frame data filled in the at least one buffer from the audio queue, so that the audio output device outputs the corresponding audio signal.

14. The method according to claim 12, characterized in that, In response to a first switching instruction instructing the third-party player to switch from the foreground to the background, the method further includes: Based on the progress of the video frame data that the native player continues to play in the native player interface, the audio frame data that is synchronized with the video frame data played in the native player interface is played.

15. The method according to claim 12, characterized in that, When sending the video frame data to the native player in the terminal device, the method further includes: The audio frame data is sent to the native player so that the native player continues to play the audio frame data according to the playback progress of the third-party player.

16. The method according to claim 15, characterized in that, Sending the audio frame data to the native player includes: Perform one of the following processes: Send the following data from the audio frame data of a predetermined duration to the native player: audio frame data whose playback time is after the first moment; Send the audio frame data of the complete playback timeline of the video file after the first moment to the native player, wherein the first moment is the moment when the third-party player switches from the foreground to the background.

17. A video playback processing device, characterized in that, A third-party media player used in terminal devices, the device comprising: The acquisition module is used to acquire video files; A decoding module is used to decode the video file; The playback module is used to play decoded video frame data in a third-party player interface; The hidden module is used to hide the interface of the third-party player in response to a first switching command that instructs the third-party player to switch from the foreground to the background. The sending module is used to send the video frame data to the native player in the terminal device, so that the native player can continue the playback progress of the third-party player and continue playing the video frame data in the native player interface, wherein the size of the native player interface is smaller than the size of the display screen of the terminal device.

18. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video playback processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the video playback processing method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the video playback processing method according to any one of claims 1 to 16 is implemented.

Images