Video playing method and related equipment

Abstract

The invention provides a video playing method and related equipment, and relates to the technical field of terminal equipment. The video playing method comprises the steps of displaying a first interface of a video application in response to a first operation; in response to a second operation on the first interface, starting to play a first picture of the first video at the first moment, and between the first moment and the second moment, image data of the first video is not processed by the first processor; processing the image data of the first video by using the first processor in response to a preset trigger event at a second moment; and after the second moment, a second picture of the first video is played based on the image data of the first video processed by the first processor, and the power consumption of the first processor between the first moment and the second moment is smaller than that of the first processor after the second moment. By using the method, the power consumption of playing the video by the terminal equipment can be reduced.

Description

[0001] This application is a divisional of a Chinese patent application with the title "Video Playback Method and Related Devices", application number "202410749805.X", and filing date "2024-06-11". Technical Field

[0002] Embodiments of this application relate to the field of terminal devices, and in particular, to a video playback method and related devices. Background Art

[0003] With the development of terminal technology, the functions of terminal devices have become increasingly rich. In particular, the video playback function has gradually become a commonly used function of terminal devices. Terminal devices can play videos through various installed applications.

[0004] To improve the smoothness and image quality of video playback, video playback applications generally use multi-buffer caching technology to cache image frames and call the graphics processing unit (GPU) to enhance the cached image frames. However, during video playback, using multiple buffers to cache image frames and using the GPU to enhance image frames will both cause an increase in the power consumption of the terminal device and reduce the battery life of the terminal device. Summary of the Invention

[0005] In view of the above, it is necessary to provide a video playback method and related devices that can solve the problem of high power consumption of terminal devices during video playback.

[0006] In a first aspect, this application provides a video playback method applied to a terminal device, including: in response to a first operation, starting a video application; in response to a second operation of playing a first video on the video application, obtaining a video stream of the first video; processing the video stream using a first processing mode, where the first processing mode includes decoding the video stream and storing the decoded video image frames in a display buffer for display; in response to a preset trigger event, processing the video stream using a second processing mode, where the second processing mode includes decoding the video stream and storing the decoded video image frames in a cache buffer, and the video image frames stored in the cache buffer can be copied to the display buffer for display.

[0007] With the above technical solution, when playing a video in a video application, the first processing mode is used by default to decode and play the video stream. Since the buffer in the cache buffer is not used to cache video image frames and the GPU is not used to process video image frames, the video playback power consumption of the terminal device can be reduced. When a preset trigger event is detected, for example, a video image frame playback lag event or an event that the video application receives a preset operation, the processing mode of the video stream is then switched from the first processing mode to the second processing mode to ensure the smoothness of video playback and enable the preset operation to be correctly executed. For example, the preset operation may include an operation that triggers the call of the graphics processing unit (GPU) to process video image frames.

[0008] In a possible implementation, the video playback method further includes: obtaining the media information of the video stream of the first video, where the media information includes the resolution of the first video; when the resolution of the first video is not the preset resolution, using the first processing mode to process the video stream; when the resolution of the first video is the preset resolution, using the second processing mode to process the video stream.

[0009] With the above technical solution, the preset resolution can be set according to actual needs. For example, the preset resolution is an ultra-high definition resolution, such as a 4K resolution. For the video playback of an ultra-high definition resolution, it may be necessary to call the GPU to adjust the resolution of the decoded video image frames. To ensure the smoothness of video playback and the smooth progress of resolution adjustment, the second processing mode is used to process the video stream. The GPU adjusts the resolution of the video image frames stored in the cache buffer (the buffer in the cache buffer), and then copies them to the display buffer (the buffer in the display buffer) for display. For the case where the resolution is not the preset resolution, since it is not necessary to call the GPU to adjust the resolution of the decoded video image frames, the first processing mode can be used to process the video stream to reduce the video playback power consumption of the terminal device.

[0010] In a possible implementation, before using the first processing mode to process the video stream, the video playback method further includes: obtaining the performance information of the terminal device, where the performance information includes one or more of the processor model, memory size, battery rated power, and battery remaining power; determining the processing mode of the video stream based on the performance information.

[0011] With the above technical solution, the processing mode default used for video playback can be determined based on the performance information of the terminal device. For example, if any of the following conditions is met: the processor performance score is low, the memory is small, the battery capacity is small, or the battery remaining power is low, the first processing mode is used by default to decode and play the video stream. Otherwise, the second processing mode is used by default to decode and play the video stream.

[0012] In a possible implementation, before processing the video stream using the first processing mode, the video playback method further includes: obtaining historical video playback related information of the terminal device, where the historical video playback related information includes one or more of the video playback frequency per unit time, the video playback duration per unit time, the number of operations of the controls in the video playback interface per unit time, and the network speed of the network connected when the terminal device plays the video; determining the processing mode of the video stream based on the historical video playback related information.

[0013] With the above technical solution, it is also possible to determine the processing mode that the video playback defaults to use based on the historical video playback related information of the terminal device. For example, if any of the following conditions is met: the video playback frequency per unit time is relatively high, the video playback duration per unit time is relatively long, the number of operations of the controls in the video playback interface per unit time is relatively small, and the network speed of the network connected when the terminal device plays the video is relatively good, the first processing mode is default used for decoding and playing the video stream. Conversely, the second processing mode is default used for decoding and playing the video stream. The unit time can be set according to actual needs. For example, the unit time is one day.

[0014] In a possible implementation, before processing the video stream using the first processing mode, the video playback method further includes: determining the processing mode of the video stream based on a preset mode selection model, where the training data of the mode selection model includes the performance information of the terminal device and the historical video playback related information of the terminal device.

[0015] With the above technical solution, it is also possible to determine the processing mode that the video playback defaults to use based on the pre-trained mode selection model. The training data of the mode selection model may include the performance information of the terminal device and / or the historical video playback related information of the terminal device.

[0016] In a possible implementation, the preset trigger event includes a video image frame playback stuttering event or an event that the video application receives a preset operation.

[0017] With the above technical solution, if a video image frame playback stuttering event or an event that the video application receives a preset operation is detected, by switching the processing mode of the video stream from the first processing mode to the second processing mode, the smoothness of the video playback can be guaranteed and the preset operation can be correctly executed. For example, the preset operation may include an operation of triggering the call of the graphics processing unit (GPU) to process the video image frame.

[0018] In a possible implementation, the preset operation includes an operation of triggering the call of the graphics processing unit to process the video image frame.

[0019] With the above technical solution, the second processing mode is configured with the logic for the graphics processor to process video image frames. In the case where an event that triggers the call to the graphics processor to process video image frames is detected, the processing mode of the video stream is switched from the first processing mode to the second processing mode, so that the GPU can smoothly process the video image frames.

[0020] In a possible implementation, in the first processing mode, the graphics processor is not used to process the decoded video image frames.

[0021] With the above technical solution, in the first processing mode, since the GPU is not used to process the video image frames, the video playback power consumption of the terminal device can be reduced.

[0022] In a possible implementation, the video playback interface provided by the video application includes multiple controls, and the preset operation includes a touch operation on a target control among the multiple controls, and the target control is a control for triggering the call to the GPU.

[0023] With the above technical solution, the second processing mode is configured with the execution logic of the task corresponding to the target control. In the case where an event of a click operation on the target control in the video playback interface is detected, the processing mode of the video stream is switched from the first processing mode to the second processing mode, so that the click operation on the target control can be smoothly responded to.

[0024] In a second aspect, the present application provides a terminal device, which includes a touch screen, a memory, and a processor; the touch screen, the memory, and the processor are coupled; the memory is used to store program instructions; the processor is used to read the program instructions stored in the memory to implement the video playback method in the first aspect above.

[0025] In a third aspect, the present application provides a computer-readable storage medium, in which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the video playback method in the first aspect above is implemented.

[0026] In a fourth aspect, the present application provides a chip, which is coupled to the memory in the terminal device, and the chip is used to control the terminal device to implement the video playback method in the first aspect above.

[0027] In addition, the technical effects brought by the second aspect to the fourth aspect can be referred to the descriptions related to the methods in each design in the above method part, and will not be elaborated here. Description of the Drawings

[0028] Figure 1 It is a schematic diagram of the video playback architecture of the terminal device provided by an embodiment of the present application;

[0029] Figure 2 Schematic diagram of an image frame stored in a display buffer provided by an embodiment of the present application;

[0030] Figure 3 Schematic diagram of an image frame stored in a display buffer provided by another embodiment of the present application;

[0031] Figure 4 Schematic diagram of the interface display of a terminal device provided by an embodiment of the present application;

[0032] Figure 5 Schematic diagram of the interface display of a terminal device provided by another embodiment of the present application;

[0033] Figure 6 Schematic diagram of the process of a terminal device decoding and playing an acquired video stream provided by an embodiment of the present application;

[0034] Figure 7 Schematic diagram of the application architecture of a mode selection framework provided by an embodiment of the present application;

[0035] Figure 8 Flowchart of a video playback method provided by an embodiment of the present application;

[0036] Figure 9 Hardware architecture diagram of a terminal device provided by an embodiment of the present application;

[0037] Figure 10 Software architecture diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0039] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, words such as "exemplary", "or", "for example", etc. are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary", "or", "for example" is intended to present related concepts in a specific manner.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the description of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that unless otherwise specified in this application, " / " means "or". For example, A / B may represent A or B. The "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, these three situations. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b, or c may represent: a, b, c, a and b, a and c, b and c, a, b, and c, these seven situations. It should be understood that the order of the steps shown in the flowcharts herein may be changed and some may be omitted.

[0041] To facilitate the understanding of the embodiments of the present application, first, the technical terms related to this application are introduced:

[0042] User interface (UI): It is a media interface for interaction and information exchange between an application or an operating system and a user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is source code written in specific computer languages such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on a terminal device and finally presented as content recognizable by the user. The common form of presentation of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations presented in a graphical manner. It can be visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. displayed on the display screen of the terminal device.

[0043] Application (APP): A software program capable of implementing one or more specific functions. For example, instant messaging applications, video applications, audio applications, image capture applications, cloud desktop applications, and so on.

[0044] Video: A dynamic visual presentation composed of consecutive image frames. An image frame is the smallest unit that makes up a video. Each image frame is a static image. When multiple image frames are played continuously at a certain rate (e.g., 24 FPS, 25 FPS, or 30 FPS), the human eye can perceive dynamic motion and changes.

[0045] In the related art, in order to ensure the smoothness of video playback and improve the picture quality, video playback applications generally use a multi-buffer caching technique to cache the image frames decoded from the video stream, and call the graphics processing unit (GPU) to perform processing such as rendering enhancement and resolution adjustment on the cached image frames. However, with the development of communication technology and video playback technology, the network environment has been greatly improved. It is very likely to ensure the smoothness of video playback without caching the image frames, and the picture quality of the decoded image frames is also very likely to meet the viewing needs of users. During video playback, if multi-buffer is always used to cache the image frames, it will bring additional computational load and power consumption (the image frames need to be copied from the cached buffer to the display buffer, and more memory space needs to be occupied), and the use of GPU will further increase the power consumption of the terminal device, thereby reducing the battery life of the terminal device.

[0046] In view of this, the embodiments of the present application provide a video playback method, which preferentially uses the first processing mode to decode and play the video stream. In the first processing mode, the multi-buffer caching technique is not used to cache the image frames decoded from the video stream, and the GPU is not called to process the image frames decoded from the video stream (e.g., rendering enhancement, resolution adjustment, etc.). When an event for triggering the second processing mode is detected, it switches from the first processing mode to the second processing mode, and uses the second processing mode to decode and play the video stream. In the second processing mode, the multi-buffer caching technique is used to cache the image frames decoded from the video stream, and / or the GPU is called to process the image frames decoded from the video stream.

[0047] Please refer to Figure 1 , which is a schematic diagram of the video playback architecture of the terminal device provided by the embodiments of the present application in different processing modes.

[0048] Taking a smart phone as an example of the terminal device, the terminal device is installed with an application capable of video playback. The application capable of video playback may refer to applications such as short video playback applications and film and television drama playback applications. The embodiments of the present application do not limit this. The terminal device may respond to a first operation of the user to start the video application. After the video application is started, the video application may respond to a second operation of the user on the application interface to play a first video specified by the user.

[0049] In some embodiments, the first video may have different picture resolutions for the user to select, and different picture resolutions may correspond to different video uniform resource locators (URLs). The video application may access the video URL corresponding to the current picture resolution of the first video through a network (such as a cellular communication network, a wireless local area network, etc.) to obtain the video stream of the first video at that picture resolution.

[0050] The video stream of the first video may be a data stream in the transport stream (TS) format or a data stream in other formats. The embodiments of the present application do not limit this.

[0051] As Figure 1 shown, in the first processing mode, the video stream of the first video may be decoded by a video codec (such as MediaCodec) to obtain image frames (hereinafter referred to as video image frames). The decoded video image frames may be directly sent for display. For example, the decoded video image frames may be directly stored in the display buffer and displayed on the display screen by the display driver.

[0052] In some implementations, the display mechanism of the terminal device may be a double-buffer mechanism or a triple-buffer mechanism. If the display mechanism of the terminal device is a double-buffer mechanism, that is, the display buffer includes two buffers for display (hereinafter referred to as display buffers for easy distinction), one display buffer is used to display the current picture, and the other display buffer is used as a backup buffer. Figure 1 Taking the display buffer including two display buffers as an example. If the display mechanism of the terminal device is a triple-buffer mechanism, that is, the display buffer includes three display buffers, one display buffer is used to display the current picture, and the other two display buffers are used as backup buffers.

[0053] In some implementations, in the first processing mode, the specific process of each decoded video image frame being displayed on the display screen may include:

[0054] i. A decoded video image frame is stored in a display buffer.

[0055] ii. The SurfaceFlinger reads the image frames to be composed from the display buffer, and the image frames to be composed include the decoded video image frames.

[0056] In some embodiments, during the process of playing the first video on the terminal device, the image frames stored in the display buffer may only include the decoded video image frames, or may also include other image frames. That is, the image frames to be composed may only include the decoded video image frames, or may also include the decoded video image frames and other image frames.

[0057] For example, as Figure 2 shown, the image frames stored in the display buffer include the decoded video image frame I1. As Figure 3 shown, the image frames stored in the display buffer include the decoded video image frame I2, the image frame I3 corresponding to the playback control (hereinafter referred to as the control image frame), and the image frame I4 corresponding to the bullet screen (hereinafter referred to as the bullet screen image frame). The control image frame I3 and the bullet screen image frame I4 can be rendered by the video application. The control image frame I3 can include one or more playback controls. For example, the pause control Ac1, the next episode control Ac2, the bullet screen control Ac3, the speed control Ac4, the screenshot control Ac5, the intelligent recognition control Ac6, etc. The bullet screen image frame I4 can include one or more bullet screen contents.

[0058] iii. The SurfaceFlinger sends the image frames to be composed read from the display buffer to the Hardware Composer (HWComposer, HWC) for image frame composition.

[0059] iv. The HWC sends the composition result to the display driver to be displayed on the display screen.

[0060] In some embodiments, if the image frames to be composed only include the decoded video image frames, for example, the playback interface is full-screen playback. In this case, the HWC does not need to compose the video image frames. The HWC can directly use the video image frames as the composition result (that is, the HWC does not need to perform any processing on the video image frames) and send them to the display driver to display the video image frames on the display screen. The display result is as Figure 4 shown. In this case, the video playback does not require the participation of the GPU, and the video playback will not cause power consumption of the GPU.

[0061] In some embodiments, if the image frames to be composed include not only the decoded video image frames but also other image frames, for example, the playback interface is non-full-screen playback, the image frames to be composed also include Figure 3The control image frame shown. In this case, HWC can synthesize the decoded video image frame and the control image frame, and then send the synthesis result to the display driver for display on the display screen. The display result is as shown in Figure 5 shown. In the video playback in this case, the video application may need to call the GPU to render the control image frame (for example, render the text in the control image frame). However, since only the GPU is called to render the control image frame and the GPU is not called to process the video image frame, the power consumption of the GPU in this case is relatively low.

[0062] As shown in Figure 1 shown, in the second processing mode, the video stream of the first video can be decoded by a video codec to obtain a video image frame, and the decoded video image frame can be stored in a buffer buffer. The buffer buffer may include n buffers (hereinafter referred to as buffer buffers for easy distinction), where n is a positive integer, and the value of n can be set according to the memory size of the terminal device and the actual buffer requirement. The embodiments of the present application do not limit this. For example, n is 40, and the buffer buffer can buffer 40 video image frames.

[0063] In the second processing mode, the video application can also call the GPU to perform image enhancement, resolution adjustment, format conversion, etc. on the video image frames stored in the buffer buffer. The processed video image frames can be transmitted to the display buffer for display. For example, image enhancement can be that the video application calls the GPU to render each decoded video image frame to adjust one or more of the brightness, contrast, color saturation, background, people, etc. of the video image frame. Resolution adjustment can be that the video application calls the GPU to adjust the resolution of the decoded video image frame to better adapt to the resolution of the display screen. Since the GPU needs to be called to process each frame of the video image frame to be played, the power consumption of the GPU in this case is relatively high.

[0064] In some embodiments, in the second processing mode, the video application can also respond to the user's click operation on the screenshot control Ac5 and call the GPU to process the intercepted video segment or the intercepted video picture. For example, the video picture in the intercepted video segment is a picture synthesized from a video image frame and a control image frame, or a picture synthesized from a video image frame, a control image frame, and a bullet screen image frame. By calling the GPU to process the intercepted video segment, the final obtained video segment picture can only include the video image frame. For example, the GPU can use the recognition algorithm or model in the related technology to process the intercepted video segment so that the final obtained video segment picture only includes the video image frame. Since each frame of the video image frame in the video segment needs to be processed, the power consumption of the GPU in this case is relatively high.

[0065] In some embodiments, in the second processing mode, the video application can also call the GPU to identify the information of each character in the current screen, the information of the actor of each character, etc. in response to the user's click operation on the intelligent recognition control Ac6. For example, the GPU can identify the information of each character in the current screen and the information of the actor of each character by running an existing generative artificial intelligence (AI) model. Since the GPU needs to be called to run the generative AI model data, the power consumption of the GPU in this case is relatively high.

[0066] In some embodiments, the first processing mode is not configured with the execution logic of the tasks corresponding to the screenshot control Ac5 and the smart recognition control Ac6, and the second processing mode is configured with the execution logic of the tasks corresponding to the screenshot control Ac5 and the smart recognition control Ac6. When a click operation on the screenshot control Ac5 or the smart recognition control Ac6 is detected, the tasks corresponding to the screenshot control Ac5 or the smart recognition control Ac6 need to be executed in the second processing mode to ensure that the tasks corresponding to the screenshot control Ac5 or the smart recognition control Ac6 are executed smoothly.

[0067] Combine the following Figure 6 The process of decoding and playing the video stream by the terminal device is described in detail. Figure 6 Taking the example of a terminal device having a video application installed, the video application can be a video playback application for playing short videos, movies and TV series, etc., and this application example is not limited to this.

[0068] S30, in response to the user's operation of playing the first video in the application interface of the video application, obtaining a video stream of the first video.

[0069] In some embodiments, after the video application is started, the user can select a video to play in the application interface of the video application, for example, select to play the first video. The definition of the first video can be the definition selected by the user or the definition preset by the video application. The video stream of the first video obtained by the terminal device is the video stream corresponding to the selected definition. For example, video streams of different definitions correspond to different video URLs, and the video application in the terminal device can access the corresponding video URL through the network to obtain the video stream corresponding to the selected definition.

[0070] S31, determining a processing mode of a video stream of a first video.

[0071] In some embodiments, after acquiring the video stream of the first video, the terminal device may first determine a processing mode of the video stream of the first video, and then process the video stream of the first video based on the determined processing mode to play the first video.

[0072] For example, if the score of the processor of the terminal device is relatively low, for example, the running score of the central processing unit (CPU) or the system on chip (SoC) is lower than a preset score, or the memory is small (for example, lower than a first preset capacity), or the battery capacity is small (for example, lower than a second preset capacity), or the remaining battery power is low (for example, lower than a preset percentage), the terminal device determines that the processing mode of the video stream of the first video is the first processing mode. If the score of the processor of the terminal device is relatively high (the running score of the CPU or SoC is greater than or equal to the preset score), or the memory is large (for example, greater than or equal to the first preset capacity), or the battery capacity is large (for example, greater than or equal to the second preset capacity), or the remaining battery power is high (for example, greater than or equal to the preset percentage), the terminal device determines that the processing mode of the video stream of the first video is the second processing mode.

[0073] Also for example, if the frequency / duration of the user using the terminal device to watch videos every day is high (for example, greater than or equal to a preset frequency / preset duration), or the number of operations on the video playback interface when watching videos is small (for example, the number of operations is less than or equal to a preset number), or the network speed when watching videos is good (for example, the network rate is greater than or equal to a preset rate), the terminal device determines that the processing mode of the video stream of the first video is the first processing mode. If the frequency / duration of the user using the terminal device to watch videos every day is low (for example, less than the preset frequency / preset duration), or the number of operations on the video playback interface when watching videos is large (for example, the number of operations is greater than the preset number), or the network speed when watching videos is poor (for example, the network rate is less than the preset rate), the terminal device determines that the processing mode of the video stream of the first video is the second processing mode.

[0074] S32, if it is determined that the processing mode of the video stream of the first video is the first processing mode, decode the video stream of the first video, and store the decoded video image frames in the display buffer.

[0075] In some embodiments, the display buffer may include two or three buffers (referred to as display buffers). If it is determined that the processing mode of the video stream of the first video is the first processing mode, the terminal device decodes the video stream of the first video, and stores the decoded video image frames in the display buffer. For example, the video codec in the terminal device decodes the video stream of the first video, and stores the decoded video image frames in the display buffer.

[0076] In some embodiments, in the first processing mode, the terminal device may also respond to a preset trigger event. For example, in response to a user's click operation on a specified control in the video playback interface, the processing mode of the video stream is switched from the first processing mode to the second processing mode. For example, the specified control may include a screenshot control Ac5, an intelligent recognition control Ac6, etc.

[0077] S33. If it is determined that the processing mode of the video stream of the first video is the second processing mode, decode the video stream of the first video and store the decoded video image frames in the buffer cache.

[0078] In some embodiments, the buffer cache may include multiple buffers (referred to as buffer caches) for caching video image frames. If it is determined that the processing mode of the video stream of the first video is the second processing mode, the terminal device decodes the video stream of the first video and stores the decoded video image frames in the buffer cache including multiple buffer caches. In the case of network latency, by sequentially copying the video image frames stored in the multiple buffer caches to the display buffer for display, the smoothness of video playback can be improved, thereby avoiding video stuttering. For example, the video codec in the terminal device decodes the video stream of the first video, and the video image frames decoded by the video codec are stored in the buffer cache.

[0079] In some embodiments, each buffer cache in the buffer cache can be used to store one frame of video image frame. The buffer cache can also store information for characterizing the frame playback order of the video image frames, such as decoding timestamps, frame numbers, etc., which is convenient for subsequently copying the video image frames cached in the buffer cache to the display buffer according to the information of the frame playback order to realize sequential playback of the video image frames.

[0080] S34. Transmit the video image frames stored in the buffer cache to the display buffer.

[0081] In some embodiments, the terminal device may copy the video image frames stored in the buffer cache to the display buffer to realize transmitting the video image frames stored in the buffer cache to the display buffer. For example, the processor in the terminal device can respond to a playback request for a certain video image frame by a video application and copy the video image frame from the buffer cache to the display buffer.

[0082] In some embodiments, the terminal device can also perform image enhancement, resolution adjustment, etc. on the video image frames stored in the buffer cache, and then transmit the processed video image frames to the display buffer.

[0083] In S35, SurfaceFlinger reads the image frames to be synthesized from the display buffer and sends the image frames to be synthesized to HWC. The image frames to be synthesized include video image frames.

[0084] In S36, HWC synthesizes the image frames to be synthesized and sends the synthesis result to the display driver.

[0085] In S37, the display driver displays the synthesis result on the display screen.

[0086] In some embodiments, if the display buffer only includes video image frames, that is, the image frames to be synthesized only include video image frames, SurfaceFlinger reads the video image frames from the display buffer and sends them to HWC. HWC can directly send the video image frames to the display driver (that is, HWC does not synthesize the video image frames), and the display driver displays the video image frames on the display screen. If the display buffer includes other image frames in addition to video image frames (for example, control image frames, barrage image frames), SurfaceFlinger reads the image frames to be synthesized (video image frames and other image frames) from the display buffer and sends them to HWC. HWC synthesizes the received image frames and sends the synthesis result to the display driver, and the display driver displays the synthesis result on the display screen.

[0087] As Figure 7 shown, in order to implement determining the processing mode of the video stream, a mode selection framework can be configured in the terminal device. The terminal device can adopt the first processing mode or the second processing mode for decoding and playing the video stream based on the mode selection information output by the mode selection framework. For example, a video application can respond to an operation of playing a first video, obtain the video stream of the first video. The first video can call a video codec to decode the video stream of the first video. The video codec can store the decoded video image frames in the display buffer (corresponding to the first processing mode) or in the cache buffer (corresponding to the second processing mode) based on the mode selection information output by the mode selection framework.

[0088] For example, taking the mode selection information represented as a numerical value as an example, in the case where the mode selection information output by the mode selection framework is Information 1, the terminal device adopts the first processing mode for decoding and playing the video stream. In the case where the mode selection information output by the mode selection framework is Information 2, the terminal device adopts the second processing mode for decoding and playing the video stream. In other embodiments, the mode selection information can also be represented by other characters, and the above examples are not limited in practical applications.

[0089] In some embodiments, the mode selection framework may output corresponding mode selection information based on the constructed device performance profile, device usage profile, video playback smoothness, or specific operations received during video playback, etc.

[0090] In some embodiments, the device performance profile may include multiple performance information, and the multiple performance information may include parameter information such as the processor, memory, and battery in the terminal device. For example, the mode selection framework may determine that the default processing mode for video playback is the first processing mode or the second processing mode based on the device performance profile. For example, for the case where the processor performance score is low, the memory is small, the battery capacity is small, or the remaining battery power is low, the mode selection information output by the mode selection framework is Information 1 to trigger the terminal device to default to the first processing mode for decoding and playing the video stream.

[0091] For example, a low processor performance score may be that the running score of the processor is lower than a preset score. The running score of the processor being lower than the preset score indicates that there may be a large time consumption for the processor (integrated with GPU) to process video image frames. Therefore, it is recommended to use the first processing mode for decoding and playing the video stream. A small memory may be that the memory capacity is lower than a first preset capacity. The memory capacity being lower than the first preset capacity indicates that there may not be enough memory space as a cache buffer. Therefore, it is recommended to use the first processing mode for decoding and playing the video stream. A small battery capacity may be that the rated battery capacity is lower than a second preset capacity. The rated battery capacity being lower than the second preset capacity indicates that the sustainable video playback time is short. Therefore, it is recommended to use the first processing mode for decoding and playing the video stream. A low remaining battery power may be that the remaining battery power is lower than a preset percentage. The remaining battery power being lower than the preset percentage indicates that the sustainable video playback time is short. Therefore, it is recommended to use the first processing mode for decoding and playing the video stream. The first preset capacity, the second preset capacity, and the preset percentage can all be set according to actual needs, and the embodiments of the present application do not limit this.

[0092] In some embodiments, if the mode selection framework determines the default processing mode for video playback based on multiple performance information, a weight coefficient may be set for each performance information. For example, based on the score of each performance information (pre-constructed scoring rules) and the corresponding weight coefficient, a total score is obtained, and then the total score is compared with a preset score value to determine that the default processing mode is the first processing mode or the second processing mode.

[0093] In some embodiments, the device usage portrait for video playback may include multiple pieces of historical video playback-related information. The multiple pieces of historical video playback-related information may include the frequency / duration of the user watching videos using the terminal device per unit time (e.g., per day), the number of times the user operates the controls in the video playback interface during video playback, the network speed when the user watches videos using the terminal device, and other information. For example, the mode selection framework may determine that the default processing mode for video playback is the first processing mode or the second processing mode based on the device usage portrait. For example, for the case where the frequency / duration of watching videos per day is relatively high (greater than a preset frequency or preset duration), or the number of times of operating the controls in the video playback interface when watching videos per day is relatively small (less than a preset number), or the network speed when watching videos is relatively good (network rate greater than a preset rate), the mode selection information output by the mode selection framework is Information 1, so as to trigger the terminal device to default to using the first processing mode for video stream decoding and playback, thereby reducing the power consumption of video playback.

[0094] For example, a relatively high frequency / duration of watching videos per day indicates that more device power is used for video playback. In order to increase the duration of the device's sustainable video playback, it is therefore recommended to use the first processing mode for video stream decoding and playback. Another example is that a relatively small number of times of operating the controls in the video playback interface when watching videos per day indicates a relatively low probability of clicking on a specified control (e.g., the specified control may include a screenshot control Ac5, an intelligent recognition control Ac6, etc.). Therefore, it is recommended to use the first processing mode for video stream decoding and playback. Another example is that a relatively good network speed when watching videos indicates that there is a relatively low probability of video playback jitter without video image frame caching. Therefore, it is recommended to use the first processing mode for video stream decoding and playback.

[0095] In some embodiments, if the mode selection framework determines the default processing mode for video playback based on multiple pieces of historical video playback-related information, a weight coefficient may be set for each piece of historical video playback-related information. For example, based on the score of each piece of historical video playback-related information (a scoring rule is pre-constructed) and the corresponding weight coefficient, a total score is obtained, and then the total score is compared with a preset score value to determine whether the default processing mode is the first processing mode or the second processing mode.

[0096] In some embodiments, video playback smoothness may refer to whether there is picture jitter during video playback. The specific operations received during video playback may refer to whether an operation to trigger the call of the GPU, or an operation to trigger the GPU to process video image frames, or whether a click operation on a specified control is received during video playback.

[0097] For example, a specific operation may be an operation that can trigger the invocation of the GPU, such as clicking on the barrage control Ac3 to enable the barrage function, clicking on the screenshot control Ac5 to capture a video segment, clicking on the intelligent recognition control Ac6 to recognize the characters in the picture, or switching the picture clarity to a preset resolution (e.g., 4K resolution). In other embodiments, the specific operation may not include the operation of clicking on the barrage control Ac3 to enable the barrage function.

[0098] For another example, the specific operation may further include operations that can trigger the invocation of the GPU to process video image frames, such as clicking on the intelligent recognition control Ac6 to recognize the characters in the picture, or switching the picture clarity to a preset resolution.

[0099] In some embodiments, when video playback experiences frame freezes, or when a specific operation is received during video playback, the mode selection information output by the mode selection framework is Information 2 to trigger the terminal device to decode and play the video stream using the second processing mode. Assume that the frame rate of video playback is 25 FPS. If the display interval between two video image frames is detected to exceed 40 ms, it can be considered that video playback experiences frame freezes. For example, when a specific operation received during video playback requires the invocation of the GPU or other hardware resources to execute the task corresponding to the specific operation, there is a certain time consumption for task processing and it occupies device resources (hardware resources or network resources). Decoding and playing the video stream using the second processing mode can ensure the smoothness of video playback to the greatest extent.

[0100] In some embodiments, the terminal device may also be configured with a control or interface for setting the processing mode. For example, the settings application of the terminal device includes an interface for setting the processing mode. The user can set in this interface to automatically select the processing mode or manually select the processing mode. Automatically selecting the processing mode may refer to automatically selecting the processing mode based on the mode selection framework, and manually selecting the processing mode may refer to the user setting the default processing mode used by the video application (e.g., defaulting to use the first processing mode or the second processing mode).

[0101] In some embodiments, a mode selection model can also be trained by constructing training data and adding labels to each piece of training data. The training data may include performance information and / or historical video playback related information. The mode selection framework can determine the default processing mode used by the video application based on the labeled training data. For example, any possible algorithm such as the decision tree algorithm or the support vector machine algorithm can be used to train the mode selection model. The selection of the machine learning algorithm can be based on actual usage requirements, and the embodiments of the present application do not make limitations.

[0102] For example, label training data a with label 1 (first processing mode), label training data b with label 2 (second processing mode), use training data a and training data b as the input of the mode selection model, use label 1 and label 2 as the expected results, and assign initial values to the connection weights between neurons in each layer of the neural network. According to the connection weights between neurons in each layer, use the neural network algorithm to obtain the actual output results corresponding to the training data. After processing the expected results and the actual output results using the error function, adjust the connection weights between neurons in each layer according to the processing results to obtain the adjusted connection weights between neurons in each layer. Repeat the above training process. After multiple rounds of iterative training, a trained mode selection model can be obtained.

[0103] For another example, taking the case where some training data includes performance information and some training data includes historical video playback related information as an example, the training and deployment of the mode selection model can be completed before the terminal device leaves the factory. After the terminal device leaves the factory, in the case where the preset number of historical video playback related information has not been collected, the mode selection framework can input the performance information of the terminal device into the mode selection model to obtain the default processing mode of the video stream. In the case where the preset number of historical video playback related information has been collected, the mode selection model deployed on the terminal device can be trained based on the collected historical video playback related information. Subsequently, the mode selection framework can input the video playback related information of the terminal device into the mode selection model to obtain the default processing mode of the video stream.

[0104] For yet another example, taking the case where the training data includes historical video playback related information as an example, in the case where the preset number of historical video playback related information has not been collected (that is, an effective mode selection model has not been trained yet), the mode selection framework can use the processing mode preset by the developer as the default processing mode of the video stream. For example, the developer can preset the first processing mode or the second processing mode as the default processing mode of the video stream based on the performance information of the terminal. In the case where the preset number of historical video playback related information has been collected, a mode selection model can be trained based on the collected historical video playback related information. Subsequently, the mode selection framework can input the video playback related information of the terminal device into the mode selection model to obtain the default processing mode of the video stream. For example, the mode selection framework can input the video playback related information of the terminal device on the previous day into the mode selection model to obtain the default processing mode of the video stream for the current day.

[0105] In some embodiments, if the mode selection framework determines that the processing mode defaultly used by the video application based on the mode selection model is the first processing mode, during the video playback process, the mode selection framework can also switch the first processing mode to the second processing mode when the picture freezes or a specific operation is received (for example, an operation that can trigger the call of the GPU or an operation that can trigger the GPU to process the video image frames).

[0106] Refer to Figure 8 As shown, it is a video playback method provided by an embodiment of the present application. The video playback method can be applied to a terminal device, and the terminal device includes a display screen. The video playback method can include:

[0107] S51, in response to a first operation, start the video application.

[0108] For example, the first operation is an operation of clicking the icon of the video application on the desktop. The terminal device can start the video application in response to the operation of clicking the icon of the video application on the desktop. In other embodiments of the present application, the first operation can also be an operation of calling the video application in other applications. For example, when the application running in the foreground of the terminal device is an instant messaging application, if the user starts the video application through a touch operation of a certain control, icon, option, etc. in the instant messaging application and enters the application interface of the video application.

[0109] S52, in response to a second operation of playing a first video on the video application, obtain the video stream of the first video.

[0110] For example, the second operation can be an operation of clicking to play the first video on the application interface of the video application. The terminal device can obtain the video stream of the first video in response to the operation of clicking to play the first video.

[0111] S53, process the video stream using the first processing mode. The first processing mode includes decoding the video stream and storing the decoded video image frames in the display buffer for display.

[0112] In some embodiments, the display buffer can include two or three display buffers. The embodiments of the present application do not limit the number of display buffers included in the display buffer.

[0113] In some embodiments, the terminal device can be configured to default to using a first processing mode to process the video stream. In the first processing mode, the GPU is not used to process the decoded video image frames, so as to save the power consumption of video playback. The first processing mode may include decoding the video stream and storing the decoded video image frames in the display buffer for display. For example, SurfaceFlinger reads the image frames to be synthesized (video image frames, or video image frames and other image frames) from the display buffer and sends them to HWC for synthesis and display.

[0114] S54. In response to a preset trigger event, a second processing mode is used to process the video stream. The second processing mode includes decoding the video stream and storing the decoded video image frames in the cache buffer. The video image frames stored in the cache buffer can be copied to the display buffer for display.

[0115] In some embodiments, the cache buffer may include multiple buffers for caching video image frames. The number of buffers included in the cache buffer can be set according to actual needs, and the embodiments of the present application do not limit this.

[0116] In some embodiments, the preset trigger event may include a video image frame playback stuttering event or an event that the video playback interface of the video application receives a preset operation. For example, the preset operation includes an operation to trigger the call of the GPU to process the video image frames, or the preset operation may include a touch operation on a target control among multiple controls in the video playback interface provided by the video application, and the target control is a control that can trigger the call of the GPU. For example, the controls that can trigger the call of the GPU include the barrage control Ac3, the screenshot control Ac5, and the intelligent recognition control Ac6.

[0117] In some embodiments, the processing mode defaultly used by the terminal device can also be determined based on the resolution of the first video. By obtaining the media information of the video stream of the first video (the media information includes the resolution of the first video), when the resolution of the first video is not the preset resolution, the first processing mode is defaultly used to process the video stream. When the resolution of the first video is the preset resolution, the second processing mode is defaultly used to process the video stream. The preset resolution can be an ultra-high definition resolution. For example, the 4K resolution. For the video playback of the ultra-high definition resolution, it may be necessary to call the GPU to adjust the resolution of the decoded video image frames. In order to ensure the smoothness of video playback and the smooth progress of resolution adjustment, the second processing mode is used to process the video stream.

[0118] In some embodiments, the defaultly used processing mode may refer to the processing mode initially used for the video stream.

[0119] In some embodiments, the processing mode that the terminal device defaults to use may also be determined based on the performance information of the terminal device. For example, by obtaining the performance information of the terminal device, the performance information may include one or more of the processor model, memory size, rated battery power, and remaining battery power, and the processing mode that defaults to use is determined based on the performance information.

[0120] In some embodiments, the processing mode that the terminal device defaults to use may also be determined based on the historical video playback related information of the terminal device. For example, by obtaining the historical video playback related information of the terminal device, the historical video playback related information includes one or more of the video playback frequency per unit time, the video playback duration per unit time, the number of operations of the controls in the video playback interface per unit time, and the network speed of the network connected when the terminal device plays the video, and the processing mode that defaults to use is determined based on the historical video playback related information.

[0121] In some embodiments, a mode selection model may also be obtained through training, and the processing mode that the terminal device defaults to use is determined based on the mode selection model. The training data of the mode selection model may include the performance information of the terminal device and / or the historical video playback related information of the terminal device.

[0122] Refer to Figure 9 As shown, the terminal device 100 involved in the embodiments of the present application will be introduced below. The terminal device in the embodiments of the present application may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a large screen, a smart TV, a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device, etc., which include touch screen devices. The specific form of the terminal device in the embodiments of the present application is not particularly limited. Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of the terminal device 100 provided by the embodiments of the present application.

[0123] The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0124] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0125] In addition, an operating system runs on the above components. For example, the iOS operating system developed by Apple Inc., the Android open-source operating system developed by Google Inc., the Windows operating system developed by Microsoft Corporation, etc.

[0126] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0127] The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of instruction fetching and execution.

[0128] A memory can also be set in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save the instructions or data just used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0129] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0130] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0131] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: Antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0132] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter, amplify, and process the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be provided in the same device.

[0133] The wireless communication module 160 can provide solutions for wireless communications such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the terminal device 100. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0134] In some embodiments, antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, enabling the terminal device 100 to communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0135] The terminal device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0136] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1. Among them, the display screen 194 in the embodiments of the present application may be a touch screen. That is, a touch sensor 180K is integrated in the display screen 194.

[0137] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0138] The random access memory may include a static random-access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally called DDR5 SDRAM), etc.;

[0139] The non-volatile memory may include a disk storage device and a flash memory.

[0140] Flash memory can be classified according to its operating principle into NOR Flash, NAND Flash, 3D NAND Flash, etc. According to the number of potential levels of storage cells, it can include single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. According to storage specifications, it can include universal flash storage (UFS), embedded multi media Card (eMMC), etc.

[0141] The random access memory can be directly read and written by the processor 110. It can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store data of users and application programs, etc.

[0142] The non-volatile memory can also store executable programs and data of users and application programs, etc. It can be pre-loaded into the random access memory for the processor 110 to directly read and write.

[0143] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function.

[0144] The video playback methods in the above embodiments can all be implemented in the terminal device 100 with the above hardware structure.

[0145] The operating system of the terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the terminal device is exemplarily described. Figure 10 It is a block diagram of the software structure of the terminal device. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate through software interfaces. Taking the Android system as an example, in some embodiments, the Android system is divided into four layers, from top to bottom are the application layer, application framework layer (Framework), hardware abstraction layer (HAL), and kernel layer (Kernel).

[0146] The application layer may include a series of application packages. For example, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0147] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The application framework layer may also include a mode selection framework, which can be used to determine the processing mode of the video stream.

[0148] The hardware abstraction layer may include HWC, which can be used for synthesizing image frames.

[0149] The kernel layer is the layer between hardware and software. The kernel layer may contain a display driver, a camera driver, an audio driver, a sensor driver, etc.

[0150] The kernel layer is the core of the operating system of the terminal device, is the first layer of software extension based on the hardware, provides the most basic functions of the operating system, is the basis for the operating system to work, and is responsible for managing the system's processes, memory, device drivers, files, and network systems, and determines the performance and stability of the system. For example, the kernel can determine the operation time of an application on a certain part of the hardware.

[0151] The kernel layer includes programs closely related to the hardware, such as interrupt handlers, device drivers, etc., and also includes basic, common, and frequently running modules, such as a clock management module, a process scheduling module, etc., and also includes key data structures. The kernel layer can be set in the processor or solidified in the internal memory.

[0152] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions run on the terminal device 100, the terminal device 100 is enabled to execute the above related method steps to implement the video playback method in the above embodiment.

[0153] This embodiment also provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above related steps to implement the video playback method in the above embodiment.

[0154] In addition, an embodiment of the present application further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the video playback method in each of the above method embodiments.

[0155] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and conciseness of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0156] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0157] The unit described as a separate component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0160] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A video playback method, applied to a terminal device, characterized in that: The method comprises: In response to the first operation, displaying a first interface of the video application; In response to a second operation on the first interface, starting to play a first frame of a first video at a first moment, and between the first moment and the second moment, image data of the first video is not processed by the first processor; At the second moment, in response to a preset trigger event, using the first processor to process the image data of the first video; After the second moment, the second frame of the first video is played based on the image data of the first video processed by the first processor, and the power consumption of the first processor between the first moment and the second moment is less than the power consumption of the first processor after the second moment.

2. The video playback method according to claim 1, wherein: The processing of the image data of the first video by using the first processor in response to a preset trigger event includes: In response to a click operation on the first control, the image data of the first video is processed by the first processor.

3. The video playback method according to claim 2, characterized in that: The first control includes any one of a barrage control, a screenshot control, a smart recognition control, and a resolution selection control.

4. The video playback method according to claim 1, wherein: The processing of the image data of the first video by using the first processor in response to a preset trigger event includes: Obtain a freeze event of playing the first video; Based on the freeze event, the image data of the first video is processed using the first processor.

5. The video playback method according to any one of claims 1 to 4, characterized in that: The display parameters of the second picture are different from the display parameters of the first picture, and the display parameters include at least one of brightness, contrast, color saturation, and resolution.

6. The video playback method according to claim 1, wherein: The method of starting to play the first frame of the first video at a first moment in response to the second operation on the first interface includes: In response to the second operation on the first interface, if the resolution of the first video to be played is not a preset resolution, the first frame of the first video starts to be played at the first moment, and the image data of the first frame is not processed by the first processor.

7. The video playback method according to claim 6, characterized in that: The method further comprises: In response to the second operation on the first interface, if the resolution of the first video to be played is the preset resolution, the third frame of the first video starts to be played at the first moment, and the image data of the third frame is processed by the first processor.

8. The video playback method according to claim 1, wherein: The method of starting to play the first frame of the first video at a first moment in response to the second operation on the first interface includes: In response to the second operation on the first interface, if the performance information of the terminal device meets the preset conditions, the first frame of the first video starts to be played at the first moment, and the image data of the first frame is not processed by the first processor. The performance information includes one or more of the processor model, memory size, battery rated power, and remaining battery power.

9. The video playback method according to claim 8, characterized in that: The method further comprises: In response to the second operation on the first interface, if the performance information of the terminal device does not meet the preset condition, the third frame of the first video starts to be played at the first moment, and the image data of the third frame is processed by the first processor.

10. The video playback method according to claim 1, characterized in that: The method of starting to play the first frame of the first video at a first moment in response to the second operation on the first interface includes: In response to the second operation on the first interface, if the historical video playback related information of the terminal device meets the preset situation, the first frame of the first video starts to be played at the first moment, and the image data of the first frame has not been processed by the first processor. The historical video playback related information includes one or more of the video playback frequency per unit time, the video playback duration per unit time, the number of operations of the controls in the video playback interface per unit time, and the network speed of the network to which the terminal device is connected when playing the video.

11. The video playback method according to claim 10, characterized in that: The method further comprises: In response to the second operation on the first interface, if the historical video playback related information of the terminal device does not meet the preset situation, the third frame of the first video starts to be played at the first moment, and the image data of the third frame is processed by the first processor.

12. A terminal device, characterized in that: The terminal device includes a display screen, a memory and a processor; The display screen, the memory and the processor are coupled; The memory is used to store program instructions; The processor is used to read the program instructions stored in the memory to implement the video playback method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the video playback method according to any one of claims 1 to 11 is implemented.

Images