Video image quality control method, electronic equipment and storage medium

Abstract

The invention provides a video image quality control method, electronic equipment and a storage medium, and relates to the technical field of terminals. The method comprises the steps that firstly, operation state information of the terminal equipment is acquired, and the operation state information of the terminal equipment comprises temperature information and electric quantity information of the terminal equipment; then acquiring operation scene information of a target video application on the terminal equipment; wherein the running scene information of the target video application comprises window state information of the target video application, and then, judging whether to perform image quality enhancement on a video of the target video application or not by utilizing the running state information of the terminal equipment and the running scene information of the target video application, according to the method, the frame rate of the video of the target video application is controlled by using at least one of the running state information of the terminal equipment and the running scene information of the target video application, so that the video image quality can be adaptively controlled along with the change of the window state, and the power consumption and the load of the terminal equipment are optimized.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for controlling video quality, an electronic device, and a storage medium. Background Art

[0002] With the continuous breakthroughs in artificial intelligence technology and the increasing popularity of various terminal devices, more and more users are using various application software functions on terminal devices to meet their daily needs, such as watching videos using various video software on terminal devices, or playing games using various game software, etc. When users use video software on terminal devices to watch videos or make video calls, the higher the video quality, the higher the power consumption and load required for downloading and buffering of the terminal device.

[0003] At present, in order to improve the quality of the video, a quality enhancement module or algorithm is usually set in the video decoding part of the terminal device to enhance the video quality and improve the user's video viewing experience. However, these quality enhancement modules or algorithms are often effective in all video scenes, resulting in waste of power consumption and increased load in some scenes. Summary of the invention

[0004] In order to solve the above problems, the present application provides a video quality control method, an electronic device and a storage medium, the purpose of which is to be able to adaptively control the video quality as the video window state changes to optimize the power consumption and load of the terminal device.

[0005] In a first aspect, the present application provides a method for controlling video quality, the method comprising: first obtaining operating status information of a terminal device, wherein the operating status information of the terminal device includes temperature information and power information of the terminal device; then obtaining operating scene information of a target video application on the terminal device; wherein the operating scene information of the target video application includes window status information of the target video application, and then, the operating status information of the terminal device and the operating scene information of the target video application can be used to determine whether to perform image quality enhancement control on the video of the target video application, and / or, using at least one of the operating status information of the terminal device and the operating scene information of the target video application, to perform frame rate control on the video of the target video application.

[0006] It can be seen that in the above-mentioned video quality control method, when the user makes the running video application a small window or hides it, this embodiment no longer directly performs image quality enhancement processing on it, but adaptively controls whether the video quality is enhanced by integrating the temperature information and power information of the terminal device and the window status information of the video application. For example, when the video window is a small window, image quality enhancement processing is not performed on it, and frame loss processing is performed. Although the image quality of the video is reduced, the power consumption and load of the terminal device can be optimized. When the video window is updated to a non-small window, it means that the user is still concerned about the video window. At this time, the image quality enhancement processing can be restored, and frame loss processing is not performed, thereby improving the image quality of the video, thereby improving the user's video viewing experience, and achieving a balance between the power consumption overhead of the terminal device and the video quality.

[0007] In one possible implementation, the running status information of the terminal device and the running scenario information of the target video application are used to determine whether to perform image quality enhancement on the video of the target video application, including: determining whether the target video application needs to perform image quality identification to obtain a determination result; when it is determined according to the determination result that image quality identification needs to be performed on the video in the target video application, the state identification result of the terminal device is determined by using the temperature information and power information of the terminal device, and the state identification result of the terminal device and the video window state information of the video application used by the user are fused and identified to obtain an identification result; and then, based on the identification result, it is determined whether to perform image quality enhancement processing on the video of the target video application, thereby ensuring that whether to perform video quality enhancement is adaptively controlled as the video window state changes, effectively avoiding waste of power consumption and increase in load of the terminal device.

[0008] In one possible implementation, based on the recognition result, it is determined whether to enhance the image quality of the video of the target video application, including: generating an image quality enhancement control instruction based on the recognition result, and sending the image quality enhancement control instruction to the video post-processing service unit Vpp, so that Vpp executes the image quality enhancement control instruction to determine whether to enhance the image quality of the video of the target video application, thereby optimizing the power consumption and load of the terminal device.

[0009] In a possible implementation, the recognition result includes but is not limited to global off, global on, or on / off of a single application record in the target video application.

[0010] In one possible implementation, the frame rate of the video of the target video application is controlled by using at least one of the running status information of the terminal device and the running scene information of the target video application, including: judging whether the window of the target video application is a small window, hidden or destroyed according to the window status information of the target video application, obtaining a judgment result, and then updating the scene recognition result record of the target video application according to the judgment result and controlling the frame rate of the video of the target video application, thereby ensuring that the video frame loss processing is adaptively performed as the video window state changes, thereby reducing the power consumption and load of the terminal device.

[0011] In one possible implementation, based on the judgment result, the scene recognition result record of the target video application is updated and the frame rate of the video of the target video application is controlled, including: based on the judgment result, the scene recognition result record of the target video application is updated and a frame rate control instruction is generated, and the frame rate control instruction is sent to the multimedia codec so that the multimedia codec executes the frame rate control instruction to achieve frame rate control of the video of the target video application, thereby optimizing the power consumption and load of the terminal device.

[0012] In a possible implementation, the target video application on the terminal device refers to a video application in which the current user is making a video call or watching a video, such as common video chatting software or video watching software.

[0013] In a possible implementation, the window status information of the target video application includes but is not limited to whether the window of the target video application is a small window or hidden, which is used as a control basis for adaptively controlling the image quality.

[0014] In a second aspect, the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, the processor being used to call and execute the computer program to implement the video quality control method described in any one of the first aspects above.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, is used to implement the video quality control method described in any one of the first aspects above.

[0016] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer executes the video quality control method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 One of the scenario diagrams provided in the embodiment of the present application;

[0018] Figure 2 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0019] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of a method for controlling video quality provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of a process for determining whether a video application needs to perform image quality recognition provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of a process for obtaining temperature information and power information of a terminal device provided in an embodiment of the present application;

[0023] Figure 7 One of the schematic diagrams of the process of obtaining the video window status information of the video application used by the user provided in the embodiment of the present application;

[0024] Figure 8 A schematic diagram of a process for fusing and identifying temperature information and power information of a terminal device and video window status information of a video application used by a user provided in an embodiment of the present application;

[0025] Fig. 9 A schematic diagram of a process for generating an image quality enhancement control instruction provided in an embodiment of the present application;

[0026] Fig.10 A schematic diagram of the execution process of the image quality enhancement control instruction provided in an embodiment of the present application;

[0027] Fig.11 A second schematic diagram of a process for obtaining video window status information of a video application used by a user provided in an embodiment of the present application;

[0028] Fig.12 A schematic diagram of a process for generating a frame rate control instruction provided in an embodiment of the present application;

[0029] Fig.13 A schematic diagram of the execution process of the frame rate control instruction provided in an embodiment of the present application;

[0030] Fig.14 A schematic diagram of the process of dropping frames according to a frame rate threshold provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0032] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0033] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0034] In order to enable those skilled in the art to more clearly understand the solution of the present application, the application scenario of the technical solution of the present application is first explained below.

[0035] See also Figure 1 , which shows a scenario schematic diagram provided in an embodiment of the present application.

[0036] In this example scenario, when a user is watching a video or making a video call using some application software, he or she may open or hide these video applications while operating other pages. At this time, the image quality (including image quality and frame rate) of these small windows or hidden videos is not the main concern of the user. Figure 1As shown in (a), when a user populates the video being watched in video software B on the mobile phone and views the display interface of application software A, the user is not primarily concerned with the image quality of the video being played in video software B, but may be more concerned with the content displayed on the interface of application software A. Figure 1 As shown in (b), when the user hides the video being watched in the B video software on the mobile phone and views the display interface of the A application software, the user is not concerned about the image quality of the video being played in the B video software, but is more concerned about the content displayed on the A application software interface.

[0037] Therefore, in scenarios similar to the above two situations, it is not actually necessary to use the terminal device to improve the image quality of the video being played in the B video software, that is, the image quality of the video does not need to be enhanced. However, in the current terminal devices, in order to provide users with high-quality videos, a quality enhancement module or algorithm is usually set in the video decoding part of the terminal device to enhance the video quality and improve the user's video viewing experience. However, since these quality enhancement modules or algorithms are usually effective in all video playback scenarios, this will result in waste of power consumption and increased load of the terminal device in some scenarios such as the above two situations, and may even cause the device to "overheat".

[0038] In order to overcome the above technical problems, the present application provides a method for controlling video quality, an electronic device and a storage medium, which can adaptively control the video quality when the user makes the video window small or hidden, thereby optimizing the power consumption and load of the terminal device and improving the user's video viewing experience.

[0039] The video quality control method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, televisions, medical instruments, personal digital assistants (PDA), desktops, laptops, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, and wearable devices.

[0040] In order to enable those skilled in the art to more clearly understand the video quality control method provided by the present application, the hardware architecture and software system architecture of the electronic device are first introduced in detail below.

[0041] See also Figure 2 , which shows a schematic diagram of an electronic device provided in an embodiment of the present application.

[0042] like Figure 2As shown, the electronic device 200 may include a processor 210, a temperature sensor 220, a motor 230, a button 240, an indicator 250, a power management module 260, a battery 261, a charging management module 262, a universal serial bus (USB) interface 263, a camera 270, a display screen 280, an audio module 290, a speaker 290A, a receiver 290B, a microphone 290C, and a headphone interface 290D.

[0043] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. The video codec can realize adaptive control of video quality as the window state changes to optimize the power consumption and load of the terminal device.

[0045] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0046] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] The temperature sensor 220 is used to detect the temperature of the electronic device 200. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 220 to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 220 exceeds a threshold value (such as 37°C, etc.), the electronic device 200 reduces the performance of the processor located near the temperature sensor 220, such as not performing video quality enhancement processing, etc., in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold value, the electronic device 200 heats the battery 261 to avoid abnormal shutdown of the electronic device 200 due to low temperature. In some other embodiments, when the temperature is lower than another threshold value, the electronic device 200 boosts the output voltage of the battery 261 to avoid abnormal shutdown caused by low temperature.

[0048] Motor 230 can generate vibration prompts. Motor 230 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as video playback, etc.) can correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0049] The key 240 includes a power button, a volume button, etc. The key 240 may be a mechanical key or a touch key. The electronic device 200 may receive key input and generate key signal input related to user settings and function control of the electronic device 200. For example, the electronic device 200 may receive key input from a user to hide a window of a video software and generate a corresponding key signal input.

[0050] The USB interface 263 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 263 can be used to connect a charger to charge the electronic device 200, and can also be used to transfer data between the electronic device 200 and a peripheral device. It can also be used to connect headphones to play the audio of a video call or video playback through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0051] The charging management module 262 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 262 may receive charging input from a wired charger through a USB interface 263. In some wireless charging embodiments, the charging management module 262 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 262 is charging the battery 261, it may also power the electronic device 200 through the power management module 260.

[0052] The power management module 260 is used to connect the battery 261, the charging management module 262 and the processor 210. The power management module 260 receives input from the battery 261 and / or the charging management module 262 to power the processor 210, the display screen 280, the camera 270, etc. The power management module 260 can also be used to monitor parameters such as the battery power of the electronic device as a control basis to determine whether to perform image quality enhancement processing on the corresponding video application. In some other embodiments, the power management module 260 can also be set in the processor 210. In other embodiments, the power management module 260 and the charging management module 262 can also be set in the same device.

[0053] The camera 270 is used to capture still images or videos, such as taking a video of the user during a video call. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 270, where N is a positive integer greater than 1.

[0054] The display screen 280 is used to display images, videos, etc., such as displaying a screen in which a user is watching a video or making a video call using a certain application software. The display screen 280 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 280, where N is a positive integer greater than 1.

[0055] The electronic device 200 implements the display function through a GPU, a display screen 280, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 280 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0056] The electronic device 200 can implement audio functions such as music playback, video calls, and voice input and output during video playback through the audio module 290, the speaker 290A, the receiver 290B, the microphone 290C, the headphone jack 290D, and the application processor.

[0057] The audio module 290 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 290 can also be used to encode and decode audio signals. In some embodiments, the audio module 290 can be arranged in the processor 210, or some functional modules of the audio module 290 can be arranged in the processor 210.

[0058] The speaker 290A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 200 can listen to music or listen to a hands-free call through the speaker 290A.

[0059] The receiver 290B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or voice message, the voice can be received by placing the receiver 290B close to the human ear.

[0060] Microphone 290C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 290C to input the sound signal into the microphone 290C. The electronic device 200 can be provided with at least one microphone 290C. In other embodiments, the electronic device 200 can be provided with two microphones 290C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 200 can also be provided with three, four or more microphones 290C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0061] The earphone interface 290D is used to connect a wired earphone and can be a USB interface 263, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0062] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0063] In addition, on top of the above components, the electronic device 200 runs an operating system, such as an iOS operating system, an Android operating system, a Windows operating system, etc. Application programs can be installed and run on the operating system.

[0064] See also Figure 3 , which shows a schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0065] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 200.

[0066] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided from top to bottom into the application layer, the framework layer, the Android runtime and system library, the hardware abstraction layer (HAL), and the kernel layer.

[0067] The application layer can include a series of application packages. Figure 3 As shown, the application package may include applications such as camera, call, music, video, WLAN, Bluetooth, and scene recognition. Among them, the video application is used to pass the video data stream to the multimedia codec (Media Codec) of the framework layer and the codec abstraction layer (Codec HAL) of the hardware decoding layer through the media player (Media Player) interface, so as to control the image quality enhancement algorithm in Vpp through the decoder video post-processing service unit (Vpp Service) to provide a dynamic start and stop interface, and control the opening and closing of the video frame rate control algorithm (i.e., control the display frame rate) through the control interface provided by the multimedia codec (Media Codec). The scene recognition application is used to identify the status of the terminal device and each application for scene-based identification (such as mobile phone temperature, power, application scenario, window status information, etc.) and management, so as to perform scene-based dynamic start and stop (i.e., start and stop of video quality enhancement and frame rate control) voting control for the video of the application in the image quality enhancement whitelist and frame rate control whitelist.

[0068] The framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3 As shown, the application framework layer may include a window manager, a content manager, a view system, a multimedia codec (Media Codec), a scene-changing dynamic start and stop interface, etc.

[0069] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0070] The view system includes visual controls. For example, controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display interface of an application can include a view for displaying text and a view for displaying video images.

[0071] The multimedia codec (Media Codec) contains a video frame rate control algorithm, which is used to control the frame rate of the video. It provides the registration information of each application to the scene recognition application by changing the scene-based dynamic start and stop interface, and receives and executes the frame rate control instructions sent by the scene recognition application to specify whether to start or stop the application video.

[0072] Among them, the specific method of using the video frame rate control algorithm to control the frame rate of the video may include but is not limited to complete frame loss and partial frame loss, and the specific implementation method of partial frame loss is not limited, and can be processed according to actual conditions and experience values. For example, it can be processed according to the implementation process of frame loss according to the frame rate threshold described in detail in the subsequent embodiments.

[0073] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0074] The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0075] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0076] The Hardware Abstraction Layer (HAL) is between the Linux kernel and the system operation layer. It is a unified encapsulation of the lower-level Linux drivers and provides an interface to the upper layer, hiding the details of the underlying layer. It can include but is not limited to multimedia hardware codecs (media.hwcodec), codec abstraction layer (Codec HAL), etc. Figure 3 As shown, the codec abstraction layer may include a video post-processing service unit (Vpp Service), which is used to use a picture quality enhancement control algorithm to achieve picture quality enhancement for various video applications. Among them, the specific methods for video picture quality enhancement may include but are not limited to resolution enhancement, color enhancement, SDR to HDR dynamic range conversion, super resolution, intelligent interpolation and other processing methods.

[0077] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0078] The technical solutions involved in the following embodiments can all be implemented in electronic devices having the above-mentioned hardware architecture and software architecture.

[0079] Next, the implementation process of the video quality control method provided by this application will be introduced in detail:

[0080] In some embodiments, to address the Figure 1 In order to solve the problem of power consumption waste and increased load of terminal devices caused by miniaturizing or hiding video applications in some scenarios, the present application provides a method for controlling video quality to adaptively control video quality when the user miniaturizes or hides the video window, thereby optimizing the power consumption and load of the terminal device. Figure 4 As shown, the specific implementation process of the video quality control method may include the following steps S401-S403:

[0081] S401: Acquire operation status information of a terminal device, wherein the operation status information of the terminal device includes temperature information and power information of the terminal device.

[0082] In this embodiment, in order to optimize the power consumption and load of the terminal device and improve the user's video viewing experience, a whitelist of applications that require adaptive video quality control can be pre-set, and the whitelist consists of at least one preset video application (such as video call and video playback applications). That is, when the user uses any video application in the whitelist (subsequently becoming the target video application) to make a video call or watch a video, the video quality can be adaptively controlled as the video window state of the target video application changes (such as full screen video, small video window or hidden video, etc.) by executing steps S401-S402, including but not limited to whether to enhance the image quality and control the transmission frame rate, so as to optimize the power consumption and load of the terminal device.

[0083] Among them, the scenarios of video small windows include but are not limited to: video chat small windows, floating small windows that come with video playback applications (Pip picture-in-picture small windows are determined by area thresholds), mini windows in self-developed floating windows, etc.

[0084] Video hiding scenarios include but are not limited to: hiding the small window of video chat, hiding the invisible video chat after closing the floating background, and other video hiding scenarios that can be expanded in the future.

[0085] Further, when it is detected that the user uses any video application in the whitelist to make a video call or watch a video, in order to achieve adaptive control of the video quality, the operating status information of the terminal device can be first obtained to perform the subsequent step S403. The operating status information of the terminal device may include but is not limited to the temperature information and power information of the terminal device. For example, the temperature information of the device can be obtained by using a temperature sensor preset on the terminal device, and the power information of the device can be obtained by using a power management module preset on the terminal device, so as to determine which type of video quality control of the preset video application can be supported by the current operating state of the terminal device, that is, only image quality enhancement, only frame loss processing, image quality enhancement and frame loss processing at the same time, or at least one type of video quality control mode of neither image quality enhancement nor frame loss processing, so as to reduce the power consumption and load of the terminal device, so as to avoid the problem of overheating of the device.

[0086] S402: Acquire running scenario information of a target video application on a terminal device; wherein the running scenario information of the target video application includes window status information of the target video application.

[0087] In this embodiment, in order to optimize the power consumption and load of the terminal device, it is also necessary to obtain the running scene information of the target video application on the terminal device to perform the subsequent step S403. The running scene information of the target video application may include the window status information of the target video application, and the video window status information may include but is not limited to the small window and hidden of the video window.

[0088] There is no limitation on the determination of a small video window, which can be set according to actual conditions and experience values. For example, when the proportion of the video window area in the entire display interface of the terminal device is less than a preset proportion threshold, it can be determined as a small video window. There is no limitation on the preset proportion threshold, which can be set according to actual conditions and experience values. For example, it can be set to 1 / 4. At this time, when the proportion of the video window area in the entire display interface of the terminal device is less than 1 / 4, the video window can be determined as a small video window.

[0089] S403: Using the operating status information of the terminal device and the operating scenario information of the target video application, determine whether to enhance the image quality of the video of the target video application, and / or, using at least one of the operating status information of the terminal device and the operating scenario information of the target video application, control the frame rate of the video of the target video application to achieve a balance between the power consumption overhead of the terminal device and the video quality.

[0090] In this embodiment, after obtaining the operating status information such as the temperature and power of the terminal device through step S401, and obtaining the operating scene information such as the window status of the preset video application through step S402, in order to optimize the power consumption and load of the terminal device, the operating status information of the terminal device and the operating scene information of the preset video application can be further used to generate and issue control instructions on whether to enhance the image quality of the video of the preset video application, and / or control the frame rate (such as global off, global on, single video application record on / off, no control requirement, etc.), so as to perform corresponding video quality control processing according to the control instruction, so as to achieve a balance between the power consumption overhead and video quality of the terminal device.

[0091] In this way, when the user shrinks or hides the video application window in which he is watching a video or making a video call while operating other pages, the image quality of the videos processed in these shrinks or hidden windows can be adaptively controlled by executing the above steps S401-S403, thereby optimizing the power consumption and load of the terminal device and improving the user's video viewing experience.

[0092] Next, the embodiment of the present application will introduce in detail the process of determining whether to enhance the image quality of a video. In some embodiments, the specific implementation process of the method for determining whether to enhance the image quality of a video may include the following steps S501-S506:

[0093] S501: Determine whether a video application program in which a user is watching a video or making a video call needs to perform image quality recognition, and obtain a determination result.

[0094] In this embodiment, in order to realize adaptive control of whether to enhance the video quality, it is firstly determined whether the video application program that the user is watching a video or making a video call needs to perform image quality recognition, and obtain a determination result. The determination process may include: Figure 5 The steps S5011-S5015 shown are specifically as follows: first, it is possible to start judging whether image quality identification is required, that is, to detect whether the user is watching a video or making a video call through a video application. If so, it is further judged whether the video application belongs to the image quality enhancement identification whitelist. If not, the judgment result is that image quality identification is not required. On the contrary, if it belongs to the image quality enhancement identification whitelist, it is also necessary to judge whether the video application is a designated video application (referring to a video application that must be subject to image quality enhancement control according to actual conditions, and the specific content is not limited) or whether the image quality enhancement in the setting menu is turned on. If the video application is not a designated video application and the image quality enhancement in the setting menu is turned off, the judgment result is that image quality identification is not required. However, if the video application is a designated video application or the image quality enhancement in the setting menu is turned on, the judgment result is that image quality identification is required.

[0095] The specific content of the image quality enhancement recognition whitelist with recognition requirements is not limited and can be pre-set according to actual conditions and experience. For designated video applications (such as a video chat software) that are not controlled by the switch status of the setting menu, if the designated video application is running in the foreground of the terminal device, the judgment result that requires image quality recognition can be directly returned.

[0096] S502: When it is determined based on the judgment result that the video in the video application needs to be image-quality recognized, the temperature information and the power information of the terminal device are obtained, and the state recognition result of the terminal device is determined by using the temperature information and the power information of the terminal device.

[0097] In this embodiment, when it is determined based on the judgment result that the image quality of the video in the video application needs to be identified, it is further necessary to obtain the temperature information and power information of the terminal device and fuse the two to determine the current state identification result of the terminal device, that is, to determine whether to globally turn on enhancement of the video quality.

[0098] This determination process may include Figure 6 Steps A1, A2 (A21-A26), A3 (A31-A34) and A4 (A41-A44) shown are specifically as follows: when it is determined that the temperature of the terminal device obtained is greater than a preset high temperature threshold (high_temp, the specific value is not limited, and can be set according to actual conditions and experience values, for example, high_temp can be set to 37°C, etc.), the current temperature control state is set to off, and the temperature control voting is globally closed and enhanced; when it is determined that the temperature of the terminal device obtained is lower than a preset low temperature threshold (low_temp, the specific value is not limited, and can be set according to actual conditions and experience values, for example, low_temp can be set to 35°C, etc.), the current temperature control state is set to on, and the temperature control voting is globally opened and enhanced; when it is determined that the temperature of the terminal device obtained is between a preset low temperature threshold (low_temp such as 35°C) and a preset high temperature threshold (high_temp such as 37°C), the previous temperature control voting state is retained.

[0099] When it is determined that the battery level of the terminal device obtained is lower than the preset low battery threshold (battery_low, the specific value is not limited and can be set according to actual conditions and experience values, for example, battery_low can be set to 10%, etc.) and the charger is not connected, the battery voting result is set to globally turn off enhancement; when it is determined that the battery level of the terminal device obtained is not lower than the preset low battery threshold (battery_low), or is lower than the preset low battery threshold but the charger is connected, the battery voting result is set to globally turn on enhancement.

[0100] On this basis, when the voting results of temperature control and power are both to turn on enhancement, the fusion result of the two is to turn on enhancement globally; when one of the voting results of temperature control and power is to turn off enhancement, the fusion result of the two is to turn off enhancement globally. In this way, whether the terminal device is low on power or high on temperature, the video quality enhancement function can be disabled to reduce power consumption or continue to heat up.

[0101] S503: Obtain video window status information of a video application used by a user on a terminal device.

[0102] In this embodiment, in order to achieve adaptive control of whether to enhance the video quality, it is also necessary to obtain the video window status information of the video application used by the user on the terminal device, and determine whether the video window is a small window or hidden. If not, it is further necessary to determine whether the video window is destroyed. The specific determination process may include Figure 7 Steps S71-S77 are shown.

[0103] S504: Fusion recognition is performed on the state recognition result of the terminal device and the video window state information of the video application used by the user to obtain a recognition result.

[0104] In this embodiment, after obtaining the temperature information and power information of the terminal device in step S502 and determining the state recognition result of the terminal device, and obtaining the video window state information of the video application used by the user in step S503 and determining whether the video window is a small window or hidden, the information can be further integrated and recognized to obtain the recognition result, including global off, global on, single application record on / off, etc. The specific recognition process may include Figure 8 Steps S81-S88 are shown.

[0105] Among them, it can be determined whether the scene recognition status of the video application has been updated. If yes, the application scene recognition result record needs to be updated. The result record of the application <application package name, image quality switch of this application>, this image quality switch is only affected by the change of the application scene.

[0106] S505: Generate an image quality enhancement control instruction according to the recognition result.

[0107] In this embodiment, after obtaining the recognition result (global off, global on, single application record on / off, etc.) through step S504, a picture quality enhancement control instruction can be further generated and sent to the Vpp module. The specific implementation process may include Fig. 9 Steps S91-S94 are shown.

[0108] Among them, the parameter format of the image quality enhancement control instruction can be: image quality enhancement: global open enhancement: number of entries: package name switch status; package name switch status; ...

[0109] For example, if the recognition result is globally off, the parameter format may be: image quality enhancement: globally turn off enhancement.

[0110] The parameter format for the recognition result of a single application record can be: Image quality enhancement: Globally open enhancement: 1: Package name Open status.

[0111] The parameter format of the recognition result of global on can be: image quality enhancement: global on enhancement: number of entries: package name switch status; package name switch status; ….

[0112] It should be noted that in the optional interface design provided in this application, a unified interface or a single independent interface can be used. Considering that the control is not high frequency, a separate control interface design for sending image quality enhancement control instructions to vpp is also possible. For example:

[0113] Globally enable a(true,null), globally disable a(false,null); single application enable a(true,"packageName"), single application disable a(false,"packageName"). When there are multiple application entries, the single application switch interface can be called multiple times.

[0114] S506: The video post-processing service unit Vpp executes the image quality enhancement control instruction to determine whether to enhance the image quality of the video.

[0115] In this embodiment, after the image quality enhancement control instruction is generated and sent to the Vpp module in step S505, the Vpp module can further execute the image quality enhancement control instruction to determine whether to enhance the image quality of the video, and when it is determined that the video quality needs to be enhanced, a pre-stored video quality enhancement algorithm is used to perform image quality enhancement processing on the video, such as resolution enhancement, color enhancement, SDR to HDR dynamic range conversion, super resolution, intelligent interpolation and other image quality enhancement processing operations on the video. The specific implementation process may include Fig.10 Steps B1-B5 shown. It should be noted that the Vpp module needs to pre-cache the state of the global switch. When the global switch is off, even if the whitelist application creates a new flow, it is necessary to ensure that the post-processing algorithm is in a disabled state; when the global switch is on, the whitelist application creates a new flow by default and the post-processing algorithm is in an enabled state.

[0116] Next, the embodiment of the present application will introduce in detail the implementation process of frame rate control of the video of the preset video application. In some embodiments, the specific implementation process of the method for frame rate control of the video of the preset video application may include the following steps S601-S603:

[0117] S601: Obtain video window status information of a video application used by a user on a terminal device, and perform recognition processing thereon to obtain a recognition result.

[0118] In this embodiment, in order to achieve adaptive processing of frame rate control of video, it is first necessary to start identifying the frame rate control scenario to determine whether the video application used by the user has a frame rate control recognition requirement. If so, after performing scene recognition, the recognition result can be returned, the application scenario recognition result record can be updated, and the frame rate control instruction can be generated and issued.

[0119] This process may include Fig.11 Steps C1-C9 shown are specifically as follows: after obtaining the video window status information of the video application used by the user on the terminal device, further determine whether the video window is a small window, hidden or destroyed, and obtain the corresponding recognition result (that is, the video window is a small window, hidden, non-small window or window destroyed). Among them, the scenarios of the small video window still include but are not limited to: small video chat windows, floating small windows that come with video playback applications (Pip picture-in-picture small windows are determined by area thresholds), mini windows in self-developed floating windows, etc. The scenarios of video hiding still include but are not limited to: small window hiding of video chat, invisible hiding of video chat when the floating background is closed and exited, and other video hiding scenarios that can be expanded in the future. Non-small window scenarios include other scenarios besides small video windows and hiding.

[0120] S602: According to the recognition result, update the application scenario recognition result record and generate a frame rate control instruction.

[0121] In this embodiment, after obtaining the recognition result (i.e., the change of the video window of the video application) through step S601, the record of the application (package name and the switch state of the frame rate control) can be deleted, added or updated according to the recognition result, and a frame rate control instruction is generated, and the control instruction is sent to the MediaCodec module. The specific implementation process may include Fig.12 Steps D1-D11 are shown.

[0122] The frame rate threshold (represented by M) indicates the minimum frame rate value for quality assurance provided by the terminal device to the user, which is obtained through testing by professional visual performance testers. It can be pre-configured in the frame rate control whitelist configuration file, and the frame rate threshold M corresponding to different video applications can be different. There is no limit on the specific value of the frame rate threshold M, which can be 15 frames or 13 frames, etc.

[0123] The parameter format of the frame rate control instruction can be: frame rate control: package name: switch status: whether all are lost: frame rate threshold M.

[0124] For example, the parameter format for turning off the transmission frame rate can be: frame rate control: package name: off status.

[0125] The format for enabling frame rate control can be: frame rate control: package name: on status: not all lost: frame rate threshold M.

[0126] The format for turning on frame rate control and losing all can be: frame rate control: package name: on status: losing all.

[0127] S603: The multimedia codec executes the frame rate control instruction to control the frame rate of the video.

[0128] In this embodiment, after the frame rate control instruction is generated and sent to the MediaCodec module in step S602, the MediaCodec module can further parse the instruction immediately after receiving the frame rate control instruction and execute the corresponding processing logic to realize the frame rate control of the video. The specific implementation process may include Fig.13 Steps E1-E7 shown. Among them, the frame rate control instruction includes three processing logics: one is to turn off the frame drop logic, that is, no frame is dropped, and the video window is not a small window. The second is to drop all frames, and the video window is hidden. The third is to drop frames according to the frame rate threshold, and the video window is a small window.

[0129] The detailed processing process of frame loss according to the frame rate threshold may include: Fig.14Steps F1-F11 shown are specifically as follows: First, the current frame rate (fps) and frame loss rate (N) can be calculated and updated according to the frame index (frameIndex), the time difference between two frame records (T2-T1) and the frame rate threshold parameter M, and it can be determined whether the frames are evenly lost according to the frame loss rate; when the current frame rate is lower than the threshold, no frame loss processing is performed. Then, the time (ms) when the current frame arrives is recorded for each N frames transmitted (for example, the initial value N=10 after receiving the command), and the current frame rate is calculated based on the time difference with the last record. Next, when the frame index frameIndex=0 and the frame rate fps is greater than the threshold M, the current frame is discarded, and the frame loss rate N=(int)((float)fps / (fps-M)+0.5) is dynamically updated to ensure that the frame rate after the frame loss is close to the threshold M and not lower than M.

[0130] In addition, the present application also provides an electronic device. For the hardware structure and software framework of the electronic device, please refer to Figure 2 and Figure 3 The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is used to call and execute the computer program to implement the video quality control method provided in the above description.

[0131] The present application also provides a computer-readable storage medium in an embodiment, on which a computer program is stored. When the computer program is executed by a processor of a terminal device, it is used to implement the video quality control method provided in the above description.

[0132] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for controlling video quality, characterized in that: The method comprises: Acquire operation status information of the terminal device, wherein the operation status information of the terminal device includes temperature information and power information of the terminal device; Acquire the running scenario information of the target video application on the terminal device; the running scenario information of the target video application includes the window state information of the target video application; Using the operating status information of the terminal device and the operating scenario information of the target video application, determine whether to enhance the image quality of the video of the target video application, and / or, using at least one of the operating status information of the terminal device and the operating scenario information of the target video application, control the frame rate of the video of the target video application.

2. The method according to claim 1, characterized in that The determining whether to perform image quality enhancement on the video of the target video application by using the operation status information of the terminal device and the operation scene information of the target video application includes: Determine whether the target video application needs to perform image quality recognition, and obtain a determination result; When it is determined according to the judgment result that the video in the target video application needs to be image-quality identified, the state identification result of the terminal device is determined by using the temperature information and the power information of the terminal device; Merge and identify the state recognition result of the terminal device and the video window state information of the video application used by the user to obtain a recognition result; According to the recognition result, it is determined whether to perform image quality enhancement on the video of the target video application.

3. The method according to claim 2, characterized in that The determining, based on the recognition result, whether to perform image quality enhancement control on the video of the target video application includes: According to the recognition result, a picture quality enhancement control instruction is generated, and the picture quality enhancement control instruction is sent to the video post-processing service unit Vpp, so that the Vpp executes the picture quality enhancement control instruction and determines whether to enhance the picture quality of the video of the target video application.

4. The method according to any one of claims 2 to 3, characterized in that: The identification result includes global off, global on, or on / off of a single application record in the target video application.

5. The method according to claim 1, characterized in that The using at least one of the running state information of the terminal device and the running scene information of the target video application to control the frame rate of the video of the target video application includes: According to the window status information of the target video application, determining whether the window of the target video application is a small window, hidden, or destroyed, and obtaining a determination result; According to the determination result, the scene recognition result record of the target video application is updated and the frame rate of the video of the target video application is controlled.

6. The method according to claim 5, characterized in that The updating of the scene recognition result record of the target video application and performing frame rate control on the video of the target video application according to the judgment result includes: According to the judgment result, the scene recognition result record of the target video application is updated and a frame rate control instruction is generated, and the frame rate control instruction is sent to the multimedia codec so that the multimedia codec executes the frame rate control instruction to achieve frame rate control of the video of the target video application.

7. The method according to claim 1, characterized in that The target video application on the terminal device is a video application in which the current user is making a video call or watching a video.

8. The method according to claim 1, characterized in that The window status information of the target video application includes whether the window of the target video application is a small window or hidden.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor is used to call and execute the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by an electronic device, the method according to any one of claims 1 to 8 is implemented.

Images