Video enhancement method and apparatus, server, and terminal device

Abstract

Embodiments of the present application provide a video enhancement method and apparatus, a server, and a terminal device. In the video enhancement method, a terminal device acquires a video playback request carrying information of a video to be played back, and downloads from a video distribution source station a video file to be played back; then, the terminal device interprets a first-level index file of said video file, obtains a second-level index address and a PQ parameter file address, and, on the basis of the PQ parameter file address, downloads a PQ parameter file from the video distribution source station; and, on the basis of the second-level index address, the terminal downloads from the video distribution source station code stream data of the video to be played back; finally, on the basis of the PQ parameter file, the terminal device obtains a PQ parameter corresponding to the terminal device, and performs a video enhancement operation on the basis of the PQ parameter and the code stream data, so as to obtain enhanced code stream data. Thus a result of adaptation between a video enhancement characteristic of the terminal device and different types of videos can be improved, and a video playback result of the terminal device is improved.

Description

Video enhancement method, device, server and terminal equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 28, 2023, with application number 202310938114.X and application name “Video Enhancement Method, Device, Server and Terminal Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of smart terminal technology, and in particular to a video enhancement method, apparatus, server, and terminal device. Background Art

[0003] To support the user's playback experience under different network conditions, reduce key indicators such as playback freezes and start-up delays, and reduce the adaptation requirements of the video file encoding format on the client side, online video platforms will encode and compress the video files output by the video shooting on the platform server, and then distribute them to the client for playback through the media distribution network using the hypertext transfer protocol (HTTP) communication form. During the encoding and compression process on the platform server, two encapsulation formats, HTTP live streaming (HLS) and dynamic adaptive streaming over HTTP (DASH), are usually used to encapsulate video files of different bitrates into a set of multi-bitrate video file structures. This supports the client to select the binary stream file with the appropriate bitrate for download and playback according to the current network conditions during the playback process, thereby achieving dynamic bitrate.

[0004] At the same time, since the platform server encodes and compresses the original recorded video, there will be a certain loss in the subjective quality of the video. Therefore, with the development of artificial intelligence (AI) technology, the platform server can optimize the video file through algorithms during the encoding and compression process to better adapt to and enhance the subjective experience of client playback. However, during the platform server enhancement process, due to the different display parameters (such as brightness and / or color) of different terminal devices, the playback rendering effect of the enhanced video stream file on different client screens may vary greatly. The actual effect on some terminal devices may even affect the user's subjective playback experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a video enhancement method, apparatus, server, and terminal device. The embodiments of the present application also provide a computer-readable storage medium to achieve the application effect of improving the video enhancement characteristics of the terminal device and improving the video playback effect of the terminal device.

[0007] In a first aspect, an embodiment of the present application provides a video enhancement method, including: obtaining a video playback request, the video playback request carrying information of a video to be played; downloading the video file to be played from a video distribution source station based on the information of the video to be played; interpreting the first-level index file of the video file to obtain a second-level index address and a perceptual quantization (PQ) parameter file address of the video to be played; downloading the PQ parameter file from the video distribution source station based on the PQ parameter file address; and downloading the code stream data of the video to be played from the video distribution source station based on the second-level index address; obtaining PQ parameters corresponding to a terminal device based on the PQ parameter file; and performing a video enhancement operation based on the PQ parameters and the code stream data to obtain enhanced code stream data.

[0008] In the above-mentioned video enhancement method, the terminal device obtains a video playback request carrying information about the video to be played, and downloads the video file to be played from the video distribution source station based on the information about the video to be played. Then, the terminal device interprets the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played, and downloads the PQ parameter file from the video distribution source station based on the PQ parameter file address; and downloads the code stream data of the video to be played from the video distribution source station based on the secondary index address. Finally, the terminal device obtains the PQ parameters corresponding to the terminal device based on the PQ parameter file, and performs a video enhancement operation based on the PQ parameters and the code stream data to obtain enhanced code stream data, thereby improving the adaptability of the terminal device's video enhancement features to different types of videos and enhancing the video playback effect of the terminal device.

[0009] In one possible implementation, after performing the video enhancement operation according to the PQ parameter and the bitstream data to obtain enhanced bitstream data, the method further includes rendering and playing the enhanced bitstream data.

[0010] In one possible implementation method, interpreting the first-level index file of the video file to obtain the second-level index address and perceptual quantization PQ parameter file address of the video to be played includes: interpreting the first-level index file of the video file according to the encapsulation protocol of the video file to obtain the second-level index address of the video to be played; if the label of the PQ parameter file exists in the first-level index file, obtaining the PQ parameter file address corresponding to the label.

[0011] In one possible implementation, obtaining the PQ parameters corresponding to the terminal device according to the PQ parameter file includes: interpreting the PQ parameter file according to the structure of the PQ parameter file; and obtaining corresponding PQ parameters from the PQ parameters included in the PQ parameter file according to information of the terminal device.

[0012] In the second aspect, an embodiment of the present application provides a video enhancement method, including: obtaining a filmed video source file; transcoding and stream slicing the video source file; encapsulating the sliced ​​video file to obtain a packaged video file, wherein the packaged video file includes a first-level index file; and constructing a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; adding a label and address of the PQ parameter file in the first-level index file; and sending the packaged video file and the PQ parameter file to a video distribution source station.

[0013] In the above-mentioned video enhancement method, the server obtains the filmed video source file, transcodes and stream slices the above-mentioned video source file, encapsulates the sliced ​​video file, obtains the encapsulated video file, and the encapsulated video file includes a first-level index file; and constructs a PQ parameter file corresponding to the sliced ​​video file. Then, the server adds the label and address of the above-mentioned PQ parameter file to the first-level index file, sends the encapsulated video file and the above-mentioned PQ parameter file to the video distribution source station, and then, the server also generates PQ parameters corresponding to the information of the terminal device, sends the PQ parameters to the video distribution source station, and the video distribution source station updates the above-mentioned PQ parameters to the above-mentioned PQ parameter file. In this way, in the process of generating PQ parameters, the parameters of the terminal device and the video transcoding parameters can be combined to effectively coordinate the parameters between the server and the terminal device, thereby improving the adaptation effect of the terminal device's video enhancement characteristics to different types of videos and improving the video playback effect of the terminal device.

[0014] In one possible implementation, after sending the encapsulated video file and the PQ parameter file to the video distribution source station, it also includes: obtaining the parameters and video transcoding parameters of the terminal device; training the parameters and video transcoding parameters of the terminal device through the PQ algorithm to obtain a PQ algorithm model; generating PQ parameters corresponding to the information of the terminal device in the PQ parameter file based on the PQ algorithm model and the encapsulated video file; and sending the PQ parameters to the video distribution source station so that the video distribution source station can update the PQ parameters to the PQ parameter file.

[0015] In a third aspect, embodiments of the present application provide a video enhancement device, which is included in a terminal device and has the function of implementing the terminal device behavior described in the first aspect and possible implementations of the first aspect. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a video playback module, a PQ parameter file interpretation module, and a video enhancement module.

[0016] In a fourth aspect, embodiments of the present application provide a video enhancement device, which is included in a server and has the functionality to implement the server behaviors described in the second aspect and possible implementations of the second aspect. The functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the aforementioned functionality. For example, a media encoding module, a stream slicing module, and a sending module.

[0017] In a fifth aspect, an embodiment of the present application provides a terminal device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the terminal device, enable the terminal device to execute the method provided in the first aspect.

[0018] In the sixth aspect, an embodiment of the present application provides a server, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the server, enable the server to execute the method provided in the second aspect.

[0019] It should be understood that the third and fifth aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0020] It should be understood that the fourth and sixth aspects of the embodiments of the present application are consistent with the technical solutions of the second aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0021] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0022] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the second aspect.

[0023] In a ninth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0024] In a tenth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the second aspect.

[0025] In one possible design, the programs in the ninth and tenth aspects may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of a file structure of a video multi-bitrate encapsulation format;

[0027] FIG2 is a schematic diagram showing the principle of algorithm enhancement of a video stream by a terminal device;

[0028] FIG3 is a schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0029] FIG4 is a schematic diagram of the structure of a server provided in one embodiment of the present application;

[0030] FIG5 is a schematic diagram of an application scenario of a video enhancement method provided by an embodiment of the present application;

[0031] FIG6 is a system architecture of a video enhancement method provided by one embodiment of the present application;

[0032] FIG7 is a schematic diagram of a video file storage structure provided by an embodiment of the present application;

[0033] FIG8 is a flowchart of a video enhancement method provided by one embodiment of the present application;

[0034] FIG9 is a flowchart of a video enhancement method provided by another embodiment of the present application;

[0035] FIG10 is a schematic diagram of the structure of a primary index file provided by an embodiment of the present application;

[0036] FIG11 is a schematic diagram of the structure of a PQ parameter file provided in one embodiment of the present application;

[0037] FIG12 is a flowchart of a video enhancement method provided by yet another embodiment of the present application;

[0038] FIG13 is a flowchart of a video enhancement method provided by yet another embodiment of the present application;

[0039] FIG14 is a flowchart of a video enhancement method provided by yet another embodiment of the present application;

[0040] FIG15 is a schematic structural diagram of a terminal device provided in another embodiment of the present application;

[0041] FIG16 is a schematic diagram of the structure of a server provided in another embodiment of the present application;

[0042] FIG17 is a schematic diagram of the structure of a server provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0043] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0044] In the process of encoding and compressing video files, the platform server usually encapsulates video files of different bit rates into a set of multi-bit rate video file structures. The file structure of the video multi-bit rate encapsulation format can be shown in Figure 1, which is a schematic diagram of the file structure of the video multi-bit rate encapsulation format.

[0045] During the process of enhancing the video files by the platform server, due to the different display parameters (such as brightness and / or color, etc.) of different terminal devices, there are large differences in the playback rendering effects of the enhanced video stream files on different client screens. Even the actual effects on some terminal devices may affect the user's subjective playback experience.

[0046] To solve this problem, existing related technologies have introduced the concept of metadata (including static metadata and dynamic metadata) in some new high dynamic range video coding technologies (high dynamic range, HDR). Metadata is used to describe some technical parameter information used by the server in the video encoding process. During the decoding and rendering process, the terminal device can understand the relevant parameters and perform targeted decoding and rendering, thereby better maintaining the end-side playback effect.

[0047] However, HDR metadata technology often has two practical application problems:

[0048] 1. The relevant metadata concept is only used in the newly designed HDR technology and is not applicable to some videos that are not suitable for HDR encoding;

[0049] 2. The metadata in HDR technology is encapsulated in the video stream file and can only express some parameters in the actual video encoding process. It cannot solve the differences in playback effects caused by screen differences on different terminal devices. If the actual playback effect of some terminal devices is poor, the problem cannot be effectively solved by adjusting the metadata.

[0050] To address the challenges of HDR metadata technology, existing technologies propose that terminal devices perform algorithmic enhancement (e.g., perceptual quantization (PQ)) on video streams during the video playback and rendering process. This approach mitigates the impact of the terminal device's screen parameters on the subjective playback experience. The specific technical principles are shown in Figure 2, which illustrates the principle behind the algorithmic enhancement of video streams by a terminal device.

[0051] As shown in Figure 2, the video production system performs one or a combination of the following operations on the video source: video super-resolution, standard dynamic range (SDR) to HDR, sharpening, contrast enhancement, noise reduction, super resolution (SR), and brightness / saturation enhancement to obtain a multi-bitrate video clip (i.e., the playback source). After the television (TV) end (i.e., the terminal device) obtains the playback source, it uses a set of PQ algorithms to perform one or a combination of the following operations on the playback source: noise reduction, SR, sharpening, contrast enhancement, and color / saturation enhancement. This reduces the impact of the TV screen parameters on the subjective playback experience.

[0052] However, the following problems also exist during the implementation of this plan:

[0053] 1. Due to the limited computing power of terminal devices, only PQ parameters trained using screen parameters can be preloaded. As a result, the on-device enhancement algorithm cannot be adjusted according to the transcoding parameters of specific videos, resulting in poor algorithm application results.

[0054] 2. If the algorithm parameters used by the platform server vary for different videos, the playback effects on the client side may vary, and even negative effects may occur due to the superposition of the platform server and client side algorithms. This results in poor adaptability of client side algorithm applications.

[0055] 3. After video encoding is produced, if it is found that the playback effect is poor on the screens of some terminal devices, the platform server can only readjust the video transcoding algorithm, which is not only costly but also easily leads to the problem of reduced playback effect on the screens of other terminal devices.

[0056] Based on the above problems, the present invention provides a video enhancement method that can solve the following problems:

[0057] 1. The video enhancement PQ parameters executed by the terminal device cannot be adaptively adjusted according to the transcoding parameters of the specific video being played.

[0058] 2. When the transcoded video does not play well on the screens of some terminal devices, the video platform server cannot resolve the known playback experience issues by adjusting the PQ parameters of the terminal devices.

[0059] The video enhancement method provided in the embodiments of the present application can be applied to terminal devices, wherein the above-mentioned terminal devices can be smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks or personal digital assistants (PDAs) and other devices; the embodiments of the present application do not impose any restrictions on the specific type of terminal devices.

[0060] For example, Figure 3 is a structural diagram of a terminal device provided by an embodiment of the present application. As shown in Figure 3, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0063] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0064] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0066] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device 100 via the power management module 141.

[0067] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0068] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0069] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0070] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0071] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0072] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0073] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0074] The terminal device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0075] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0076] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0077] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0079] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0080] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows terminal device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0081] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in the terminal device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0082] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0083] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0084] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0086] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0087] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0088] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0089] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0090] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0091] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0092] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0093] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the terminal device 100 by inserting or removing it from the SIM card interface 195. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0094] In addition, the implementation of the video enhancement method provided in the embodiments of the present application involves not only the terminal device side, but also the server side. The server can be a video platform server, which can also be called a video online platform server or platform server. The server can be set up in the cloud, and the embodiments of the present application do not impose any restrictions on the specific type of server.

[0095] For example, FIG4 is a schematic diagram of the structure of a server provided in one embodiment of the present application. As shown in FIG4 , server 400 includes a processor 410 and a communication interface 420. Optionally, server 400 may further include a memory 430. The processor 410, communication interface 420, and memory 430 may communicate with each other via an internal connection path to transmit control and / or data signals. The memory 430 is used to store computer programs, and the processor 410 is used to call and execute the computer programs from the memory 430.

[0096] The processor 410 and the memory 430 may be combined into a processing device, or more commonly, they are independent components. The processor 410 is configured to execute program codes stored in the memory 430. In a specific implementation, the memory 430 may also be integrated into the processor 410 or independent of the processor 410.

[0097] In addition, in order to make the functions of the server 400 more complete, the server 400 may further include one or more of an input unit 460 and a display unit 470 .

[0098] Optionally, the server 400 may further include a power supply 450 for providing power to various devices or circuits in the server 400 .

[0099] It should be understood that the processor 410 in the server 400 shown in Figure 4 can be a system on a chip SOC, which can include a central processing unit (CPU) and can further include other types of processors, such as a graphics processing unit (GPU).

[0100] For ease of understanding, the following embodiments of this application will take the terminal device 100 having the structure shown in Figure 2 and the server 400 having the structure shown in Figure 4 as examples, and combine the accompanying drawings and application scenarios to specifically explain the video enhancement method provided in the embodiments of this application.

[0101] The application scenario involved in the embodiment of the present application is that after the server 400 encodes the video source file uploaded to the server 400, the encoded video is stored in the video distribution source station / content delivery network (CDN), and the video distribution source station distributes the encoded video to the terminal device 100 through the CDN service. The terminal device 100 provides video enhancement capabilities while playing the video, thereby improving the subjective playback experience of the video on the terminal device 100. Figure 5 is a schematic diagram of the application scenario of the video enhancement method provided by an embodiment of the present application. As shown in Figure 5, after the server 400 performs video enhancement, multi-bitrate encoding and / or video stream slicing operations on the video source file through media encoding and stream segmentation capabilities, it is encapsulated into a storage format of multiple video binary files described by an index file (commonly encapsulated by HLS protocol and DASH protocol).

[0102] After the terminal device 100 obtains the video index file and video binary file from the video distribution source station through HTTP communication, it performs video decoding and rendering. During the video decoding and rendering process, the video enhancement module of the terminal device 100 improves the video playback experience on the terminal device 100 by executing the video enhancement PQ algorithm.

[0103] In summary, the video enhancement method provided in the embodiments of the present application may include:

[0104] 1. First, by expanding the structure definition of video files, a set of PQ parameter files is added without affecting existing video encoding and decoding methods. These files are used to describe the PQ parameter definitions for performing end-side enhancement on different models of terminal devices 100.

[0105] 2. The video platform server 400 outputs the corresponding PQ parameter file and then updates it to the video distribution source station. This allows adjustments to the client-side algorithm without having to update the basic video index file and video source (.ts / .mp4) files, thus greatly ensuring the reliability of the basic video playback process.

[0106] 3. The terminal device 100 needs to increase the ability to interpret the PQ parameter file, which may include:

[0107] 1) Interpret all files in the index file and obtain the storage address of the PQ parameter file on the video distribution source station;

[0108] 2) Download the PQ parameter file from the video distribution source station;

[0109] 3) Interpret the PQ parameter file and obtain the applicable PQ parameters from the PQ parameter file according to the device model of the currently running video decoding;

[0110] 4) The interpreted PQ parameters are input into the decoder together with the video source file for video decoding, and the PQ parameters are used to drive the end-side enhancement algorithm in the decoding process.

[0111] Figure 6 is the system architecture of the video enhancement method provided by an embodiment of the present application. As shown in Figure 6, the system architecture of the embodiment of the present application provides a video file storage structure. After the server 400 encodes the video source file uploaded to the server 400, the encoded video is stored in the video distribution source station / CDN. At the same time, the server 400 provides a PQ algorithm training and PQ parameter file generation module to support the production of the PQ parameter file in the video file storage structure provided by the embodiment of the present application. The video distribution source station distributes the encoded video file and the PQ parameter file to the terminal device 100 through the CDN service. While playing the video, the terminal device 100 provides a PQ parameter file interpretation module to support obtaining the PQ parameter file from the video file storage structure provided by the embodiment of the present application, which is used for the video enhancement module in the terminal device 100 to perform algorithm enhancement input to provide video enhancement capabilities, thereby improving the video playback experience on the terminal device 100.

[0112] The system architecture shown in FIG6 is described in detail below.

[0113] 1. PQ algorithm training module: The PQ algorithm training module can support training corresponding PQ parameter models based on corresponding terminal parameters for video categories or specific videos.

[0114] 2. PQ parameter file generation module: Generates video-related PQ parameter files based on the PQ parameter model output by the PQ algorithm training module and synchronizes them to the video distribution source station.

[0115] 3. Video file storage: This refers to the structure and implementation of video file storage for video distribution origins and their CDNs. The video file storage structure can include index files, video binary stream files (usually with a .ts or .mp4 file extension), and PQ parameter files. The video file storage structure is a file storage structure composed of these three files through structural relationships.

[0116] 4. Video Playback Module: The video playback module is a module in the terminal device 100 that performs video file downloading, interpretation, decoding, and rendering. It downloads video files from the video distribution source station, interprets the video files according to the video file storage structure, and then plays the video files on the terminal side through decoding and rendering. In this embodiment of the application, the video playback module is responsible for triggering the PQ parameter file interpretation module to interpret the PQ parameters applicable to the terminal for the video to be played, and then inputs them into the video enhancement module to trigger the video enhancement operation during the video decoding and rendering process.

[0117] 5. PQ parameter file interpretation module: The PQ parameter file interpretation module provides the function of interpreting the PQ parameter file, is responsible for downloading and interpreting the PQ parameter file, and matches the PQ parameters applicable to the video to be played from the PQ parameter file according to the information of the terminal device 100.

[0118] 6. Video enhancement module: performs end-side video enhancement operations based on the PQ enhancement parameters and the binary stream information of the video to be played. In this embodiment of the present application, the video enhancement module will receive the PQ parameters obtained from the PQ parameter file interpretation module as input parameters of the enhancement algorithm.

[0119] The schematic diagram of the video file storage structure, which is the key technology of an embodiment of the present application, can be shown in Figure 7. Based on the multi-level video file storage structure of HLS or DASH, this embodiment of the present application provides a new end-side PQ parameter file definition and modifies the first-level index file definition. Figure 7 is a schematic diagram of the video file storage structure provided by an embodiment of the present application.

[0120] The following describes the terminal-side PQ parameter file and the primary index file in the video file storage structure shown in FIG. 7 .

[0121] 1. End-side PQ parameter file: This is the end-side PQ algorithm file shown in Figure 7. This file describes the PQ parameters used for end-device enhancement of video files on different types of end devices 100. Therefore, the basic structure of the PQ parameter file is a data storage structure that uses the series or model of the end device 100 as a keyword and PQ parameters as the primary storage information. This file supports expanding the PQ parameter list corresponding to the end device 100 to dynamically adjust the input of the end-device enhancement PQ algorithm executed on different end-devices for video files. PQ parameters may include parameters and curve values ​​for video enhancement algorithms, such as clarity, noise reduction, contrast, color, HDR, and / or light control.

[0122] 2. Primary Index File: The primary index file uses tags to identify different video encoding formats, resolutions, and bitrates, as well as the relative addresses of the corresponding secondary index files. In this embodiment, an optional tag that identifies the client-side PQ parameter file is extended to identify the relative addresses of the client-side PQ parameter file.

[0123] The implementation process of the embodiment of the present application involves the transcoding, algorithm training and video file distribution processes of the server 400, as well as the video downloading, interpretation, terminal enhancement and decoding rendering processes of the terminal device 100. The processes involved in the embodiment of the present application are explained below.

[0124] FIG8 is a flow chart of a video enhancement method provided by one embodiment of the present application. As shown in FIG8 , the video enhancement method may include:

[0125] In step 801, the server 400 adjusts the video file storage data structure, adds a data management model for the PQ parameter file, publishes the transcoded video file to the video distribution source station, and adjusts the video file storage of the video distribution source station (CDN) during video transcoding and stream file encapsulation.

[0126] In step 802 , a PQ algorithm training module is added to the server 400 to support algorithm engineers in performing terminal-side PQ algorithm training for specific videos and terminal device 100 models.

[0127] In step 803 , after the PQ algorithm training is completed, the server 400 generates a PQ parameter file through the PQ parameter file generation module, and then updates the PQ parameter file to the video distribution source station (CDN).

[0128] In step 804 , the video playback module in the terminal device 100 obtains the video file to be played from the video distribution source station.

[0129] In step 805 , the video playback module in the terminal device 100 interprets the primary index file of the video file to obtain a secondary index address and a PQ parameter file address.

[0130] Step 806 : The video playback module in the terminal device 100 drives the PQ parameter file interpretation module in the terminal device 100 to download and interpret the PQ parameter file.

[0131] Step 807: The PQ parameter file interpretation module downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address interpreted by the video playback module.

[0132] In step 808 , the PQ parameter file interpretation module obtains the PQ parameters corresponding to the terminal device 100 according to the model of the terminal device 100 .

[0133] Step 809 : The PQ parameter file interpretation module sends the PQ parameters corresponding to the terminal device 100 to the video playback module.

[0134] In step 810 , the video playback module downloads the stream data of the video to be played from the video distribution source station according to the secondary index address.

[0135] In step 811, the video playback module sends the PQ parameters and the bitstream data to the video enhancement module in the terminal device 100. The video enhancement module performs a video enhancement operation based on the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0136] The video enhancement method provided in the embodiment of the present application is described below from three aspects: the server 400, the video distribution source station and the terminal device 100.

[0137] (1) Implementation process of server 400.

[0138] In an embodiment of the present application, the server 400 (video platform server) provides encoding and stream slicing operations for the video source file, produces a video file and uploads it to the video distribution source station, performs PQ algorithm training based on the parameters of the terminal device 100 and the video transcoding parameters, generates a PQ parameter file and uploads it to the video distribution source station. FIG9 is a flowchart of a video enhancement method provided in another embodiment of the present application. As shown in FIG9, the process executed by the server 400 may include:

[0139] In step 901, the server 400 obtains the recorded video source file.

[0140] In step 902, the server 400 encodes the video source file into video files of different bit rates through the video encoding module and the video stream slicing module. After transcoding and slicing, the video files are encapsulated using protocols such as HLS or DASH to form a complete video file structure. The server 400 encapsulates the video file using protocols such as HLS or DASH. In the process of generating the first-level index file, the server 400 constructs the basic framework of the PQ parameter file of the video file based on the PQ parameter file basic framework (as shown in Figure 11), and adds the PQ parameter file tag and the storage address of the corresponding PQ parameter file in the first-level index file (as shown in Figure 10).

[0141] In step 903, the server 400 sends the PQ parameter file and the video file to the video distribution source station (CDN is an acceleration method used by the video distribution source station in the video distribution process, which is a common method used by video online platforms and will not be described in detail in the implementation of this application).

[0142] At this point, the video files in the video distribution source station are available for download and playback by the terminal device 100. However, since the PQ parameter file only has a basic framework at this time, the specific PQ parameters cannot be obtained from the PQ parameter file and input into the video enhancement module of the terminal device 100 during the playback process of the terminal device 100.

[0143] In step 904, the PQ algorithm training module in the server 400 uses the parameters of the terminal device 100 as input and performs PQ algorithm training in combination with the video transcoding parameters (which can be a single video transcoding parameter or a class of video transcoding parameters) to form a PQ algorithm model that matches the video transcoding parameters and the parameters of the terminal device 100.

[0144] In step 905, the PQ parameter file generation module in server 400 generates PQ parameters for the video file in a PQ parameter file based on the PQ algorithm model and the video file generated in step 902. The PQ parameters are then updated into the PQ parameter file generated in step 902. After step 905, the PQ parameter file includes all information about the basic framework and the PQ parameters used by the terminal device 100.

[0145] When there are playback experience issues with a video in a certain terminal device 100 or a new terminal device 100 is added, the server 400 can execute steps 904 to 905 again to retrain the PQ algorithm model and update the PQ parameters used by the terminal device 100 in the above-mentioned PQ parameter file, thereby achieving better adaptation of the playback experience of the video file on different terminal devices 100 without adjusting the video file.

[0146] (2) Implementation process of video distribution source station.

[0147] As the distribution module of the online video operation platform, the video distribution source station provides video storage and video distribution and download service capabilities based on HTTP communication. In an embodiment of the present application, the video distribution source station provides a video file storage function that can be used to support the storage and update of videos and corresponding PQ parameter files. Among them, the video file storage structure in the video distribution source station can be shown in Figure 7. The end-side PQ parameter file is newly added to the video file storage structure, and an optional tag that identifies the end-side PQ parameter file is added to the first-level index file.

[0148] Figure 10 is a schematic diagram of the structure of a first-level index file provided by an embodiment of the present application. The structure shown in Figure 10 is illustrated by taking the m3u8 first-level index file of the HLS encapsulation protocol as an example; Figure 11 is a schematic diagram of the structure of a PQ parameter file provided by an embodiment of the present application. The structure shown in Figure 11 refers to the tag definition format of the m3u8 index file of the HLS encapsulation protocol to illustrate the structure of the PQ parameter file. Of course, the embodiment of the present application is not limited to this. In the specific implementation process, different file formats based on tag descriptions can be used to implement it, for example: the tag description method based on the extensible markup language (XML) format adopted by the DASH encapsulation protocol index file (.MPD file).

[0149] Referring to Figure 10, in the m3u8 first-level index file, without modifying the basic file structure and tag definition of the index file defined in the HLS protocol, the relative storage address of the PQ parameter file is identified by extending the custom tag "#EXT-X-HW-TERMINALPQ-INF". The "TERMINALPQ-LIST-INFO.PQI" in Figure 10 is the address information of the PQ parameter file relative to the m3u8 file storage directory. Among them, the file name of the PQ parameter file only needs to conform to the file name definition specification of the mainstream operating system. Adjustment of the file name and file suffix definition of the PQ parameter file does not affect the implementation effect of the embodiment of the present application.

[0150] As shown in Figure 11 , the PQ parameter file uses a tag description file structure to store the PQ parameters corresponding to the terminal device 100. Typically, a tag description file includes a file header, a file version number, a file end tag, and a data tag. The PQ parameter file name and storage path must be consistent with the file name and storage address described in the relative address of the PQ parameter file in the primary index file.

[0151] Figure 11 shows an example of the structure of a PQ parameter file defined with reference to the m3u8 tag definition format. The left image in Figure 11 shows the basic file framework, including the file header, file version number, and file footer tag information. The right image in Figure 11 adds a list of terminal devices 100 and their corresponding PQ parameters to the basic file framework. "#EXT-HW-TERMINALNUM" is a tag definition, "ABC" and "DEF" are examples of information corresponding to terminal devices 100, and "THISISVALUEOFPQPARAMETFORTERMINALABC" is an example of the textual expression of the PQ parameters corresponding to terminal devices 100.

[0152] The label definition, information of the terminal device 100 and PQ parameters shown in Figure 11 can be expressed in text form. The specific naming method and expression method do not affect the implementation effect of the embodiment of the present application.

[0153] The text definition of the PQ parameters may include parameters and curve values ​​for dimensions such as clarity, noise reduction, contrast, color, HDR, and / or dimming. The text definition of the PQ parameters may be expanded and adjusted based on the input parameter requirements of the actual enhancement algorithm of the terminal device 100. The organization order and organization format of each parameter text may be consistent with the interpretation logic of the specific implementation of the PQ parameter file interpretation module. Adjusting the interpretation logic of the PQ parameter file interpretation module and the organization format of the parameter text does not affect the implementation effect of the embodiments of the present application.

[0154] (3) Implementation process of terminal device 100 (video playback terminal).

[0155] After the user selects a video to be played, the terminal device 100 can download the video file of the video to be played from the video distribution source station via HTTP communication, decode, render, and play it. During this process, video enhancement operations can be triggered based on the capabilities of the terminal device 100 to improve the video playback experience. In this embodiment, the terminal device 100 uses the PQ parameter file interpretation module to complete the download and interpretation of the PQ parameter file.

[0156] FIG12 is a flowchart of a video enhancement method provided by yet another embodiment of the present application. As shown in FIG12 , the process executed by the terminal device 100 may include:

[0157] Step 1201: After receiving the user's video playback request, the video playback module downloads the video file to be played from the video distribution source station according to the information of the video to be played selected by the user, and interprets the video file according to the encapsulation protocol of the video file; interprets the first-level index file of the video file to obtain the second-level index address of the video to be played, and adds the interpretation logic of the PQ parameter file tag to the logic of interpreting the first-level index file of the video file. If the first-level index file contains the PQ parameter file tag, the PQ parameter file address is obtained.

[0158] In step 1202, the video playback module inputs the PQ parameter file address to the PQ parameter file interpretation module, and downloads the stream data of the video to be played from the video distribution source station according to the secondary index address. The video playback module sends the stream data of the video to be played to the video enhancement module.

[0159] Step 1203: After receiving the PQ parameter file address input by the video playback module, the PQ parameter file interpretation module downloads the PQ parameter file of the video to be played from the video distribution source station.

[0160] In step 1204 , the PQ parameter file interpretation module interprets the PQ parameter file according to its structure (as shown in FIG. 11 ) to obtain PQ parameters corresponding to the terminal device 100 .

[0161] Step 1205 : The PQ parameter file interpretation module sends the PQ parameters corresponding to the terminal device 100 to the video playback module.

[0162] Step 1206 : After receiving the PQ parameters corresponding to the terminal device 100 , the video playback module sends the PQ parameters corresponding to the terminal device 100 to the video enhancement module.

[0163] In step 1207, the video enhancement module performs a video enhancement operation according to the PQ parameter and the bitstream data of the video to be played to obtain enhanced bitstream data. The video enhancement module sends the enhanced bitstream data to the video playback module.

[0164] Step 1208: The video playback module decodes, renders, and plays the enhanced bitstream data.

[0165] During the specific implementation process, in order to improve the compatibility of video enhancement of the terminal device 100, a set of default PQ parameters can be pre-made in the terminal device 100. If the PQ parameter file interpretation module cannot obtain the PQ parameters used by the terminal device 100 from the PQ parameter file of the video, the video enhancement module can use the default PQ parameters to perform the video enhancement operation. This operation does not affect the implementation effect of the embodiment of the present application.

[0166] As described above, the embodiment of the present application provides a video enhancement method that can solve the problem faced in online video playback services that the terminal video enhancement parameters cannot be adapted to the transcoding parameters of the specific playback video, and the problem that the transcoded video cannot be solved by quickly adjusting the playback parameters when it is poorly adapted to the new terminal device. The video enhancement method provided by the embodiment of the present application can effectively apply the application effect of the terminal video enhancement feature and improve the video playback experience of the terminal device 100.

[0167] FIG13 is a flow chart of a video enhancement method provided in yet another embodiment of the present application. As shown in FIG13 , the video enhancement method may include:

[0168] In step 1301, the terminal device 100 obtains a video play request, which carries information about the video to be played.

[0169] In step 1302 , the terminal device 100 downloads the video file to be played from the video distribution source station according to the information of the video to be played.

[0170] In step 1303, the terminal device 100 interprets the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played.

[0171] Specifically, the terminal device 100 interprets the first-level index file of the above-mentioned video file and obtains the second-level index address and PQ parameter file address of the video to be played. This can be as follows: the terminal device 100 interprets the first-level index file of the above-mentioned video file according to the encapsulation protocol of the above-mentioned video file and obtains the second-level index address of the video to be played; if the label of the above-mentioned PQ parameter file exists in the above-mentioned first-level index file, then obtains the PQ parameter file address corresponding to the above-mentioned label.

[0172] In step 1304 , the terminal device 100 downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloads the stream data of the video to be played from the video distribution source station according to the secondary index address.

[0173] Step 1305 : The terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 according to the above PQ parameter file.

[0174] Specifically, the terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 according to the above-mentioned PQ parameter file: the terminal device 100 interprets the PQ parameter file according to the structure of the above-mentioned PQ parameter file, and obtains the corresponding PQ parameters from the PQ parameters included in the above-mentioned PQ parameter file according to the information of the terminal device 100.

[0175] In this embodiment, the basic structure of the PQ parameter file is a set of data storage structures with the information of the terminal device 100 as the keyword and the PQ parameters as the main storage information. The above-mentioned information of the terminal device 100 may include the series to which the terminal device 100 belongs and / or the model of the terminal device 100.

[0176] In step 1306, the terminal device 100 performs a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0177] Furthermore, after step 1306, the following steps may also be included:

[0178] Step 1307: The terminal device 100 renders and plays the enhanced bitstream data.

[0179] In the above-mentioned video enhancement method, the terminal device 100 obtains a video playback request carrying information about the video to be played, and downloads the video file to be played from the video distribution source station based on the information about the video to be played. Then, the terminal device 100 interprets the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played, and downloads the PQ parameter file from the video distribution source station based on the PQ parameter file address; and downloads the bitstream data of the video to be played from the video distribution source station based on the secondary index address. Finally, the terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 based on the PQ parameter file, and performs a video enhancement operation based on the PQ parameters and the bitstream data to obtain enhanced bitstream data, thereby improving the adaptability of the video enhancement characteristics of the terminal device 100 to different types of videos and improving the video playback effect of the terminal device 100.

[0180] FIG14 is a flow chart of a video enhancement method provided in yet another embodiment of the present application. As shown in FIG14 , the video enhancement method may include:

[0181] In step 1401, the server 400 obtains a recorded video source file.

[0182] In step 1402 , the server 400 performs transcoding and stream slicing operations on the video source file.

[0183] In step 1403 , the server 400 encapsulates the sliced ​​video file to obtain an encapsulated video file, which includes a first-level index file; and constructs a PQ parameter file corresponding to the sliced ​​video file.

[0184] The PQ parameter file includes information about the terminal device 100 , and the information about the terminal device 100 may include the series to which the terminal device 100 belongs and / or the model of the terminal device 100 .

[0185] In step 1404 , the server 400 adds the label and address of the PQ parameter file to the primary index file.

[0186] In step 1405 , the server 400 sends the encapsulated video file and the PQ parameter file to the video distribution source station.

[0187] Furthermore, after step 1405, the following steps may also be included:

[0188] In step 1406 , the server 400 obtains parameters of the terminal device 100 and video transcoding parameters.

[0189] In step 1407 , the server 400 trains the parameters of the terminal device 100 and the video transcoding parameters using the PQ algorithm to obtain a PQ algorithm model.

[0190] In step 1408 , the server 400 generates PQ parameters corresponding to the information of the terminal device 100 in the PQ parameter file according to the PQ algorithm model and the encapsulated video file.

[0191] In step 1409 , the server 400 sends the PQ parameters to the video distribution source station, so that the video distribution source station updates the PQ parameters to the PQ parameter file.

[0192] In the above-mentioned video enhancement method, the server 400 obtains a recorded video source file, transcodes and stream slices the video source file, encapsulates the sliced ​​video file to obtain an encapsulated video file, which includes a first-level index file; and constructs a PQ parameter file corresponding to the sliced ​​video file. The server 400 then adds the tag and address of the PQ parameter file to the first-level index file and sends the encapsulated video file and the PQ parameter file to the video distribution source station. Furthermore, the server 400 also generates PQ parameters corresponding to the information of the terminal device 100 and sends the PQ parameters to the video distribution source station, which updates the PQ parameters to the PQ parameter file. This allows the server 400 and the terminal device 100 to effectively coordinate parameters during the generation of the PQ parameters by combining the parameters of the terminal device 100 and the video transcoding parameters, thereby improving the adaptability of the terminal device 100's video enhancement features to different types of videos and improving the video playback effect of the terminal device 100.

[0193] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in a different order than those presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0194] It is understandable that, in order to implement the above functions, the terminal device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0195] This embodiment can divide the terminal device into functional modules based on the above method embodiment. For example, each functional module can be divided according to its function, or two or more functions can be integrated into a single module. The above integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division. In actual implementation, other division methods may be used.

[0196] FIG15 is a schematic diagram of the structure of a terminal device provided in another embodiment of the present application. In the case where each functional module is divided according to each function, FIG15 shows a possible schematic diagram of the composition of a terminal device 1500 involved in the above embodiment. As shown in FIG15 , the terminal device 1500 may include: a video playback module 1501, a PQ parameter file interpretation module 1502, and a video enhancement module 1503.

[0197] The video playback module 1501 is configured to obtain a video playback request, which carries information about the video to be played; download the video file to be played from a video distribution source station based on the information about the video to be played; interpret the primary index file of the video file to obtain a secondary index address and PQ parameter file address of the video to be played; and download the stream data of the video to be played from the video distribution source station based on the secondary index address.

[0198] The PQ parameter file interpretation module 1502 is configured to download the PQ parameter file from the video distribution source station according to the PQ parameter file address; and obtain the PQ parameters corresponding to the terminal device according to the PQ parameter file;

[0199] The video enhancement module 1503 is configured to perform a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0200] Furthermore, the video playing module 1501 is further configured to render and play the enhanced code stream data after the video enhancing module 1503 performs a video enhancing operation to obtain enhanced code stream data.

[0201] In this embodiment, the PQ parameter file interpretation module 1502 is specifically used to interpret the first-level index file of the above-mentioned video file according to the encapsulation protocol of the above-mentioned video file to obtain the second-level index address of the video to be played; if there is a PQ parameter file tag in the first-level index file, then obtain the PQ parameter file address corresponding to the above-mentioned tag.

[0202] The PQ parameter file interpretation module 1502 is specifically configured to interpret the PQ parameter file according to its structure, and obtain corresponding PQ parameters from the PQ parameters included in the PQ parameter file according to information of the terminal device.

[0203] It should be noted that all relevant contents of each step involved in the method embodiment shown in Figure 13 of this application can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0204] The terminal device 1500 provided in this embodiment is used to execute the video enhancement method provided in the embodiment shown in Figure 13 of this application, and thus can achieve the same effect as the above method.

[0205] It should be understood that the terminal device 1500 may correspond to the terminal device 100 shown in FIG3 . The functions of the video playback module 1501 may be implemented by the processor 110, antenna 1, and mobile communication module 150 in the terminal device 100 shown in FIG3 , and / or by the processor 110, antenna 2, and wireless communication module 160, as well as by the processor 110 and display screen 194 in the terminal device 100 shown in FIG3 ; the functions of the PQ parameter file interpretation module 1502 and the video enhancement module 1503 may be implemented by the processor 110 in the terminal device 100 shown in FIG1 .

[0206] In the case of adopting an integrated unit, the terminal device 1500 may include a processing module, a storage module, and a communication module.

[0207] The processing module can be used to control and manage the operations of the terminal device 1500. For example, it can be used to support the terminal device 1500 in executing the steps performed by the video playback module 1501, the PQ parameter file interpretation module 1502, and the video enhancement module 1503. The storage module can be used to support the terminal device 1500 in storing program code and data. The communication module can be used to support communication between the terminal device 1500 and other devices.

[0208] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0209] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 1500 involved in this embodiment may be a device having the structure shown in FIG. 3 .

[0210] Similarly, it is understandable that, in order to implement the above functions, the server includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0211] This embodiment can divide the server into functional modules based on the above-mentioned method embodiment. For example, each functional module can be divided into corresponding functional modules, or two or more functions can be integrated into a single module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division. In actual implementation, other division methods may be used.

[0212] FIG16 is a schematic diagram of the structure of a server provided in another embodiment of the present application. In the case where each functional module is divided according to each function, FIG16 shows a possible schematic diagram of the composition of a server 1600 involved in the above embodiment. As shown in FIG16 , the server 1600 may include: a media encoding module 1601, a stream slicing module 1602, and a sending module 1603.

[0213] The media encoding module 1601 is used to obtain the recorded video source file and perform transcoding operation on the video source file;

[0214] The stream slicing module 1602 is configured to perform a stream slicing operation on the transcoded video file, encapsulate the sliced ​​video file to obtain an encapsulated video file, wherein the encapsulated video file includes a first-level index file; and construct a PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information about the terminal device; and add a label and address of the PQ parameter file to the first-level index file.

[0215] The sending module 1603 is used to send the encapsulated video file and the above-mentioned PQ parameter file to the video distribution source station.

[0216] FIG17 is a schematic diagram of the structure of a server provided in yet another embodiment of the present application. Compared with the server shown in FIG16 , the difference is that the server 1600 shown in FIG17 may further include: a PQ algorithm training module 1604 and a PQ parameter file generation module 1605;

[0217] The PQ algorithm training module 1604 is configured to obtain the terminal device parameters and video transcoding parameters after the sending module 1603 sends the encapsulated video file and the PQ parameter file to the video distribution source station, and train the terminal device parameters and the video transcoding parameters using the PQ algorithm to obtain a PQ algorithm model.

[0218] A PQ parameter file generating module 1605 is configured to generate PQ parameters corresponding to the terminal device information in the PQ parameter file according to the PQ algorithm model and the encapsulated video file;

[0219] The sending module 1603 is configured to send the PQ parameters to the video distribution source station, so that the video distribution source station updates the PQ parameters into a PQ parameter file.

[0220] It should be noted that all relevant contents of each step involved in the method embodiment shown in Figure 14 of this application can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0221] The server 1600 provided in this embodiment is used to execute the video enhancement method provided in the embodiment shown in FIG. 14 of the present application, and thus can achieve the same effect as the above method.

[0222] It should be understood that server 1600 may correspond to server 400 shown in FIG4 . The functions of sending module 1603 may be implemented by processor 410 and communication interface 420 in server 400 shown in FIG4 ; the functions of media encoding module 1601 , stream slicing module 1602 , PQ algorithm training module 1604 , and PQ parameter file generation module 1605 may be implemented by processor 410 in server 400 shown in FIG4 .

[0223] In the case of adopting an integrated unit, the server 1600 may include a processing module, a storage module, and a communication module.

[0224] The processing module can be used to control and manage the actions of the server 1600. For example, it can be used to support the server 1600 in executing the steps performed by the aforementioned media encoding module 1601, stream slicing module 1602, sending module 1603, PQ algorithm training module 1604, and PQ parameter file generation module 1605. The storage module can be used to support the server 1600 in storing program code and data. The communication module can be used to support communication between the server 1600 and other devices.

[0225] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0226] In one embodiment, when the processing module is a processor and the storage module is a memory, the server 1600 involved in this embodiment may be a device having the structure shown in FIG. 4 .

[0227] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided in the embodiment shown in Figure 13 of the present application.

[0228] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided in the embodiment shown in Figure 14 of the present application.

[0229] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided by the embodiment shown in Figure 13 of the present application.

[0230] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided by the embodiment shown in Figure 14 of the present application.

[0231] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0232] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0233] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0234] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0235] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A video enhancement method, characterized in that: include: Obtaining a video playback request, wherein the video playback request carries information about the video to be played; Download the video file to be played from the video distribution source station according to the information of the video to be played; Interpreting the primary index file of the video file to obtain the secondary index address and the perceptual quantization (PQ) parameter file address of the video to be played; Downloading the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloading the code stream data of the to-be-played video from the video distribution source station according to the secondary index address; According to the PQ parameter file, obtain the PQ parameters corresponding to the terminal device; A video enhancement operation is performed according to the PQ parameter and the bitstream data to obtain enhanced bitstream data.

2. The method according to claim 1, characterized in that After performing the video enhancement operation according to the PQ parameter and the bitstream data to obtain enhanced bitstream data, the method further includes: The enhanced code stream data is rendered and played.

3. The method according to claim 1, characterized in that The step of interpreting the primary index file of the video file to obtain the secondary index address and the perceptual quantization PQ parameter file address of the video to be played includes: According to the encapsulation protocol of the video file, interpret the primary index file of the video file to obtain the secondary index address of the video to be played; If the tag of the PQ parameter file exists in the primary index file, the PQ parameter file address corresponding to the tag is obtained.

4. The method according to claim 1, characterized in that: The obtaining, according to the PQ parameter file, a PQ parameter corresponding to the terminal device comprises: Interpreting the PQ parameter file according to the structure of the PQ parameter file; From the PQ parameters included in the PQ parameter file, corresponding PQ parameters are obtained according to the information of the terminal device.

5. A video enhancement method, characterized in that: include: Get the recorded video source file; Performing transcoding and stream slicing operations on the video source file; Encapsulating the sliced ​​video files to obtain encapsulated video files, wherein the encapsulated video files include a primary index file; and constructing a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; Add the label and address of the PQ parameter file in the primary index file; The encapsulated video file and the PQ parameter file are sent to a video distribution source station.

6. The method according to claim 5, characterized in that After sending the encapsulated video file and the PQ parameter file to the video distribution source station, the method further includes: Get the parameters of the terminal device and video transcoding parameters; The parameters of the terminal device and the video transcoding parameters are trained by the PQ algorithm to obtain a PQ algorithm model; Generate PQ parameters corresponding to the information of the terminal device in the PQ parameter file according to the PQ algorithm model and the encapsulated video file; The PQ parameters are sent to the video distribution source station, so that the video distribution source station updates the PQ parameters into the PQ parameter file.

7. A video enhancement device, characterized in that: Set in the terminal device, the video enhancement device includes: A video playback module is used to obtain a video playback request, wherein the video playback request carries information about the video to be played; download the video file to be played from a video distribution source station according to the information about the video to be played; and interpret the primary index file of the video file to obtain a secondary index address and a perceptual quantization PQ parameter file address of the video to be played; and download the bitstream data of the video to be played from the video distribution source station according to the secondary index address; A PQ parameter file interpretation module, configured to download the PQ parameter file from the video distribution source station according to the PQ parameter file address; and obtain PQ parameters corresponding to the terminal device according to the PQ parameter file; A video enhancement module is used to perform a video enhancement operation according to the PQ parameter and the bit stream data to obtain an enhanced bit stream. data.

8. A video enhancement device, characterized in that: Set in the server, the video enhancement device includes: A media encoding module, used to obtain the recorded video source file and perform transcoding operation on the video source file; A stream slicing module is used to perform a stream slicing operation on the video file after the transcoding operation, encapsulate the sliced ​​video file, obtain the encapsulated video file, wherein the encapsulated video file includes a primary index file; and construct a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; and add a label and address of the PQ parameter file to the primary index file; The sending module is used to send the encapsulated video file and the PQ parameter file to the video distribution source station.

9. A terminal device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the terminal device, enable the terminal device to execute the method according to any one of claims 1 to 4.

10. A server, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the server, enable the server to perform the method according to any one of claims 5-6.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 4.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 5 to 6.

Images