Video enhancement method and device, server and terminal equipment

Abstract

The embodiment of the invention provides a video enhancement method and device, a server and terminal equipment, in the video enhancement method, the terminal equipment obtains a video playing request carrying information of a video to be played, downloads a video file to be played from a video distribution source station, then explains a primary index file of the video file, and sends the explained primary index file to the terminal equipment; obtaining a secondary index address and a PQ parameter file address, and downloading a PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloading code stream data of the video to be played from the video distribution source station according to the secondary index address. And finally, the terminal equipment obtains a PQ parameter corresponding to the terminal equipment according to the PQ parameter file, and performs video enhancement operation according to the PQ parameter and the code stream data to obtain enhanced code stream data, so that the video enhancement characteristic of the terminal equipment and the adaptation effect of different types of videos can be improved, and the user experience is improved. And the video playing effect of the terminal equipment is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent terminal technology, and in particular to a video enhancement method, apparatus, server and terminal device. Background Art

[0002] In order to support the user's playback experience under different network conditions, reduce key indicators such as user playback freezes and start delays, and reduce the adaptation requirements of the video file encoding format on the terminal side, the online video platform will encode and compress the video files output by the video shooting on the platform server, and then distribute them to the client for playback through the media distribution network using the hypertext transfer protocol (HTTP) communication form. In the process of encoding and compression on the platform server, two encapsulation formats, HTTP livestreaming (HLS) and dynamic adaptive streaming over HTTP (DASH), are usually used to encapsulate video files of different bit rates into a set of multi-bit rate video file structures to support the client to select binary stream files with appropriate bit rates for download and playback according to the current network conditions during playback, so as to achieve dynamic bit rates.

[0003] At the same time, since the platform server encodes and compresses the original recorded video, the subjective quality of the video will be lost to a certain extent. Therefore, with the development of artificial intelligence (AI) technology, the platform server can optimize the video file through algorithms during the encoding and compression process to better adapt and improve the subjective experience of client playback. In the process of enhancement by the platform server, due to the different display parameters (such as brightness and / or color, etc.) of different terminal devices, the playback rendering effect of the enhanced video stream file on different client screens has great differences, and even the actual effect on some terminal devices may affect the user's subjective playback experience. Summary of the invention

[0004] The embodiments of the present application provide a video enhancement method, apparatus, server and terminal device. The embodiments of the present application also provide a computer-readable storage medium to improve the application effect of the video enhancement feature of the terminal device and improve the video playback effect of the terminal device.

[0005] In a first aspect, an embodiment of the present application provides a video enhancement method, including: obtaining a video playback request, the video playback request carrying information of a video to be played; downloading the video file to be played from a video distribution source station according to the information of the video to be played; interpreting the primary index file of the video file to obtain a secondary index address and a perceptual quantization (PQ) parameter file address of the video to be played; downloading the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloading the bitstream data of the video to be played from the video distribution source station according to the secondary index address; obtaining PQ parameters corresponding to a terminal device according to the PQ parameter file; performing a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0006] In the above-mentioned video enhancement method, the terminal device obtains a video playback request carrying information of the video to be played, and downloads the video file to be played from the video distribution source station according to the information of the video to be played. Then, the terminal device interprets the first-level index file of the video file to obtain the second-level index address and PQ parameter file address of the video to be played, and downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloads the bitstream data of the video to be played from the video distribution source station according to the second-level index address. Finally, the terminal device obtains the PQ parameters corresponding to the terminal device according to the PQ parameter file, and performs a video enhancement operation according to the PQ parameters and the bitstream data to obtain the enhanced bitstream data, so as to achieve the improvement of the adaptation effect of the video enhancement characteristics of the terminal device and different types of videos, and improve the video playback effect of the terminal device.

[0007] In one possible implementation manner, after performing the video enhancement operation according to the PQ parameter and the bitstream data to obtain enhanced bitstream data, the method further includes: rendering and playing the enhanced bitstream data.

[0008] In one possible implementation, interpreting the first-level index file of the video file to obtain the second-level index address and perceptual quantization PQ parameter file address of the video to be played includes: interpreting the first-level index file of the video file according to the encapsulation protocol of the video file to obtain the second-level index address of the video to be played; if the label of the PQ parameter file exists in the first-level index file, obtaining the PQ parameter file address corresponding to the label.

[0009] In one possible implementation, obtaining PQ parameters corresponding to the terminal device according to the PQ parameter file includes: interpreting the PQ parameter file according to the structure of the PQ parameter file; and obtaining corresponding PQ parameters from the PQ parameters included in the PQ parameter file according to information of the terminal device.

[0010] In a second aspect, an embodiment of the present application provides a video enhancement method, including: obtaining a filmed video source file; transcoding and stream slicing the video source file; encapsulating the sliced ​​video file to obtain a packaged video file, wherein the packaged video file includes a first-level index file; and constructing a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of a terminal device; adding a label and an address of the PQ parameter file in the first-level index file; and sending the packaged video file and the PQ parameter file to a video distribution source station.

[0011] In the above-mentioned video enhancement method, the server obtains the filmed video source file, performs transcoding and stream slicing operations on the above-mentioned video source file, encapsulates the sliced ​​video file, obtains the encapsulated video file, and the encapsulated video file includes a first-level index file; and constructs a PQ parameter file corresponding to the sliced ​​video file. Then, the server adds the label and address of the above-mentioned PQ parameter file in the first-level index file, sends the encapsulated video file and the above-mentioned PQ parameter file to the video distribution source station, and then, the server also generates PQ parameters corresponding to the information of the terminal device, sends the PQ parameters to the video distribution source station, and the video distribution source station updates the above-mentioned PQ parameters to the above-mentioned PQ parameter file, so that in the process of generating PQ parameters, the parameters of the terminal device and the video transcoding parameters can be combined to effectively coordinate the parameters between the server and the terminal device, thereby improving the adaptation effect of the terminal device video enhancement characteristics and different types of videos, and improving the video playback effect of the terminal device.

[0012] In one possible implementation, after sending the encapsulated video file and the PQ parameter file to the video distribution source station, it also includes: obtaining parameters and video transcoding parameters of the terminal device; training the parameters and video transcoding parameters of the terminal device through the PQ algorithm to obtain a PQ algorithm model; generating PQ parameters corresponding to the information of the terminal device in the PQ parameter file according to the PQ algorithm model and the encapsulated video file; and sending the PQ parameters to the video distribution source station so that the video distribution source station can update the PQ parameters into the PQ parameter file.

[0013] In a third aspect, an embodiment of the present application provides a video enhancement device, which is included in a terminal device, and has the function of implementing the behavior of the terminal device in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a video playback module, a PQ parameter file interpretation module, and a video enhancement module.

[0014] In a fourth aspect, an embodiment of the present application provides a video enhancement device, which is included in a server, and has the function of implementing the server behavior in the second aspect and the possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a media encoding module, a stream slicing module, and a sending module.

[0015] In a fifth aspect, an embodiment of the present application provides a terminal device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the terminal device, enable the terminal device to execute the method provided in the first aspect.

[0016] In the sixth aspect, an embodiment of the present application provides a server, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the server, enable the server to execute the method provided in the second aspect.

[0017] It should be understood that the third aspect and the fifth aspect of the embodiment of the present application are consistent with the technical solution of the first aspect of the embodiment of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0018] It should be understood that the fourth and sixth aspects of the embodiments of the present application are consistent with the technical solutions of the second aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0019] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0020] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the second aspect.

[0021] In a ninth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0022] In a tenth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the second aspect.

[0023] In one possible design, the programs in the ninth and tenth aspects may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the file structure of a video multi-bitrate encapsulation format;

[0025] Figure 2 Schematic diagram of the principle of algorithm enhancement of video stream for terminal equipment;

[0026] Figure 3 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a server provided for one embodiment of the present application;

[0028] Figure 5 A schematic diagram of an application scenario of a video enhancement method provided by an embodiment of the present application;

[0029] Figure 6 A system architecture of a video enhancement method provided in one embodiment of the present application;

[0030] Figure 7 A schematic diagram of a video file storage structure provided by an embodiment of the present application;

[0031] Figure 8 A flowchart of a video enhancement method provided by an embodiment of the present application;

[0032] Fig. 9 A flowchart of a video enhancement method provided by another embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of a primary index file provided for one embodiment of the present application;

[0034] Fig.11 A schematic diagram of the structure of a PQ parameter file provided for one embodiment of the present application;

[0035] Fig.12 A flowchart of a video enhancement method provided in yet another embodiment of the present application;

[0036] Fig.13 A flowchart of a video enhancement method provided in yet another embodiment of the present application;

[0037] Fig.14 A flowchart of a video enhancement method provided in yet another embodiment of the present application;

[0038] Fig.15 A schematic diagram of the structure of a terminal device provided in another embodiment of the present application;

[0039] Fig.16 A schematic diagram of the structure of a server provided in another embodiment of the present application;

[0040] Fig.17 A schematic diagram of the structure of a server provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0041] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0042] When the platform server encodes and compresses the video files, it usually encapsulates the video files with different bit rates into a set of multi-bit rate video file structures. The file structure of the video multi-bit rate encapsulation format can be as follows: Figure 1 As shown, Figure 1 This is a schematic diagram of the file structure of the video multi-bitrate encapsulation format.

[0043] During the process of enhancing the video files by the platform server, due to the different display parameters (such as brightness and / or color) of different terminal devices, there are large differences in the playback rendering effects of the enhanced video stream files on different client screens. Even the actual effects on some terminal devices may affect the user's subjective playback experience.

[0044] To solve this problem, existing related technologies have introduced the concept of metadata (including static metadata and dynamic metadata) in some new high dynamic range video encoding technologies (high dynamic range, HDR). Metadata is used to describe some technical parameter information used by the server in the video encoding process. During the decoding and rendering process, the terminal device can understand the relevant parameters and perform targeted decoding and rendering, thereby better maintaining the end-side playback effect.

[0045] However, HDR metadata technology usually has two practical application problems:

[0046] 1. The relevant metadata concept is only used in the newly designed HDR technology, and is not applicable to some videos that are not suitable for HDR encoding;

[0047] 2. The metadata in HDR technology is encapsulated in the video stream file and can only express some parameters in the actual video encoding process. It cannot solve the differences in playback effects caused by screen differences on different terminal devices. If the actual playback effect of some terminal devices is not good, the problem cannot be effectively solved by adjusting the metadata.

[0048] In order to solve the problems faced by HDR metadata technology, existing related technologies have proposed that the terminal device perform algorithm enhancement (for example, perceptual quantizer (PQ) algorithm) on the video stream during the video playback rendering process to reduce the impact of the subjective playback experience caused by the screen parameters of the terminal device. The specific technical principles are as follows Figure 2 As shown, Figure 2 Schematic diagram of the principle of algorithmic enhancement of video stream for terminal devices.

[0049] from Figure 2 It can be seen that the video production system performs one or a combination of the following operations on the video source: video super-resolution, standard dynamic range (SDR)->HDR, sharpening, contrast enhancement, noise reduction, super resolution (SR) and brightness / saturation enhancement operations to obtain a multi-bitrate video film (i.e., playback source). After the television (TV) side (i.e., terminal device side) obtains the playback source, a set of PQ algorithms is used to perform one or a combination of the following operations on the playback source: noise reduction, SR, sharpening, contrast enhancement and color / saturation enhancement, thereby reducing the impact of the subjective playback experience caused by the TV screen parameters.

[0050] However, the following problems also exist in the implementation of this plan:

[0051] 1. Due to the limited computing power of the terminal device, only the PQ parameters trained using screen parameters can be preloaded in advance, resulting in the inability of the end-side enhancement algorithm to make associated adjustments based on the transcoding parameters of the specific video, resulting in poor algorithm application effects;

[0052] 2. For different videos, if the algorithm parameters used by the platform server are different, it will lead to different playback effects on the end side, and even negative effects after the superposition of the platform server and end-side algorithms. The adaptability of the end-side algorithm application is poor.

[0053] 3. After video encoding is produced, if it is found that the playback effect is poor on the screens of some models of terminal devices, the platform server can only readjust the video transcoding algorithm, which is not only costly but also easily leads to a decline in the playback effect on the screens of other models of terminal devices.

[0054] Based on the above problems, the present application provides a video enhancement method that can solve the following problems:

[0055] 1. The video enhancement PQ parameters executed by the terminal device cannot be adaptively adjusted according to the transcoding parameters of the specific video being played.

[0056] 2. When the transcoded video does not play well on the screens of some terminal devices, the video platform server cannot solve the known playback experience problem by adjusting the PQ parameters of the terminal device.

[0057] The video enhancement method provided in the embodiment of the present application can be applied to a terminal device, wherein the terminal device can be a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA) and the like; the embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0058] For example, Figure 3 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application is shown in FIG. Figure 3As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0059] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0060] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0061] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0062] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0064] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device 100 through the power management module 141.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0066] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0068] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0069] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0071] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0072] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0073] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0074] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0075] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0076] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0077] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0078] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0079] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0081] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0082] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0083] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0084] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0085] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0086] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0087] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0088] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0089] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0090] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0091] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0092] In addition, the implementation of the video enhancement method provided in the embodiment of the present application involves not only the terminal device side, but also the server side. The above-mentioned server can be a video platform server, which can also be called a video online platform server or a platform server. The above-mentioned server can be set in the cloud, and the embodiment of the present application does not impose any restrictions on the specific type of the server.

[0093] For example, Figure 4 A schematic diagram of the structure of a server provided in one embodiment of the present application, such as Figure 4 As shown, the server 400 includes a processor 410 and a communication interface 420. Optionally, the server 400 may also include a memory 430. The processor 410, the communication interface 420 and the memory 430 may communicate with each other through an internal connection path to transmit control and / or data signals, the memory 430 is used to store a computer program, and the processor 410 is used to call and run the computer program from the memory 430.

[0094] The processor 410 and the memory 430 may be combined into a processing device, or more commonly, they are independent components, and the processor 410 is used to execute the program code stored in the memory 430. In specific implementation, the memory 430 may also be integrated into the processor 410, or may be independent of the processor 410.

[0095] In addition, in order to make the functions of the server 400 more complete, the server 400 may further include one or more of an input unit 460 and a display unit 470 .

[0096] Optionally, the server 400 may further include a power supply 450 for providing power to various devices or circuits in the server 400 .

[0097] It should be understood that Figure 4 The processor 410 in the server 400 shown may be a system on chip SOC, and the processor 410 may include a central processing unit (CPU), and may further include other types of processors, such as a graphics processing unit (GPU) and the like.

[0098] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 2 The terminal device 100 of the structure shown in FIG. Figure 4 Taking the server 400 of the shown structure as an example, the video enhancement method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0099] The application scenario involved in the embodiment of the present application is that the server 400 encodes the video source file uploaded to the server 400, and then stores the encoded video in a video distribution source station / content delivery network (CDN). The video distribution source station distributes the encoded video to the terminal device 100 through the CDN service. The terminal device 100 provides video enhancement capabilities while playing the video, thereby improving the subjective playback experience of the video on the terminal device 100. Figure 5 A schematic diagram of an application scenario of a video enhancement method provided by an embodiment of the present application. Figure 5 As shown, after the server 400 performs video enhancement, multi-bitrate encoding and / or video stream slicing operations on the video source file through media encoding and stream segmentation capabilities, it is encapsulated into a storage format of multiple video binary files described by an index file (commonly encapsulated using the HLS protocol and the DASH protocol).

[0100] After the terminal device 100 obtains the video index file and the video binary file from the video distribution source station through HTTP communication, it performs video decoding and rendering. During the video decoding and rendering process, the video enhancement module of the terminal device 100 improves the video playback experience on the terminal device 100 by executing the video enhancement PQ algorithm.

[0101] In summary, the video enhancement method provided in the embodiment of the present application may include:

[0102] 1. First, by extending the structure definition of the video file, a set of PQ parameter files is added without affecting the existing video encoding and decoding method, so as to describe the definition of PQ parameters for performing end-side enhancement on different types of terminal devices 100 of the video;

[0103] 2. The video platform server 400 outputs the corresponding PQ parameter file and then updates it to the video distribution source station. This allows adjustment of the terminal-side algorithm without having to update the basic video index file and video source (.ts / .mp4) file, thereby greatly ensuring the reliability of the basic video playback process.

[0104] 3. The terminal device 100 needs to increase the ability to interpret the PQ parameter file, which may include:

[0105] 1) Explain all the files in the index file and obtain the storage address of the PQ parameter file on the video distribution source station;

[0106] 2) Download the PQ parameter file from the video distribution source station;

[0107] 3) interpret the PQ parameter file and obtain applicable PQ parameters from the PQ parameter file according to the device model of the currently running video decoding;

[0108] 4) The interpreted PQ parameters are input into the decoder together with the video source file for video decoding, and the end-side enhancement algorithm in the decoding process is driven by the PQ parameters.

[0109] Figure 6 The system architecture of the video enhancement method provided by one embodiment of the present application is as follows: Figure 6As shown, the system architecture of the embodiment of the present application provides a video file storage structure. After the server 400 encodes the video source file uploaded to the server 400, the encoded video is stored in the video distribution source station / CDN. At the same time, a PQ algorithm training and PQ parameter file generation module is provided on the server 400 to support the production of the PQ parameter file in the video file storage structure provided in the embodiment of the present application. The video distribution source station distributes the encoded video file and the PQ parameter file to the terminal device 100 through the CDN service. While playing the video, the terminal device 100 provides a PQ parameter file interpretation module on the terminal device 100 to support the acquisition of the PQ parameter file from the video file storage structure provided in the embodiment of the present application, and provides video enhancement capability for the input of the video enhancement module in the terminal device 100 to perform algorithm enhancement, thereby improving the video playback experience on the terminal device 100.

[0110] Below Figure 6 The system architecture shown is described in detail.

[0111] 1. PQ algorithm training module: The PQ algorithm training module can support training corresponding PQ parameter models according to corresponding terminal parameters for video categories or specific videos.

[0112] 2. PQ parameter file generation module: Generates video-related PQ parameter files based on the PQ parameter model output by the PQ algorithm training module and synchronizes them to the video distribution source station.

[0113] 3. Video file storage: It is a data video file storage structure and implementation of the video distribution source station and its CDN. The video file storage structure can include index files, video binary stream files (usually .ts or .mp4 suffix files) and PQ parameter files. The video file storage structure is a file storage structure composed of the above three files through structural relationships.

[0114] 4. Video playback module: The video playback module is a module in the terminal device 100 that performs video file downloading, interpretation, decoding and rendering, downloads video files from the video distribution source station, interprets video files according to the video file storage structure, and then decodes and renders to achieve the playback operation of video files on the terminal side. In the embodiment of the present application, the video playback module is responsible for triggering the PQ parameter file interpretation module to interpret the PQ parameters applicable to the terminal for the video to be played, and then inputs them into the video enhancement module to trigger the video enhancement operation during the video decoding and rendering process.

[0115] 5. PQ parameter file interpretation module: The PQ parameter file interpretation module provides a PQ parameter file interpretation function, is responsible for downloading and interpreting the PQ parameter file, and matches the PQ parameters applicable to the video to be played from the PQ parameter file according to the information of the terminal device 100.

[0116] 6. Video enhancement module: performs end-side video enhancement operations according to the PQ enhancement parameters and the binary stream information of the video to be played. In an embodiment of the present application, the video enhancement module receives the PQ parameters obtained from the PQ parameter file interpretation module as input parameters of the enhancement algorithm.

[0117] The schematic diagram of the video file storage structure as the key technology of the embodiment of the present application can be as follows: Figure 7 As shown, the embodiment of the present application provides a newly added end-side PQ parameter file definition and modifies the first-level index file definition based on the multi-level video file storage structure of HLS or DASH. Figure 7 A schematic diagram of a video file storage structure provided for one embodiment of the present application.

[0118] Below Figure 7 The end-side PQ parameter file and the first-level index file in the video file storage structure shown are explained.

[0119] 1. End-side PQ parameter file: Figure 7 The terminal-side PQ algorithm file in the terminal-side PQ parameter file is used to describe the terminal enhancement PQ parameter information used by the video file in different types of terminal devices 100. Therefore, the basic structure of the PQ parameter file is a set of data storage structures with the series of the terminal device 100 or the model of the terminal device 100 as the keyword and the PQ parameter as the main storage information. It can support the PQ parameter list corresponding to the expansion terminal device 100 to dynamically adjust the input of the terminal enhancement PQ algorithm executed by the video file on different terminals. The PQ parameters may include: video enhancement algorithm parameter values ​​such as parameters and curve values ​​of dimensions such as clarity, noise reduction, contrast, color, HDR and / or light control.

[0120] 2. Primary index file: The primary index file identifies different video encoding formats, clarity, bit rate and other information and the corresponding relative address information of the secondary index file in the form of tags. In the embodiment of the present application, the optional tag of the terminal-side PQ parameter file is extended to identify the relative address information of the terminal-side PQ parameter file.

[0121] The implementation process of the embodiment of the present application involves the transcoding, algorithm training and video file distribution processes of the server 400, as well as the video downloading, interpretation, terminal enhancement and decoding rendering processes of the terminal device 100. The processes involved in the embodiment of the present application are described below.

[0122] Figure 8 A flowchart of a video enhancement method provided by an embodiment of the present application is shown in FIG. Figure 8 As shown, the above video enhancement method may include:

[0123] Step 801, during the implementation of video transcoding and stream file encapsulation, the server 400 adjusts the video file storage data structure, adds a data management model for the PQ parameter file, publishes the transcoded video file to the video distribution source station, and adjusts the video file storage of the video distribution source station (CDN).

[0124] In step 802 , a PQ algorithm training module is added to the server 400 to support algorithm engineers to perform terminal-side PQ algorithm training for specific videos and terminal device 100 models.

[0125] Step 803: After the PQ algorithm training is completed, the server 400 generates a PQ parameter file through a PQ parameter file generation module, and then updates the PQ parameter file to the video distribution source station (CDN).

[0126] Step 804: the video playback module in the terminal device 100 obtains the video file to be played from the video distribution source station.

[0127] Step 805: The video playback module in the terminal device 100 interprets the primary index file of the video file to obtain a secondary index address and a PQ parameter file address.

[0128] Step 806: The video playback module in the terminal device 100 drives the PQ parameter file interpretation module in the terminal device 100 to download and interpret the PQ parameter file.

[0129] Step 807: The PQ parameter file interpretation module downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address interpreted by the video playback module.

[0130] Step 808 : The PQ parameter file interpretation module obtains the PQ parameters corresponding to the terminal device 100 according to the model of the terminal device 100 .

[0131] Step 809: The PQ parameter file interpretation module sends the PQ parameters corresponding to the terminal device 100 to the video playback module.

[0132] Step 810: The video playback module downloads the code stream data of the video to be played from the video distribution source station according to the secondary index address.

[0133] Step 811, the video playback module sends the PQ parameters and the bitstream data to the video enhancement module in the terminal device 100, and the video enhancement module performs a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0134] The video enhancement method provided in the embodiment of the present application is described below from three aspects: the server 400, the video distribution source station and the terminal device 100.

[0135] (I) Implementation process of server 400.

[0136] In the embodiment of the present application, the server 400 (video platform server) provides encoding and stream slicing operations for the video source file, produces a video file and uploads it to the video distribution source station, performs PQ algorithm training according to the parameters of the terminal device 100 and the video transcoding parameters, generates a PQ parameter file and uploads it to the video distribution source station, Fig. 9 A flowchart of a video enhancement method provided by another embodiment of the present application is shown in FIG. Fig. 9 As shown, the process executed by the server 400 may include:

[0137] Step 901: the server 400 obtains the recorded video source file.

[0138] Step 902: The server 400 encodes the video source file into video files of different bit rates through the video encoding module and the video stream slicing module. After the transcoded and sliced ​​video files are encapsulated using protocols such as HLS or DASH, a complete video file structure is formed. The server 400 encapsulates the video file using protocols such as HLS or DASH. In the process of generating the primary index file, the server 400 constructs the basic framework of the PQ parameter file of the video file according to the basic framework of the PQ parameter file (such as Fig.11 ), and add the PQ parameter file tag and the storage address of the corresponding PQ parameter file in the primary index file (as shown in Fig.10 shown).

[0139] Step 903, the server 400 sends the PQ parameter file and the video file to the video distribution source station (CDN is an acceleration method used by the video distribution source station in the video distribution process, which is a common method of the video online platform and is not described in detail in the implementation of this application).

[0140] At this point, the video files in the video distribution source station can be downloaded and played by the terminal device 100, but because the PQ parameter file only has a basic framework at this time, during the playback process of the terminal device 100, specific PQ parameters cannot be obtained from the PQ parameter file and input into the video enhancement module of the terminal device 100.

[0141] In step 904, the PQ algorithm training module in the server 400 uses the parameters of the terminal device 100 as input and performs PQ algorithm training in combination with the video transcoding parameters (which can be a single video transcoding parameter or a class of video transcoding parameters) to form a PQ algorithm model that matches the video transcoding parameters and the parameters of the terminal device 100.

[0142] In step 905, the PQ parameter file generation module in the server 400 generates the PQ parameters used by the terminal device 100 in the PQ parameter file for the video file according to the PQ algorithm model and the video file generated in step 902, and updates the PQ parameter file generated in step 902. After completing step 905, the PQ parameter file will include the basic framework and the full amount of information of the PQ parameters used by the terminal device 100.

[0143] When there are playback experience issues with a video in a certain terminal device 100 or a new terminal device 100 is added, the server 400 can execute steps 904 to 905 again to retrain the PQ algorithm model and update the PQ parameters used by the terminal device 100 in the above-mentioned PQ parameter file, thereby achieving better adaptation of the playback experience of the video file on different terminal devices 100 without adjusting the video file.

[0144] (2) Implementation process of video distribution source station.

[0145] As a distribution module of the online video operation platform, the video distribution source station provides video storage and video distribution and download service capabilities based on HTTP communication. In the embodiment of the present application, the video distribution source station provides a video file storage function, which can be used to support the storage and update of videos and corresponding PQ parameter files. Among them, the video file storage structure in the video distribution source station can be as follows: Figure 7 As shown, a terminal-side PQ parameter file is newly added to the video file storage structure, and an optional tag for identifying the terminal-side PQ parameter file is added to the primary index file.

[0146] Fig.10 A schematic diagram of the structure of a primary index file provided in one embodiment of the present application, Fig.10 The structure shown is explained using the m3u8 first-level index file of the HLS packaging protocol as an example; Fig.11 A schematic diagram of the structure of a PQ parameter file provided in one embodiment of the present application, Fig.11 The structure shown refers to the tag definition format of the m3u8 index file of the HLS encapsulation protocol to illustrate the structure of the PQ parameter file. Of course, the embodiment of the present application is not limited to this. In the specific implementation process, different file formats based on tag description can be used to implement it, for example: the tag description method based on the extensible markup language (XML) format adopted by the DASH encapsulation protocol index file (.MPD file).

[0147] See also Fig.10In the m3u8 first-level index file, without modifying the basic file structure and tag definition of the index file defined in the HLS protocol, the relative storage address of the PQ parameter file is identified by extending the custom tag "#EXT-X-HW-TERMINALPQ-INF". Fig.10 The "TERMINALPQ-LIST-INFO.PQI" in the file is the address information of the PQ parameter file relative to the m3u8 file storage directory. The file name of the PQ parameter file only needs to conform to the file name definition specification of the mainstream operating system, and the adjustment of the file name and file suffix definition of the PQ parameter file does not affect the implementation effect of the embodiment of the present application.

[0148] See also Fig.11 The PQ parameter file uses a tag description file structure to store the PQ parameters corresponding to the terminal device 100. Usually, the tag description file includes a file header, a file version number, a file end tag, and a data tag. The PQ parameter file name and storage path need to be consistent with the file name and storage address described in the relative address of the PQ parameter file in the primary index file.

[0149] Fig.11 The following is an example of the structure of a PQ parameter file defined in the m3u8 tag definition format. Fig.11 The left picture in the figure is the basic framework of the file, which includes the label information of the file header, file version number and file end; Fig.11 The right picture in the figure adds a list of terminal devices 100 and their corresponding PQ parameters to the basic framework of the file, where "#EXT-HW-TERMINALNUM" is a label definition, "ABC" and "DEF" are information examples of the corresponding terminal device 100, and "THISISVALUEOFPQPARAMETFORTERMINALABC" is a text expression example of the PQ parameters corresponding to the terminal device 100.

[0150] Fig.11 The tag definition, information of the terminal device 100 and PQ parameters shown in the figure can be expressed in text form, and the specific naming method and expression method do not affect the implementation effect of the embodiment of the present application.

[0151] Among them, the text definition of the PQ parameter may include parameters and curve values ​​of dimensions such as clarity, noise reduction, contrast, color, HDR and / or dimming, and the text definition of the PQ parameter may be expanded and adjusted according to the input parameter requirements of the actual terminal device 100 enhancement algorithm. The organization order and organization form of each parameter text may be consistent with the interpretation logic of the specific implementation of the PQ parameter file interpretation module, and adjusting the interpretation logic of the PQ parameter file interpretation module and the organization form of the parameter text does not affect the implementation effect of the embodiment of the present application.

[0152] (III) Implementation process of terminal device 100 (video playback terminal).

[0153] After the user selects a video to be played, the terminal device 100 can download the video file of the video to be played from the video distribution source station through HTTP communication, decode, render and play it, and in this process, the video enhancement operation can be triggered according to the capability of the terminal device 100 to improve the video playback experience. In this embodiment, the terminal device 100 completes the download and interpretation of the PQ parameter file through the PQ parameter file interpretation module.

[0154] Fig.12 A flowchart of a video enhancement method provided in yet another embodiment of the present application is shown in FIG. Fig.12 As shown, the process executed by the terminal device 100 may include:

[0155] Step 1201, after receiving the user's video playback request, the video playback module downloads the video file to be played from the video distribution source station according to the information of the video to be played selected by the user, and interprets the video file according to the encapsulation protocol of the video file; interprets the first-level index file of the video file to obtain the second-level index address of the video to be played, and adds the interpretation logic of the PQ parameter file tag to the logic of interpreting the first-level index file of the video file. If the first-level index file contains the tag of the PQ parameter file, the PQ parameter file address is obtained.

[0156] Step 1202, the video playback module inputs the PQ parameter file address to the PQ parameter file interpretation module, and downloads the bitstream data of the video to be played from the video distribution source station according to the secondary index address, and the video playback module sends the bitstream data of the video to be played to the video enhancement module.

[0157] Step 1203, after receiving the PQ parameter file address input by the video playback module, the PQ parameter file interpretation module downloads the PQ parameter file of the video to be played from the video distribution source station.

[0158] Step 1204, the PQ parameter file interpretation module interprets the structure of the PQ parameter file (such as Fig.11 ) interpret the PQ parameter file to obtain the PQ parameters corresponding to the terminal device 100.

[0159] Step 1205: The PQ parameter file interpretation module sends the PQ parameters corresponding to the terminal device 100 to the video playback module.

[0160] Step 1206: After receiving the PQ parameters corresponding to the terminal device 100, the video playback module sends the PQ parameters corresponding to the terminal device 100 to the video enhancement module.

[0161] Step 1207: the video enhancement module performs a video enhancement operation according to the PQ parameter and the bitstream data of the video to be played to obtain enhanced bitstream data, and the video enhancement module sends the enhanced bitstream data to the video playback module.

[0162] Step 1208: The video playback module decodes, renders, and plays the enhanced bitstream data.

[0163] During the specific implementation process, in order to improve the compatibility of video enhancement of the terminal device 100, a set of default PQ parameters can be pre-made in the terminal device 100. If the PQ parameter file interpretation module cannot obtain the PQ parameters used by the terminal device 100 from the PQ parameter file of the video, the video enhancement module can use the default PQ parameters to perform video enhancement operations. This operation does not affect the implementation effect of the embodiment of the present application.

[0164] As described above, the embodiment of the present application provides a video enhancement method that can solve the problem faced in the online video playback business that the terminal video enhancement parameters cannot be adapted to the transcoding parameters of the specific playback video, and the problem that the transcoded video is not well adapted to the new terminal device and cannot be solved by quickly adjusting the playback parameters. The video enhancement method provided in the embodiment of the present application can effectively apply the application effect of the terminal video enhancement feature and improve the video playback experience of the terminal device 100.

[0165] Fig.13 A flowchart of a video enhancement method provided in yet another embodiment of the present application is shown in FIG. Fig.13 As shown, the above video enhancement method may include:

[0166] Step 1301: The terminal device 100 obtains a video play request, which carries information about the video to be played.

[0167] Step 1302: The terminal device 100 downloads the video file to be played from the video distribution source station according to the information of the video to be played.

[0168] Step 1303: The terminal device 100 interprets the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played.

[0169] Specifically, the terminal device 100 interprets the first-level index file of the above-mentioned video file, and obtains the second-level index address and PQ parameter file address of the video to be played. The terminal device 100 interprets the first-level index file of the above-mentioned video file according to the encapsulation protocol of the above-mentioned video file, and obtains the second-level index address of the video to be played; if the label of the above-mentioned PQ parameter file exists in the above-mentioned first-level index file, the PQ parameter file address corresponding to the above-mentioned label is obtained.

[0170] Step 1304: the terminal device 100 downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloads the code stream data of the video to be played from the video distribution source station according to the secondary index address.

[0171] Step 1305: The terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 according to the above PQ parameter file.

[0172] Specifically, the terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 according to the above-mentioned PQ parameter file: the terminal device 100 interprets the PQ parameter file according to the structure of the above-mentioned PQ parameter file, and obtains the corresponding PQ parameters from the PQ parameters included in the above-mentioned PQ parameter file according to the information of the terminal device 100.

[0173] In this embodiment, the basic structure of the PQ parameter file is a set of data storage structures that uses the information of the terminal device 100 as keywords and PQ parameters as the main storage information. The information of the terminal device 100 may include the series to which the terminal device 100 belongs and / or the model of the terminal device 100.

[0174] Step 1306: The terminal device 100 performs a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0175] Furthermore, after step 1306, the following steps may also be included:

[0176] Step 1307: The terminal device 100 renders and plays the enhanced bitstream data.

[0177] In the above-mentioned video enhancement method, the terminal device 100 obtains a video playback request carrying information of the video to be played, and downloads the video file to be played from the video distribution source station according to the information of the video to be played. Then, the terminal device 100 interprets the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played, and downloads the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloads the bitstream data of the video to be played from the video distribution source station according to the secondary index address. Finally, the terminal device 100 obtains the PQ parameters corresponding to the terminal device 100 according to the PQ parameter file, and performs a video enhancement operation according to the PQ parameters and the bitstream data to obtain the enhanced bitstream data, so as to achieve the improvement of the adaptation effect of the video enhancement characteristics of the terminal device 100 and different types of videos, and improve the video playback effect of the terminal device 100.

[0178] Fig.14 A flowchart of a video enhancement method provided in yet another embodiment of the present application is shown in FIG. Fig.14 As shown, the above video enhancement method may include:

[0179] Step 1401: the server 400 obtains the recorded video source file.

[0180] Step 1402: The server 400 performs transcoding and stream slicing operations on the above-mentioned video source file.

[0181] Step 1403, the server 400 encapsulates the sliced ​​video file to obtain an encapsulated video file, which includes a primary index file; and constructs a PQ parameter file corresponding to the sliced ​​video file.

[0182] The PQ parameter file includes information about the terminal device 100 , and the information about the terminal device 100 may include the series to which the terminal device 100 belongs and / or the model of the terminal device 100 .

[0183] Step 1404: the server 400 adds the label and address of the PQ parameter file in the primary index file.

[0184] Step 1405: the server 400 sends the packaged video file and the above-mentioned PQ parameter file to the video distribution source station.

[0185] Furthermore, after step 1405, the following steps may also be included:

[0186] Step 1406 , the server 400 obtains parameters of the terminal device 100 and video transcoding parameters.

[0187] In step 1407 , the server 400 trains the parameters of the terminal device 100 and the video transcoding parameters through the PQ algorithm to obtain a PQ algorithm model.

[0188] Step 1408: The server 400 generates PQ parameters corresponding to the information of the terminal device 100 in the PQ parameter file according to the PQ algorithm model and the encapsulated video file.

[0189] Step 1409: the server 400 sends the PQ parameters to the video distribution source station, so that the video distribution source station updates the PQ parameters into the PQ parameter file.

[0190] In the above-mentioned video enhancement method, the server 400 obtains the filmed video source file, transcodes and stream slices the above-mentioned video source file, encapsulates the sliced ​​video file, obtains the encapsulated video file, and the encapsulated video file includes a primary index file; and constructs a PQ parameter file corresponding to the sliced ​​video file. Then, the server 400 adds the label and address of the above-mentioned PQ parameter file in the primary index file, sends the encapsulated video file and the above-mentioned PQ parameter file to the video distribution source station, and then, the server 400 also generates PQ parameters corresponding to the information of the terminal device 100, sends the PQ parameters to the video distribution source station, and the video distribution source station updates the above-mentioned PQ parameters to the above-mentioned PQ parameter file, so that in the process of generating PQ parameters, the parameters of the terminal device 100 and the video transcoding parameters can be combined to effectively coordinate the parameters between the server 400 and the terminal device 100, thereby improving the adaptation effect of the video enhancement characteristics of the terminal device 100 and different types of videos, and improving the video playback effect of the terminal device 100.

[0191] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0192] It is understandable that, in order to implement the above functions, the terminal device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0193] This embodiment can divide the terminal device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0194] Fig.15 A schematic diagram of the structure of a terminal device provided in another embodiment of the present application, in which each functional module is divided according to each function, Fig.15 A possible schematic diagram of the composition of the terminal device 1500 involved in the above embodiment is shown. Fig.15 As shown, the terminal device 1500 may include: a video playback module 1501, a PQ parameter file interpretation module 1502 and a video enhancement module 1503;

[0195] The video playback module 1501 is used to obtain a video playback request, which carries information about the video to be played; download the video file to be played from the video distribution source station according to the information about the video to be played; interpret the primary index file of the video file to obtain the secondary index address and PQ parameter file address of the video to be played; and download the bitstream data of the video to be played from the video distribution source station according to the secondary index address;

[0196] A PQ parameter file interpretation module 1502 is used to download the PQ parameter file from the video distribution source station according to the PQ parameter file address; and obtain the PQ parameters corresponding to the terminal device according to the PQ parameter file;

[0197] The video enhancement module 1503 is used to perform a video enhancement operation according to the PQ parameters and the bitstream data to obtain enhanced bitstream data.

[0198] Furthermore, the video playing module 1501 is also used for rendering and playing the enhanced code stream data after the video enhancing module 1503 performs a video enhancing operation to obtain the enhanced code stream data.

[0199] In this embodiment, the PQ parameter file interpretation module 1502 is specifically used to interpret the first-level index file of the above-mentioned video file according to the encapsulation protocol of the above-mentioned video file, and obtain the second-level index address of the video to be played; if there is a label of the PQ parameter file in the first-level index file, then obtain the PQ parameter file address corresponding to the above-mentioned label.

[0200] The PQ parameter file interpretation module 1502 is specifically used to interpret the PQ parameter file according to the structure of the PQ parameter file, and obtain corresponding PQ parameters from the PQ parameters included in the PQ parameter file according to the information of the terminal device.

[0201] It should be noted that this application Fig.13 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0202] The terminal device 1500 provided in this embodiment is used to execute the present application Fig.13 The video enhancement method provided by the illustrated embodiment can therefore achieve the same effect as the above method.

[0203] It should be understood that the terminal device 1500 may correspond to Figure 3 The terminal device 100 shown in FIG. The function of the video playback module 1501 can be Figure 3 The processor 110, antenna 1 and mobile communication module 150 in the terminal device 100 shown in the figure, and / or, implemented by the processor 110, antenna 2 and wireless communication module 160, and by Figure 3 The processor 110 and the display screen 194 in the terminal device 100 are implemented; the functions of the PQ parameter file interpretation module 1502 and the video enhancement module 1503 can be implemented by Figure 1 The processor 110 in the terminal device 100 is implemented.

[0204] In the case of adopting an integrated unit, the terminal device 1500 may include a processing module, a storage module and a communication module.

[0205] The processing module can be used to control and manage the actions of the terminal device 1500, for example, it can be used to support the terminal device 1500 to execute the steps executed by the above-mentioned video playback module 1501, PQ parameter file interpretation module 1502 and video enhancement module 1503. The storage module can be used to support the terminal device 1500 to store program codes and data, etc. The communication module can be used to support the communication between the terminal device 1500 and other devices.

[0206] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0207] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 1500 involved in this embodiment may be a Figure 3 Device of the structure shown.

[0208] Similarly, it is understandable that, in order to implement the above functions, the server includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0209] This embodiment can divide the server into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0210] Fig.16 A schematic diagram of the structure of a server provided in another embodiment of the present application, in which each functional module is divided into corresponding functional modules, Fig.16 A possible composition diagram of the server 1600 involved in the above embodiment is shown. Fig.16 As shown, the server 1600 may include: a media encoding module 1601, a stream slicing module 1602 and a sending module 1603;

[0211] The media encoding module 1601 is used to obtain the recorded video source file and perform transcoding operation on the video source file;

[0212] The stream slicing module 1602 is used to perform a stream slicing operation on the video file after the transcoding operation, encapsulate the sliced ​​video file, obtain the encapsulated video file, wherein the encapsulated video file includes a primary index file; and construct a PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; and add a label and address of the PQ parameter file to the primary index file;

[0213] The sending module 1603 is used to send the encapsulated video file and the above-mentioned PQ parameter file to the video distribution source station.

[0214] Fig.17 A schematic diagram of the structure of a server provided in another embodiment of the present application, Fig.16 Compared to the server shown, the difference is that Fig.17 The server 1600 shown may also include: a PQ algorithm training module 1604 and a PQ parameter file generation module 1605;

[0215] The PQ algorithm training module 1604 is used to obtain the parameters of the terminal device and the video transcoding parameters after the sending module 1603 sends the encapsulated video file and the PQ parameter file to the video distribution source station, and train the parameters of the terminal device and the video transcoding parameters through the PQ algorithm to obtain the PQ algorithm model;

[0216] A PQ parameter file generating module 1605 is used to generate PQ parameters corresponding to the information of the terminal device in the above PQ parameter file according to the PQ algorithm model and the encapsulated video file;

[0217] The sending module 1603 is used to send the PQ parameters to the video distribution source station, so that the video distribution source station updates the PQ parameters into the PQ parameter file.

[0218] It should be noted that this application Fig.14 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0219] The server 1600 provided in this embodiment is used to execute the application Fig.14 The video enhancement method provided by the illustrated embodiment can therefore achieve the same effect as the above method.

[0220] It should be understood that the server 1600 may correspond to Figure 4 The server 400 shown in FIG. 1600 includes a sending module 1603 and a sending module 1604. Figure 4 The processor 410 and the communication interface 420 in the server 400 are implemented; the functions of the media encoding module 1601, the stream slicing module 1602, the PQ algorithm training module 1604 and the PQ parameter file generation module 1605 can be implemented by Figure 4 The processor 410 in the server 400 is shown implemented.

[0221] In the case of adopting an integrated unit, the server 1600 may include a processing module, a storage module, and a communication module.

[0222] The processing module may be used to control and manage the actions of the server 1600, for example, it may be used to support the server 1600 in executing the steps performed by the above-mentioned media encoding module 1601, stream slicing module 1602, sending module 1603, PQ algorithm training module 1604, and PQ parameter file generation module 1605. The storage module may be used to support the server 1600 in storing program codes and data, etc. The communication module may be used to support the communication between the server 1600 and other devices.

[0223] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0224] In one embodiment, when the processing module is a processor and the storage module is a memory, the server 1600 involved in this embodiment may be a server having Figure 4 Device of the structure shown.

[0225] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Fig.13 The method provided by the illustrated embodiment.

[0226] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Fig.14 The method provided by the illustrated embodiment.

[0227] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Fig.13 The method provided by the illustrated embodiment.

[0228] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Fig.14 The method provided by the illustrated embodiment.

[0229] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0230] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0231] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0232] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video enhancement method, characterized in that: include: Obtaining a video playback request, wherein the video playback request carries information about the video to be played; Download the video file to be played from the video distribution source station according to the information of the video to be played; Interpreting the primary index file of the video file to obtain the secondary index address and the perceptual quantization (PQ) parameter file address of the video to be played; Downloading the PQ parameter file from the video distribution source station according to the PQ parameter file address; and downloading the code stream data of the to-be-played video from the video distribution source station according to the secondary index address; According to the PQ parameter file, obtain the PQ parameters corresponding to the terminal device; A video enhancement operation is performed according to the PQ parameter and the bitstream data to obtain enhanced bitstream data.

2. The method according to claim 1, characterized in that After performing the video enhancement operation according to the PQ parameter and the bitstream data to obtain enhanced bitstream data, the method further includes: The enhanced code stream data is rendered and played.

3. The method according to claim 1, characterized in that The step of interpreting the primary index file of the video file to obtain the secondary index address and the perceptual quantization PQ parameter file address of the video to be played includes: According to the encapsulation protocol of the video file, interpret the primary index file of the video file to obtain the secondary index address of the video to be played; If the tag of the PQ parameter file exists in the primary index file, the PQ parameter file address corresponding to the tag is obtained.

4. The method according to claim 1, characterized in that: The obtaining, according to the PQ parameter file, a PQ parameter corresponding to the terminal device comprises: Interpreting the PQ parameter file according to the structure of the PQ parameter file; From the PQ parameters included in the PQ parameter file, corresponding PQ parameters are obtained according to the information of the terminal device.

5. A video enhancement method, characterized in that: include: Get the recorded video source file; Performing transcoding and stream slicing operations on the video source file; Encapsulating the sliced ​​video files to obtain encapsulated video files, wherein the encapsulated video files include a primary index file; and constructing a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; Add the label and address of the PQ parameter file in the primary index file; The encapsulated video file and the PQ parameter file are sent to a video distribution source station.

6. The method according to claim 5, characterized in that After sending the encapsulated video file and the PQ parameter file to the video distribution source station, the method further includes: Get the parameters of the terminal device and video transcoding parameters; The parameters of the terminal device and the video transcoding parameters are trained by the PQ algorithm to obtain a PQ algorithm model; Generate PQ parameters corresponding to the information of the terminal device in the PQ parameter file according to the PQ algorithm model and the encapsulated video file; The PQ parameters are sent to the video distribution source station, so that the video distribution source station updates the PQ parameters into the PQ parameter file.

7. A video enhancement device, characterized in that: Set in the terminal device, the video enhancement device includes: A video playback module is used to obtain a video playback request, wherein the video playback request carries information about the video to be played; download the video file to be played from a video distribution source station according to the information about the video to be played; and interpret the primary index file of the video file to obtain a secondary index address and a perceptual quantization PQ parameter file address of the video to be played; and download the bitstream data of the video to be played from the video distribution source station according to the secondary index address; A PQ parameter file interpretation module, configured to download the PQ parameter file from the video distribution source station according to the PQ parameter file address; and obtain PQ parameters corresponding to the terminal device according to the PQ parameter file; The video enhancement module is used to perform a video enhancement operation according to the PQ parameter and the bit stream data to obtain enhanced bit stream data.

8. A video enhancement device, characterized in that: Set in the server, the video enhancement device includes: A media encoding module, used to obtain the recorded video source file and perform transcoding operation on the video source file; A stream slicing module is used to perform a stream slicing operation on the video file after the transcoding operation, encapsulate the sliced ​​video file, obtain the encapsulated video file, wherein the encapsulated video file includes a primary index file; and construct a perceptual quantization PQ parameter file corresponding to the sliced ​​video file; wherein the PQ parameter file includes information of the terminal device; and add a label and address of the PQ parameter file to the primary index file; The sending module is used to send the encapsulated video file and the PQ parameter file to the video distribution source station.

9. A terminal device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the terminal device, enable the terminal device to execute the method according to any one of claims 1 to 4.

10. A server, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the server, enable the server to perform the method according to any one of claims 5-6.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 4.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 5 to 6.

Images