Terminal device, video enhancement method, computer device and medium

Abstract

Embodiments of the present application disclose a terminal device, a video enhancement method, a computer device and a medium. In a specific embodiment, the terminal device comprises: an acquisition module configured to acquire video data; a decoding module configured to perform frame-by-frame decoding processing on the video data; a color enhancement module configured to perform color enhancement processing on each video frame obtained by the decoding processing in decoding order; a super-resolution module configured to perform super-resolution processing on each video frame after the color enhancement processing in decoding order; and an encoding module configured to perform encoding processing on each video frame after the super-resolution processing in decoding order; wherein the decoding module, the color enhancement module, the super-resolution module and the encoding module perform multi-thread concurrent processing based on a multi-thread pipeline processing framework, the decoding module calls resources of a CPU or a GPU, the color enhancement module calls resources of the GPU, the super-resolution module calls resources of an NPU, and the encoding module calls resources of the CPU or the GPU of the terminal device.

Description

Technical Field

[0001] This invention relates to the field of computer technology. More specifically, it relates to a terminal device, a video enhancement method, a computer device, and a medium. Background Technology

[0002] Currently, mobile phones, tablets, laptops, and other terminal devices can perform video enhancement to improve video quality and provide users with a better viewing experience. The inventors discovered that because user terminal devices often have limited data processing capabilities, they cannot simultaneously achieve both efficiency and effectiveness in video enhancement. If the video enhancement effect is reduced to accommodate the terminal device's data processing capabilities, the improvement in video quality is limited. Conversely, if the video enhancement effect is guaranteed, the efficiency is compromised, making real-time video quality enhancement impossible and resulting in latency and stuttering during playback. Therefore, existing video enhancement methods limit the ability of terminal devices to perform real-time and effective video enhancement, especially for terminal devices with limited data processing capabilities. Summary of the Invention

[0003] The purpose of this invention is to provide a terminal device, a video enhancement method, a computer device, and a medium to solve at least one of the problems existing in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] The first aspect of the present invention provides a terminal device, comprising:

[0006] The acquisition module is used to acquire video data;

[0007] The decoding module is used to perform frame-by-frame decoding processing on the video data;

[0008] The color enhancement module is used to enhance the color of each video frame obtained from the decoding process according to the decoding process order;

[0009] The super-resolution module is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order.

[0010] The encoding module is used to encode each video frame processed by super-resolution according to the decoding processing order;

[0011] The decoding module, color enhancement module, super-resolution module, and encoding module are executed concurrently in a multi-threaded pipeline processing framework. The decoding module calls the resources of the central processing unit or graphics processing unit of the terminal device, the color enhancement module calls the resources of the graphics processing unit of the terminal device, the super-resolution module calls the resources of the neural network processor of the terminal device, and the encoding module calls the resources of the central processing unit or graphics processing unit of the terminal device.

[0012] Optionally, the decoding module, the color enhancement module, the super-resolution module, and the encoding module are configured to use semaphores for synchronization control in their multi-threaded concurrent execution processing based on a multi-threaded pipeline processing framework.

[0013] Optionally, for the synchronization control of the first and second modules among the decoding module, the color enhancement module, the super-resolution module, and the encoding module, semaphores are used. The terminal device allocates at least one memory address, which is configured with a read semaphore and a write semaphore. The first module is used to write the processed video frame data to the memory address according to the write semaphore, and the second module is used to read the video frame data in the memory address according to the read semaphore.

[0014] Optionally, the read semaphore is initially in a waiting state, and the write semaphore is initially in a writable state. The first module is used to write the processed video frame data to the memory address according to the write semaphore, and the second module is used to read the video frame data in the memory address according to the read semaphore, including:

[0015] The first module is used to write the processed video frame data to the memory address when the wait function is called to determine that the write semaphore of the memory address is in a writable state, and to call the release function to set the read semaphore of the memory address to a readable state after the processed video frame data is written to the memory address. The wait function sets the write semaphore of the memory address to a wait state when the first module writes the processed video frame data to the memory address.

[0016] The second module is used to read video frame data in the memory address when the wait function is called to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data in the memory address, to call the release function to set the write semaphore of the memory address to a writable state. In this case, when the second module reads the video frame data in the memory address, the wait function sets the read semaphore of the memory address to a wait state.

[0017] Optionally, the first module is the decoding module and the second module is the color enhancement module, or the first module is the color enhancement module and the second module is the super-resolution module, or the first module is the super-resolution module and the second module is the encoding module.

[0018] Optionally, the color enhancement module is used to perform color enhancement processing on each obtained video frame according to the decoding processing order based on off-screen rendering technology.

[0019] Optionally, the super-resolution module is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order based on the deep learning inference framework.

[0020] A second aspect of the present invention provides a video enhancement method applied to a terminal device, the method comprising:

[0021] Acquire video data;

[0022] The video data is decoded frame by frame, and each video frame obtained from the decoding process is sequentially subjected to color enhancement, super-resolution processing, and encoding processing according to the decoding processing order. The decoding, color enhancement, super-resolution processing, and encoding processing are executed concurrently in a multi-threaded pipeline processing framework. The decoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device, the color enhancement process calls on the resources of the graphics processing unit of the terminal device, the super-resolution processing calls on the resources of the neural network processor of the terminal device, and the encoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device.

[0023] Optionally, the decoding process, the color enhancement process, the super-resolution process, and the encoding process are executed concurrently in a multi-threaded manner based on a multi-threaded pipeline processing framework, and synchronization is controlled by semaphores.

[0024] Optionally, for the first and second processes in the decoding process, the color enhancement process, the super-resolution process, and the encoding process, semaphore synchronization control is adopted. The terminal device allocates at least one memory address, which is configured with a read semaphore and a write semaphore. A first module for performing the first process writes the processed video frame data to the memory address according to the write semaphore, and a second module for performing the second process reads the video frame data in the memory address according to the read semaphore.

[0025] Optionally, the read semaphore is initially in a waiting state, the write semaphore is initially in a writable state, the first module for performing the first processing writes the processed video frame data to the memory address according to the write semaphore, and the second module for performing the second processing reads the video frame data from the memory address according to the read semaphore, including:

[0026] The first module for performing the first processing writes the first processed video frame data to the memory address when it calls the wait function to determine that the write semaphore of the memory address is in a writable state, and after writing the first processed video frame data to the memory address, it calls the release function to set the read semaphore of the memory address to a readable state. The wait function sets the write semaphore of the memory address to a wait state when the first processed video frame data is written to the memory address.

[0027] The second module for performing the second processing reads video frame data from the memory address when it calls the wait function to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data from the memory address, it calls the release function to set the write semaphore of the memory address to a writable state. The wait function sets the read semaphore of the memory address to a wait state when reading the video frame data from the memory address.

[0028] Optionally, the first process is the decoding process and the second process is the color enhancement process; or, the first process is the color enhancement process and the second process is the super-resolution process; or, the first process is the super-resolution process and the second process is the encoding process.

[0029] A third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video enhancement method provided in the second aspect of the present invention.

[0030] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video enhancement method provided in the second aspect of the present invention.

[0031] The beneficial effects of this invention are as follows:

[0032] The technical solution described in this invention enhances video data on terminal devices through multi-threaded concurrency using a multi-threaded pipeline processing framework. This allows for the simultaneous and orderly processing of multiple video frames at different stages, eliminating the need to wait until one frame is fully processed before moving on to the next. This improves efficiency while maintaining the desired video enhancement effect. Specifically, color enhancement improves color saturation, corrects dullness, and delivers a more vibrant and bright picture. Super-resolution processing (super-stretching) increases video resolution, enhancing detail and making it more suitable for large screens. The design of performing color enhancement before super-resolution processing on video frames avoids increasing the processing load of color enhancement, ensuring efficiency and speed. Furthermore, considering the characteristics of each processing stage and the individual features of the CPU, GPU, and NPU, this invention employs a multi-threaded pipelined processing approach. Decoding utilizes CPU or GPU resources, color enhancement utilizes GPU resources, super-resolution utilizes NPU resources, and encoding utilizes CPU or GPU resources. This fully leverages the resources of the terminal device's CPU, GPU, and NPU, effectively freeing up CPU resources and avoiding excessive consumption. While ensuring video enhancement effects and improving efficiency, it also maintains the terminal device's operating speed. The technical solution described in this invention is particularly suitable for terminal devices with limited data processing capabilities. Attached Figure Description

[0033] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] Figure 1 This diagram illustrates a multi-threaded pipeline processing framework for video enhancement in a terminal device provided by an embodiment of the present invention.

[0035] Figure 2 This diagram illustrates color enhancement processing based on off-screen rendering technology.

[0036] Figure 3 This diagram illustrates the synchronization control of the color enhancement module in a synchronization control system employing semaphores.

[0037] Figure 4 A schematic diagram of the structure of a computer system implementing the terminal device provided in the embodiments of the present invention is shown. Detailed Implementation

[0038] To more clearly illustrate the present invention, the following description, in conjunction with embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0039] Currently, mobile phones, tablets, laptops, and other terminal devices can perform video enhancement to improve video quality and provide users with a better viewing experience. The inventors discovered that because user terminal devices often have limited data processing capabilities, they cannot simultaneously achieve both efficiency and effectiveness in video enhancement. If the video enhancement effect is reduced to accommodate the terminal device's data processing capabilities, the improvement in video quality is limited. Conversely, if the video enhancement effect is guaranteed, the efficiency is compromised, making real-time video quality enhancement impossible and resulting in latency and stuttering during playback. Therefore, existing video enhancement methods limit the ability of terminal devices to perform real-time and effective video enhancement, especially for terminal devices with limited data processing capabilities.

[0040] In view of this, such as Figure 1 As shown, one embodiment of the present invention provides a terminal device, including:

[0041] Acquisition module 101 is used to acquire video data;

[0042] Decoding module 102 is used to perform frame-by-frame decoding processing on the video data;

[0043] Color enhancement module 103 is used to perform color enhancement processing on each video frame obtained by decoding according to the decoding processing order;

[0044] The super-resolution module 104 is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order.

[0045] Encoding module 105 is used to encode each super-resolution processed video frame according to the decoding processing order;

[0046] The decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 are executed concurrently in a multi-threaded pipeline processing framework. The decoding module 102 calls the resources of the terminal device's central processing unit or graphics processing unit, the color enhancement module 103 calls the resources of the terminal device's graphics processing unit, the super-resolution module 104 calls the resources of the terminal device's neural network processor, and the encoding module 105 calls the resources of the terminal device's central processing unit or graphics processing unit.

[0047] The terminal device provided in this embodiment enhances video data through multi-threaded concurrency based on a multi-threaded pipeline processing framework. The decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 can process multiple video frames at different stages simultaneously in the order of the video frame sequence. Each module in the data concatenated decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 can immediately receive data from the previous module for processing after it becomes idle, without having to wait until all video frames have been processed before starting processing the next video frame. This can improve the efficiency of video enhancement while ensuring the video enhancement effect.

[0048] The color enhancement module 103 performs color enhancement processing to improve the color saturation of the video image, changing the dullness of the image quality and making the user experience a more vivid and bright picture. The super-resolution module 104 performs super-resolution processing (super-stretching) to improve the resolution of the video, making its details clearer and more suitable for playback on large screens. The design of performing color enhancement processing by the color enhancement module 103 first and then super-resolution processing by the super-resolution module 104 can avoid increasing the processing load of the color enhancement module 103, ensuring the efficiency and speed of color enhancement processing. For example, for a 360p resolution video frame, the color enhancement module 103 performs color enhancement processing first and then the super-resolution module 104 super-stretches it to 1080p. The color enhancement module 103 is designed for 360p video frames, which avoids increasing the processing load of the color enhancement module 103 caused by super-stretching first and then color enhancement. The enhancement effect of the method of color enhancement first and then super-stretching is guaranteed. For example, the color enhancement module 103 can enhance or adjust at least one of the contrast, saturation, brightness, and color vibrancy of a video frame. Different parameters can be set according to actual needs (such as playback scene requirements, playback content requirements, or user requirements) to achieve different color enhancement effects.

[0049] Furthermore, this embodiment comprehensively considers the characteristics of each stage of processing and the characteristics of the central processing unit, graphics processing unit, and neural network processor. It adopts a method in which the decoding module 102 calls the resources of the central processing unit or graphics processing unit, the color enhancement module 103 calls the resources of the graphics processing unit, the super-resolution module 104 calls the resources of the neural network processor, and the encoding module 105 calls the resources of the central processing unit or graphics processing unit. Combined with a multi-threaded pipeline processing method, it makes full use of the resources of the terminal device's CPU, GPU, and NPU. The CPU can only complete the data synchronization and transmission between the GPU, CPU, and NPU and the scheduling work between multiple threads, effectively releasing CPU resources and avoiding excessive consumption of CPU resources. While ensuring the video enhancement effect, it improves the efficiency of video enhancement and ensures the operating speed of the terminal device.

[0050] This embodiment is particularly suitable for terminal devices with low data processing capabilities.

[0051] For example, the terminal device in this embodiment can be a mobile phone, tablet computer, laptop computer, etc., and various client applications can be installed on the terminal device, such as video playback applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0052] In a specific example, the video data acquired by the acquisition module 101 comes from a video stream or video file, or directly from the camera capturing the video. The encoding module 105 encodes the video frames according to the decoding processing order, which can form a video stream or video file, or be directly rendered and played on the display screen.

[0053] In a specific example, assuming that the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 process a video frame for a similar or even the same duration, then the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 perform multi-threaded concurrent processing based on a multi-threaded pipeline processing framework. For example:

[0054] After acquiring the video data, the acquisition module 101 sends it to the decoding module 102;

[0055] After decoding the first video frame, the decoding module 102 sends it to the color enhancement module 103. The color enhancement module 103 receives the first video frame, performs color enhancement processing on the first video frame, and then sends it to the super-resolution module 104. At the same time that the color enhancement module 103 receives the first video frame and performs color enhancement processing on the first video frame, the decoding module 102 decodes the second video frame.

[0056] The super-resolution module 104 receives the first video frame and performs super-resolution processing on the first video frame before sending it to the encoding module 105. While the super-resolution module 104 receives the first video frame and performs super-resolution processing on the first video frame, the decoding module 102 decodes the third video frame and the color enhancement module 103 receives the second video frame and performs color enhancement processing on the second video frame.

[0057] Encoding module 105 receives the first video frame and encodes it. While encoding module 105 receives the first video frame and encodes it, decoding module 102 decodes the fourth video frame, color enhancement module 103 receives the third video frame and performs color enhancement on it, and super-resolution module 104 receives the second video frame and performs super-resolution on it.

[0058] And so on.

[0059] As can be seen, in this example, the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 adopt a pipelined parallel mechanism. During the processing of video frames according to their order in the video frame sequence, at the same time, the decoding module 102 processes the nth video frame, the color enhancement module 103 processes the (n-1)th video frame, the super-resolution module 104 processes the (n-2)th video frame, and the encoding module 105 processes the (n-3)th video frame, greatly improving video enhancement efficiency. Furthermore, in practice, the processing speeds of the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 may differ. Therefore, the overall processing speed is determined by the module with the longest processing time among the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105. Synchronous control of the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 can be achieved through a designed data waiting and transmission mechanism, which will be explained in detail later.

[0060] In a specific example, when the encoding module 105 calls the resources of the GPU, the video frames processed by the super-resolution module 104, which calls the resources of the NPU, are first sent to the CPU and then forwarded by the CPU to the encoding module 105 that calls the resources of the GPU.

[0061] In one possible implementation, the color enhancement module 103 is used to perform color enhancement processing on each video frame obtained according to the decoding processing order based on off-screen rendering technology.

[0062] In a specific example, to efficiently utilize GPU resources, this implementation employs an OpenGL (Open Graphics Library) acceleration framework. OpenGL is generally used for rendering and playback; in this embodiment, to render video frames and copy them for the super-resolution module 104 to perform further super-resolution processing, an off-screen rendering technique based on FBO (Frame Buffer Object) is used. The processing flow is as follows: Figure 2 As shown, according to the method of this embodiment, it is possible to render to an FBO using OpenGL shaders and then copy the data out of the FBO.

[0063] In one possible implementation, the super-resolution module 104 is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order based on a deep learning inference framework.

[0064] For example, the terminal device is equipped with a Rockchip RK3399Pro chip, which includes the RKNN deep learning inference framework. The super-resolution module 104 is used to perform super-resolution processing on video frames according to the RKNN deep learning inference framework. The RKNN deep learning inference framework can quantize AI (Artificial Intelligence) models to INT8 (meaning that one data point occupies 8 bits, and the data consumes fewer resources) for model inference, thereby improving inference speed.

[0065] Continuing with the previous example, the processing speed of color enhancement performed on the CPU in existing technologies is approximately 0.2 seconds. In this embodiment, the color enhancement processing speed of the OpenGL off-screen rendering technology can reach 0.03 seconds. The super-resolution module 104 can compress the model according to actual needs, achieving a super-resolution processing speed of 0.04 seconds under the RKNN deep learning inference framework.

[0066] In one possible implementation, the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 are configured to perform multi-threaded concurrent execution based on a multi-threaded pipeline processing framework, and are synchronized using semaphores.

[0067] In one possible implementation, semaphore synchronization control is used between the first and second modules of the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105. The terminal device allocates at least one memory address, which is configured with a read semaphore and a write semaphore. The first module is used to write processed video frame data to the memory address according to the write semaphore, and the second module is used to read video frame data from the memory address according to the read semaphore.

[0068] In one possible implementation, the read semaphore is initially in a waiting state, and the write semaphore is initially in a writable state. The first module is used to write the processed video frame data to the memory address according to the write semaphore, and the second module is used to read the video frame data from the memory address according to the read semaphore, including:

[0069] The first module is used to write the processed video frame data to the memory address when the wait function is called to determine that the write semaphore of the memory address is in a writable state, and to call the release function to set the read semaphore of the memory address to a readable state after the processed video frame data is written to the memory address. The wait function sets the write semaphore of the memory address to a wait state when the first module writes the processed video frame data to the memory address.

[0070] The second module is used to read video frame data in the memory address when the wait function is called to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data in the memory address, to call the release function to set the write semaphore of the memory address to a writable state. In this case, when the second module reads the video frame data in the memory address, the wait function sets the read semaphore of the memory address to a wait state.

[0071] In one possible implementation, the first module is a decoding module 102 and the second module is a color enhancement module 103, or the first module is a color enhancement module 103 and the second module is a super-resolution module 104, or the first module is a super-resolution module 104 and the second module is an encoding module 105.

[0072] In a specific example, combining the above implementation methods, the data waiting and transmission mechanism implemented by the semaphore synchronization control of the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 is as follows:

[0073] A first memory address var1 is allocated to the decoding module 102 and the color enhancement module 103. The first memory address var1 serves as both the output memory address of the decoding module 102 and the input memory address of the color enhancement module 103. A second memory address var2 is allocated to the color enhancement module 103 and the super-resolution module 104. The second memory address var2 serves as both the output memory address of the color enhancement module 103 and the input memory address of the super-resolution module 104. A third memory address var3 is allocated to the super-resolution module 104 and the encoding module 105. The third memory address var3 serves as both the output memory address of the super-resolution module 104 and the input memory address of the encoding module 105.

[0074] The first memory address var1, the second memory address var2, and the third memory address var3 are each controlled by two semaphores: the read semaphore ReadSem and the write semaphore WriteSem. The initial value of the read semaphore ReadSem is 0, and the initial value of the write semaphore WriteSem is 1.

[0075] The `sem_wait` function checks the value of a semaphore (both for reading and writing): if the semaphore value is 0, the thread waits (or is blocked); if the semaphore value is 1, the thread starts, and `sem_wait` decrements the semaphore value by 1. The `sem_post` function increments the semaphore value by 1.

[0076] Based on the above, the data processing of the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105, as well as the synchronization control process between the modules using semaphores, is as follows:

[0077] After the acquisition module 101 acquires the video data and sends it to the decoding module 102, the decoding module 102 decodes the data n1 of the first video frame. At this time, the write semaphore WriteSem at the first memory address var1 is initially set to 1. The decoding module 102 calls the wait function sem_wait to determine that the value of the write semaphore WriteSem at the first memory address var1 is 1, and writes the data n1 of the first video frame to the first memory address var1. At the same time, the wait function sem_wait decrements the value of the write semaphore WriteSem at the first memory address var1 by 1 to 0. After writing the data n1 of the first video frame to the first memory address var1, the decoding module 102 calls the release function sem_post to increment the value of the read semaphore ReadSem at the first memory address var1 by 1, so that the read semaphore ReadSem at the first memory address var1 changes from its initial value of 0 to 1, so that the color enhancement module 103 can know that it can read data from the first memory address var1 by calling the wait function sem_wait.

[0078] like Figure 3As shown, the color enhancement module 103 calls the wait function sem_wait to determine the value of the read semaphore ReadSem at the first memory address var1. When it is determined that the value of the read semaphore ReadSem at the first memory address var1 is 1, the color enhancement module 103 reads the data n1 of the first video frame in the first memory address var1. At the same time, the wait function sem_wait decrements the value of the read semaphore ReadSem at the first memory address var1 by 1 to 0. After reading the data n1 of the first video frame from the first memory address var1, the color enhancement module 103 calls the release function sem_post to increment the value of the write semaphore WriteSem at the first memory address var1 by 1 to 1, so that the decoding module 102 can know that it can write data to the first memory address var1 by calling the wait function sem_wait.

[0079] The color enhancement module 103 performs color enhancement processing on the data n1 of the first video frame to obtain the color-enhanced data n of the first video frame. 1` At this point, the write semaphore WriteSem at the second memory address var2 is initialized to 1. The color enhancement module 103 calls the wait function sem_wait to determine that the value of the write semaphore WriteSem at the second memory address var2 is 1, and then processes the data n of the first video frame after color enhancement. 1` Write to the second memory address var2, and simultaneously wait for the function sem_wait to decrement the value of the write semaphore WriteSem at the second memory address var2 to 0. Then, process the data n of the first video frame after color enhancement. 1` After writing to the second memory address var2, the color enhancement module 103 calls the release function sem_post to increment the value of the read semaphore ReadSem of the second memory address var2 by 1, so that the read semaphore ReadSem of the second memory address var2 changes from the initial value 0 to 1, so that the super-resolution module 104 can know that it can read data from the second memory address var2 by calling the wait function sem_wait.

[0080] The super-resolution module 104 calls the wait function sem_wait to check the value of the read semaphore ReadSem at the second memory address var2. When the value of the read semaphore ReadSem at the second memory address var2 is found to be 1, the super-resolution module 104 reads the data n of the first video frame after color enhancement from the second memory address var2. 1` Meanwhile, the waiting function sem_wait decrements the value of the read semaphore ReadSem at the second memory address var2 by 1, making it 0, and then processes the data n of the first video frame after color enhancement. 1`After reading from the second memory address var2, the super-resolution module 104 calls the release function sem_post to increment the value of the write semaphore WriteSem of the second memory address var2 by 1, so that the color enhancement module 103 can know that it can write data to the second memory address var2 by calling the wait function sem_wait.

[0081] Super-resolution module 104 processes the data n of the first video frame after color enhancement. 1` Perform super-resolution processing to obtain the data n of the first super-resolution video frame. 1`` At this point, the write semaphore WriteSem at the third memory address var3 is initialized to 1. The super-resolution module 104 calls the wait function sem_wait to determine that the value of the write semaphore WriteSem at the third memory address var3 is 1, and then transfers the data n of the first video frame after super-resolution. 1`` Write to the third memory address var3, and simultaneously wait for the function sem_wait to decrement the value of the write semaphore WriteSem at the third memory address var3 to 0. Then, write the data n of the first video frame after super-resolution. 1`` After writing to the third memory address var3, the super-resolution module 104 calls the release function sem_post to increment the value of the read semaphore ReadSem at the third memory address var3 by 1, so that the read semaphore ReadSem at the third memory address var3 changes from the initial value 0 to 1, so that the encoding module 105 can know that it can read data from the third memory address var3 by calling the wait function sem_wait.

[0082] Encoding module 105 calls the wait function sem_wait to check the value of the read semaphore ReadSem at the third memory address var3. When the value of the read semaphore ReadSem at the third memory address var3 is found to be 1, encoding module 105 reads the data n of the first super-resolution video frame from the third memory address var3. 1`` Encoding is performed, and the waiting function sem_wait decrements the value of the read semaphore ReadSem at the third memory address var3 to 0, before transferring the data n of the first video frame after super-resolution. 1`` After reading from the third memory address var3, the encoding module 105 calls the release function sem_post to increment the value of the write semaphore WriteSem of the third memory address var3 by 1, so that the super-resolution module 104 can know that it can write data to the third memory address var3 by calling the wait function sem_wait.

[0083] And so on.

[0084] As can be seen, in this example, the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 adopt a pipelined parallel mechanism. During the process of processing video frames in the order of the video frame sequence, the decoding module 102, color enhancement module 103, super-resolution module 104, and encoding module 105 can process different video frames at the same time. For example, at the same time, the decoding module 102 processes the nth video frame, the color enhancement module 103 processes the (n-1)th video frame, the super-resolution module 104 processes the (n-2)th video frame, and the encoding module 105 processes the (n-3)th video frame, which greatly improves the video enhancement efficiency.

[0085] Another embodiment of the present invention provides a video enhancement method applied to a terminal device, the method comprising:

[0086] Acquire video data;

[0087] The video data is decoded frame by frame, and each video frame obtained from the decoding process is sequentially subjected to color enhancement, super-resolution processing, and encoding processing according to the decoding processing order. The decoding, color enhancement, super-resolution processing, and encoding processing are executed concurrently in a multi-threaded pipeline processing framework. The decoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device, the color enhancement process calls on the resources of the graphics processing unit of the terminal device, the super-resolution processing calls on the resources of the neural network processor of the terminal device, and the encoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device.

[0088] In one possible implementation, the decoding process, the color enhancement process, the super-resolution process, and the encoding process are executed concurrently in a multi-threaded pipelined processing framework and synchronized using semaphores.

[0089] In one possible implementation, semaphore synchronization control is employed between the first and second processes in the decoding process, the color enhancement process, the super-resolution process, and the encoding process. The terminal device allocates at least one memory address, which is configured with a read semaphore and a write semaphore. A first module for performing the first process writes the processed video frame data to the memory address according to the write semaphore, and a second module for performing the second process reads the video frame data from the memory address according to the read semaphore.

[0090] In one possible implementation, the read semaphore is initially in a waiting state, the write semaphore is initially in a writable state, the first module for performing the first processing writes the processed video frame data to the memory address according to the write semaphore, and the second module for performing the second processing reads the video frame data from the memory address according to the read semaphore, including:

[0091] The first module for performing the first processing writes the first processed video frame data to the memory address when it calls the wait function to determine that the write semaphore of the memory address is in a writable state, and after writing the first processed video frame data to the memory address, it calls the release function to set the read semaphore of the memory address to a readable state. The wait function sets the write semaphore of the memory address to a wait state when the first processed video frame data is written to the memory address.

[0092] The second module for performing the second processing reads video frame data from the memory address when it calls the wait function to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data from the memory address, it calls the release function to set the write semaphore of the memory address to a writable state. The wait function sets the read semaphore of the memory address to a wait state when reading the video frame data from the memory address.

[0093] In one possible implementation, the first process is the decoding process and the second process is the color enhancement process, or the first process is the color enhancement process and the second process is the super-resolution process, or the first process is the super-resolution process and the second process is the encoding process.

[0094] In one possible implementation, the color enhancement process includes color enhancement based on off-screen rendering technology.

[0095] In one possible implementation, the super-resolution processing includes performing super-resolution processing based on a deep learning inference framework.

[0096] It should be noted that the video enhancement method provided in this embodiment is similar in principle and workflow to the video enhancement implemented by the terminal device provided in the above embodiments. The relevant parts can be referred to the above description, and will not be repeated here.

[0097] like Figure 4As shown, a computer system suitable for implementing a terminal device such as a laptop computer provided in the above embodiments includes a central processing module (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM). Various programs and data required for the operation of the computer system are also stored in the RAM. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0098] The following components are connected to the I / O interface: input sections including keyboards, mice, etc.; output sections including liquid crystal displays (LCDs) and speakers, etc.; storage sections including hard disks, etc.; and communication sections including network interface cards such as LAN cards and modems. The communication sections perform communication processing via networks such as the Internet. Drives are also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required.

[0099] Specifically, according to this embodiment, the process described in the flowchart above can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.

[0100] The flowcharts and schematic diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product of this embodiment. In this regard, each block in the flowchart or schematic diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the schematic diagram and / or flowchart, and combinations of blocks in the schematic diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] In another aspect, this embodiment also provides a non-volatile computer storage medium. This non-volatile computer storage medium can be the non-volatile computer storage medium included in the above-described device in the above embodiments, or it can be a separate non-volatile computer storage medium not assembled into the terminal. The non-volatile computer storage medium stores one or more programs. When the one or more programs are executed by a device, the device: acquires video data; performs frame-by-frame decoding processing on the video data, and sequentially performs color enhancement processing, super-resolution processing, and encoding processing on each video frame obtained by the decoding processing according to the decoding processing order. The decoding processing, color enhancement processing, super-resolution processing, and encoding processing are executed concurrently using a multi-threaded pipeline processing framework. The decoding processing calls resources from the central processing unit or graphics processing unit of the terminal device; the color enhancement processing calls resources from the graphics processing unit of the terminal device; the super-resolution processing calls resources from the neural network processor of the terminal device; and the encoding processing calls resources from the central processing unit or graphics processing unit of the terminal device.

[0102] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0103] It should also be noted that in the description of this invention, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A terminal device, characterized in that, include: The acquisition module is used to acquire video data; The decoding module is used to perform frame-by-frame decoding processing on the video data; The color enhancement module is used to enhance the color of each video frame obtained from the decoding process according to the decoding process order; The super-resolution module is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order. The encoding module is used to encode each video frame processed by super-resolution according to the decoding processing order; The decoding module, color enhancement module, super-resolution module, and encoding module are executed concurrently in a multi-threaded pipeline processing framework. The decoding module calls the resources of the central processing unit or graphics processing unit of the terminal device, the color enhancement module calls the resources of the graphics processing unit of the terminal device, the super-resolution module calls the resources of the neural network processor of the terminal device, and the encoding module calls the resources of the central processing unit or graphics processing unit of the terminal device. The super-resolution module is used to perform super-resolution processing on each color-enhanced video frame according to the decoding processing order based on the deep learning inference framework; the deep learning inference framework is the RKNN deep learning inference framework. The color enhancement module is used to perform color enhancement processing on each video frame obtained according to the decoding processing order based on off-screen rendering technology; The decoding module, the color enhancement module, the super-resolution module, and the encoding module are configured to use semaphores for synchronization control in their multi-threaded concurrent execution based on a multi-threaded pipeline processing framework. For the synchronization control of the first and second modules among the decoding module, the color enhancement module, the super-resolution module and the encoding module, the terminal device allocates at least one memory address, the memory address is configured with a read semaphore and a write semaphore, the first module is used to write the processed video frame data to the memory address according to the write semaphore, and the second module is used to read the video frame data in the memory address according to the read semaphore; The memory addresses include: a first memory address allocated to the decoding module and the color enhancement module, which also serves as the output memory address of the decoding module and the input memory address of the color enhancement module; a second memory address allocated to the color enhancement module and the super-resolution module, which also serves as the output memory address of the color enhancement module and the input memory address of the super-resolution module; and a third memory address allocated to the super-resolution module and the encoding module, which also serves as the output memory address of the super-resolution module and the input memory address of the encoding module.

2. The terminal device according to claim 1, characterized in that, The read semaphore is initially in a waiting state, and the write semaphore is initially in a writable state. The first module is used to write the processed video frame data to the memory address according to the write semaphore. The second module is used to read the video frame data from the memory address according to the read semaphore, including: The first module is used to write the processed video frame data to the memory address when the wait function is called to determine that the write semaphore of the memory address is in a writable state, and to call the release function to set the read semaphore of the memory address to a readable state after the processed video frame data is written to the memory address. The wait function sets the write semaphore of the memory address to a wait state when the first module writes the processed video frame data to the memory address. The second module is used to read video frame data in the memory address when the wait function is called to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data in the memory address, to call the release function to set the write semaphore of the memory address to a writable state. In this case, when the second module reads the video frame data in the memory address, the wait function sets the read semaphore of the memory address to a wait state.

3. The terminal device according to claim 1 or 2, characterized in that, The first module is the decoding module and the second module is the color enhancement module, or the first module is the color enhancement module and the second module is the super-resolution module, or the first module is the super-resolution module and the second module is the encoding module.

4. A video enhancement method using the terminal device according to any one of claims 1-3, characterized in that, Applied to a terminal device, the method includes: Acquire video data; The video data is decoded frame by frame, and each video frame obtained from the decoding process is sequentially subjected to color enhancement, super-resolution processing, and encoding processing according to the decoding processing order. The decoding process, color enhancement process, super-resolution processing, and encoding processing are executed concurrently in a multi-threaded pipeline processing framework. The decoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device, the color enhancement process calls on the resources of the graphics processing unit of the terminal device, the super-resolution processing calls on the resources of the neural network processor of the terminal device, and the encoding process calls on the resources of the central processing unit or graphics processing unit of the terminal device. The decoding process, the color enhancement process, the super-resolution process, and the encoding process are executed concurrently in a multi-threaded pipeline processing framework and synchronized using semaphores. For the first and second processes in the decoding process, the color enhancement process, the super-resolution process and the encoding process, semaphore synchronization control is adopted. The terminal device allocates at least one memory address. The memory address is configured with a read semaphore and a write semaphore. The first module for executing the first process writes the processed video frame data into the memory address according to the write semaphore. The second module for executing the second process reads the video frame data in the memory address according to the read semaphore. The memory addresses include: a first memory address allocated to the decoding module and the color enhancement module, which also serves as the output memory address of the decoding module and the input memory address of the color enhancement module; a second memory address allocated to the color enhancement module and the super-resolution module, which also serves as the output memory address of the color enhancement module and the input memory address of the super-resolution module; and a third memory address allocated to the super-resolution module and the encoding module, which also serves as the output memory address of the super-resolution module and the input memory address of the encoding module.

5. The method according to claim 4, characterized in that, The read semaphore is initially in a waiting state, and the write semaphore is initially in a writable state. The first module for performing the first processing writes the processed video frame data to the memory address according to the write semaphore. The second module for performing the second processing reads the video frame data from the memory address according to the read semaphore, including: The first module for performing the first processing writes the first processed video frame data to the memory address when it calls the wait function to determine that the write semaphore of the memory address is in a writable state, and after writing the first processed video frame data to the memory address, it calls the release function to set the read semaphore of the memory address to a readable state. The wait function sets the write semaphore of the memory address to a wait state when the first processed video frame data is written to the memory address. The second module for performing the second processing reads video frame data from the memory address when it calls the wait function to determine that the read semaphore of the memory address is in a readable state, and after reading the video frame data from the memory address, it calls the release function to set the write semaphore of the memory address to a writable state. The wait function sets the read semaphore of the memory address to a wait state when reading the video frame data from the memory address.

6. The method according to claim 4 or 5, characterized in that, The first process is the decoding process and the second process is the color enhancement process, or the first process is the color enhancement process and the second process is the super-resolution process, or the first process is the super-resolution process and the second process is the encoding process.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 4-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 4-6.

Images