A method, device, medium and electronic equipment for mobile terminal video quality enhancement

Abstract

The application discloses a mobile terminal video quality enhancement method, device, medium and electronic equipment. First, the image of the first frame of the video is acquired as a first image, and the first image is input into a pre-trained image enhancement model to acquire a first enhanced image output by the image enhancement model. The image of the next frame is acquired as a second image, and the first image and the second image are input into a pre-trained optical flow estimation model to acquire a dense optical flow and an optical flow evaluation value output by the optical flow estimation model. If the optical flow evaluation value exceeds a threshold value, the first enhanced image, the second image and the dense optical flow are input into a pre-trained optical flow enhancement model to determine a second enhanced image. Then, the second image is taken as the first image, the second enhanced image is taken as the first enhanced image, the image of the next frame is acquired, the image of the next frame is taken as the second image, the above process of optimizing the second image is repeated, and the process is stopped until all images are optimized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, medium and electronic device for enhancing video quality on a mobile device. Background Technology

[0002] To improve the user experience of watching videos, video enhancement technologies are often used to improve the clarity of the videos viewed by users. However, improving video clarity requires the use of complex neural networks, which consume a lot of computing resources, making it difficult to improve video clarity on the client devices used by users.

[0003] Currently, the main way to improve the clarity of videos viewed by users is to upload the video to a cloud server, which then uses a complex neural network to improve the video's clarity before sending the improved video to the user's client.

[0004] While this method can improve the clarity of the video seen by the user, it requires uploading the video to a cloud server first, and then downloading the video from the cloud server. Due to limitations in the client's network status and the cloud server's load status, this may cause video playback to stutter.

[0005] Therefore, how to improve video clarity while ensuring smooth video playback is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, medium, and electronic device for enhancing video quality on a mobile device, thereby partially solving the technical problem of improving video clarity while ensuring smooth video playback.

[0007] The technical solution adopted in this application is as follows:

[0008] This application provides a method for enhancing video quality on mobile devices, including:

[0009] The image of the first frame of the video is acquired as the first image, and the first image is input into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model;

[0010] The image of the next frame is acquired as the second image, and the first image and the second image are input into a pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model;

[0011] If the optical flow evaluation value exceeds the threshold, the first enhanced image, the second image, and the dense optical flow are input into a pre-trained optical flow enhancement model to determine the second enhanced image;

[0012] The second image is used as the first image, the second enhanced image is used as the first enhanced image, and the image of the next frame is obtained. The image of the next frame is used as the second image. The above process of optimizing the second image is repeated to obtain the newly determined enhanced image of the second image, until all images are optimized and then the process stops.

[0013] Optionally, if the optical flow evaluation value exceeds a threshold, the first enhanced image, the second image, and the dense optical flow are input into a pre-trained optical flow enhancement model to determine the second enhanced image, specifically including:

[0014] If the optical flow evaluation value does not exceed the threshold, the second image is input into the pre-trained image enhancement model to determine the second enhanced image.

[0015] Optionally, the image of the first frame of the video is acquired as the first image, specifically including:

[0016] Acquire the image of the first frame of the video and compress the image of the first frame of the video to an image of a preset resolution;

[0017] The compressed image is used as the first compressed image.

[0018] Optionally, training the optical flow enhancement model specifically includes:

[0019] Training samples are obtained, including: a real image of the third image, a real image of the fourth image, a degraded fourth image, and dense optical flow. The real images of the third image and the fourth image are obtained by extracting two consecutive frames from a high-definition video. The degraded fourth image is obtained by adding noise, blurring, and reducing the resolution of the real image of the fourth image. The dense optical flow is obtained by using a preset algorithm with the degraded third image and the degraded fourth image as input.

[0020] The real image of the third image, the fourth degraded image, and the dense optical flow are input into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model.

[0021] The optical flow enhancement model is trained with the optimization objective of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image.

[0022] Optionally, training the optical flow estimation model specifically includes:

[0023] Obtain training samples, wherein the training samples include: the third degraded image, the fourth degraded image, and the true value of the dense optical flow, wherein the true value of the dense optical flow is obtained by using the third degraded image and the fourth degraded image as input through a preset algorithm;

[0024] The third and fourth degraded images are input into the optical flow estimation model to be trained to obtain the predicted dense optical flow output by the optical flow estimation model.

[0025] The optical flow enhancement model is trained with the optimization objective of minimizing the difference between the predicted dense optical flow and the true value of the dense optical flow.

[0026] Optionally, training the optical flow estimation model and training the optical flow enhancement model specifically include:

[0027] Obtain training samples, wherein the training samples include: the real image of the third image, the third degraded image, the real image of the fourth image, and the fourth degraded image;

[0028] The third and fourth degraded images are input into the optical flow estimation model to be trained to obtain the predicted dense optical flow and the predicted optical flow score output by the optical flow estimation model.

[0029] The predicted dense optical flow, the real image of the third image, and the fourth degraded image are input into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model.

[0030] Based on the predicted fourth enhanced image and the true image of the fourth image, the true value of the optical flow evaluation value is determined;

[0031] The optical flow estimation model and the optical flow enhancement model are jointly trained with the optimization objectives of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image, and minimizing the predicted optical flow evaluation value and the real value of the optical flow evaluation value.

[0032] Optionally, based on the predicted fourth enhanced image and the true image of the fourth image, the true value of the optical flow evaluation value is determined, specifically including:

[0033] Based on the predicted fourth enhanced image and the real image of the fourth image, determine the average color similarity of pixels at each position in the predicted fourth enhanced image and the real image of the fourth image;

[0034] The true value of the optical flow evaluation value is determined based on the average color similarity.

[0035] This application provides a device for enhancing video quality on a mobile device, comprising:

[0036] The first acquisition module is used to acquire the image of the first frame of the video as the first image, and input the first image into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model;

[0037] The second acquisition module is used to acquire the image of the next frame as the second image, and input the first image and the second image into the pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model;

[0038] The determination module is used to input the first enhanced image, the second image, and the dense optical flow into a pre-trained optical flow enhancement model to determine the second enhanced image if the optical flow evaluation value exceeds a threshold.

[0039] The loop module is used to take the second image as the first image, the second enhanced image as the first enhanced image, and obtain the image of the next frame, take the image of the next frame as the second image, and repeat the above process of optimizing the second image to obtain the newly determined enhanced image of the second image, until all images are optimized and then stop.

[0040] This application provides a computer-readable medium storing a computer program that, when executed by a processor, implements the above-described method for enhancing video quality on a mobile device.

[0041] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for enhancing video quality on a mobile device.

[0042] The above-mentioned technical solution adopted in this application can achieve the following beneficial effects:

[0043] The mobile video quality enhancement method provided in this application first acquires the image of the first frame of the video as the first image, and inputs the first image into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model. Then, it acquires the image of the next frame as the second image, and inputs the first image and the second image into a pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model. If the optical flow evaluation value exceeds a threshold, the first enhanced image, the second image, and the dense optical flow are input into the pre-trained optical flow enhancement model to determine the second enhanced image. Then, the second image is used as the first image, the second enhanced image is used as the first enhanced image, and the image of the next frame is acquired and used as the second image. The above process of optimizing the second image is repeated to obtain the enhanced image of the newly determined second image, until all images are optimized and then the process stops.

[0044] As can be seen from the above method, the optical flow evaluation value obtained from the optical flow estimation model can be used to determine whether the images in the video are enhanced by an optical flow enhancement model or an image enhancement model. This allows for quality enhancement of most images in the video through a less computationally intensive method by controlling the threshold of the optical flow evaluation value and designing computationally efficient optical flow enhancement and estimation models. Therefore, the entire video enhancement process can be performed on the client side without uploading images to a cloud server, thus improving video clarity while avoiding playback stuttering caused by network factors. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a method for enhancing video quality on a mobile device provided in this application.

[0047] Figure 2 A schematic diagram illustrating the working principle of an image enhancement model provided in this application;

[0048] Figure 3 A schematic diagram illustrating the working principle of an optical flow enhancement model provided in this application;

[0049] Figure 4 A flowchart illustrating a mobile video quality enhancement method provided in this application is presented below.

[0050] Figure 5 A schematic diagram of a mobile video quality enhancement device provided in this application;

[0051] Figure 6 A corresponding to the present application Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a flowchart illustrating a method for enhancing video quality on a mobile device provided in this application, including the following steps:

[0055] S101: Obtain the image of the first frame of the video as the first image, and input the first image into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model.

[0056] As can be seen from the above description, the core innovation of the mobile video quality enhancement method provided in this application is to determine the optimization method of the current frame image by judging the optical flow evaluation value between the current frame image and the previous frame image. Therefore, in this application, the execution subject of the mobile video quality enhancement method can be a client set on the user's desktop computer, laptop computer, mobile phone and other terminal devices. For ease of description, the client is used as the execution subject of the mobile video quality enhancement method provided in this application.

[0057] Currently, the main way to increase video clarity is to upload images to a cloud server, where a neural network optimizes the uploaded images for higher resolution. The cloud server then sends the optimized image to the client. While this method improves video clarity, it is susceptible to buffering and stuttering issues due to limitations in the client's network connection.

[0058] Based on this, this application provides a method for enhancing video quality on mobile devices, in order to at least partially solve the above-mentioned problems.

[0059] First, after initialization, the client can obtain the image of the first frame of the video, which serves as the image of the current frame, i.e., the first image. It should be noted that this image can be an unprocessed video image. Of course, to reduce the computational burden on the client, the client can also compress the image to a preset resolution (e.g., 480*480) after obtaining it, thereby reducing the computational resources required for subsequent image processing, and using the compressed first image as the first compressed image.

[0060] After acquiring the first image, the client can input it into a pre-trained image augmentation model to obtain the first augmented image output by the model. It should be noted that since not all images require optimization using the image augmentation model, the training process can aim to achieve the highest possible image sharpness.

[0061] The following is combined with Figure 2 A schematic diagram illustrating the working principle of an image enhancement model provided in this application is provided below:

[0062] exist Figure 2 In this process, the client can input the compressed low-quality image of the current frame into the encoder of the image enhancement model. The encoder outputs the depth feature map of the image and inputs the depth feature map into the decoder. Then, the decoder obtains the enhanced image corresponding to the image based on the depth feature map.

[0063] In the method of training the image enhancement model in this application, the terminal device can first acquire some high-resolution images, and use a degradation algorithm to convert the high-resolution images into low-resolution images by adding noise, blurring, or reducing resolution. Then, the high-resolution images can be used as label values, and the low-resolution images as training samples to train the image enhancement model.

[0064] S102: Obtain the image of the next frame as the second image, and input the first image and the second image into the pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model.

[0065] After obtaining the first enhanced image, the client can obtain the image of the next frame as the second image. Similarly, the client can first compress the image of the next frame to a preset resolution and then use the compressed image as the second compressed image.

[0066] The client can input the first compressed image and the second compressed image into a pre-trained optical flow estimation model to obtain the dense optical flow output by the optical flow estimation model and the optical flow evaluation values ​​of the first compressed image and the second compressed image.

[0067] In this application, the method for training the optical flow estimation model can be as follows: the terminal device can first obtain training samples, wherein the training samples include: a third degraded image, a fourth degraded image, and the true value of dense optical flow and dense optical flow. The true value of dense optical flow is obtained by using the third degraded image and the fourth degraded image as input through a preset algorithm. The third degraded image and the fourth degraded image are obtained by adding noise, blurring, and reducing the resolution of the true images of the third image and the fourth image.

[0068] It should be noted that the preset algorithm can be set in the accurate model, which is a model trained to obtain accurate dense optical flow. At the same time, the accurate model does not need to be implemented in the client. Therefore, the accurate model does not need to consider the computing resources it occupies and can be a model with a relatively complex structure, so that accurate dense optical flow can be obtained based on the accurate model.

[0069] The terminal device can input the third and fourth degraded images into the optical flow estimation model to be trained, obtain the predicted dense optical flow output by the optical flow estimation model, and then train the optical flow estimation model with the optimization objective of minimizing the difference between the predicted dense optical flow and the true value of the dense optical flow.

[0070] To obtain an optical flow estimation model with optical flow evaluation values, the client can jointly train the optical flow estimation model using a pre-trained optical flow enhancement model. Specifically, the optical flow enhancement model can be trained as follows:

[0071] First, the terminal device can acquire training samples, which include: the real image of the third image, the real image of the fourth image, the degraded fourth image, and dense optical flow. The real images of the third and fourth images are obtained by extracting two consecutive frames from the high-definition video. The dense optical flow is obtained by inputting the degraded third and fourth images into an accurate optical flow model. Next, the terminal device can input the real image of the third image, the degraded fourth image, and the dense optical flow into the optical flow enhancement model to be trained, obtaining the predicted fourth enhanced image output by the optical flow enhancement model. Finally, the optical flow enhancement model can be trained with the optimization objective of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image.

[0072] After training the optical flow enhancement model, the client can train the optical flow estimation model based on the trained optical flow enhancement model and the training samples. Specifically, the client can obtain the training samples, which include: the real image of the third image, the third degraded image, the real image of the fourth image, and the fourth degraded image.

[0073] The client can then input the third and fourth degraded images into the optical flow estimation model to be trained, obtain the predicted dense optical flow and the predicted optical flow score output by the optical flow estimation model, and input the predicted dense optical flow, the real image of the third image, and the fourth degraded image into the optical flow enhancement model to be trained, obtain the predicted fourth enhanced image output by the optical flow enhancement model, so as to determine the true value of the optical flow evaluation value based on the predicted fourth enhanced image and the real image of the fourth image.

[0074] Finally, the terminal device can jointly train the optical flow estimation model and the optical flow enhancement model with the optimization objectives of minimizing the difference between the predicted fourth enhanced image and the real fourth image, and minimizing the predicted optical flow evaluation value and the real optical flow evaluation value.

[0075] Specifically, the terminal device can determine the average color similarity of pixels at each location in the predicted fourth enhanced image and the real image of the fourth image, and then determine the true value of the optical flow evaluation value based on the average color similarity. Specifically, the following formula can be used as a reference:

[0076]

[0077] Where M can be the optical flow evaluation value, n is the number of pixels in the predicted fourth enhanced image or the real image of the fourth image, R1, G1 and B1 are the red, green and blue color values ​​of a certain pixel in the fourth enhanced image, and R2, G2 and B2 are the red, green and blue color values ​​of the corresponding pixel in the real image of the fourth image.

[0078] S103: If the optical flow evaluation value exceeds the threshold, the first enhanced image, the second image, and the dense optical flow are input into the pre-trained optical flow enhancement model to determine the second enhanced image.

[0079] If the client determines that the optical flow evaluation values ​​of the first image and the second image do not exceed the preset threshold, it can determine the second enhanced image based on the first enhanced image, the second image, and the dense optical flow.

[0080] In this application, the client can input a first enhanced image, a second image, and dense optical flow into a pre-trained optical flow enhancement model to obtain the second enhanced image output by the optical flow enhancement model.

[0081] The following is combined with Figure 3 A schematic diagram illustrating the working principle of an optical flow enhancement model provided in this application is provided.

[0082] exist Figure 3In this process, the client can input the second compressed image into the trained neural network to obtain the feature map of the second compressed image. At the same time, the dense optical flow and the first enhanced image are input into the trained neural network to obtain the feature map of the second enhanced image predicted by the optical flow. The feature map of the second enhanced image and the feature map of the second image are fused to obtain the second enhanced image output by the optical flow enhancement model.

[0083] It should be noted that the optical flow enhancement model does not perform well when the difference between the first and second images is too large. Therefore, if the optical flow evaluation value determined by the client exceeds the preset threshold, the second image is input into the pre-trained image enhancement model to obtain the second enhanced image output by the image enhancement model.

[0084] S104: Use the second image as the first image, use the second enhanced image as the first enhanced image, and obtain the image of the next frame, use the image of the next frame as the second image, repeat the above process of optimizing the second image to obtain the newly determined enhanced image of the second image, until all images are optimized and then stop.

[0085] After obtaining the second enhanced image, the client can use the second image as the image of the current frame, obtain the image of the next frame, and input the second image and the image of the next frame into the optical flow estimation model. Repeat the above process to obtain the enhanced image of the next frame.

[0086] The following is combined with Figure 4 This application provides a flowchart illustrating a mobile video quality enhancement process.

[0087] exist Figure 4 In this process, the client can first acquire the image of the current frame and compress it to a preset resolution. After acquiring the current frame image, the client inputs it into the image enhancement model to obtain the enhanced image of the current frame.

[0088] The next frame image is then acquired and compressed to a preset resolution. This image is then used as the next frame image. The current frame image and the next frame image are input into the optical flow estimation model to determine the optical flow evaluation value and dense optical flow output by the optical flow estimation model.

[0089] If the client determines that the optical flow evaluation value exceeds a preset threshold, it inputs the current frame image, the next frame image, and the current frame enhanced image into the optical flow enhancement model to obtain the next frame enhanced image. If the client determines that the optical flow evaluation value does not exceed a preset value, it inputs the next frame image into the image enhancement model to obtain the next frame enhanced image.

[0090] Once the client obtains the next enhanced image, it can use that image as the current frame image and obtain the next frame image, repeating the above process until all images have been optimized.

[0091] As can be seen from the above method, the optical flow evaluation value obtained from the optical flow estimation model can be used to determine whether the images in the video are enhanced by an optical flow enhancement model or an image enhancement model. This allows for quality enhancement of most images in the video through a less computationally intensive method by controlling the threshold of the optical flow evaluation value and designing computationally efficient optical flow enhancement and estimation models. Therefore, the entire video enhancement process can be performed on the client side without uploading images to a cloud server, thus improving video clarity while avoiding playback stuttering caused by network factors.

[0092] The above describes one or more embodiments of a mobile video quality enhancement method provided in this application. Based on the same concept, this application also provides a corresponding mobile video quality enhancement apparatus, such as... Figure 5 As shown.

[0093] Figure 5 A schematic diagram of a mobile video quality enhancement device provided in this application includes:

[0094] The first acquisition module 501 is used to acquire the image of the first frame of the video as the first image, and input the first image into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model.

[0095] The second acquisition module 502 is used to acquire the image of the next frame as the second image, and input the first image and the second image into a pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model;

[0096] The determination module 503 is used to input the first enhanced image, the second image, and the dense optical flow into a pre-trained optical flow enhancement model to determine the second enhanced image if the optical flow evaluation value exceeds a threshold.

[0097] The loop module 504 is used to take the second image as the first image, take the second enhanced image as the first enhanced image, and obtain the image of the next frame, take the image of the next frame as the second image, and repeat the above process of optimizing the second image to obtain the newly determined enhanced image of the second image, until all images are optimized and then stop.

[0098] Optionally, the first acquisition module 501 is specifically used to input the second image into a pre-trained image enhancement model to determine the second enhanced image if the optical flow evaluation value does not exceed the threshold.

[0099] Optionally, the second acquisition module 502 is further configured to acquire the image of the first frame of the video and compress the image of the first frame of the video into an image of a preset resolution; and use the compressed image as the first compressed image.

[0100] Optionally, the determining module 503 is specifically used to: acquire training samples, wherein the training samples include: a real image of the third image, a real image of the fourth image, a degraded fourth image, and dense optical flow; wherein the real images of the third image and the fourth image are acquired by extracting two consecutive frames from a high-definition video; the fourth degraded image is obtained by adding noise, blurring, and reducing resolution to the real image of the fourth image; the dense optical flow is obtained by using a preset algorithm with the degraded third image and the fourth degraded image as input; inputting the real image of the third image, the fourth degraded image, and the dense optical flow into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model; and training the optical flow enhancement model with the optimization objective of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image.

[0101] Optionally, the determining module 503 is further configured to: acquire training samples, wherein the training samples include: the third degraded image, the fourth degraded image, and the true value of the dense optical flow; input the third degraded image and the fourth degraded image into the optical flow estimation model to be trained, and acquire the predicted dense optical flow output by the optical flow estimation model; and train the optical flow enhancement model with the optimization objective of minimizing the difference between the predicted dense optical flow and the true value of the dense optical flow.

[0102] Optionally, training samples are obtained, wherein the training samples include: the real image of the third image, the third degraded image, the real image of the fourth image, and the fourth degraded image; the third degraded image and the fourth degraded image are input into the optical flow estimation model to be trained to obtain the predicted dense optical flow and the predicted optical flow score output by the optical flow estimation model; the predicted dense optical flow, the real image of the third image, and the fourth degraded image are input into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model; the true value of the optical flow evaluation value is determined based on the predicted fourth enhanced image and the real image of the fourth image; the optical flow estimation model and the optical flow enhancement model are jointly trained with the optimization objective of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image, and minimizing the predicted optical flow evaluation value and the true value of the optical flow evaluation value.

[0103] Optionally, based on the predicted fourth enhanced image and the real image of the fourth image, the average color similarity of pixels at each position in the predicted fourth enhanced image and the real image of the fourth image is determined; the true value of the optical flow evaluation value is determined based on the average color similarity.

[0104] This application also provides a computer-readable medium storing a computer program that can be used to perform the above-described actions. Figure 1 This provides a method for enhancing video quality on mobile devices.

[0105] This application also provides Figure 6 One of the corresponding Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 6 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method for enhancing video quality on mobile devices is described above. Of course, besides software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0106] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0107] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0108] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0109] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer media, including storage devices.

[0120] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0121] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for enhancing video quality on a mobile device, characterized in that: The image of the first frame of the video is acquired as the first image, and the first image is input into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model; The image of the next frame is acquired as the second image, and the first image and the second image are input into a pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model; If the optical flow evaluation value exceeds the threshold, the first enhanced image, the second image, and the dense optical flow are input into a pre-trained optical flow enhancement model to determine the second enhanced image; If the optical flow evaluation value does not exceed the threshold, the second image is input into the pre-trained image enhancement model to determine the second enhanced image; The second image is used as the first image, the second enhanced image is used as the first enhanced image, and the image of the next frame is obtained. The image of the next frame is used as the second image. The above process of optimizing the second image is repeated to obtain the newly determined enhanced image of the second image, until all images are optimized and then the process stops.

2. The method according to claim 1, characterized in that, Obtain the image of the first frame of the video as the first image, specifically including: Acquire the image of the first frame of the video and compress the image of the first frame of the video to an image of a preset resolution; The compressed image is used as the first compressed image, and the first compressed image is used as the first image.

3. The method according to claim 1, characterized in that, Training the optical flow enhancement model specifically includes: Training samples are obtained, including: a real image of the third image, a real image of the fourth image, a degraded fourth image, and dense optical flow. The real images of the third image and the fourth image are obtained by extracting two consecutive frames from a high-definition video. The degraded fourth image is obtained by adding noise, blurring, and reducing the resolution of the real image of the fourth image. The dense optical flow is obtained by using a preset algorithm with the degraded third image and the degraded fourth image as input. The real image of the third image, the fourth degraded image, and the dense optical flow are input into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model. The optical flow enhancement model is trained with the optimization objective of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image.

4. The method according to claim 3, characterized in that, Training the optical flow estimation model specifically includes: Obtain training samples, wherein the training samples include: the third degraded image, the fourth degraded image, and the true value of the dense optical flow; The third and fourth degraded images are input into the optical flow estimation model to be trained to obtain the predicted dense optical flow output by the optical flow estimation model. The optical flow enhancement model is trained with the optimization objective of minimizing the difference between the predicted dense optical flow and the true value of the dense optical flow.

5. The method according to claim 4, characterized in that, Training the optical flow estimation model and training the optical flow enhancement model specifically include: Obtain training samples, wherein the training samples include: the real image of the third image, the third degraded image, the real image of the fourth image, and the fourth degraded image; The third and fourth degraded images are input into the optical flow estimation model to be trained to obtain the predicted dense optical flow and the predicted optical flow score output by the optical flow estimation model. The predicted dense optical flow, the real image of the third image, and the fourth degraded image are input into the optical flow enhancement model to be trained to obtain the predicted fourth enhanced image output by the optical flow enhancement model. Based on the predicted fourth enhanced image and the true image of the fourth image, the true value of the optical flow evaluation value is determined; The optical flow estimation model and the optical flow enhancement model are jointly trained with the optimization objectives of minimizing the difference between the predicted fourth enhanced image and the real image of the fourth image, and minimizing the predicted optical flow evaluation value and the real value of the optical flow evaluation value.

6. The method according to claim 5, characterized in that, Based on the predicted fourth enhanced image and the true image of the fourth image, the true value of the optical flow evaluation value is determined, specifically including: Based on the predicted fourth enhanced image and the real image of the fourth image, determine the average color similarity of pixels at each position in the predicted fourth enhanced image and the real image of the fourth image; The true value of the optical flow evaluation value is determined based on the average color similarity.

7. A device for enhancing video quality on a mobile device, characterized in that, include: The first acquisition module is used to acquire the image of the first frame of the video as the first image, and input the first image into a pre-trained image enhancement model to obtain the first enhanced image output by the image enhancement model; The second acquisition module is used to acquire the image of the next frame as the second image, and input the first image and the second image into the pre-trained optical flow estimation model to obtain the dense optical flow and optical flow evaluation value output by the optical flow estimation model; The determination module is used to input the first enhanced image, the second image, and the dense optical flow into a pre-trained optical flow enhancement model to determine the second enhanced image if the optical flow evaluation value exceeds a threshold. If the optical flow evaluation value does not exceed the threshold, the second image is input into the pre-trained image enhancement model to determine the second enhanced image; The loop module is used to take the second image as the first image, the second enhanced image as the first enhanced image, and obtain the image of the next frame, take the image of the next frame as the second image, and repeat the above process of optimizing the second image to obtain the newly determined enhanced image of the second image, until all images are optimized and then stop.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images