Image quality enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides an image quality enhancement method and apparatus, an electronic device and a storage medium. The enhancement method comprises the steps of obtaining a target image of a first resolution; performing resolution compression on the target image with the first resolution to obtain a target image with a second resolution; performing residual extraction on the target image with the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a target enhancement residual image with the second resolution; converting the target enhanced residual image of the second resolution into a target enhanced residual image of the first resolution; and superposing the target enhanced residual image of the first resolution and the target image of the first resolution to obtain an image quality enhanced target image of the first resolution. Compared with the prior art, the method has the advantages that through two-time size change operation and one-time superposition addition operation, the reasoning duration of the model is not increased, image detail enhancement and image quality improvement can be realized for the high-resolution live broadcast image, and better watching experience is provided for a user.

Description

Technical Field

[0001] The present invention relates to the field of image data processing, and more specifically, to an image quality enhancement method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of technology, the popularization of high-resolution devices and 5G network technology, users have an increasing demand for ultra-high-definition smooth live broadcasts. Users hope that the live broadcast resolution can be as high as possible, with minimal lag and reduced latency. However, most ordinary users' viewing devices currently have a resolution of 2K or even 4K, and the corresponding video or live stream resolution has also reached 2K or 4K. On this basis, it is difficult to improve the user's viewing experience by increasing the resolution of the image. Summary of the invention

[0003] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide an image quality enhancement method, device, electronic device and storage medium for improving the quality of the image to enhance the user's viewing experience.

[0004] According to one aspect of the present application, a method for enhancing image quality is provided, the method comprising:

[0005] Acquire a target image of a first resolution;

[0006] Compressing the target image at the first resolution to obtain the target image at a second resolution;

[0007] Performing residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution;

[0008] Converting the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation;

[0009] The target enhanced residual image of the first resolution is superimposed with the target image of the first resolution to obtain a target image of the first resolution with enhanced image quality.

[0010] Optionally, performing residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution specifically includes:

[0011] The image quality enhancement residual extraction model includes a plurality of residual blocks;

[0012] Performing convolution processing on the target image of the second resolution to obtain convolution features of the target image;

[0013] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0014] Passing the nonlinear features through the residual blocks of the image quality enhancement residual extraction model in sequence to obtain residual extraction features of the residual blocks;

[0015] The residual extraction feature output by the last residual block is used as the target enhanced residual image of the second resolution.

[0016] Optionally, the training step of the image quality enhancement residual extraction model specifically includes:

[0017] Obtain a high-definition training image set and a corresponding low-definition training image set;

[0018] Acquire a training residual image set according to the image quality enhancement residual extraction model and the low-definition training image set;

[0019] Acquire an enhanced residual image set according to the low-definition training image set and the training residual image set;

[0020] The loss of the image quality enhancement residual extraction model is obtained according to the high-definition training image set and the enhanced residual image set, and the image quality enhancement residual extraction model is updated according to the loss.

[0021] Optionally, the high-definition training image set includes a first high-definition training image set and a second high-definition training image set, and the low-definition training image set includes a first low-definition training image set and a second low-definition training image set;

[0022] The obtaining of the high-definition training image set and the corresponding low-definition training image set specifically includes:

[0023] Collecting high-definition video data with a resolution of the third resolution;

[0024] Keep the resolution unchanged and reduce the bit rate of high-definition video data to obtain the corresponding low-definition video data;

[0025] Respectively extracting a number of corresponding video frames from the high-definition video data and the low-definition video data to obtain a first high-definition training image set and a first low-definition training image set;

[0026] The resolutions of the image data in the first high-definition training image set and the first low-definition training image set are compressed to a fourth resolution, thereby obtaining a second high-definition training image set and a second low-definition training image set respectively.

[0027] Optionally, the acquiring a training residual image set according to the image quality enhancement residual extraction model and the low-definition training image set specifically includes:

[0028] Inputting the second low-definition training image set into the image quality enhancement residual extraction model to be trained to obtain an output second training residual image set;

[0029] The resolution of the image data in the second training residual image set is converted into a third resolution by interpolation to obtain a first training residual image set.

[0030] Optionally, the step of obtaining an enhanced residual image set according to the low-definition training image set and the training residual image set specifically includes:

[0031] Superimposing the image data in the first low-definition training image set with the corresponding image data in the first training residual image set to obtain a first enhanced residual image set;

[0032] The image data in the second low-definition training image set is superimposed with the corresponding image data in the second training residual image set to obtain a second enhanced residual image set.

[0033] Optionally, obtaining the loss of the image quality enhancement residual extraction model according to the high-definition training image set and the enhanced residual image set, and updating the image quality enhancement residual extraction model according to the loss specifically includes:

[0034] Calculate a first loss between the first enhanced residual image set and the first high-definition training image set, and calculate a second loss between the second enhanced residual image set and the second high-definition training image set, and update the image quality enhancement residual extraction model according to the first loss and the second loss.

[0035] According to a second aspect of the present application, there is provided an image quality enhancement device, the enhancement device comprising:

[0036] A target image acquisition module, used to acquire a target image with a first resolution;

[0037] An image compression module, used for performing resolution compression on the target image of the first resolution to obtain the target image of the second resolution;

[0038] A residual processing module, used to perform residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution;

[0039] A residual enhancement module, used for converting the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation;

[0040] The image quality enhancement module is used to superimpose the target enhanced residual image of the first resolution with the target image of the first resolution to obtain the image quality enhanced target image of the first resolution.

[0041] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement an image quality enhancement method described in the first aspect above.

[0042] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer-readable program is stored. When the computer-readable program is executed, the image quality enhancement method described in the first aspect is implemented.

[0043] According to any one of the above aspects, an image quality enhancement method, device, electronic device and storage medium provided in this embodiment extract a target enhanced residual image of the target image of the second resolution by using an image quality enhancement residual extraction model, and the target image of the second resolution is obtained by compressing the target image of the first resolution through resolution compression, so the target enhanced residual image corresponds to the target image of the first resolution, and the enhancement information in the target enhanced residual image can be directly applied to the target image of the first resolution with high resolution, thereby being able to effectively improve the quality of the target image without changing the resolution of the target image and without losing the image details of the target image of high resolution, thereby improving the user's viewing experience;

[0044] At the same time, in the image quality enhancement method, device, electronic device and storage medium provided by the present embodiment, only two size change operations and one addition operation are added, wherein one size change operation is resolution compression, the other size change operation is interpolation conversion, and the addition operation is the superposition of the target enhanced residual image and the target image. The added change operation and addition operation will not increase the inference time of the image quality enhancement residual extraction model too much, thereby ensuring the efficiency of the image quality enhancement operation;

[0045] Furthermore, the image quality enhancement residual extraction model in the image quality enhancement method, device, electronic device and storage medium provided in this embodiment processes lower resolutions, which can effectively avoid the increase in computational complexity caused by directly processing high-resolution images, further reducing the efficiency of image quality enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of an application scenario of the enhancement method provided in this embodiment.

[0048] Figure 2 This is a flowchart of the steps of the enhancement method provided in this embodiment.

[0049] Figure 3 A flowchart of the steps for training the image quality enhancement residual extraction model provided in this embodiment.

[0050] Figure 4 This is a structural diagram of the enhancement device provided in this embodiment.

[0051] Figure 5 This is a device structure diagram of the electronic device provided in this embodiment.

[0052] Figure annotations: server 100, terminal 200, target image acquisition module 11, image compression module 12, residual processing module 13, residual enhancement module 14, image quality enhancement module 15, model training module 16, memory 21, processor 22, bus 23, communication interface 24. DETAILED DESCRIPTION

[0053] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] With the continuous development of technology, the popularization of high-resolution devices and 5G network technology, users have an increasing demand for ultra-high-definition smooth live broadcasts and videos. Users hope that the resolution of live broadcasts and videos can be as high as possible, with as little lag as possible and reduced latency. Nowadays, 2K resolution is already the basic configuration of most mobile devices, and the resolution of many live streams and video streams has also been correspondingly increased from 1080P to 2K or even 4K. For these high-resolution live broadcasts and videos, further increasing the resolution does not make much sense. First of all, most users' devices today do not support higher resolution displays, so even if the resolution of live broadcasts and videos is increased, they cannot be presented. Secondly, the difficulty and cost of further increasing the resolution are very high, including the configuration cost of the equipment and the operating cost of the corresponding platform.

[0057] In fact, even if the picture resolution is not increased, the picture effect can be significantly improved by simply enhancing the picture quality at the original resolution. Picture quality enhancement technology mainly improves the details and clarity of the image, making the image more realistic and vivid. For example, detail enhancement can make the facial features of the characters in the live broadcast clearer and the background scenery richer, which is obvious for the improvement of user experience. The application of picture quality enhancement technology is particularly important in live broadcast scenes that require high-definition picture quality, such as sports events and concerts, or in the process of watching high-definition movies. High-definition picture quality can not only make users feel immersive, but also improve the competitiveness and user stickiness of live video platforms. In addition, picture quality enhancement technology can also improve the clarity and brightness of the picture in low-light environments, solving the common problems of picture blur and noise in traditional live broadcasts.

[0058] On the other hand, after the image quality is improved, the redundant bit rate of the video can be compressed, thereby reducing bandwidth. Bandwidth is an important resource for live video platforms, especially in high-resolution live broadcasts and videos. High-bitrate live streams and video streams will occupy a lot of bandwidth resources. If the redundant bit rate can be compressed while ensuring the image quality, it will greatly reduce bandwidth consumption and improve network transmission efficiency. Through image quality enhancement technology, video quality can be improved without increasing the bit rate, and the bit rate can even be appropriately reduced while maintaining the image quality. This can not only reduce the operating costs of live video platforms, but also ensure that users can watch live broadcasts smoothly under poor network conditions, reducing the occurrence of freezes and delays.

[0059] In order to ensure real-time performance and reduce latency and freezes, a single-image quality enhancement model is usually used, that is, the quality of the decoded live video stream is enhanced frame by frame. However, the inference time of the existing quality enhancement model increases with the increase of the input image resolution. Compared with the common 1080P live broadcast resolution, the number of pixels that need to be processed for 2K and 4K high resolutions increases significantly. If the same quality enhancement model as 1080P is used, the inference time will increase significantly.

[0060] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0061] For example, a schematic diagram of an application scenario of an image quality enhancement method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal 200 has a streaming media playback function and can also have an image processing function.

[0062] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0063] In one practicable manner, the server 100 and the terminal 200 may respectively execute an image quality enhancement method provided in an embodiment of the present application, or, optionally, the image quality enhancement method provided in an embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0064] like Figure 2 As shown, this embodiment provides an image quality enhancement method, the enhancement method comprising:

[0065] S1: Acquire a target image with a first resolution;

[0066] In this embodiment, the target image of the first resolution may be a live image of the first resolution, or a video image of the first resolution. Specifically, the corresponding live stream or video stream may be acquired, the live stream or video stream may be decoded into continuous single-frame images, and the decoded single-frame images may be used as the target image.

[0067] In order to better improve the user's viewing experience, the resolution of the target image needs to match the playback devices on the market with video live broadcast or video playback functions, such as mobile phones, tablets, computers and TVs. The first resolution is the highest resolution supported by the playback device, or the most common resolution. It can be understood that in the context of prior art, the first resolution can be set to 2k / 4k resolution.

[0068] S2: compressing the target image of the first resolution to obtain the target image of the second resolution;

[0069] In this embodiment, the resolution of the target image is compressed mainly to match it with the subsequent pre-trained model. At the same time, compared with directly using a higher first resolution, using a lower second resolution can reduce the amount of calculation of the model and reduce the model's reasoning market, thereby improving the efficiency and timeliness of the enhancement method of this embodiment.

[0070] Taking into account that the setting of the second resolution needs to be able to effectively reduce the calculation amount of the model, the second resolution cannot be too close to the first resolution. At the same time, in order to facilitate processing and calculation, the second resolution can be set to a resolution size commonly used on the market. In this embodiment, based on the first resolution being set to 2k / 4k resolution, the second resolution can be set to 1080p.

[0071] S3: performing residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution;

[0072] In this embodiment, the image quality enhancement residual extraction model is set based on the existing super-resolution network EDRN (Enhanced Deep Residual Networks for Single Image Super-Resolution), wherein the super-resolution network EDRN mainly optimizes and removes unnecessary modules in the residual network in the traditional convolutional neural network model, and at the same time, in the process of stable training, improves the performance of the model by expanding the size of the model; conventionally, the super-resolution network EDRN model mainly includes: a convolutional network for obtaining convolutional features of the input image, an activation network for performing nonlinear mapping, a residual network for extracting residual features, an enhancement network for outputting an enhanced quality image according to the residual features, and an upsampling module for improving the image resolution output by the model; wherein the residual network contains a plurality of residual blocks, and residual blocks are used to perform residual extraction on the input features in sequence;

[0073] In this embodiment, the image quality enhancement residual extraction model is structurally improved according to the super-resolution network EDRN, so that the image quality enhancement residual extraction model of this embodiment can perform residual extraction on the image input therein to obtain the corresponding image quality enhancement residual.

[0074] Specifically, the image quality enhancement residual extraction model at least includes a convolution module, an activation module, a residual module and a residual connection module;

[0075] The convolution module can be set based on the convolution network of the super-resolution network EDRN. In this embodiment, the convolution module is used to perform convolution processing on the input image input into the image quality enhancement residual extraction model;

[0076] The activation module may be set based on the activation network of the super-resolution network EDRN. In this embodiment, the activation module is used to perform a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0077] The activation module is set based on the activation network of the super-resolution network EDRN. Furthermore, this embodiment reduces the number of residual blocks in the residual module in the activation network to 10, and also reduces the number of channels of the image quality enhancement residual extraction model to one-fourth of the number of channels of the existing super-resolution network EDRN, thereby greatly reducing the computational complexity of the image quality enhancement residual extraction model and improving the real-time processing capability of the image quality enhancement residual extraction model. In this embodiment, the residual module performs residual extraction on the nonlinear features in turn through 10 of the residual blocks to obtain residual features.

[0078] The image quality enhancement residual extraction model of the present embodiment does not directly output the image after image quality enhancement, but obtains the residual feature image through the residual connection module, and obtains the residual feature image as the output result of the image quality enhancement residual extraction model. Compared with the existing super-resolution network EDRN, the time of model training and reasoning can be reduced; in addition, unlike the super-resolution network EDRN, in the present embodiment, the image quality enhancement residual extraction model removes the upsampling module, so that the residual feature image output by the image quality enhancement residual extraction model is consistent with the resolution of the input image. This is because the present embodiment does not hope to improve the resolution of the image during the model reasoning process, but hopes to achieve image quality enhancement at the same resolution.

[0079] It can be understood that the residual represents the difference between the actual observed value and the estimated value. In this embodiment, the residual feature represents the difference between the enhanced image and the original image. This example does not directly output the enhanced image, but uses the extracted residual feature image as the output result of the image enhancement residual extraction model. It can effectively retain the image enhancement information in the output residual feature image, so that the image enhancement residual extraction model can better learn the detailed information of image enhancement.

[0080] In this example, the specific steps of step S3 may include:

[0081] Performing convolution processing on the target image of the second resolution to obtain convolution features of the target image;

[0082] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0083] Passing the nonlinear features through the residual blocks of the image quality enhancement residual extraction model in sequence to obtain residual extraction features of the residual blocks;

[0084] The residual extraction feature output by the last residual block is used as the target enhanced residual image of the second resolution.

[0085] In this embodiment, the image quality enhancement residual extraction model improves the super-resolution network EDRN. At the same time, the specific use reasoning process of the model is also different. This embodiment also optimizes and improves the training process of the image quality enhancement residual extraction model; Figure 3 As shown, the training process of the image quality enhancement residual extraction model may specifically include:

[0086] A1: Obtain a high-definition training image set and a corresponding low-definition training image set;

[0087] In this embodiment, high definition and low definition represent the image quality of corresponding images. The description of high definition and low definition is relative. For example, if an image of a certain image quality is used as a high definition image, an image with a lower image quality than the high definition image can be used as a low definition image corresponding to the high definition image.

[0088] The high-definition training image set includes a first high-definition training image set and a second high-definition training image set, and the low-definition training image set includes a first low-definition training image set and a second low-definition training image set; wherein the resolutions of the image data in the first high-definition training image set and the first high-definition training image set are both the third resolution, and the resolutions of the image data in the second high-definition training image set and the second low-definition training image set are both the fourth resolution. The fourth resolution is smaller than the third resolution. Similar to the first resolution and the second resolution, the third resolution and the fourth resolution can be set to the resolution sizes commonly used in the market, the purpose of which is to be able to improve the image quality improvement capability of the image quality enhancement residual extraction model obtained by training for images of various resolutions.

[0089] Specifically, obtaining the high-definition training image set and the low-definition training image set may include:

[0090] The high-definition video data having a third resolution is acquired; it is understandable that the high-definition video data may be recorded video data, such as high-definition movies and high-definition videos, or may be real-time video data, such as high-definition live video, etc. This embodiment does not limit the form of the high-definition video data and the specific acquisition means.

[0091] After obtaining the high-definition video data, the resolution of the high-definition video data is kept unchanged, and the bit rate of the high-definition video data is reduced to obtain corresponding low-definition video data; specifically, in a preferred embodiment, a bit rate lower than the high-definition video data can be randomly selected, and while keeping the resolution unchanged, the high-definition video data is encoded and degraded at the bit rate, random noise is added, or random Gaussian blur is added to degrade, so as to obtain the low-definition video data of the bit rate; in the actual training process, since the bit rate is randomly selected, the bit rates selected for different batches of training are different, so that the image quality enhancement residual extraction model can adapt to the image quality under various network conditions, so as to still provide better image quality under bandwidth limited conditions, and at the same time, the trained image quality enhancement residual extraction model can also have better robustness.

[0092] Then, a number of corresponding video frames are extracted from the high-definition video data and the low-definition video data respectively to obtain a first high-definition training image set and a first low-definition training image set; since the low-definition video data is directly obtained by reducing the bit rate of the high-definition video data, the number of frames in the high-definition video data and the low-definition video data is corresponding, and the video frames therein are also one-to-one corresponding;

[0093] The resolutions of the image data in the first high-definition training image set and the first low-definition training image set are compressed to the fourth resolution, thereby obtaining a second high-definition training image set and a second low-definition training image set respectively.

[0094] In a preferred implementation of the present embodiment, in order to better train the image quality enhancement residual extraction model so that the image quality enhancement residual extraction model can better perform model inference for the first resolution and the second resolution, the third resolution can be set to be the same as the first resolution, and the fourth resolution can be set to be the same as the second resolution, so that the trained image quality enhancement residual extraction model can better perform residual extraction for the target image of the second resolution, and at the same time effectively improve the residual extraction efficiency of the image quality enhancement residual extraction model.

[0095] A2: acquiring a training residual image set according to the image quality enhancement residual extraction model and the low-definition training image set;

[0096] In this embodiment, the training residual image set corresponding to the high-definition training image set and the low-definition training image set includes a first training residual image set and a second training residual image set; then on the basis of obtaining the first high-definition training image set, the second high-definition training image set, the first low-definition training image set and the second low-definition training image set in step A1, step A2 respectively obtains the training residual image sets corresponding to the first high-definition training image set, the second high-definition training image set, the first low-definition training image set and the second low-definition training image set, specifically including:

[0097] Inputting the second low-definition training image set into the image quality enhancement residual extraction model to be trained, and obtaining an output second training residual image set; in this embodiment, the second low-definition training image set is input into the image quality enhancement residual extraction model for training. Since the resolution of the second low-definition training image set is the lower fourth resolution, compared with the image set using the third resolution, the amount of calculation for training the image quality enhancement residual extraction model can be effectively reduced, thereby reducing the training time;

[0098] The resolution of the image data in the second training residual image set is then converted into a third resolution through interpolation to obtain a first training residual image set; the obtained first training residual image set and second training residual image set can effectively retain enhanced information.

[0099] A3: Obtaining an enhanced residual image set according to the low-definition training image set and the training residual image set;

[0100] Since the resolutions of the image data in the first training residual image set and the second training residual image set are the third resolution and the fourth resolution respectively, and they match the first low-definition training image set and the second low-definition training image set respectively, the training residual image set can be directly superimposed with the corresponding low-definition training image set;

[0101] Specifically, in this embodiment, obtaining the enhanced residual image set according to the low-definition training image set and the training residual image set may specifically include:

[0102] Superimposing the image data in the first low-definition training image set with the corresponding image data in the first training residual image set to obtain a first enhanced residual image set;

[0103] The image data in the second low-definition training image set is superimposed with the corresponding image data in the second training residual image set to obtain a second enhanced residual image set.

[0104] A4: Obtain the loss of the image quality enhancement residual extraction model according to the high-definition training image set and the enhanced residual image set, and update the image quality enhancement residual extraction model according to the loss.

[0105] In this embodiment, the enhanced residual image set includes the first enhanced residual image set and the second enhanced residual image set, so the losses need to be calculated respectively;

[0106] Specifically, the loss calculation may include:

[0107] A first loss between the first enhanced residual image set and the first high-definition training image set is calculated, and a second loss between the second enhanced residual image set and the second high-definition training image set is calculated, and the image quality enhancement residual extraction model can be updated according to the first loss and the second loss.

[0108] In a preferred implementation of the present embodiment, the first loss and the second loss may be L1 loss, i.e., calculating the mean absolute error between the first enhanced residual image set and the first high-definition training image set, and calculating the mean absolute error between the second enhanced residual image set and the second high-definition training image set; in another preferred implementation of the present embodiment, the first loss and the second loss may also be L2 loss; the specific loss setting may be set according to the actual situation, and its purpose is to be able to better update the image quality enhancement residual extraction model and make the model converge.

[0109] S4: converting the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation;

[0110] Similar to the training step A3 of the image quality enhancement residual extraction model, the resolution of the target enhanced residual image is converted from the second resolution to the first resolution, so as to ensure that the enhancement information extracted by the image quality enhancement residual extraction model can be effectively applied to the original target image of the first resolution;

[0111] S5: Superimposing the target enhanced residual image of the first resolution with the target image of the first resolution to obtain a quality enhanced target image of the first resolution.

[0112] The target enhanced residual image of the first resolution effectively retains the enhancement information and matches the target image of the first resolution. The target enhanced residual image of the first resolution can be seamlessly integrated into the target image of the first resolution by superposition, thereby enhancing the image quality of the target image of the first resolution, while also retaining the image details of the original target image of the first resolution.

[0113] This embodiment uses two size change operations and one addition operation, without increasing the model inference time, combined with the bit rate compression method, to achieve image detail enhancement and overall picture quality improvement for high-resolution live images, significantly improving the picture quality of high-resolution live broadcasts and videos, providing users with a better viewing experience, while ensuring real-time performance, thereby achieving the goal of low-latency and smooth viewing of ultra-high-definition live videos.

[0114] Based on the same inventive concept, this embodiment also provides an image quality enhancement device, such as Figure 4 As shown, the enhancement device may include:

[0115] A target image acquisition module 11, used to acquire a target image with a first resolution;

[0116] In this embodiment, the target image acquisition module 11 can be used to perform Figure 3 As shown in step S1 , for a detailed description of the target image acquisition module 11 , reference may be made to the description of step S1 .

[0117] An image compression module 12, configured to perform resolution compression on the target image of the first resolution to obtain the target image of the second resolution;

[0118] In this embodiment, the image compression module 12 can be used to perform Figure 3 As shown in step S2, for the specific description of the image compression module 12, reference may be made to the description of step S2.

[0119] The residual processing module 13 is used to perform residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution;

[0120] In this embodiment, the residual processing module 13 can be used to perform Figure 3 As shown in step S3, for the detailed description of the residual processing module 13, reference may be made to the description of step S3.

[0121] A residual enhancement module 14, configured to convert the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation;

[0122] In this embodiment, the residual enhancement module 14 can be used to perform Figure 3 As shown in step S4, for the detailed description of the residual enhancement module 14, reference may be made to the description of step S4.

[0123] The image quality enhancement module 15 is used to superimpose the target enhanced residual image of the first resolution with the target image of the first resolution to obtain the image quality enhanced target image of the first resolution;

[0124] In this embodiment, the image quality enhancement module 15 can be used to perform Figure 3 As shown in step S5, for the detailed description of the residual image quality enhancement module 15, reference may be made to the description of step S5.

[0125] Furthermore, the enhancement device further comprises a model training module 16, which is used to train the image quality enhancement residual extraction model;

[0126] In this embodiment, the model training module 16 can be used to perform Figure 4For the steps A1-A4 shown, the specific description of the model training module 16 can refer to the description of the steps A1-A4.

[0127] Based on the same inventive concept, this embodiment also provides an electronic device, Figure 5 The structural diagram of the electronic device of this embodiment is shown, which includes a memory 21 and a processor 22. The memory 21 stores computer-readable instructions, and the processor 22 executes the computer-readable instructions to implement the image quality enhancement method of this embodiment.

[0128] Preferably, the electronic device further includes a bus 23 and a communication interface 24 , and the processor 22 , the communication interface 24 and the memory 21 are connected via the bus 23 .

[0129] The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 24 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 23 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus 23 may be divided into an address bus, a data bus, a control bus, etc. (not fully drawn in the figure).

[0130] The processor 22 may be an integrated circuit chip with signal processing capability. In the specific implementation process, each step in the above method embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor 22, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the method of the above embodiment in combination with its hardware.

[0131] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor 22, the computer-executable instructions prompt the processor 22 to implement the above-mentioned image quality enhancement method. The specific implementation can be found in the embodiment, which will not be repeated here.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0133] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for enhancing image quality, characterized in that: The enhancement method comprises: Acquire a target image of a first resolution; Compressing the target image at the first resolution to obtain the target image at a second resolution; Performing residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution; Converting the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation; The target enhanced residual image of the first resolution is superimposed with the target image of the first resolution to obtain a target image of the first resolution with enhanced image quality.

2. The method for enhancing image quality according to claim 1, characterized in that: The step of performing residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution specifically includes: The image quality enhancement residual extraction model includes a plurality of residual blocks; Performing convolution processing on the target image of the second resolution to obtain convolution features of the target image; Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature; Passing the nonlinear features through the residual blocks of the image quality enhancement residual extraction model in sequence to obtain residual extraction features of the residual blocks; The residual extraction feature output by the last residual block is used as the target enhanced residual image of the second resolution.

3. The image quality enhancement method according to claim 2, characterized in that: The training steps of the image quality enhancement residual extraction model specifically include: Obtain a high-definition training image set and a corresponding low-definition training image set; Acquire a training residual image set according to the image quality enhancement residual extraction model and the low-definition training image set; Acquire an enhanced residual image set according to the low-definition training image set and the training residual image set; The loss of the image quality enhancement residual extraction model is obtained according to the high-definition training image set and the enhanced residual image set, and the image quality enhancement residual extraction model is updated according to the loss.

4. The method for enhancing image quality according to claim 3, characterized in that: The high-definition training image set includes a first high-definition training image set and a second high-definition training image set, and the low-definition training image set includes a first low-definition training image set and a second low-definition training image set; The obtaining of the high-definition training image set and the corresponding low-definition training image set specifically includes: Collecting high-definition video data with a resolution of the third resolution; Keep the resolution unchanged and reduce the bit rate of high-definition video data to obtain the corresponding low-definition video data; Respectively extracting a number of corresponding video frames from the high-definition video data and the low-definition video data to obtain a first high-definition training image set and a first low-definition training image set; The resolutions of the image data in the first high-definition training image set and the first low-definition training image set are compressed to a fourth resolution, thereby obtaining a second high-definition training image set and a second low-definition training image set respectively.

5. The method for enhancing image quality according to claim 4, characterized in that: The step of obtaining a training residual image set according to the image quality enhancement residual extraction model and the low-definition training image set specifically includes: Inputting the second low-definition training image set into the image quality enhancement residual extraction model to be trained to obtain an output second training residual image set; The resolution of the image data in the second training residual image set is converted into a third resolution by interpolation to obtain a first training residual image set.

6. The method for enhancing image quality according to claim 5, characterized in that: The step of obtaining the enhanced residual image set according to the low-definition training image set and the training residual image set specifically includes: Superimposing the image data in the first low-definition training image set with the corresponding image data in the first training residual image set to obtain a first enhanced residual image set; The image data in the second low-definition training image set is superimposed with the corresponding image data in the second training residual image set to obtain a second enhanced residual image set.

7. The method for enhancing image quality according to claim 6, characterized in that: The obtaining the loss of the image quality enhancement residual extraction model according to the high-definition training image set and the enhanced residual image set, and updating the image quality enhancement residual extraction model according to the loss, specifically includes: Calculate a first loss between the first enhanced residual image set and the first high-definition training image set, and calculate a second loss between the second enhanced residual image set and the second high-definition training image set, and update the image quality enhancement residual extraction model according to the first loss and the second loss.

8. An image quality enhancement device, characterized in that: The enhancement device comprises: A target image acquisition module, used to acquire a target image with a first resolution; An image compression module, used for performing resolution compression on the target image of the first resolution to obtain the target image of the second resolution; A residual processing module, used to perform residual extraction on the target image of the second resolution through a pre-trained image quality enhancement residual extraction model to obtain a corresponding target enhanced residual image of the second resolution; A residual enhancement module, used for converting the target enhanced residual image of the second resolution into the target enhanced residual image of the first resolution by interpolation; The image quality enhancement module is used to superimpose the target enhanced residual image of the first resolution with the target image of the first resolution to obtain the image quality enhanced target image of the first resolution.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement an image quality enhancement method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer-readable program is stored thereon, and when the computer-readable program is executed, an image quality enhancement method as described in any one of claims 1 to 7 is implemented.

Images