Image quality enhancement method and device, user equipment, server, storage medium and computer program product

Abstract

The invention provides an image quality enhancement method and device, user equipment, a server, a storage medium and a computer program product, and the method comprises the steps: receiving a model index from the server, and enabling the user equipment to deploy a plurality of image quality enhancement models, the model index indicates that at least one image quality enhancement model among the plurality of image quality enhancement models is used for performing image quality enhancement on the target video; and performing image quality enhancement on the target video based on the at least one image quality enhancement model indicated by the model index. Therefore, the invention provides an end-cloud collaborative image quality enhancement scheme, complex computing tasks which consume computing power and are selected for the image quality enhancement model are unloaded to the cloud for processing, so that the terminal can directly call the corresponding image quality enhancement model to execute image quality enhancement. Therefore, the power consumption of executing image quality enhancement at the terminal side can be reduced, and the effect of image quality enhancement is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to an image quality enhancement method, an apparatus, a user equipment, a server, a storage medium, and a computer program product. Background Art

[0002] With the development of technology, terminals have more and more functions. For example, users can use terminals to browse news, watch videos, shop online, etc. When users use terminals to watch videos, the image quality of the images presented on the terminal display will directly affect the user's video consumption experience. Especially for some image video consumption platforms, the image quality of the images presented on the terminal will directly affect the daily activity of the platform.

[0003] In the related art, the image enhancement and restoration algorithm is generally deployed on the terminal side. On the one hand, it can save computing resources and bandwidth on the server side, which can reduce costs for software companies; on the other hand, as the last link of user consumption, directly enhancing and repairing the image on the screen through the terminal can ensure that there are no other degradation links in the future, and thus ensure that users can enjoy a better consumer experience brought by high-quality picture quality. However, considering the limited computing power on the terminal side, in order to balance low power consumption and real-time performance in the related art, the image enhancement and restoration algorithm deployed on the terminal side is usually a single type and small size algorithm. This algorithm is used for enhancement and restoration processing for different images and videos, and the effect of enhancement and restoration processing is relatively limited, making it difficult to provide users with a tangible picture quality enhancement experience. Summary of the invention

[0004] Examples of the present disclosure provide image quality enhancement methods, apparatuses, user equipment, servers, storage media, and computer program products.

[0005] According to a first aspect of the present disclosure, there is provided an image quality enhancement method, which is applied to a user device, and the image quality enhancement method comprises: receiving a model index from a server, wherein the user device is deployed with multiple image quality enhancement models, and the model index indicates that at least one image quality enhancement model among the multiple image quality enhancement models is used to perform image quality enhancement on a target video; and performing image quality enhancement on the target video based on the at least one image quality enhancement model indicated by the model index.

[0006] Optionally, each of the multiple image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0007] Optionally, the multiple image quality enhancement models include at least one of the following items: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharpening model for reducing the degree of image sharpness.

[0008] Optionally, the server and the user equipment both deploy image quality enhancement models for the same type of image quality issues.

[0009] Optionally, the user equipment pre-deploys the multiple image quality enhancement models; or deploys the multiple image quality enhancement models in response to a deployment instruction received from the server.

[0010] According to a second aspect of the present disclosure, there is provided an image quality enhancement method, which is applied to a server, and the image quality enhancement method comprises: performing image quality enhancement on a target video using a plurality of image quality enhancement models respectively to obtain a plurality of target videos after image quality enhancement, wherein the server is deployed with the plurality of image quality enhancement models; obtaining quality indicator data for each target video after image quality enhancement; determining at least one image quality enhancement model whose quality indicator data meets preset conditions as a model for performing image quality enhancement on the target video; and sending a model index corresponding to the at least one image quality enhancement model to a user device.

[0011] Optionally, each of the multiple image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0012] Optionally, the obtaining of quality indicator data for each target video that has undergone image quality enhancement includes: for each target video that has undergone image quality enhancement, calculating a quality indicator score for the target video that has undergone image quality enhancement based on at least one quality indicator; and determining at least one image quality enhancement model whose quality indicator data meets preset conditions as a model for performing image quality enhancement on the target video includes: determining at least one image quality enhancement model with the best quality indicator score as a model for performing image quality enhancement on the target video.

[0013] Optionally, the multiple image quality enhancement models include at least one of the following items: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharpening model for reducing the degree of image sharpness.

[0014] Optionally, both the server and the user equipment are deployed with image quality enhancement models for the same type of image quality problems.

[0015] According to a third aspect of the present disclosure, there is provided an image quality enhancement device, which is applied to a user device, and the image quality enhancement device comprises: an index receiving module, configured to: receive a model index from a server, wherein the user device is deployed with multiple image quality enhancement models, and the model index indicates that at least one image quality enhancement model among the multiple image quality enhancement models is used to perform image quality enhancement on a target video; an image enhancement module, configured to: perform image quality enhancement on the target video based on the at least one image quality enhancement model indicated by the model index.

[0016] Optionally, each of the multiple image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0017] Optionally, the multiple image quality enhancement models include at least one of the following items: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharpening model for reducing the degree of image sharpness.

[0018] Optionally, both the server and the user equipment deploy image quality enhancement models for the same type of image quality issues.

[0019] Optionally, the user equipment pre-deploys the multiple image quality enhancement models; or deploys the multiple image quality enhancement models in response to a deployment instruction received from the server.

[0020] According to a fourth aspect of the present disclosure, there is provided an image quality enhancement device, which is applied to a server, and the image quality enhancement device comprises: an image enhancement module, configured to respectively use a plurality of image quality enhancement models to perform image quality enhancement on a target video to obtain a plurality of target videos after image quality enhancement, wherein the server is deployed with the plurality of image quality enhancement models; an indicator acquisition module, configured to obtain quality indicator data for each target video after image quality enhancement; a model determination module, configured to determine at least one image quality enhancement model whose quality indicator data meets preset conditions as a model for performing image quality enhancement on the target video; and an index sending module, configured to send a model index corresponding to the at least one image quality enhancement model to a user device.

[0021] Optionally, each of the multiple image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0022] Optionally, the indicator acquisition module is configured to: for each target video that has undergone image quality enhancement, calculate the quality indicator score of the target video that has undergone image quality enhancement based on at least one quality indicator; the model determination module is configured to: determine at least one image quality enhancement model with the best quality indicator score as the model for performing image quality enhancement on the target video.

[0023] Optionally, the multiple image quality enhancement models include at least one of the following items: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharpening model for reducing the degree of image sharpness.

[0024] Optionally, both the server and the user equipment are deployed with image quality enhancement models for the same type of image quality problems.

[0025] According to a fifth aspect of the present disclosure, a user device is provided, comprising: one or more processors; and a memory, coupled to the one or more processors and configured to store instructions that can be executed by the one or more processors, wherein the one or more processors are configured to execute the image quality enhancement method described in the first aspect of the present disclosure when executing the instructions.

[0026] According to the sixth aspect of the present disclosure, a server is provided, comprising: one or more processors; and a memory, coupled to the one or more processors and configured to store instructions that can be executed by the one or more processors, wherein the one or more processors are configured to execute the image quality enhancement method described in the second aspect of the present disclosure when executing the instructions.

[0027] According to a seventh aspect of the present disclosure, a non-transitory computer-readable storage medium for storing computer-executable instructions is provided, wherein the computer-executable instructions, when executed by one or more computer processors, cause the one or more computer processors to perform the image quality enhancement method according to the present disclosure.

[0028] According to an eighth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the image quality enhancement method according to the present disclosure.

[0029] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0030] In the present disclosure, a terminal-cloud collaborative image quality enhancement solution is provided, which offloads the complex computing tasks of selecting image quality enhancement models that consume computing power to the cloud for processing, so that the terminal can directly use the corresponding image quality enhancement model to perform image quality enhancement, which can not only reduce the power consumption of performing image quality enhancement on the terminal side, but also ensure the effect of image quality enhancement. In addition, a variety of lightweight models for image quality enhancement for a single type of image quality problem can be deployed on both the terminal and the cloud. With the help of the powerful computing power of the cloud side (i.e., the server side), at least one type of image quality enhancement model that is most suitable for the target video can be searched, so that the terminal can directly call the corresponding image quality enhancement model to perform image quality enhancement on the target video, which can improve the robustness of the terminal when facing different types of image quality problems and achieve a better image quality enhancement and repair effect.

[0031] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate examples according to the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0033] Figure 1 is a schematic diagram showing an end-cloud collaborative image quality enhancement solution according to an exemplary embodiment of the present disclosure.

[0034] Figure 2 is a flowchart illustrating an image quality enhancement method applied to a user equipment according to an exemplary embodiment of the present disclosure.

[0035] Figure 3 is a flowchart illustrating an image quality enhancement method applied to a server according to an exemplary embodiment of the present disclosure.

[0036] Figure 4 It is a flowchart showing a method for enhancing image quality through end-cloud collaboration according to an exemplary embodiment of the present disclosure.

[0037] Figure 5 is a block diagram illustrating an image quality enhancement apparatus applied to a user equipment according to an exemplary embodiment of the present disclosure.

[0038] Figure 6 is a block diagram illustrating an image quality enhancement apparatus applied to a server according to an exemplary embodiment of the present disclosure.

[0039] Figure 7 is a block diagram illustrating an image quality enhancement system according to an exemplary embodiment of the present disclosure.

[0040] Figure 8is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0041] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0043] As mentioned above, although there are many benefits to deploying image enhancement and restoration algorithms on the terminal side, it is not easy to design such an algorithm that can run on the terminal side. For example, for short video applications, on the one hand, it is necessary to consider the power consumption of the terminal and not let the mobile phone overheat; on the other hand, it is also necessary to consider that the processing speed of the algorithm cannot be too slow to avoid jamming during video playback; on the third hand, it is also necessary to consider the effect of the image enhancement and restoration algorithm, that is, the algorithm needs to be able to improve the image quality while also having strong robustness, that is, the algorithm needs to be able to handle a variety of complex video quality problems to avoid distortion of the picture. In fact, the first two aspects have constrained the algorithm model from being designed too large, because compared with the server, the computing power of the terminal is limited after all; for the third aspect mentioned above, according to conventional experience, the algorithm model needs to be large enough to achieve a better image enhancement effect. It can be seen that in the related technology, avoiding the image enhancement and restoration algorithm from being too large and improving the image enhancement effect are contradictory.

[0044] In order to solve the above-mentioned problems existing in the related art, the present disclosure provides an end-cloud collaborative image quality enhancement solution, which offloads the complex computing tasks that consume computing power for selecting image quality enhancement models to the cloud for processing, so that the terminal can directly use the corresponding image quality enhancement model to perform image quality enhancement, which can not only reduce the power consumption of performing image quality enhancement on the terminal side, but also ensure the effect of image quality enhancement. In addition, a variety of lightweight models for image quality enhancement for a single type of image quality problem can be deployed on both the end and the cloud. With the help of the powerful computing power of the cloud side (i.e., the server side), at least one type of image quality enhancement model that is most suitable for the target video can be searched, so that the terminal can directly call the corresponding image quality enhancement model to perform image quality enhancement on the target video, which can improve the robustness of the terminal when facing different types of image quality problems and achieve a better image quality enhancement and repair effect.

[0045] Next, we will refer to Figures 1 to 8 The specific implementation process of the image quality enhancement method provided in the present disclosure is described in detail.

[0046] Figure 1 is a schematic diagram showing an end-cloud collaborative image quality enhancement solution according to an exemplary embodiment of the present disclosure. Figure 1 , multiple image quality enhancement models can be deployed on both the server side (i.e., the "cloud" of "end-cloud collaboration") and the mobile side (i.e., the "end" of "end-cloud collaboration"). On the server side, for the sent videos, they can pass through these image quality enhancement models respectively, and the optimal at least one image quality enhancement model for the video can be determined by calculating the objective score reflecting the image quality improvement index of the image quality enhancement model. Then, the server can send the index of at least one image quality enhancement model to the mobile side. The mobile side only needs to call at least one corresponding image quality enhancement model to process the video according to the index sent by the server. This allows the mobile side to only download the image quality enhancement model, and does not need to perform complex and computationally intensive model selection calculations, which can not only reduce the power consumption of image quality enhancement on the terminal side, but also ensure the effect of image quality enhancement.

[0047] On the other hand, since the algorithm model deployed on the mobile terminal is small, its capabilities are limited, but the images and videos played on the mobile terminal are diverse, that is, the mobile terminal may face many different types of image quality problems. In this regard, the problem can be disassembled, that is, in the present disclosure, multiple small models can be designed, and each model can have the ability to focus on solving a certain type of image quality problem while having a weaker comprehensive enhancement and repair capability. For example, a model that focuses on solving blur and distortion, a model that focuses on solving over-sharp problems, and a model that focuses on solving noise distortion. Different images and videos have different characteristics. For example, some pictures are blurry, and their appeal is to deblur; some pictures are already clear, and their appeal is to add some details; some pictures are sharp, and their appeal is to desharpen. Therefore, through the end-cloud collaborative image quality enhancement solution disclosed in the present disclosure, the mobile terminal can use the computing power of the cloud to easily and targetedly call the optimal model to deal with a certain type of image quality problem. In other words, if storage allows, the enhancement and restoration tasks can be broken down into smaller pieces, and the number of small models can be increased on the mobile side to replace computing with storage, allowing different small models to specialize in a certain type of image quality problem in a more refined manner, so as to further improve the image quality enhancement effect.

[0048] Figure 2 1 is a flow chart showing an image quality enhancement method according to an exemplary embodiment of the present disclosure, which is applied to a user equipment. Here, the user equipment may also be referred to as a mobile terminal, a terminal, etc.

[0049] Reference Figure 2 In step 201, a model index may be received from a server, wherein the model index may indicate at least one image quality enhancement model among a plurality of image quality enhancement models for performing image quality enhancement on a target video.

[0050] According to an exemplary embodiment of the present disclosure, as described above, each of the above-mentioned multiple image quality enhancement models can be trained as a lightweight model for performing image quality enhancement for a single type of image quality problem. Exemplarily, the lightweight model for performing image quality enhancement for a single type of image quality problem may include, but is not limited to: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, an anti-oversharp model for reducing the degree of image sharpness, and the like.

[0051] According to an exemplary embodiment of the present disclosure, both the server and the user equipment may deploy image quality enhancement models for the same type of image quality issues.

[0052] According to an exemplary embodiment of the present disclosure, a user device may pre-deploy multiple image quality enhancement models; or, it may also deploy multiple image quality enhancement models in response to a deployment instruction received from a server. Exemplarily, the user device may temporarily deploy the above-mentioned multiple image quality enhancement models when receiving an instruction from a server to perform image quality enhancement on a video; or, when adding a new image quality enhancement model to the server, the user device may also synchronously add the new image quality enhancement model based on the instruction issued by the server; or, when updating an image quality enhancement model deployed on the server, the user device may also synchronously update the corresponding image quality enhancement model based on the update instruction issued by the server, and so on.

[0053] It should be noted that each of the above-mentioned multiple image quality enhancement models can be any lightweight network structure that can meet the needs of image enhancement and restoration business. The present disclosure does not make any specific limitations on this, and the above-mentioned implementation is merely an exemplary description.

[0054] In step 202, image quality enhancement can be performed on the target video based on at least one image quality enhancement model indicated by the model index. That is, the user device can call the corresponding small model downloaded in advance to perform image quality enhancement on the current video based on the model index sent by the server. In this way, in the present disclosure, by proposing a lightweight model deployment solution for end-cloud collaboration, a better and more robust image quality enhancement and restoration effect can be achieved while taking into account power consumption and time consumption.

[0055] Figure 3 is a flowchart illustrating an image quality enhancement method according to an exemplary embodiment of the present disclosure, which is applied to a server.

[0056] Reference Figure 3 In step 301, multiple image quality enhancement models can be used to perform image quality enhancement on the target video to obtain multiple target videos with enhanced image quality, wherein multiple image quality enhancement models can be deployed on the server.

[0057] According to an exemplary embodiment of the present disclosure, each of the above-mentioned multiple image quality enhancement models may be trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0058] In this way, by designing multiple small models and making these lightweight small models focus on the image processing effects, it is possible to meet the processing requirements of different types of videos, thereby improving the robustness of user devices when facing different types of image quality problems, that is, to achieve better, more universal and more robust image quality enhancement and restoration effects.

[0059] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple image quality enhancement models may include at least one of the following items: a deblurring model for removing picture blur, a detail adding model for increasing picture details, a denoising model for removing noise, and an anti-oversharp model for reducing the degree of picture sharpness.

[0060] According to an exemplary embodiment of the present disclosure, both the server and the user equipment may be deployed with image quality enhancement models for the same type of image quality issues.

[0061] In step 302, quality indicator data for each target video after image quality enhancement may be obtained. For example, in the present disclosure, some open source or self-developed subjective quality assessment algorithms may be used to formulate objective image quality indicators. For example, the open source subjective quality assessment algorithm may be, but is not limited to: Blind Referenceless Image Spatial Quality Evaluator (BRISQUE), etc.; the self-developed subjective quality assessment algorithm may be, but is not limited to: Kwai Video Quality Assessment (KVQ), etc.

[0062] It should be noted that in the present disclosure, in addition to considering subjective image quality, other winning rules can be designed according to actual application scenarios when screening image quality enhancement models. For example, the image quality enhancement models can be screened by comprehensively considering factors in multiple dimensions such as the computational complexity of the model, subjective image quality, and user sensitivity to image quality.

[0063] In step 303, at least one image quality enhancement model whose quality index data meets preset conditions may be determined as a model for performing image quality enhancement on the target video. The "preset conditions" may include but are not limited to: the corresponding quality index is optimal, the corresponding quality index ranks in the top few positions, etc.

[0064] In addition, in the present disclosure, there may be a situation where the quality index of any image quality enhancement model is unqualified. In this case, if the hardware performance of the terminal side allows, the indexes of multiple small models with unqualified quality indexes can also be sent to the user device. At this time, by using multiple small models to perform image enhancement on the current video together, each small model contributes a corresponding role in its respective image enhancement field, which can ensure that the image quality enhancement effect of the current video is improved as much as possible.

[0065] According to an exemplary embodiment of the present disclosure, for each target video after image quality enhancement, a quality index score of the target video after image quality enhancement can be calculated based on at least one quality index, and the higher the quality index score, the higher the image quality, that is, the image processing effect of the corresponding image quality enhancement model is better. Exemplarily, the value range of the quality index score can be, but is not limited to: 0 to 5. Then, at least one image quality enhancement model with the best quality index score can be determined as a model for performing image quality enhancement on the target video.

[0066] In this way, by determining the image quality enhancement model with the best quality index score as the model used to perform image quality enhancement on the target video, the best picture quality enhancement and restoration effect can be guaranteed, that is, the video picture with the best viewing experience can be guaranteed.

[0067] In step 304, a model index corresponding to at least one image quality enhancement model may be sent to the user device, that is, the model index corresponding to one or more image quality enhancement models with the best image processing effect may be sent to the user device. Thus, in the present disclosure, by using the pre-analysis capability of the server to screen the image quality enhancement model, the computing pressure of the user device may be reduced, thereby reducing the power consumption of the user device and releasing the computing resources of the user device.

[0068] Figure 4 is a flowchart illustrating an end-cloud collaborative image quality enhancement method according to an exemplary embodiment of the present disclosure.

[0069] Reference Figure 4 In step 401, the server may use multiple image quality enhancement models to perform image quality enhancement on the target video to obtain multiple target videos with enhanced image quality. Multiple image quality enhancement models may be deployed on both the server and the user device.

[0070] According to an exemplary embodiment of the present disclosure, as described above, each of the above-mentioned multiple image quality enhancement models can be trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0071] In this way, in the present disclosure, multiple small models can be designed, and these lightweight small models can each focus on the image processing effect, thereby meeting the processing requirements of different types of videos, thereby improving the robustness of user devices when facing different types of image quality problems, that is, achieving better, more universal, and more robust image quality enhancement and restoration effects.

[0072] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple image quality enhancement models may include at least one of the following items: a deblurring model for removing picture blur, a detail adding model for increasing picture details, a denoising model for removing noise, and an anti-oversharp model for reducing the degree of picture sharpness.

[0073] According to an exemplary embodiment of the present disclosure, both the server and the user equipment may be deployed with image quality enhancement models for the same type of image quality issues.

[0074] In step 402, the server may obtain quality index data for each target video after image quality enhancement. Specifically, in the present disclosure, some open source or self-developed subjective quality assessment algorithms may be used to formulate objective image quality indicators. For example, the open source subjective quality assessment algorithm may be, but not limited to, BRISQUE, etc.; the self-developed subjective quality assessment algorithm may be, but not limited to, KVQ, etc.

[0075] In step 403, the server may determine at least one image quality enhancement model whose quality indicator data meets a preset condition as a model for performing image quality enhancement on the target video.

[0076] According to an exemplary embodiment of the present disclosure, the server may calculate the quality index score of each target video after image quality enhancement based on at least one quality index, and the higher the quality index score, the higher the image quality, that is, the better the image processing effect of the corresponding image quality enhancement model. Exemplarily, the value range of the quality index score may be, but is not limited to, 0 to 5. Then, the server may determine at least one image quality enhancement model with the best quality index score as the model for performing image quality enhancement on the target video.

[0077] In this way, by determining the image quality enhancement model with the best quality index score as the model used to perform image quality enhancement on the target video, the best picture quality enhancement and restoration effect can be guaranteed, that is, the video picture with the best viewing experience can be guaranteed.

[0078] In step 404, the server may send a model index corresponding to at least one image quality enhancement model to the user device. That is, the server may send a model index corresponding to one or more image quality enhancement models with the best image processing effect to the user device. In this way, in the present disclosure, by using the pre-analysis capability of the server to screen the image quality enhancement model, the computing pressure of the user device may be reduced, thereby reducing the power consumption of the user device and releasing the computing resources of the user device.

[0079] In step 405, the user equipment may receive a model index, that is, the user equipment may receive a model index corresponding to one or more image quality enhancement models with the best image processing effect sent by the server.

[0080] According to an exemplary embodiment of the present disclosure, a user device may pre-deploy multiple image quality enhancement models; or, it may also deploy multiple image quality enhancement models in response to a deployment instruction received from a server. Exemplarily, the user device may temporarily deploy the above-mentioned multiple image quality enhancement models when receiving an instruction from a server to perform image quality enhancement on a video; or, when adding a new image quality enhancement model to the server, the user device may also synchronously add the new image quality enhancement model based on the instruction issued by the server; or, when updating an image quality enhancement model deployed on the server, the user device may also synchronously update the corresponding image quality enhancement model based on the update instruction issued by the server, and so on.

[0081] In step 406, the user device may perform image quality enhancement on the target video based on at least one image quality enhancement model indicated by the model index. That is, the user device may call the corresponding small model downloaded in advance to perform image quality enhancement on the current video based on the model index sent by the server.

[0082] Figure 5 is a block diagram showing an image quality enhancement apparatus 500 according to an exemplary embodiment of the present disclosure, which is applied to a user equipment.

[0083] Reference Figure 5 , the image quality enhancement device 500 may include an index receiving module 501 and an image enhancement module 502.

[0084] The index receiving module 501 can receive a model index from a server, wherein a plurality of image quality enhancement models can be deployed on a user device, and the model index can indicate at least one image quality enhancement model among the plurality of image quality enhancement models for performing image quality enhancement on a target video.

[0085] In this way, by using the pre-analysis capability of the server to screen the image quality enhancement model, the computing pressure of the user device can be reduced, thereby reducing the power consumption of the user device and releasing the computing resources of the user device.

[0086] According to an exemplary embodiment of the present disclosure, each of the above-mentioned multiple image quality enhancement models may be trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0087] In this way, in the present disclosure, multiple small models can be designed, and these lightweight small models can each focus on the image processing effect, thereby meeting the processing requirements of different types of videos, thereby improving the robustness of user devices when facing different types of image quality problems, that is, achieving better, more universal, and more robust image quality enhancement and restoration effects.

[0088] According to an exemplary embodiment of the present disclosure, a lightweight model for performing image quality enhancement for a single type of image quality problem may include, but is not limited to: a deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, an anti-oversharpening model for reducing the degree of image sharpness, and the like.

[0089] According to an exemplary embodiment of the present disclosure, both the server and the user equipment may deploy image quality enhancement models for the same type of image quality issues.

[0090] According to an exemplary embodiment of the present disclosure, a user device may pre-deploy multiple image quality enhancement models; or, it may also deploy multiple image quality enhancement models in response to a deployment instruction received from a server. Exemplarily, the user device may temporarily deploy the above-mentioned multiple image quality enhancement models when receiving an instruction from a server to perform image quality enhancement on a video; or, when adding a new image quality enhancement model to the server, the user device may also synchronously add the new image quality enhancement model based on the instruction issued by the server; or, when updating an image quality enhancement model deployed on the server, the user device may also synchronously update the corresponding image quality enhancement model based on the update instruction issued by the server, and so on.

[0091] The image enhancement module 502 can perform image quality enhancement on the target video based on at least one image quality enhancement model indicated by the model index. That is, the user device can call the corresponding small model downloaded in advance to perform image quality enhancement on the current video based on the model index sent by the server.

[0092] In this way, in the present disclosure, by proposing a lightweight model deployment solution for end-cloud collaboration, it is possible to achieve a better and more robust image quality enhancement and restoration effect while taking into account power consumption and time consumption.

[0093] Figure 6 is a block diagram showing an image quality enhancement apparatus 600 according to an exemplary embodiment of the present disclosure, which is applied to a server.

[0094] Reference Figure 6 The image quality enhancement device 600 may include an image enhancement module 601, an indicator acquisition module 602, a model determination module 603 and an index sending module 604.

[0095] The image enhancement module 601 can use multiple image quality enhancement models to perform image quality enhancement on the target video to obtain multiple target videos with enhanced image quality, wherein multiple image quality enhancement models can be deployed on the server.

[0096] According to an exemplary embodiment of the present disclosure, as described above, each of the above-mentioned multiple image quality enhancement models can be trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

[0097] In this way, by designing multiple small models and making these lightweight small models focus on the image processing effects, it is possible to meet the processing requirements of different types of videos, thereby improving the robustness of user devices when facing different types of image quality problems, that is, to achieve better, more universal and more robust image quality enhancement and restoration effects.

[0098] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple image quality enhancement models may include at least one of the following items: a deblurring model for removing picture blur, a detail adding model for increasing picture details, a denoising model for removing noise, and an anti-oversharp model for reducing the degree of picture sharpness.

[0099] According to an exemplary embodiment of the present disclosure, both the server and the user equipment may be deployed with image quality enhancement models for the same type of image quality issues.

[0100] The index acquisition module 602 can obtain quality index data for each target video after image quality enhancement. Specifically, in the present disclosure, some open source or self-developed subjective quality assessment algorithms can be used to formulate objective image quality indicators. For example, the open source subjective quality assessment algorithm can be but not limited to: BRISQUE, etc.; the self-developed subjective quality assessment algorithm can be but not limited to: KVQ, etc.

[0101] The model determination module 603 may determine at least one image quality enhancement model whose quality indicator data meets a preset condition as a model for performing image quality enhancement on the target video.

[0102] According to an exemplary embodiment of the present disclosure, for each target video that has undergone image quality enhancement, the indicator acquisition module 602 can calculate the quality indicator score of the target video that has undergone image quality enhancement based on at least one quality indicator, and the higher the quality indicator score, the higher the image quality, that is, the image processing effect of the corresponding image quality enhancement model is better. Exemplarily, the value range of the quality indicator score can be, but is not limited to, 0 to 5. Then, the model determination module 603 can determine at least one image quality enhancement model with the best quality indicator score as the model for performing image quality enhancement on the target video.

[0103] In this way, by determining the image quality enhancement model with the best quality index score as the model used to perform image quality enhancement on the target video, the best picture quality enhancement and restoration effect can be guaranteed, that is, the video picture with the best viewing experience can be guaranteed.

[0104] The index sending module 604 can send a model index corresponding to at least one image quality enhancement model to the user device, that is, the model index corresponding to one or more image quality enhancement models with the best image processing effect can be sent to the user device. In this way, in the present disclosure, by using the pre-analysis capability of the server to screen the image quality enhancement model, the computing pressure of the user device can be reduced, and the power consumption of the user device can be reduced and the computing resources of the user device can be released.

[0105] Figure 7 is a block diagram showing an image quality enhancement system 700 according to an exemplary embodiment of the present disclosure. The image quality enhancement system 700 may include a server 701 and a user device 702, and multiple image quality enhancement models may be deployed on both the server 701 and the user device 702.

[0106] Reference Figure 7 , server 701, can be configured as:

[0107] Use multiple image quality enhancement models to perform image quality enhancement on the target video respectively, and obtain multiple target videos with enhanced image quality. Then, quality indicator data for each target video with enhanced image quality can be obtained. Next, at least one image quality enhancement model whose quality indicator data meets the preset conditions can be determined as a model for performing image quality enhancement on the target video. Then, a model index corresponding to the at least one image quality enhancement model can be sent to the user equipment 702.

[0108] The user equipment 702 may be configured as follows:

[0109] Receive the model index sent by the server 701. Then, based on at least one image quality enhancement model indicated by the model index sent by the server 701, image quality enhancement may be performed on the target video.

[0110] Figure 8 A computing environment 810 is shown coupled to a user interface 850. The computing environment 810 may be part of a data processing server. The computing environment 810 includes a processor 820, a memory 830, and an input / output (I / O) interface 840.

[0111] The processor 820 generally controls the overall operation of the computing environment 810, such as operations associated with display, data acquisition, data communication, and image processing. The processor 820 may include one or more processors for executing instructions to perform all or some steps in the above method. In addition, the processor 820 may include one or more modules that facilitate the interaction between the processor 820 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0112] The memory 830 is configured to store various types of data to support the operation of the computing environment 810. The memory 830 may include predetermined software 832. Examples of such data include instructions for any application or method operating on the computing environment 810, video data sets, image data, etc. The memory 830 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0113] The I / O interface 840 provides an interface between the processor 820 and peripheral interface modules such as a keyboard, a click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scanning button, and a stop scanning button.

[0114] In an embodiment, a user device is also provided, comprising: one or more processors; and a memory coupled to the one or more processors and configured to store instructions that can be executed by the one or more processors, wherein the one or more processors are configured to perform the above-mentioned image quality enhancement method when executing the instructions.

[0115] In an embodiment, a server is also provided, comprising: one or more processors; and a memory coupled to the one or more processors and configured to store instructions that can be executed by the one or more processors, wherein the one or more processors are configured to perform the above-mentioned image quality enhancement method when executing the instructions.

[0116] In an embodiment, a non-transitory computer-readable storage medium for storing computer-executable instructions is also provided. When the computer-executable instructions are executed by one or more computer processors, the one or more computer processors perform the above-mentioned image quality enhancement method.

[0117] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 820); and a non-transitory computer-readable storage medium or memory 830 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0118] In an embodiment, a computer program product including, for example, a plurality of programs in a memory 830 is also provided, which can be executed by a processor 820 in a computing environment 810 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0119] In an embodiment, the computing environment 810 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0120] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0121] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0122] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A method for enhancing image quality, characterized in that: Applied to a server, the image quality enhancement method comprises: Using a plurality of image quality enhancement models to perform image quality enhancement on a target video respectively, to obtain a plurality of target videos with enhanced image quality, wherein the server is deployed with the plurality of image quality enhancement models; Obtaining quality indicator data for each target video after image quality enhancement; Determining at least one image quality enhancement model whose quality indicator data meets a preset condition as a model for performing image quality enhancement on the target video; A model index corresponding to the at least one image quality enhancement model is sent to a user device.

2. The image quality enhancement method according to claim 1, characterized in that: Each of the plurality of image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

3. The image quality enhancement method according to claim 1, characterized in that: The obtaining of quality indicator data for each target video after image quality enhancement includes: For each target video after image quality enhancement, calculating a quality index score of the target video after image quality enhancement based on at least one quality index; The step of determining at least one image quality enhancement model whose quality indicator data meets a preset condition as a model for performing image quality enhancement on the target video comprises: At least one image quality enhancement model with an optimal quality indicator score is determined as a model for performing image quality enhancement on the target video.

4. The image quality enhancement method according to claim 1, characterized in that: The plurality of image quality enhancement models include at least one of the following: A deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharp model for reducing the degree of image sharpness.

5. The image quality enhancement method according to claim 1, characterized in that: The server and the user equipment are both deployed with image quality enhancement models for the same type of image quality problems.

6. A method for enhancing image quality, characterized in that: Applied to user equipment, the image quality enhancement method includes: Receiving a model index from a server, wherein the user equipment is deployed with a plurality of image quality enhancement models, and the model index indicates that at least one image quality enhancement model among the plurality of image quality enhancement models is used to perform image quality enhancement on a target video; Image quality enhancement is performed on the target video based on the at least one image quality enhancement model indicated by the model index.

7. The image quality enhancement method according to claim 6, characterized in that: Each of the plurality of image quality enhancement models is trained as a lightweight model for performing image quality enhancement for a single type of image quality problem.

8. The image quality enhancement method according to claim 6, characterized in that: The plurality of image quality enhancement models include at least one of the following: A deblurring model for removing image blur, a detail adding model for increasing image details, a denoising model for removing noise, and an anti-oversharp model for reducing the degree of image sharpness.

9. The image quality enhancement method according to claim 6, characterized in that: The server and the user equipment both deploy image quality enhancement models for the same type of image quality problems.

10. The image quality enhancement method according to claim 6, characterized in that: The user equipment deploys the multiple image quality enhancement models in advance; or deploys the multiple image quality enhancement models in response to a deployment instruction received from the server.

11. An image quality enhancement device, characterized in that: Applied to user equipment, the image quality enhancement device comprises: An index receiving module is configured to: receive a model index from a server, wherein the user equipment is deployed with a plurality of image quality enhancement models, and the model index indicates that at least one image quality enhancement model among the plurality of image quality enhancement models is used to perform image quality enhancement on a target video; The image enhancement module is configured to perform image quality enhancement on the target video based on the at least one image quality enhancement model indicated by the model index.

12. An image quality enhancement device, characterized in that: Applied to a server, the image quality enhancement device comprises: An image enhancement module is configured to: use a plurality of image quality enhancement models to perform image quality enhancement on a target video respectively, to obtain a plurality of target videos with enhanced image quality, wherein the server is deployed with the plurality of image quality enhancement models; The index acquisition module is configured to: acquire quality index data for each target video after image quality enhancement; A model determination module is configured to: determine at least one image quality enhancement model whose quality indicator data meets a preset condition as a model for performing image quality enhancement on the target video; The index sending module is configured to send a model index corresponding to the at least one image quality enhancement model to a user device.

13. A server, comprising: one or more processors; as well as a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, Wherein, when executing the instructions, the one or more processors are configured to perform the image quality enhancement method according to any one of claims 1-5.

14. A user equipment, comprising: one or more processors; as well as a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, Wherein, when executing the instructions, the one or more processors are configured to perform the image quality enhancement method according to any one of claims 6-10.

15. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the image quality enhancement method according to any one of claims 1-10.

16. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the image quality enhancement method according to any one of claims 1 to 10 is implemented.

Images