Image processing method, device, program product, computer equipment and storage medium

By performing feature extraction and fusion processing on the image and training the image processing model, the problem of difficulty in taking into account both the deblocking effect and super-resolution technology in the existing technology is solved, and the image quality is improved.

CN116385267BActive Publication Date: 2024-11-08TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310362728.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-29
Publication Date
2024-11-08
Estimated Expiration
2043-03-29

AI Technical Summary

Technical Problem

In the combined application of deblocking effect and super-resolution technology, it is difficult to simultaneously eliminate the blocking effect and retain super-resolution details, resulting in limited image quality improvement.

Method used

By performing the first feature extraction process and the second feature extraction process on the image, the features used for deblocking effect and strengthening image details are obtained, and then the super-resolution image restoration process is performed, and the image processing model is used for training. Reduce the difference between the output image and the reference image, and achieve the removal of block effects and the retention of details.

Benefits of technology

While removing block effects, retaining super-resolution details improves image quality and ensures high resolution and sharpness of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116385267B_ABST
    Figure CN116385267B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, apparatus, program product, computer equipment and storage medium, the method comprising: performing a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing a blocking effect; performing a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details; fusing the first image feature and the second image feature to obtain a fused image feature; performing a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image. By adopting the present application, the super-resolution details can be retained while removing the blocking effect, thereby improving the image quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, program product, computer equipment and storage medium. Background Art

[0002] With the continuous development of computer and artificial intelligence (AI) technology, image processing has become an important direction in the field of computer science and artificial intelligence. Deblocking and super-resolution are the two most critical technologies in image processing and coding transmission. The main function of deblocking is to remove the block effect caused by the video encoding process through AI. Super-resolution technology mainly upsamples low-resolution videos through AI so as to upsample low-resolution transmitted videos, thereby saving bandwidth and ensuring the subjective quality of the video. However, deblocking will remove image details, and super-resolution will bring additional noise while enlarging the image to increase details. Therefore, it is usually difficult to reconcile deblocking and super-resolution technologies when they are used in combination.

[0003] Therefore, how to eliminate block effects and retain super-resolution details to improve image quality has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides an image processing method, apparatus, program product, computer device and storage medium, which can remove block effects while retaining super-resolution details, thereby improving image quality.

[0005] On the one hand, the present application provides an image processing method, including: performing a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects; performing a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details; fusing the first image feature and the second image feature to obtain a fused image feature; performing a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image.

[0006] Optionally, the first image feature and the second image feature are fused to obtain a fused image feature, including: performing dimensionality reduction processing on the first image feature to obtain a first offset feature; performing correction processing on the second image feature based on the first offset feature to obtain a corrected second image feature; performing dimensionality reduction processing on the second image feature to obtain a second offset feature; performing correction processing on the first image feature based on the second offset feature to obtain a corrected first image feature; and fusing the corrected first image feature with the corrected second image feature to obtain a fused image feature.

[0007] Optionally, the processed image is obtained by calling an image processing model for processing, and the training method of the image processing model includes: obtaining a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than the resolution of the sample image; calling an initial image processing model to extract a first sample image feature of the sample image, wherein the first sample image feature includes an image feature for deblocking effect; calling the initial image processing model to extract a second sample image feature of the sample image, wherein the second sample image feature includes an image feature for enhancing image details; fusing the first sample image feature and the second sample image feature to obtain a fused sample image feature; performing super-resolution image restoration processing on the fused sample image feature to obtain an output image, wherein the resolution of the output image matches the resolution of the reference image; training the initial image processing model with the training goal of reducing the difference between the output image and the reference image to obtain the image processing model.

[0008] Optionally, obtaining the sample image includes: performing a first resolution reduction process on the reference image to obtain the sample image.

[0009] Furthermore, the image processing method also includes: performing a second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image matches the resolution of the sample image and the clarity of the intermediate reference image is higher than the clarity of the sample image; performing image restoration processing on the first sample image feature to obtain an intermediate image; training the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain the image processing model, including: training the initial image processing model with the goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain the image processing model.

[0010] Optionally, obtaining a sample image and a reference image includes: performing super-resolution processing on a source reference image to obtain an enlarged image; performing deblocking processing on the enlarged image to obtain a deblocked image; and performing resolution reduction processing on the deblocked image to obtain a reference image.

[0011] On the one hand, the present application provides an image processing device, including: a feature extraction module, used to perform a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects; the feature extraction module is also used to perform a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details; the fusion module is also used to fuse the first image feature and the second image feature to obtain a fused image feature; the processing module is also used to perform a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image

[0012] Optionally, the fusion module is used to fuse the first image feature and the second image feature to obtain a fused image feature, including: performing dimensionality reduction processing on the first image feature to obtain a first offset feature; performing correction processing on the second image feature based on the first offset feature to obtain a corrected second image feature; performing dimensionality reduction processing on the second image feature to obtain a second offset feature; performing correction processing on the first image feature based on the second offset feature to obtain a corrected first image feature; and fusing the corrected first image feature with the corrected second image feature to obtain a fused image feature.

[0013] Optionally, the processed image is obtained by the processing module calling the image processing model for processing, and the training method of the image processing model includes: obtaining a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than the resolution of the sample image; calling the initial image processing model to extract a first sample image feature of the sample image, wherein the first sample image feature includes an image feature for deblocking effect; calling the initial image processing model to extract a second sample image feature of the sample image, wherein the second sample image feature includes an image feature for enhancing image details; fusing the first sample image feature and the second sample image feature to obtain a fused sample image feature; performing super-resolution image restoration processing on the fused sample image feature to obtain an output image, wherein the resolution of the output image matches the resolution of the reference image; training the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain the image processing model.

[0014] Optionally, the processing module obtains the sample image, including: performing a first resolution reduction process on the reference image to obtain the sample image.

[0015] Optionally, the processing module is also used to: perform a second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image matches the resolution of the sample image and the clarity of the intermediate reference image is higher than the clarity of the sample image; perform image restoration processing on the first sample image feature to obtain an intermediate image; and train the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain an image processing model, including: training the initial image processing model with the goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain an image processing model.

[0016] Optionally, the processing module obtains a sample image and a reference image, including: performing super-resolution processing on the source reference image to obtain an enlarged image; performing deblocking processing on the enlarged image to obtain a deblocked image; and performing resolution reduction processing on the deblocked image to obtain a reference image.

[0017] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.

[0018] In one aspect, the present application provides a computer-readable storage medium storing a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method in the above aspect.

[0019] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional modes such as the above-mentioned one aspect.

[0020] The present application performs a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing the blocking effect; performs a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for strengthening the image details; fuses the first image feature and the second image feature to obtain a fused image feature; performs a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image. It can be seen that in the method proposed in the present application, the extracted first image feature helps to remove the blocking effect of the image, and the extracted second image feature can strengthen the image details and help perform super-resolution processing. After fusing the first image feature and the second image feature, by performing a super-resolution restoration process on the fused image feature, the blocking effect can be eliminated while retaining the super-resolution details, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a schematic diagram of the architecture of an image processing system provided by the present application;

[0023] Figure 2 It is a flowchart of a training method for an image processing model provided in this application;

[0024] Figure 3 It is a flowchart of a training method of an image processing model provided in this application;

[0025] Figure 4 is a block diagram of an example of a method for obtaining a reference image provided by the present application;

[0026] Figure 5 is a block diagram of an example of obtaining a difference between a sample image and an intermediate reference image provided by the present application;

[0027] Figure 6 This is an example block diagram of the training of an image processing model provided by this application;

[0028] Figure 7 is a flow chart of an image processing method provided by the present application;

[0029] Figure 8is a flowchart of an example of another image processing method provided by the present application;

[0030] Fig. 9 It is a structural schematic diagram of an image processing device provided by the present application;

[0031] Fig.10 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. With the development of technology, artificial intelligence technology has been applied in more fields and played an increasingly important role.

[0034] This application involves artificial intelligence related technologies. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] This application mainly involves machine learning in artificial intelligence. Among them, machine learning (ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines, specializing in how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0037] The machine learning involved in this application mainly refers to using machine learning to obtain a neural network for image processing, so as to perform image processing through the neural network, remove block effects while retaining super-resolution details, and improve image quality. For details, please refer to the description in the following embodiments.

[0038] The image processing method provided in the embodiment of the present application can be applied in a computer device such as a terminal or a server. After acquiring an image, the computer device can perform deblocking and super-resolution processing on the image to remove the block effect while retaining the super-resolution details, thereby acquiring an image with better image quality.

[0039] Specifically, the computer device may perform a first feature extraction process on the image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects. In addition, the computer device may also perform a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details. Then, the computer device may fuse the first image feature and the second image feature to obtain a fused image feature, and perform a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image.

[0040] In other words, after receiving a low-resolution image, the computer device can perform deblocking and super-resolution processing on the low-resolution image to obtain a high-resolution image, thereby saving bandwidth and ensuring the subjective quality of the image.

[0041] Optionally, the above-mentioned image may be acquired by the computer device from a local memory of the computer device, or may be acquired from the cloud, or may be acquired from other devices.

[0042] See also Figure 1 , Figure 1Schematic diagram of the architecture of an image processing system provided by this application. Figure 1 Taking the schematic diagram of the image processing system shown in FIG. 1 as an example, the image processing system may include a server 100 and a terminal device cluster. The terminal device cluster may include one or more terminal devices. The number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 101, terminal device 102, terminal device 103, ..., terminal device 104; Figure 1 As shown, terminal device 101, terminal device 102, terminal device 103, ..., terminal device 104 can all be connected to the server 100 through a network, so that each terminal device can exchange data with the server 100 through the network connection.

[0043] like Figure 1 The server 100 shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smart terminal such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc.

[0044] The following is a specific description of the embodiment of the present application, taking the communication between the terminal device 101 and the server 100 as an example. If the computer device is the terminal device 101, the terminal device 101 can obtain images from the server 100, for example, the user can download images from the server 100 using the terminal device 101. If the computer device is the server 100, the server 100 can obtain images from the terminal device 101, for example, the user can upload images to the server 100 using the terminal device 101.

[0045] In general, super-resolution methods are divided into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods (i.e., deep learning methods). Deep learning-based methods can be divided into SR methods based on Convolutional Neural Networks (CNN), SR methods based on Residual Networks (ResNet), and SR methods based on Generative Adversarial Networks (GAN).

[0046] Optionally, the computer device may perform deblocking and super-resolution processing on the image through an AI-based deep learning method. For example, the computer device may call an image processing model based on a convolutional neural network to perform deblocking and super-resolution processing on the image. The computer device may obtain sample images and reference images in advance, and then the computer device may train the image processing model based on these sample images and reference images, and use the trained image processing model to process the received low-resolution image.

[0047] See also Figure 2 , Figure 2 is a flowchart of a training method for an image processing model provided in this application. Figure 2 As shown, the computer device acquires a sample image and a reference image, the reference image corresponds to the sample image, and the resolution of the reference image is greater than the resolution of the sample image. These sample images and the reference image constitute a training set 201 of the image processing model. The image processing model can extract a first sample image feature of the sample image including an image feature for deblocking, and extract a second sample image feature of the sample image including an image feature for enhancing image details. After fusing the first sample image feature with the second sample image feature, super-resolution image restoration processing is performed on the fused sample image feature to obtain a high-resolution image, and based on the obtained high-resolution image and the reference image, a trained image processing model 202 is obtained, and the trained image processing model 202 is used to perform image processing tasks.

[0048] Furthermore, the computer device can use the trained image processing model 202 to process the image, and finally obtain a processed image 203, thereby ensuring that the processed image 203 can remove the block effect and retain the super-resolution details, thereby improving the image quality.

[0049] See also Figure 3 , Figure 3 : is a flowchart of a training method for an image processing model provided by the present application. The execution subject in the embodiment of the present application may be the above-mentioned computer device. The computer device may call the initial image processing model and train it to obtain the image processing model provided by the present application. Figure 3 The training method of the image processing model may include:

[0050] Step S301 : acquiring a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than the resolution of the sample image.

[0051] The computer device may obtain a training set of the image processing model, and the training set may include a sample image and a reference image corresponding to the sample image. Specifically, the sample image may be a low-resolution image, and the reference image may be a high-resolution image corresponding to the sample image.

[0052] In some exemplary scenarios, when a computer device is training an image processing model, the quality of the high-resolution images obtained may not be clear enough. The computer device can preprocess these high-resolution images with unclear quality, i.e., source reference images, to obtain reference images with clear quality, so as to facilitate the training of the initial image processing model.

[0053] Please also see Figure 4 , Figure 4 is a block diagram of an example of a method for obtaining a reference image provided by the present application. Figure 4 As shown, the computer device obtains the reference image corresponding to the sample image, which may include: performing super-resolution processing on the source reference image to obtain an enlarged image; performing deblocking processing on the enlarged image to obtain an image with deblocking effect; performing resolution reduction processing on the image with deblocking effect to obtain a reference image.

[0054] Optionally, the resolution reduction algorithm may be a Lanczos resolution reduction algorithm.

[0055] Furthermore, the computer device performs resolution reduction processing on the deblocked image to obtain a reference image, which may also include: performing sharpening processing and denoising processing on the deblocked image, and then performing resolution reduction processing to obtain a reference image.

[0056] For example, Figure 4 As shown, the computer device can perform super-resolution processing 402 on a less clear high-resolution source reference image 401 to increase image details; then perform deblocking processing 403 to remove blocking effects in the image; optionally, since the deblocking algorithm will smoothly remove some image details, the computer device can also perform sharpening processing 404 to enhance image edge details, and then perform noise reduction processing 405 to remove image noise; finally, perform Lanczos resolution reduction processing 406 using, for example, a Lanczos resolution reduction algorithm, so as to obtain a high-resolution reference image 407 with clear image quality.

[0057] It can be found that for some images that are high-resolution but not necessarily clear, the computer device directly uses them as reference images in the image processing model training process, which does not necessarily guide the image processing model to learn better parameters. Using the above method, the computer device increases image details through the super-resolution algorithm, and then uses the deblocking algorithm to remove the block effect. The image obtained by the resolution reduction algorithm is conducive to improving the image quality of the reference image in the image processing model training process. Furthermore, the computer device uses a sharpening algorithm to enhance the edge details of the image that has been processed by the super-resolution algorithm and the deblocking algorithm, and then uses a noise reduction algorithm to remove image noise. Finally, the computer device processes it through the resolution reduction algorithm, which can obtain a reference image with better image quality, which is more conducive to guiding the image processing model to learn better model parameters.

[0058] In some exemplary scenarios, when a computer device trains an image processing model, the acquired high-resolution image may be an image with clear picture quality, and the computer device may train the initial image processing model based on the high-resolution image with clear picture quality.

[0059] Optionally, the computer device may use the high-resolution image with clear image quality as a reference image, and obtain a sample image based on the reference image. The computer device obtaining the sample image may include: performing a first resolution reduction process on the reference image to obtain the sample image.

[0060] Specifically, the computer device can directly obtain a sample image with a lower resolution corresponding to the reference image by performing a first resolution reduction processing on a high-resolution reference image with clear image quality. In this way, the operation of finding a sample image-reference image pair for training an image processing model can be simplified, and a sample image with a lower resolution can be obtained from a high-resolution reference image with clear image quality using a resolution reduction algorithm, so that the training data used to train the image processing model can only retain high-resolution images with clear image quality, thereby reducing the storage space requirement of the training set used to train the image processing model. Generally, the algorithm used in the first resolution reduction processing can be an algorithm with lower complexity and higher computing performance, such as Figure 5 As shown, the computer device may use a Bicubic resolution reduction algorithm to perform Bicubic resolution reduction 502 on the reference image 501 to obtain a sample image 503. The first resolution reduction algorithm may also be any other resolution reduction algorithm with lower complexity and higher computational performance.

[0061] It can be found that by downscaling the reference image, a matching sample image can be directly obtained, which can simplify the operation of finding sample image-reference image pairs for training image processing models, and can reduce the storage space requirements of the training set for training image processing models. In addition, by utilizing a resolution reduction algorithm with lower complexity and higher computational performance, sample images for training image processing models can be obtained more efficiently, thereby improving the efficiency of image processing model training.

[0062] Step S302: calling the initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking.

[0063] The initial image processing model may include a first image feature extraction module, and the computer device may call the initial image processing model, and perform first feature extraction processing on the sample image through the first image feature extraction module to obtain a first sample image feature of the sample image. For example, the first image feature extraction module may include a convolution (Convolution, Conv) plus a parametric rectified linear unit (PReLU), that is, the computer device may extract the first sample image feature of the acquired sample image through an artifact remove (AR) link composed of a convolution plus a parametric rectified linear unit in the initial image processing model.

[0064] In some exemplary scenarios, when a computer device trains an image processing model, the high-resolution image acquired may be an image with clear quality. The training of the image processing model may also include: performing a second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image matches the resolution of the sample image and the clarity of the intermediate reference image is higher than the clarity of the sample image; performing image restoration processing on the features of the first sample image to obtain an intermediate image.

[0065] Specifically, in order to prevent the AR link of the image processing model from removing too many image details during the deblocking process, the computer device can provide image quality constraints to the AR link of the image processing model during the training process. The computer device can perform a second resolution reduction process on a high-resolution reference image with clear image quality to obtain an intermediate reference image, which matches the resolution of the sample image and has a higher definition than the sample image, as a constraint on the first sample image feature extracted in the AR link.

[0066] Usually, as a quality constraint, the intermediate reference image obtained by the second resolution reduction process has relatively high clarity and good image quality, so that the trained image processing model can have a clear output. Therefore, the algorithm used in the second resolution reduction process can be different from the algorithm used in the first resolution reduction process, and the second resolution reduction algorithm can obtain a clear image.

[0067] For example, Figure 5 As shown, the computer device can use the Lanczos resolution reduction algorithm to perform Lanczos resolution reduction 504 on the reference image 501 to obtain an intermediate reference image 505. The second resolution reduction algorithm can also be any other resolution reduction algorithm that can obtain an image with clear image quality.

[0068] Optionally, the resolution of the intermediate reference image matches that of the sample image, which may include: the resolution of the intermediate reference image is the same or substantially the same as that of the sample image. For example, the difference between the resolution of the intermediate reference image and that of the sample image is within a preset threshold range.

[0069] Furthermore, the computer device may perform image restoration processing on the first sample image features extracted in the AR link of the image processing model to obtain an intermediate image, and provide image quality constraints for the AR link of the image processing model based on the intermediate reference image.

[0070] Optionally, providing image quality constraints for the AR link of the image processing model is achieved based on controlling the difference between the intermediate image and the intermediate reference image.

[0071] Optionally, the difference between the intermediate image and the intermediate reference image may include a corresponding AR loss function between the intermediate image and the intermediate reference image. The AR loss function may include, for example, an L2 loss function, an entropy function having a defined form, or any other function capable of representing the difference between the intermediate image and the intermediate reference image.

[0072] For example, Figure 5 As shown, the AR link of the image processing model can extract a first sample image feature 506 from the sample image 503, and the first sample image feature 506 includes an image feature for deblocking. The AR link of the image processing model can perform image restoration processing on the first sample image feature 506 using, for example, a 1×1 convolution to obtain an intermediate image 507. Furthermore, the computer device can obtain an AR L2 loss 508 based on the intermediate image 507 and the intermediate reference image 505.

[0073] Step S303: calling the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details.

[0074] The initial image processing model may include a second image feature extraction module, and the computer device may call the initial image processing model, and perform second feature extraction processing on the sample image through the second image feature extraction module to obtain a second sample image feature of the sample image. For example, the second image feature extraction module may include a convolution plus a parameter-corrected linear unit, that is, the computer device may extract the second sample image feature of the acquired sample image through a super-resolution (SR) link composed of a convolution plus a parameter-corrected linear unit in the initial image processing model.

[0075] It can be known that the order of step S302 and step S303 is not limited here. For example, the computer device may first perform a first feature extraction process on the sample image to obtain a first sample image feature of the sample image, and then perform a second feature extraction process on the sample image to obtain a second sample image feature of the sample image; for another example, the computer device may first perform a second feature extraction process on the sample image to obtain a second sample image feature of the sample image, and then perform a first feature extraction process on the sample image to obtain a first sample image feature of the sample image; for another example, the computer device may simultaneously perform a first feature extraction process on the sample image to obtain a first sample image feature of the sample image, and perform a second feature extraction process on the sample image to obtain a second sample image feature of the sample image.

[0076] Specifically, the computer device can first use the AR link of the initial image processing model to extract the first sample image feature of the acquired sample image, and then use the SR link of the initial image processing model to extract the second sample image feature of the acquired sample image; it can also first use the SR link of the initial image processing model to extract the second sample image feature of the acquired sample image, and then use the AR link of the initial image processing model to extract the first sample image feature of the acquired sample image; it can also use the AR link and SR link of the initial image processing model to simultaneously perform feature extraction processing on the acquired sample image to obtain the first sample image feature and the second sample image feature of the acquired sample image, respectively.

[0077] Step S304: fuse the first sample image feature and the second sample image feature to obtain a fused sample image feature.

[0078] The computer device may call the initial image processing model to fuse, for example, splice, the first sample image feature with the second sample image feature to obtain a fused image feature.

[0079] Stitching may refer to the stitching of channels based on image features. For example, the first sample image feature may be an image feature of the L channel, and the second sample image feature may be an image feature of the M channel. The computer device may stitch the first sample image feature and the second sample image feature based on the channels to obtain a fused image feature of the (L+M) channel.

[0080] Optionally, the computer device may call the initial image processing model to connect or splice the first sample image feature obtained in the AR link with the second sample image feature obtained in the SR link to obtain a fused image feature.

[0081] Step S305 , performing super-resolution image restoration processing on the fused sample image features to obtain an output image, wherein the resolution of the output image matches the resolution of the reference image.

[0082] The computer device may call the initial image processing model and use a suitable super-resolution algorithm to perform super-resolution image restoration processing on the fused sample image features to obtain an output image. For example, the computer device may call the initial image processing model and use a Pixel Shuffle function or any other suitable super-resolution algorithm function to perform super-resolution image restoration processing on the fused image to obtain a high-resolution output image. The resolution of the output image matches the resolution of the reference image.

[0083] Optionally, matching the resolution of the output image with the resolution of the reference image may include: the resolution of the output image is the same or substantially the same as the resolution of the reference image. For example, the difference between the resolution of the output image and the resolution of the reference image is within a preset threshold range.

[0084] Step S306, training the initial image processing model with reducing the difference between the output image and the reference image as the training goal to obtain the image processing model.

[0085] The computer device can train the initial image processing model with the goal of reducing the difference between the output image and the reference image. When the difference is reduced to within the target range, a trained image processing model can be obtained.

[0086] Optionally, the difference between the output image and the reference image may be a loss value calculated by a loss function. The loss function may include, for example, an L2 loss function, an entropy function having a defined form, or any other function capable of representing the difference between the output image and the reference image.

[0087] Optionally, the computer device trains the initial image processing model with the training goal of reducing the difference between the output image and the reference image, which may include: adjusting the parameters of the initial image processing model so that the difference between the output image and the reference image is within a preset threshold range. For example, the computer device may adjust the parameters of the convolution plus parameterized linear unit in the called initial image processing model so that the difference between the output image and the reference image is reduced to within a preset threshold range.

[0088] Optionally, the computer device trains the initial image processing model with the training goal of reducing the difference between the output image and the reference image to obtain the image processing model, which may include training the initial image processing model with the training goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain the image processing model.

[0089] Optionally, the computer device trains the initial image processing model with the goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain the image processing model, which may include reducing the difference between the output image and the reference image and the weighted sum of the difference between the intermediate image and the intermediate reference image. For example, the computer device may adjust the parameters of the convolution plus parameterized corrected linear unit in the called image processing model so that the difference between the output image and the reference image and the weighted sum of the difference between the intermediate image and the intermediate reference image are within a preset threshold range.

[0090] In the embodiment of the present application, the computer device calls the initial image processing model to extract the first sample image feature, which helps to remove the block effect of the sample image. The extracted second sample image feature can enhance the image details and help perform super-resolution processing. After fusing the first sample image feature and the second sample image feature, the fused image feature is subjected to super-resolution restoration processing, which can remove the block effect while retaining the super-resolution details, thereby improving the image quality. It can also be found that the intermediate reference image obtained by using a resolution reduction algorithm with higher clarity and better image quality can provide better image quality constraints during the training process of the image processing model, and can avoid the image features extracted by the AR link during the training process of the image processing model, which makes the image too smooth and loses too many image details; and the initial image processing model is trained with the training goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image, so that the training of the image processing model can be more finely adjusted, which is conducive to obtaining an image processing model with better performance.

[0091] See also Figure 6 , Figure 6 This is an example block diagram of the training of an image processing model provided in this application.

[0092] like Figure 6 As shown, this example block diagram illustrates a method for training an image processing model.

[0093] First, a computer device obtains a low-resolution input image with a resolution of (h, w), which is a single-channel sample image. The AR link and the SR link of the image processing model respectively use convolution + parameter rectified linear unit (Conv+PReLU) to extract 4-channel first sample image features and 4-channel second sample image features. The first sample image features include image features for deblocking effects, and the second sample image features include image features for enhancing image details.

[0094] In the AR link, the image processing model can use, for example, Figure 5 The described method performs image restoration processing on the first sample image features of the 4 channels to obtain an intermediate image, and can be used with, for example, Figure 5 The intermediate reference image obtained by the described method is solved together with the L2 loss to obtain the loss loss-ar-1.

[0095] At the same time, the image processing model can use the G(β) function to reduce the first sample image features of the 4 channels into the first sample image offset features of the 1 channel. For example, the G(β) function can be a 1×1 convolution, which directly reduces the dimension of the first sample image features of the AR link. And use the first sample image offset features of the 1 channel to correct the 4-channel second sample image features of the SR link, such as directly splicing to obtain the corrected second sample image features of the 5 channels. The G(β) function can be implemented in other ways. This application mainly uses the G(β) function to integrate the information of the deblocking effect of the AR link, and merges it into the SR link through a splicing operation. It can be understood that through the G(β) function, the number of channels of the AR feature map to be fused is less than the number of channels of the feature map of the SR link to be fused, which can ensure that the main function of the fused SR link is still focused on the processing of the super-resolution task, and by using 1×1 convolution, the features of multiple channels are synthesized into 1 channel for single-pixel level fusion, which can offset the potential noise that the SR link may amplify in the process of increasing details.

[0096] On the SR link, the image processing model can use the F(α) function to reduce the dimension of the second sample image features of the 4 channels into the second sample image offset features of the 1 channel. For example, the F(α) function can be a 3×3 convolution to reduce the dimension of the second sample image features of the SR link. And use the second sample image offset features of the 1 channel to correct the 4-channel first sample image features of the AR link, such as directly splicing to obtain the corrected first sample image features of the 5 channels. The F(α) function can be implemented in other ways. This application mainly uses the F(α) function to integrate the detail information of the second sample image features of the super-resolution task of the SR link, and merges it into the AR link through a splicing operation. It can be understood that through the F(α) function, the number of channels of the SR feature map to be fused is less than the number of channels of the feature map of the AR link to be fused, which can ensure that the main function of the fused AR link is still focused on the processing of the deblocking effect task, and by using 3×3 convolution, the features of multiple channels are synthesized into 1 channel, and the relatively rich details in the feature map of the SR link can be used to make up for the image details gradually lost in the AR link.

[0097] Furthermore, the AR link and SR link of the image processing model can repeat the above operations. The difference is that the input of the above operations is a single channel, while the inputs of the AR link and the SR link in subsequent repeated operations are 5 channels respectively. At the same time, loss-ar-2 to loss-ar-n can be obtained on the AR link, which will not be repeated here.

[0098] Secondly, the image processing model concatenates the 5-channel features of the AR link and the 5-channel features of the SR link into 10-channel features, and obtains 4-channel features through 3×3 convolution to facilitate super-resolution processing using the Pixel Shuffle function.

[0099] Next, the image processing model uses the Pixel Shuffle function to perform super-resolution restoration on the above 4-channel features to obtain a high-resolution output image with a resolution of (2h, 2w), which can be used with, for example, Figure 4 The reference image obtained by the described method is solved together with the L2 loss to obtain the loss loss-sr.

[0100] Finally, the computer device can get the final L2 loss. The final loss function is L total =W ar *(∑loss-ar-n) / n+W sr *loss-sr, for example, W ar =0.5,W sr= 0.5 means that the weight of AR link and SR link is half each, indicating the degree of integration of AR and SR. In addition, α and β can also be used to adjust the degree of integration of AR and SR. For example, α and β can indicate the number of channels after dimensionality reduction. The computer device can be based on the final loss L total For example, let L total When the value is less than a preset threshold, the parameters of the image processing model are adjusted to complete the training of the image processing model.

[0101] Based on the trained image processing model, the image processing method of the present application can be applied. Exemplarily, a computer device obtains a low-resolution input image with a resolution of (h, w), and the image processing model trained by the above method can be used to process the input image.

[0102] On the AR link, the trained image processing model performs first feature extraction processing on the input image to obtain 4-channel first image features of the image, wherein the first image features include image features for deblocking effects.

[0103] On the SR link, the trained image processing model performs second feature extraction processing on the image to obtain 4-channel second image features of the image, where the second image features include image features for enhancing image details.

[0104] At the same time, the trained image processing model uses the G(β) function to reduce the dimension of the first image features of the four channels into the first image offset features of the one channel, and based on the first image offset features, the second image features are fused and corrected, such as splicing, to obtain the corrected second image features of the five channels. The trained image processing model uses the F(α) function to reduce the dimension of the second image features of the four channels into the second image offset features of the one channel, and based on the second image offset features, the first image features are fused and corrected, such as splicing, to obtain the corrected first image features of the five channels.

[0105] Furthermore, the AR link and SR link of the image processing model can repeat the above operations. The difference is that the input of the above operations is a single channel, while the input of the AR link and the SR link in subsequent repeated operations are 5 channels respectively, which will not be repeated here.

[0106] Next, the trained image processing model concatenates the 5-channel features of the AR link and the 5-channel features of the SR link into 10-channel features, and obtains 4-channel features through 3×3 convolution to facilitate super-resolution processing using the Pixel Shuffle function.

[0107] Finally, the trained image processing model uses the Pixel Shuffle function to perform super-resolution restoration on the above 4-channel features to obtain a high-resolution output image with a resolution of (2h, 2w), that is, the processed image.

[0108] See also Figure 7 , Figure 7 1 is a flowchart of an image processing method provided by the present application. The execution subject in the embodiment of the present application may be a computer device, which may include, for example, a server and a terminal device. Figure 7 As shown, the image processing method may include:

[0109] Step S701, acquiring an image.

[0110] In an exemplary scenario, if a user wants to process an image using the image processing method mentioned in an embodiment of the present application, the user can send an image processing instruction for the image to a computer device. After detecting the image processing instruction, the computer device can acquire the image to execute subsequent steps S702 to S705.

[0111] In another exemplary scenario, the image processing method mentioned in the embodiment of the present application can be applied in a video scenario. For example, after a computer device acquires a video, if the image quality of the images in the video is low, then the computer device can acquire each image in the video. For any image, by executing subsequent steps S702 to S705, a processed image corresponding to any image can be obtained, and then the processed images corresponding to each image can be spliced ​​to obtain the encoded video corresponding to the video, so as to realize the encoding of the video and ensure that the image quality of each image in the encoded video is high.

[0112] Step S702: Perform a first feature extraction process on the image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects.

[0113] The computer device can call the image processing model to perform first feature extraction processing on the image to obtain the first image feature of the image, wherein the first image feature includes an image feature for removing the blocking effect, so as to facilitate removing the blocking effect of the image.

[0114] Specifically, the image processing model may include a first image feature extraction module, and the computer device may call the image processing model, and perform first feature extraction processing on the image through the first image feature extraction module to obtain the first image feature of the image. For example, the first image feature extraction module may include a convolution plus parameterized corrected linear unit, that is, the computer device may extract the first image feature of the acquired image through an AR link composed of a convolution plus parameterized corrected linear unit in the image processing model.

[0115] Step S703: performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features include image features for enhancing image details.

[0116] The computer device can call the image processing model to perform second feature extraction processing on the image to obtain second image features of the image, wherein the second image features include image features for enhancing image details to facilitate super-resolution processing of the image.

[0117] Specifically, the image processing model may include a second image feature extraction module, and the computer device may call the image processing model, and perform second feature extraction processing on the image through the second image feature extraction module to obtain the second image feature of the image. For example, the second image feature extraction module may include a convolution plus parameterized rectified linear unit, that is, the computer device may extract the second image feature of the acquired image through an SR link composed of a convolution plus parameterized rectified linear unit in the image processing model.

[0118] It can be known that the order of step S702 and step S703 is not limited here. For example, the computer device may first perform a first feature extraction process on the image to obtain a first image feature of the image, and then perform a second feature extraction process on the image to obtain a second image feature of the image; for another example, the computer device may first perform a second feature extraction process on the image to obtain a second image feature of the image, and then perform a first feature extraction process on the image to obtain a first image feature of the image; for another example, the computer device may simultaneously perform a first feature extraction process on the image to obtain a first image feature of the image, and perform a second feature extraction process on the image to obtain a second image feature of the image.

[0119] Specifically, the computer device can first use the AR link of the image processing model to extract the first image feature of the acquired image, and then use the SR link of the image processing model to extract the second image feature of the acquired image; it can also first use the SR link of the image processing model to extract the second image feature of the acquired image, and then use the AR link of the image processing model to extract the first image feature of the acquired image; it can also use the AR link and SR link of the image processing model to simultaneously perform feature extraction processing on the acquired image to obtain the first image feature and the second image feature of the acquired image respectively.

[0120] Step S704: fuse the first image feature and the second image feature to obtain a fused image feature.

[0121] The computer device may fuse, for example, stitch, the first image feature with the second image feature to obtain a fused image feature.

[0122] Stitching may refer to the stitching of channels based on image features. For example, the first image feature may be an image feature of the L channel, and the second image feature may be an image feature of the M channel. The computer device may stitch the first image feature and the second image feature based on the channels to obtain a fused image feature of the (L+M) channel.

[0123] Optionally, the computer device may use an image processing model to connect or splice the first image feature obtained in the AR link with the second image feature obtained in the SR link to obtain a fused image feature.

[0124] Step S705 , performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image.

[0125] The computer device may call the image processing model and use a suitable super-resolution algorithm to perform super-resolution image restoration processing on the fused sample image features to obtain an output image. For example, the computer device may call the image processing model and use the Pixel Shuffle function or any other suitable super-resolution algorithm function to perform super-resolution image restoration processing on the fused image to obtain a high-resolution SR image.

[0126] In an embodiment of the present application, a computer device calls an image processing model to extract a first image feature that helps remove the block effect of the image, and the extracted second image feature can enhance the image details and help perform super-resolution processing. After fusing the first image feature with the second image feature, super-resolution restoration processing is performed on the fused image feature, which can eliminate the block effect while retaining the super-resolution details, thereby improving the image quality.

[0127] See also Figure 8 , Figure 8 1 is a flowchart of another example of an image processing method provided by the present application. The execution subject in the embodiment of the present application may be a computer device, which may include, for example, a server and a terminal device. Figure 8 As shown, the image processing method may include:

[0128] Step S801, acquiring an image.

[0129] Step S802: performing a first feature extraction process on the image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects.

[0130] Step S803: performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features include image features for enhancing image details.

[0131] For details of steps S801 to S803, please refer to Figure 7 The description of steps S701 to S703 will not be repeated here.

[0132] Step S804: perform dimensionality reduction processing on the first image feature to obtain a first offset feature.

[0133] The computer device may utilize an image processing model to perform dimensionality reduction processing on the first image feature to obtain a first offset feature.

[0134] Specifically, the first image feature obtained by the image processing model AR link can be a multi-channel or multi-dimensional image feature. The image processing model can perform dimensionality reduction processing on the multi-channel first image feature based on, for example, a 1×1 convolution, and integrate it to obtain, for example, a first offset feature of one channel.

[0135] Step S805: Correct the second image feature based on the first offset feature to obtain a corrected second image feature.

[0136] The computer device can perform correction processing on the second image feature based on the first offset feature to obtain a corrected second image feature.

[0137] Specifically, the image processing model can connect or splice the first offset feature with the second image feature, wherein the number of channels of the first offset feature is smaller than the number of channels of the second image feature, and the second image feature of the SR link is corrected by fusing the first offset feature with the second image feature to obtain a corrected second image feature.

[0138] Step S806: Perform dimensionality reduction processing on the second image feature to obtain a second offset feature.

[0139] The computer device may perform dimensionality reduction processing on the second image feature to obtain a second offset feature.

[0140] Specifically, the second image features acquired by the SR link of the image processing model may be multi-channel or multi-dimensional image features. The image processing model may perform dimensionality reduction processing on the multi-channel second image features based on, for example, a 3×3 convolution, and integrate them to obtain, for example, a second offset feature of one channel.

[0141] Step S807: Correct the first image feature based on the second offset feature to obtain a corrected first image feature.

[0142] The computer device can perform correction processing on the first image feature based on the second offset feature to obtain a corrected first image feature.

[0143] Specifically, the image processing model can connect or splice the second offset feature with the first image feature, wherein the number of channels of the second offset feature is smaller than the number of channels of the first image feature, and the first image feature of the AR link is corrected by fusing the second offset feature with the first image feature to obtain a corrected first image feature.

[0144] It is easy to understand that the number of channels of the first offset feature is smaller than the number of channels of the second image feature, which can ensure that the main function of the second image feature corrected by the SR link after fusion is still focused on the processing of the super-resolution task. In addition, the image features for deblocking included in the first image feature in the AR link can have the characteristics of smoothing the image, so the first image feature can be directly reduced in dimension using, for example, a 1×1 convolution to obtain the first offset feature without further smoothing, and the obtained first offset feature can still have the characteristics of smoothing the image. Therefore, using the first offset feature to correct the second image feature can offset the potential noise that may be amplified in the second image feature during the process of enhancing image details in the SR link.

[0145] Similarly, it can be seen that the number of channels of the second offset feature is less than the number of channels of the first image feature, which can ensure that the main function of the first image feature corrected by the fused AR link is still focused on the processing of the deblocking effect task. In addition, the image features used to enhance image details included in the second image feature in the SR link can enrich the details of the image. Therefore, the second image feature can be reduced in dimension and smoothed at the same time using, for example, a 3×3 convolution to obtain a second offset feature. The obtained second offset feature can be relatively smooth and still have the characteristics of enriching image details. Therefore, using the second offset feature to correct the first image feature can make up for the image details that are gradually lost in the first image feature during the deblocking process of the AR link.

[0146] It can be known that the order of steps S804, S805 and steps S806, S807 is not limited here. The computer device can use the image processing model to simultaneously perform dimensionality reduction processing on the first image feature of the AR link and the second image feature of the SR link to obtain the first offset feature and the second offset feature, and simultaneously use the first offset feature and the second offset feature to correct the second image feature and the first image feature to obtain the corrected second image feature and the corrected first image feature.

[0147] Step S808: Fusing the corrected first image feature with the corrected second image feature to obtain a fused image feature.

[0148] The computer device may fuse the corrected first image feature with the corrected second image feature to obtain a fused image feature.

[0149] Step S809, performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image.

[0150] For details on step S809, please refer to Figure 7 The description of step S705 is not repeated here.

[0151] It can be found that using the first offset feature to correct the second image feature can offset the potential noise that may be amplified in the second image feature during the process of strengthening image details in the SR link; using the second offset feature to correct the first image feature can make up for the image details gradually lost in the first image feature during the process of removing the blocking effect in the AR link. The corrected first image feature is fused with the corrected second image feature to obtain the fused image feature, which is beneficial to the subsequent super-resolution image restoration process to obtain an image that not only removes the blocking effect but also improves the image quality.

[0152] See also Fig. 9 , Fig. 9 900 is a schematic diagram of the structure of an image processing device provided in the present application. The image processing device may be a computer program (including program code) running in a computer device. For example, the image processing device is an application software. The image processing device may be used to execute the corresponding steps of the method provided in the embodiment of the present application. Fig. 9 As shown, the image processing device 900 may include: an acquisition module 901 , a feature extraction module 902 , a fusion module 903 and a processing module 904 .

[0153] The acquisition module 901 is used to acquire images.

[0154] The feature extraction module 902 is used to perform a first feature extraction process on the image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing blocking effects.

[0155] The feature extraction module 902 is further used to perform a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details.

[0156] The fusion module 903 is used to fuse the first image feature and the second image feature to obtain a fused image feature;

[0157] The processing module 904 is used to perform super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image.

[0158] Optionally, the fusion module 903 fuses the first image feature and the second image feature to obtain a fused image feature, including: performing dimensionality reduction processing on the first image feature to obtain a first offset feature; performing correction processing on the second image feature based on the first offset feature to obtain a corrected second image feature; performing dimensionality reduction processing on the second image feature to obtain a second offset feature; performing correction processing on the first image feature based on the second offset feature to obtain a corrected first image feature; and fusing the corrected first image feature with the corrected second image feature to obtain a fused image feature.

[0159] Optionally, the processed image is obtained by processing the processing module 904 by calling the image processing model, and the training method of the image processing model includes: obtaining a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than the resolution of the sample image; calling the initial image processing model to extract a first sample image feature of the sample image, wherein the first sample image feature includes an image feature for deblocking effect; calling the initial image processing model to extract a second sample image feature of the sample image, wherein the second sample image feature includes an image feature for enhancing image details; fusing the first sample image feature and the second sample image feature to obtain a fused sample image feature; performing super-resolution image restoration processing on the fused sample image feature to obtain an output image, wherein the resolution of the output image matches the resolution of the reference image; training the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain the image processing model.

[0160] Optionally, the processing module 904 acquires a sample image, including: performing a first resolution reduction process on the reference image to obtain the sample image.

[0161] Optionally, the processing module 904 is also used to: perform a second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image matches the resolution of the sample image and the clarity of the intermediate reference image is higher than the clarity of the sample image; perform image restoration processing on the first sample image feature to obtain an intermediate image; and train the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain an image processing model, including: training the initial image processing model with the goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain an image processing model.

[0162] Optionally, the processing module 904 obtains a reference image corresponding to the sample image, including: performing super-resolution processing on the source reference image to obtain an enlarged image; performing deblocking processing on the enlarged image to obtain an image with deblocking effect; performing resolution reduction processing on the image with deblocking effect to obtain a reference image.

[0163] According to one embodiment of the present application, Figure 7 The steps involved in the image processing method shown can be represented by Fig. 9 The various modules in the image processing device 900 shown in the figure are executed. For example, Figure 7 The step S701 shown in FIG. 1 can be performed by Fig. 9 The acquisition module 901 in is executed, Figure 7 The steps S702-S703 shown in FIG. 1 can be performed by Fig. 9 The feature extraction module 902 is used to perform, Figure 7 The step S304 shown in FIG. 1 can be performed by Fig. 9 The fusion module 903 in the embodiment is used to perform the Figure 7 The step S705 shown in FIG. 1 can be performed by Fig. 9 The processing module 904 in is executed.

[0164] The present application can perform a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for removing the blocking effect; perform a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for strengthening the image details; fuse the first image feature and the second image feature to obtain a fused image feature; perform a super-resolution image restoration process on the fused image feature to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image. It can be seen that in the method proposed in the present application, the extracted first image feature helps to remove the blocking effect of the image, the extracted second image feature can strengthen the image details and help perform super-resolution processing, and after fusing the first image feature and the second image feature, by performing a super-resolution restoration process on the fused image feature, the blocking effect can be eliminated while retaining the super-resolution details, thereby improving the image quality.

[0165] According to one embodiment of the present application, Fig. 9 The various modules in the image processing device 900 shown can be separately or all combined to be composed of one or more units, or one (some) of the units can be further divided into multiple smaller sub-units in function, and the same operation can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of a module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the image processing device 900 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0166] According to one embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 7 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9 The image processing apparatus 900 shown in the figure can be used to implement the image processing method of the embodiment of the present application. The above computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above computing device through the computer-readable storage medium, and run therein.

[0167] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of a computer device 1000 provided by the present application. Fig. 9As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0168] exist Fig.10 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the above Figure 7 The description of the above-mentioned image processing method in the corresponding embodiment can also be performed Fig. 9 The description of the image processing device 900 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0169] In addition, it should be pointed out that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the image processing device 900 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 7 The description of the image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0170] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0171] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0172] The present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Figure 7 The description of the above-mentioned image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0173] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0176] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing a first feature extraction process on the image to obtain a first image feature of the image, wherein the first image feature includes an image feature for deblocking, and the first image feature is a multi-channel image feature; Performing a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details, and the second image feature is a multi-channel image feature; fusing the first image feature and the second image feature to obtain a fused image feature; Performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image, The fusing the first image feature and the second image feature to obtain the fused image feature includes: Performing dimensionality reduction processing on the first image feature to obtain a first offset feature; Correcting the second image feature based on the first offset feature to obtain a corrected second image feature, wherein the correcting the second image feature based on the first offset feature includes splicing the first offset feature with the second image feature; Performing dimensionality reduction processing on the second image feature to obtain a second offset feature; Correcting the first image feature based on the second offset feature to obtain a corrected first image feature, wherein the correcting the first image feature based on the second offset feature includes splicing the second offset feature with the first image feature; fusing the modified first image feature with the modified second image feature to obtain the fused image feature; and The dimensionality reduction processing includes reducing the number of channels of image features, and the fusion includes splicing between image features.

2. The image processing method according to claim 1, characterized in that: The processed image is obtained by calling an image processing model for processing, and the training method of the image processing model includes: Acquire a sample image and a reference image corresponding to the sample image, wherein a resolution of the reference image is higher than a resolution of the sample image; Calling an initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking; Calling the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details; fusing the first sample image feature and the second sample image feature to obtain a fused sample image feature; Performing super-resolution image restoration processing on the fused sample image features to obtain an output image, wherein the resolution of the output image matches the resolution of the reference image; The initial image processing model is trained with the goal of reducing the difference between the output image and the reference image to obtain the image processing model.

3. The image processing method according to claim 2, characterized in that: The obtaining of the sample image comprises: A first resolution reduction process is performed on the reference image to obtain the sample image.

4. The image processing method according to claim 3, characterized in that: The method further comprises: Performing a second resolution reduction process on the reference image to obtain an intermediate reference image, wherein the resolution of the intermediate reference image matches that of the sample image, and the clarity of the intermediate reference image is higher than that of the sample image; Performing image restoration processing on the first sample image feature to obtain an intermediate image; The step of training the initial image processing model with the goal of reducing the difference between the output image and the reference image to obtain the image processing model comprises: The initial image processing model is trained with the goal of reducing the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image to obtain the image processing model.

5. The image processing method according to any one of claims 2 to 4, characterized in that: The obtaining of a reference image corresponding to the sample image comprises: Performing super-resolution processing on the source reference image to obtain an enlarged image; Performing a deblocking process on the enlarged image to obtain a deblocked image; The deblocked image is subjected to resolution reduction processing to obtain the reference image.

6. An image processing device, characterized in that: The device comprises: A feature extraction module, configured to perform a first feature extraction process on an image to obtain a first image feature of the image, wherein the first image feature includes an image feature for deblocking, and the first image feature is a multi-channel image feature; The feature extraction module is further used to perform a second feature extraction process on the image to obtain a second image feature of the image, wherein the second image feature includes an image feature for enhancing image details, and the second image feature is a multi-channel image feature; A fusion module, used for fusing the first image feature and the second image feature to obtain a fused image feature; a processing module, configured to perform super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than the resolution of the image, Wherein, the fusion module is specifically used for: Performing dimensionality reduction processing on the first image feature to obtain a first offset feature; Correcting the second image feature based on the first offset feature to obtain a corrected second image feature, wherein the correcting the second image feature based on the first offset feature includes splicing the first offset feature with the second image feature; Performing dimensionality reduction processing on the second image feature to obtain a second offset feature; Correcting the first image feature based on the second offset feature to obtain a corrected first image feature, wherein the correcting the first image feature based on the second offset feature includes splicing the second offset feature with the first image feature; fusing the modified first image feature with the modified second image feature to obtain the fused image feature; and The dimensionality reduction processing includes reducing the number of channels of image features, and the fusion includes splicing between image features.

7. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

8. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 5.