Image processing method, device, computer equipment and readable storage medium

Abstract

The present application relates to an image processing method, device, computer equipment and readable storage medium. The method comprises: obtaining an image to be processed, obtaining a trained image quality processing model; the image quality processing model comprises a feature processing network and a global feature projection layer; the global feature projection layer is shared by the feature processing network; the image to be processed is input into the image quality processing model, and based on the feature processing network, the image features to be processed corresponding to the image to be processed are extracted; based on the global feature projection layer, the image features to be processed are processed to obtain feature similarity; based on the feature processing network, the image features to be processed and the feature similarity are fused to obtain fused image features; based on the fused image features, the target image corresponding to the image to be processed is determined. The use of this method can improve image quality.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment and readable storage medium. Background Art

[0002] With the continuous development of computer technology, image processing technology that improves image quality is applied to more and more fields. For example, super-resolution technology is widely used in real-time video processing, intelligent monitoring, and augmented reality.

[0003] However, existing image processing technologies for improving image quality cannot effectively exploit non-local self-similarity within images, and their analysis performance on image content is limited, resulting in low quality of processed images. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product that can improve image quality in response to the above technical problems.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Obtain an image to be processed and obtain a trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the global feature projection layer is shared by the feature processing network;

[0007] The image to be processed is input into the image quality processing model, and based on the feature processing network, the image features to be processed corresponding to the image to be processed are extracted; based on the global feature projection layer, the image features to be processed are processed to obtain feature similarity; based on the feature processing network, the image features to be processed and the feature similarity are fused to obtain fused image features;

[0008] Based on the fused image features, the target image corresponding to the image to be processed is determined.

[0009] In a second aspect, the present application further provides an image processing device, comprising:

[0010] An acquisition module is used to acquire an image to be processed and to acquire a trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the global feature projection layer is shared by the feature processing network;

[0011] The processing module is used to input the image to be processed into the image quality processing model, extract the image features to be processed corresponding to the image to be processed based on the feature processing network; process the image features to be processed based on the global feature projection layer to obtain feature similarity; and fuse the image features to be processed and the feature similarity based on the feature processing network to obtain fused image features;

[0012] The determination module is used to determine the target image corresponding to the image to be processed based on the fused image features.

[0013] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0015] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0016] The above-mentioned image processing method, device, computer equipment, computer-readable storage medium and computer program product obtain a trained image quality processing model including a feature processing network and a global feature projection layer shared by the feature processing network, input the obtained image to be processed into the image quality processing model, extract the image to be processed features corresponding to the image to be processed based on the feature processing network, process the image to be processed features based on the global feature projection layer to obtain feature similarity, fuse the image to be processed features and feature similarity based on the feature processing network to obtain fused image features, realize global projection sharing of the global feature projection layer, do not need to set multiple feature projection layers in each feature processing network, reduce the model parameter amount to reduce the model calculation complexity, and can effectively mine the non-local self-similarity in the image, thereby decoupling the feature extraction in the feature processing network and the calculation of feature similarity in the global feature projection layer, and better consider the distribution similarity between different layers; finally, based on the fused image features, determine the target image corresponding to the image to be processed, so that the image quality of the target image finally obtained is well improved, thereby greatly reducing the model parameters and the amount of calculation, and well improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 An application environment diagram of an image processing method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a model structure provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of an experimental comparison result provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a visual comparison of multiple algorithms provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of another visual comparison of multiple algorithms provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the results of a different algorithm in an image restoration task provided by an embodiment of the present application;

[0025] Figure 8 A structural block diagram of an image processing device provided in an embodiment of the present application;

[0026] Fig. 9 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0027] Fig.10 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0028] Fig.11 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0031] Both the terminal and the server can be used independently to execute the image processing method provided in the embodiments of the present application.

[0032] For example, the terminal obtains an image to be processed and obtains a trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the image to be processed is input into the image quality processing model, and based on the feature processing network, the image features to be processed corresponding to the image to be processed are extracted; based on the global feature projection layer, the image features to be processed are processed to obtain feature similarity; based on the feature processing network, the image features to be processed and the feature similarity are fused to obtain fused image features; based on the fused image features, a target image corresponding to the image to be processed is determined.

[0033] For example, the server obtains an image to be processed and a trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the image to be processed is input into the image quality processing model, and based on the feature processing network, the image features to be processed corresponding to the image to be processed are extracted; based on the global feature projection layer, the image features to be processed are processed to obtain feature similarity; based on the feature processing network, the image features to be processed and the feature similarity are fused to obtain fused image features; based on the fused image features, a target image corresponding to the image to be processed is determined.

[0034] In some exemplary embodiments, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 The terminal or server in the example is used to illustrate, including the following steps 200 to 204. Among them:

[0035] Step S200, obtaining an image to be processed and obtaining a trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the global feature projection layer is shared by the feature processing network.

[0036] Among them, the image quality processing model refers to a neural network model used to improve image quality, which can be but is not limited to an improved model of the Transformer model, an improved model of the window-based attention model, and can be other image quality processing models including the Transformer structure; each feature processing network in the unimproved Transformer has multiple feature projection layers, and the feature projection layers between different feature processing networks are separate. Each feature processing network focuses on the local self-similarity in the image, and cannot effectively mine the non-local self-similarity in the image, and does not consider the distribution similarity between different layers. The global feature projection layer included in the image quality processing model of the present application is shared by each feature processing network, and no feature projection layer is set in the feature processing network. The present application adopts a global feature projection layer that is globally shared. The projection method calculates feature similarity, and performs layer-by-layer feature optimization according to each layer of the feature processing network, so that there is no synchronization relationship between feature similarity optimization and feature optimization, effectively reducing the coupling of feature optimization and feature similarity optimization, reducing the optimization complexity, and can effectively mine the non-local self-similarity in the image and better consider the distribution similarity between different layers.

[0037] The feature processing network refers to a neural network that extracts, optimizes and aggregates image features. The global feature projection layer refers to a neural network layer that calculates the self-similarity of feature projections; self-similarity means that the features of the entire image or the local image are similar from different spatial or temporal scales.

[0038] Exemplarily, the image to be processed may be an image frame captured from a video, the video may be a video shot by intelligent monitoring or other equipment, and the video quality does not meet the actual requirements, the image to be processed may be an image obtained by various means and does not meet the actual image quality requirements, and this application does not limit the acquisition method of the image to be processed. In order to effectively mine the non-local self-similarity in the image to be processed, a trained image quality processing model including a feature processing network and a global feature projection layer may be obtained. Each feature processing network in the image quality processing model interacts with the global feature projection layer for data. The improved design of the global projection layer is conducive to reducing the number of model parameters, reducing the model calculation complexity, and realizing global projection sharing of image features, and better considering the distribution similarity between different layers.

[0039] Step S202, input the image to be processed into the image quality processing model, extract the image features to be processed corresponding to the image to be processed based on the feature processing network; process the image features to be processed based on the global feature projection layer to obtain feature similarity; and fuse the image features to be processed and the feature similarity based on the feature processing network to obtain fused image features.

[0040] The image feature to be processed refers to the feature of the image to be processed. The feature similarity refers to the self-similarity between the features in the image feature to be processed. The fused image feature refers to the image feature obtained by aggregating the image to be processed according to the feature similarity.

[0041] Exemplarily, each feature processing network included in the image quality processing model is connected in series, each feature processing network includes a feature extraction layer and a feature fusion layer, and each feature processing network is bidirectionally connected to the global feature projection layer; when the image to be processed is input into the image quality processing model for processing, the image to be processed can be firstly subjected to preliminary feature extraction based on the convolutional neural network in the image quality processing model, and then the image features obtained by the preliminary extraction are input into the initial feature processing network for processing, and the feature processing network currently processing the image features of the image to be processed is used as the current feature processing network, and the image features of the image to be processed are further subjected to deep feature extraction according to the feature extraction layer in the current feature processing network to obtain the features of the image to be processed, and then the features of the image to be processed are input into the global feature projection layer for processing, and feature similarity is output, and the feature similarity is input into the feature aggregation layer in the feature processing network, and the features of the image to be processed and the feature similarity are fused to obtain fused image features, and the fused image features are input into the backward feature processing network corresponding to the current feature processing network, and the backward feature processing network is used as the current feature processing network, and so on, until the fused image features output by the last feature processing network are obtained.

[0042] Step S204: determining a target image corresponding to the image to be processed based on the fused image features.

[0043] The target image refers to an image whose image quality is optimized, and the image quality of the target image is higher than the image quality of the image to be processed.

[0044] Exemplarily, the fused image features can be processed based on the output neural network layer of the image quality processing model to output a target image representing the fused image features. The image quality of the target image is higher than that of the image to be processed, which may be, but is not limited to, that the resolution of the target image is higher than that of the image to be processed, the dark light area of ​​the target image is less than that of the image to be processed, and the noise of the target image is less than that of the image to be processed.

[0045] In some embodiments, the image processing method of the present application can be applied to a variety of low-level visual tasks, including but not limited to lightweight image super-resolution, image denoising, image dark light enhancement, etc.

[0046] The above-mentioned image processing method obtains a trained image quality processing model including a feature processing network and a global feature projection layer shared by the feature processing network, inputs the acquired image to be processed into the image quality processing model, extracts the image to be processed features corresponding to the image to be processed based on the feature processing network, processes the image to be processed features based on the global feature projection layer to obtain feature similarity, fuses the image to be processed features and feature similarity based on the feature processing network to obtain fused image features, realizes global projection sharing of the global feature projection layer, does not need to set multiple feature projection layers in each feature processing network, reduces the model parameter amount to reduce the model calculation complexity, and can effectively mine the non-local self-similarity in the image, thereby decoupling the feature extraction in the feature processing network and the calculation of feature similarity in the global feature projection layer, and better considers the distribution similarity between different layers; finally, based on the fused image features, determines the target image corresponding to the image to be processed, so that the image quality of the target image finally obtained is well improved, thereby greatly reducing the model parameters and the amount of calculation, and well improving the image quality.

[0047] In some embodiments, the feature processing network includes a feature extraction layer and a feature aggregation layer; in step S202, based on the feature processing network, extracting the image features to be processed corresponding to the image to be processed includes:

[0048] Based on the feature extraction layer, extract the image features to be processed corresponding to the image to be processed;

[0049] Based on the feature processing network, the image features to be processed and the feature similarity are fused to obtain the fused image features, including:

[0050] Based on the feature aggregation layer, the features of the image to be processed and the feature similarity are fused to obtain the fused image features.

[0051] The feature extraction layer refers to the neural network layer for extracting features. The number of layers of the feature extraction layer is within the preset number, which can play the role of deep extraction and optimization of features. The feature extraction layer can be a convolutional layer. The feature aggregation layer refers to the neural network layer for aggregating image features, including but not limited to matrix product state neural network and ReLU (Rectified Linear Unit) activation function.

[0052] Exemplarily, the feature extraction layer in the feature processing network deeply extracts the image features of the image to be processed, and the feature aggregation layer aggregates the feature similarity output by the global feature projection layer and the features of the image to be processed output by the feature extraction layer. The feature processing networks at different layers all optimize the features of the image to be processed deeply extracted by this layer, and during the feature optimization aggregation process, the feature similarity analyzed according to the global feature projection layer is integrated to ensure the performance of feature optimization of the image quality processing model.

[0053] In the above embodiment, by extracting the features of the image to be processed corresponding to the image to be processed based on the feature extraction layer, and fusing the features of the image to be processed and the feature similarity based on the feature aggregation layer, the fused image features are obtained, thereby ensuring the performance of the image quality processing model in feature optimization.

[0054] In some embodiments, the feature processing network includes multiple feature processing networks, and each feature processing network is connected in series; the image processing method further includes:

[0055] Step S300, for each feature processing network, based on the feature extraction layer in the feature processing network, extract the image features to be processed corresponding to the image to be processed; input the image features to be processed into the global feature projection layer for processing to obtain feature similarity; based on the feature aggregation layer in the feature processing network, fuse the image features to be processed and the feature similarity to obtain fused image features.

[0056] Step S302, inputting the fused image features into a backward feature processing network corresponding to the feature processing network.

[0057] The backward feature processing network refers to the next feature processing network connected to the feature processing network.

[0058] For example, the image quality processing model may include multiple feature processing networks, and each feature processing network is connected in series. Figure 3 As shown, Figure 3 FEA refers to the network layer of deep feature optimization and feature similarity optimization, which includes feature processing network and global feature projection layer. Please note that Figure 3 PS u Only one, the PS u It is shared by the global projection; FE is the feature extraction layer; X1~X i With V1~V i is the image feature to be processed output by the feature extraction layer of a feature processing network, which can be represented in the form of a data list; F1~F i and Y1~Y i The fused image features output by the feature aggregation layer of a feature processing network can be represented in the form of a data list; PSu is the global feature projection layer; Q1~Q i is the feature similarity; S1 is the result of the cosine dot product (i.e., Cosine) of Q1 and the image feature to be processed, S i Similarly; Figure 3 The “global optimization” here means that global optimization training of feature similarity can be performed, without the need to connect each feature extraction layer FE with multiple feature projection layers as in the prior art to optimize local feature similarity when the model parameters are large.

[0059] Understandably, when the image to be identified is input into the image quality processing model, the convolutional neural network of the input layer of the image quality processing model can first perform preliminary feature extraction on the image to be processed, and then input the image features obtained by the preliminary feature extraction into the initial feature processing network, and use the initial feature processing network as the current feature processing network. Based on the feature extraction layer in the current feature processing network, the input image features are deeply extracted to obtain the features of the image to be processed, and the features of the image to be processed are input into the global feature projection layer for processing, so that the global feature projection layer can effectively mine the non-local self-similarity in the image to be processed to obtain feature similarity, and then the feature similarity is input into the feature aggregation layer in the feature processing network to aggregate the features of the image to be processed and the features of the image to be processed. The feature similarity is fused to obtain the fused image features corresponding to the current feature processing network; the fused features are input into the backward feature processing network of the current feature processing network, and the backward feature processing network is used as the current feature processing network. Similarly, the input fused image features are further deeply extracted according to the feature extraction layer of the current feature processing network, and the image features to be processed obtained by the deep extraction are input into the global feature projection layer, and the feature similarity is output. The feature similarity is then input into the feature aggregation layer of the current feature processing network to obtain the fused image features corresponding to the current feature processing network, and the fused features are then input into the backward feature processing network of the current feature processing network, and so on, until the fused features output by the final feature processing network are obtained.

[0060] In the above embodiment, for each feature processing network connected in series, the input image features are deeply extracted based on the feature extraction layer in the feature processing network, and the output image features to be processed are input into the global feature projection layer for processing to obtain feature similarity, and then the image features to be processed and the feature similarity are fused based on the feature aggregation layer in the feature processing network to obtain fused image features, and the output of the current feature processing network connected in series is used as the input of the next feature processing network, so that the features are deeply extracted layer by layer according to the feature processing network, ensuring the performance of feature optimization, and the feature similarity aggregated in each layer is obtained by processing the global feature projection layer, effectively mining the non-local self-similarity in the image to be processed, considering the distribution similarity between different network layers, while reducing the calculation complexity of the model, so that there is no synchronization relationship between feature similarity optimization and feature optimization, reducing the coupling of feature optimization and feature similarity optimization, which is beneficial to improving the quality of the target image finally generated.

[0061] In some embodiments, the image processing method further includes:

[0062] Step S400, performing an image block operation on the image to be processed to obtain a plurality of image blocks; based on the feature extraction layer, extracting the image features to be processed corresponding to each image block.

[0063] Step S402: input the to-be-processed image features corresponding to each image block into the global feature projection layer for processing to obtain the feature similarity corresponding to each image block.

[0064] Step S404: Based on the feature aggregation layer, the image features to be processed and the feature similarities corresponding to each image block are fused to obtain fused image features corresponding to the image to be processed.

[0065] Exemplarily, in order to more accurately extract the local features in the image to be processed and to subsequently perform better self-attention calculation on the image to be processed, the image to be processed can be first segmented, and then based on the feature extraction layer, the features of the image to be processed corresponding to each divided image block are extracted, and then the features of the image to be processed corresponding to each image block are respectively input into the global feature projection layer for processing to obtain the feature similarity corresponding to each image block, and then the features of the image to be processed corresponding to each image block and the feature similarity are fused based on the feature aggregation layer to obtain the fused image features.

[0066] In the above embodiment, the image to be processed is divided into blocks for feature extraction and feature similarity calculation of each image block, and each image block is aggregated according to the extracted features and the calculated feature similarities to obtain corresponding fused image features, which is conducive to more accurate local features in the image to be processed and more accurate self-attention calculation of the features.

[0067] In some embodiments, the image quality of the target image is higher than the image quality of the image to be processed; wherein the image quality of the target image is higher than the image quality of the image to be processed includes: the resolution of the target image is higher than the resolution of the image to be processed.

[0068] In some embodiments, a lightweight image super-resolution processing of global projection similarity of an image is described as an example, and a low-resolution image can be used as an image to be processed, and can be combined with Figure 3 To describe, you can Figure 3 The model shown is used as an image quality processing model. The image to be processed is input into the image quality processing model. The image to be processed is divided into blocks based on the input layer, and preliminary feature processing is performed on each divided image block. Then, the preliminary extracted features of each image block are input into the current feature processing network. Based on the feature extraction layer in the current feature processing network, the preliminary extracted features of each input image block are deeply extracted to obtain the image features to be processed corresponding to each image block; the image features to be processed corresponding to each image block are respectively input into the global feature projection layer for processing, and the feature similarity corresponding to each image block is output; the feature similarity of each image block is input into the feature aggregation layer in the feature processing network, and the image features to be processed and the feature similarity of each image block are fused to obtain the fused image features corresponding to the current feature processing network.

[0069] Furthermore, the fused image features are input into the backward feature processing network of the current feature processing network, and the backward feature processing network is used as the current feature processing network. The input fused image features are deeply extracted based on the feature extraction layer in the current feature processing network to obtain the image features to be processed after deep extraction. The image features to be processed after deep extraction are then input into the global feature projection layer for processing to obtain the corresponding feature similarity. The feature similarity is input into the feature aggregation layer in the current feature processing network for processing to obtain the corresponding fused image features, and so on, until the fused image features corresponding to the last feature processing network are obtained. The fused image features are then processed based on the output layer to obtain a target image with a higher resolution than the image to be processed.

[0070] To more intuitively demonstrate the effectiveness of the image quality processing model of this application, Figure 4 , Figure 5 and Figure 6 As shown, Figure 4 The image quality processing model for this application (i.e. Figure 4 UPS, UPS-S) and other existing baseline models on the same dataset (i.e. Figure 4The performance indicators of the models are compared under the processing of Set5, Set14, BSD100, Urban100 and Manga109. The performance indicators include PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity); PSNR is the peak signal-to-noise ratio, which is an indicator to measure image quality. The higher the value, the better the image quality after denoising; SSIM is the structural similarity, which is a full-reference image quality evaluation indicator. It measures the similarity of two images from three aspects: brightness, contrast and structure. The larger the value, the better the image quality. Figure 4 It can be seen that the image quality processing model of the present application can achieve the best PSNR / SSIM performance under the same data set compared with other comparison models, and also has certain advantages in model complexity (the fewer the number of parameters, the lower the complexity); in addition, Figure 7 It also intuitively shows the comparison results of the performance of this application and other baseline models and the visualization of depth map super-resolution. Figure 7 The displayed graph also shows intuitively that the super-resolution effect of the depth map corresponding to the UPS in this application is relatively good, where RMSE (Root Mean Squared Error) is the root mean square error. Figure 7 Medium reality refers to images in real and clear conditions, LR (Low Resolution) refers to low resolution, and HR (High Resolution) refers to high resolution. Figure 5 This is a visual comparison of this application with some other benchmark models. Figure 5 It can also be intuitively seen that the super-resolution effect of the present application on the image is relatively better. Furthermore, Figure 6 It also intuitively shows the comparison and visualization results of this application with some baseline models in other image restoration tasks, among which, Figure 6 The upper left corner of each picture shows the image effect of a certain part of the whole picture after being cut and processed (except the real-time picture). Figure 6 It can be seen intuitively that the image quality after processing by this application (ie UPS) is relatively good.

[0071] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0072] Based on the same inventive concept, the embodiment of the present application also provides an image processing device. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0073] like Figure 8 As shown, the embodiment of the present application provides an image processing device 800, including:

[0074] The acquisition module 802 is used to acquire the image to be processed and acquire the trained image quality processing model; the image quality processing model includes a feature processing network and a global feature projection layer; the global feature projection layer is shared by the feature processing network;

[0075] The processing module 804 is used to input the image to be processed into the image quality processing model, extract the image features to be processed corresponding to the image to be processed based on the feature processing network; process the image features to be processed based on the global feature projection layer to obtain feature similarity; fuse the image features to be processed and the feature similarity based on the feature processing network to obtain fused image features;

[0076] The determination module 806 is used to determine the target image corresponding to the image to be processed based on the fused image features.

[0077] In some embodiments, the feature processing network includes a feature extraction layer and a feature aggregation layer; in terms of extracting the features of the image to be processed corresponding to the image to be processed based on the feature processing network, the processing module 804 is specifically used to: extract the features of the image to be processed corresponding to the image to be processed based on the feature extraction layer; in terms of fusing the features of the image to be processed and the feature similarity based on the feature processing network to obtain the fused image features, the processing module 804 is specifically used to: fuse the features of the image to be processed and the feature similarity based on the feature aggregation layer to obtain the fused image features.

[0078] In some embodiments, the feature processing network includes multiple feature processing networks, each of which is connected in series; the processing module 804 is specifically used to:

[0079] For each feature processing network, based on the feature extraction layer in the feature processing network, the image features to be processed corresponding to the image to be processed are extracted; the image features to be processed are input into the global feature projection layer for processing to obtain feature similarity; based on the feature aggregation layer in the feature processing network, the image features to be processed and the feature similarity are fused to obtain fused image features;

[0080] The fused image features are input into the backward feature processing network corresponding to the feature processing network.

[0081] In some embodiments, the processing module 804 is specifically configured to:

[0082] Perform image block operation on the image to be processed to obtain multiple image blocks; based on the feature extraction layer, extract the image features to be processed corresponding to each image block;

[0083] The image features to be processed corresponding to each image block are input into the global feature projection layer for processing to obtain the feature similarity corresponding to each image block;

[0084] Based on the feature aggregation layer, the image features to be processed and the feature similarities corresponding to each image block are fused to obtain the fused image features corresponding to the image to be processed.

[0085] In some embodiments, the image quality of the target image is higher than the image quality of the image to be processed; wherein the image quality of the target image is higher than the image quality of the image to be processed includes: the resolution of the target image is higher than the resolution of the image to be processed.

[0086] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0087] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to image processing. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0088] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0089] Those skilled in the art will understand that Fig. 9 or Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0091] In some embodiments, Fig.11 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0092] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0094] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0096] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method is applied to image processing scenarios such as lightweight image super-resolution, image denoising or image dark light enhancement, and the method comprises: Acquire an image to be processed and acquire a trained image quality processing model; the image quality processing model includes N feature processing networks and a global feature projection layer connected in series; the global feature projection layer is shared by the N feature processing networks; each of the feature processing networks includes a feature extraction layer and a feature aggregation layer; The image to be processed is input into the image quality processing model, and the feature extraction layer FE based on the i-th feature processing network i Process the input data of the i-th feature processing network to obtain the i-th image feature to be processed; input the i-th image feature to be processed into the global feature projection layer PS u Processing is performed to obtain the i-th feature similarity Q i Based on the feature aggregation layer of the i-th feature processing network, the i-th image feature to be processed and the i-th feature similarity Q i Perform cosine dot product and element multiplication in sequence to obtain the i-th fused image feature; Wherein, when i is equal to 1, the input data of the i-th feature processing network is the image to be processed; When i is greater than 1 and less than N, the i-th fused image feature is used as the input data of the i+1-th feature processing network, i=i+1 is reassigned, and the feature extraction layer FE based on the i-th feature processing network is returned to execute. i The step of processing the input data of the i-th feature processing network; When i=N, the i-th fused image feature is used as the fused image feature; Based on the fused image features, a target image corresponding to the image to be processed is determined.

2. The method according to claim 1, characterized in that The image quality processing model is an improved model of the Transformer model.

3. The method according to claim 1, characterized in that When i=1, the feature extraction layer FE based on the i-th feature processing network i Process the input data of the i-th feature processing network to obtain the i-th image feature to be processed; input the i-th image feature to be processed into the global feature projection layer PS u Processing is performed to obtain the i-th feature similarity Q i Based on the feature aggregation layer of the i-th feature processing network, the i-th image feature to be processed and the i-th feature similarity Q i Perform cosine dot product and element multiplication in sequence to obtain the i-th fused image feature, including: The image to be processed is divided into blocks, and the divided image blocks are processed with preliminary features. The features extracted from each image block are input into the i-th feature processing network. Based on the feature extraction layer FE in the i-th feature processing network, i Performing deep extraction on the initially extracted features of each of the input image blocks to obtain image features to be processed corresponding to each of the image blocks; The image features to be processed corresponding to each image block are input into the global feature projection layer PS u Processing is performed to obtain the feature similarity corresponding to each image block; Based on the feature aggregation layer of the i-th feature processing network, the image features to be processed and the feature similarities corresponding to each image block are fused to obtain fused image features corresponding to the image to be processed.

4. The method according to claim 1, characterized in that The image quality of the target image is higher than that of the image to be processed; wherein, the image quality of the target image is higher than that of the image to be processed, including: the resolution of the target image is higher than that of the image to be processed.

5. An image processing device, characterized in that: The device is applied to image processing scenarios such as lightweight image super-resolution, image denoising or image dark light enhancement, and comprises: An acquisition module is used to acquire an image to be processed and to acquire a trained image quality processing model; the image quality processing model includes N feature processing networks connected in series and a global feature projection layer; the global feature projection layer is shared by the N feature processing networks; each of the feature processing networks includes a feature extraction layer and a feature aggregation layer; A processing module is used to input the image to be processed into the image quality processing model, based on the feature extraction layer FE of the i-th feature processing network i Process the input data of the i-th feature processing network to obtain the i-th image feature to be processed; input the i-th image feature to be processed into the global feature projection layer PS u Processing is performed to obtain the i-th feature similarity Q i Based on the feature aggregation layer of the i-th feature processing network, the i-th image feature to be processed and the i-th feature similarity Q i Perform cosine dot product and element multiplication in sequence to obtain the i-th fused image feature; Among them, when i is equal to 1, the input data of the i-th feature processing network is the image to be processed; when i is greater than 1 and less than N, the i-th fused image feature is used as the input data of the i+1-th feature processing network, i=i+1 is reassigned, and the feature extraction layer FE based on the i-th feature processing network is returned to execute i The step of processing the input data of the i-th feature processing network; when i=N, taking the i-th fused image feature as the fused image feature; A determination module is used to determine a target image corresponding to the image to be processed based on the fused image features.

6. The device according to claim 5, characterized in that The image quality processing model is an improved model of the Transformer model.

7. The device according to claim 5, characterized in that When i=1, the processing module is further used for: The image to be processed is divided into blocks, each divided image block is processed with preliminary features, and the features initially extracted from each image block are input into the i-th feature processing network. Based on the feature extraction layer FE in the i-th feature processing network, i Performing deep extraction on the initially extracted features of each of the image blocks to obtain the image features to be processed corresponding to each of the image blocks; The image features to be processed corresponding to each image block are input into the global feature projection layer PS u Processing is performed to obtain the feature similarity corresponding to each image block; Based on the feature aggregation layer of the i-th feature processing network, the image features to be processed and the feature similarities corresponding to each image block are fused to obtain fused image features corresponding to the image to be processed.

8. The device according to claim 5, characterized in that The image quality of the target image is higher than that of the image to be processed; wherein, the image quality of the target image is higher than that of the image to be processed, including: the resolution of the target image is higher than that of the image to be processed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images