Image processing method and apparatus, and electronic device

Abstract

The present disclosure provides an image processing method and apparatus, and an electronic device. The image processing method comprises: acquiring an image to be processed, and inputting said image into a feature extraction module for feature extraction to obtain an intermediate image feature; inputting the intermediate image feature into a colorization module for colorization processing to obtain a colorized image feature; fusing the colorized image feature and the intermediate image feature to obtain a first fused feature; inputting the first fused feature into an inpainting module for image inpainting to obtain an inpainted image feature; and inputting the inpainted image feature into a resolution enhancement module for image resolution enhancement processing to obtain a target image. In the present disclosure, the colorized image feature output by the colorization module is used for processing by the inpainting module and the resolution enhancement module, and feature information sharing among the colorization module, the inpainting module and the resolution enhancement module is realized, thereby improving the image quality of the obtained target image.

Description

Image processing method, device and electronic equipment

[0001] The present disclosure claims priority to the application with the application number 202411063636.0 and the title "Image processing method, device and electronic equipment" filed with the China Patent Office on August 2, 2024, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of image processing, in particular to an image processing method, device and electronic equipment. BACKGROUND

[0003] At present, in order to convert low-resolution old images into high-definition high-resolution color images, independent steps are adopted for coloring, repairing and resolution enhancement. Among them, image coloring can automatically color black and white photos; image repair can identify and correct damaged parts in the image, such as scratches, spots, fading and blank areas; and super-resolution technology is used to improve the resolution of the image, making the image look clearer and more delicate.

[0004] However, using independent steps for coloring, repairing and resolution enhancement of images leads to the problem that the image obtained after image processing has low quality. SUMMARY

[0005] The present disclosure provides an image processing method, device and electronic equipment, which solves the problem that the image obtained after image processing has low quality.

[0006] The first aspect of the embodiment of the present disclosure provides an image processing method, comprising: acquiring a to-be-processed image, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into a repair module for image repair to obtain repaired image features; and inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0007] The second aspect of the embodiment of the present disclosure provides an image processing method applied to a cloud server, the image processing method comprising:

[0008] receiving a to-be-processed image sent by a terminal device, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features;

[0009] inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features;

[0010] fusing the colored image features and the intermediate image features to obtain first fused features;

[0011] input the first fusion feature into the image inpainting module to perform image inpainting, to obtain an inpainted image feature;

[0012] input the inpainted image feature into the resolution enhancement module to perform image resolution enhancement processing, to obtain a target image;

[0013] send the target image to a terminal device.

[0014] A third aspect of the embodiments of the present disclosure provides an image processing apparatus, comprising:

[0015] an extraction module configured to acquire a to-be-processed image, and input the to-be-processed image into the feature extraction module to perform feature extraction, to obtain an intermediate image feature;

[0016] a processing module configured to input the intermediate image feature into the coloring module to perform coloring processing, to obtain a colored image feature;

[0017] a fusion module configured to fuse the colored image feature and the intermediate image feature, to obtain a first fusion feature;

[0018] an image inpainting module configured to input the first fusion feature into the image inpainting module to perform image inpainting, to obtain an inpainted image feature;

[0019] an enhancement module configured to input the inpainted image feature into the resolution enhancement module to perform image resolution enhancement processing, to obtain a target image.

[0020] A fourth aspect of the embodiments of the present disclosure provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the image processing method of the first aspect and / or the image processing method of the second aspect when executing the computer program.

[0021] A fifth aspect of the embodiments of the present disclosure provides a computer program product, the computer program product storing a computer program, and the computer program causes the processor to implement the image processing method of the first aspect and / or the image processing method of the second aspect when the computer program is executed by the processor.

[0022] The embodiment of the present disclosure is applied to the scene of improving image quality, and the present disclosure comprises: acquiring a to-be-processed image, inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into a repair module for image repair to obtain repaired image features; and inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image. The present disclosure uses the colored image features output by the coloring module for processing of the repair module and the resolution enhancement module, realizes sharing of feature information among the coloring module, the repair module and the resolution enhancement module, that is, image repair is performed on the basis of the image being colored, and resolution enhancement is performed on the basis of the image being colored and repaired, and the subsequent image processing steps consider the obtained features, thereby improving the image quality of the obtained target image. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are included to provide a further understanding of the present disclosure and constitute a part of the present disclosure, illustrate the exemplary embodiments of the present disclosure and specifically explain the present disclosure together with the description. In the drawings:

[0024] FIG. 1 is an application scenario provided by an exemplary embodiment of the present disclosure;

[0025] FIG. 2 is a step flowchart of an image processing method provided by an exemplary embodiment of the present disclosure;

[0026] FIG. 3 is a schematic diagram of an image processing model provided by an exemplary embodiment of the present disclosure;

[0027] FIG. 4 is a step flowchart of another image processing method provided by an exemplary embodiment of the present disclosure;

[0028] FIG. 5 is a schematic diagram of another image processing model provided by an exemplary embodiment of the present disclosure;

[0029] FIG. 6 is a schematic diagram of a cross-attention unit provided by an exemplary embodiment of the present disclosure;

[0030] FIG. 7 is a schematic diagram of a repaired image provided by an exemplary embodiment of the present disclosure;

[0031] FIG. 8 is a structural block diagram of an image processing device provided by an exemplary embodiment of the present disclosure;

[0032] FIG. 9 is a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described clearly and completely below in connection with specific embodiments of the present disclosure and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present disclosure.

[0034] With the rapid development of artificial intelligence technology, the field of image inpainting has undergone tremendous changes. In traditional image inpainting methods, the processing process mostly requires manual operation by image processing experts, which not only consumes time and effort, but also the final inpainting effect largely depends on the skills and experience of the experts. At present, with the use of artificial intelligence technology, image inpainting has become more automated and intelligent.

[0035] Further, in the related art, the resolution enhancement, inpainting and coloring of an image are usually performed sequentially as independent steps, that is, the image is respectively subjected to resolution enhancement, inpainting and coloring, and then the final image is obtained after merging, which has its limitations. On the one hand, due to the non-sharing of feature information between steps, a poor result may be obtained.

[0036] Based on the above problems, the present disclosure provides an image processing method, which obtains a to-be-processed image, inputs the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputs the intermediate image features into a coloring module for coloring processing to obtain colored image features; fuses the colored image features and the intermediate image features to obtain first fused features; inputs the first fused features into an inpainting module for image inpainting to obtain inpainted image features; inputs the inpainted image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image, so that the feature information can be shared among the modules in the entire processing process, and the quality of the final image, such as the visual effect, can be enhanced by coordinating the steps.

[0037] In the present embodiment, the execution device of the image processing method is not limited. The image processing method can realize the overall image processing method by means of a cloud computing system. For example, the image processing method can be applied to a cloud server so as to run various models by means of the advantages of resources on the cloud; relative to the application to the cloud, the image processing method can also be applied to a regular server, a cloud server or a server array, etc.

[0038] In addition, referring to FIG. 1, an application scenario of the present disclosure is shown. The application scenario includes a terminal and a cloud server. The cloud server is deployed with an image processing model. The terminal sends a poor-quality image to be processed to the cloud server. The cloud server inputs the image to be processed into the image processing model for image processing, and outputs a high-quality target image. Then, the cloud server sends the target image to the terminal.

[0039] FIG. 1 is only an exemplary application scenario of the present disclosure. The present disclosure can also be applied to other related scenarios, which are not limited herein.

[0040] The technical solutions provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0041] FIG. 2 is a step flowchart of an image processing method according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the image processing method includes the following steps:

[0042] S201, obtaining an image to be processed, and inputting the image to be processed into a feature extraction module for feature extraction to obtain intermediate image features.

[0043] Referring to FIG. 3, an image processing model 30 is shown. The image processing model 30 includes a feature extraction module, a coloring module, a repair module, and a resolution enhancement module. The image processing model is pre-trained.

[0044] In the embodiments of the present disclosure, the image to be processed can be sent by a terminal to a server. The image to be processed includes at least one of the following features: black and white, low resolution, and damaged area. The damaged area includes scratches, spots, discoloration, or blank areas.

[0045] Further, the image to be processed is input into the feature extraction module shown in FIG. 3 for feature extraction to obtain intermediate image features of the image to be processed. As shown in FIG. 5, the feature extraction module includes a convolution unit and a cross-attention unit. The convolution unit includes multiple convolution layers.

[0046] S202, inputting the intermediate image features into the coloring module for coloring processing to obtain colored image features.

[0047] Referring to FIG. 3, the image features output by the coloring module are colored image features. The colored image features refer to the colored image features after coloring the image to be processed. The colored image features can obtain the colored image after being imaged.

[0048] S203, fusing the colored image features and the intermediate image features to obtain first fused features.

[0049] The fusion of the colored image features and the intermediate image features can make the first fused features have features of the image to be processed and features of the image after coloring.

[0050] Referring to FIG. 3, the first fused features are obtained after the fusion of the colored image features and the intermediate image features.

[0051] Further, in order to further ensure the coordination of colors, the server fuses the colored image features after coloring processing with the original gray image features (the intermediate image features of the image to be processed), and such feature fusion ensures that the smooth transition and reasonable matching of image colors are achieved in the repairing process.

[0052] S204, input the first fused features into a repairing module for image repairing to obtain repaired image features.

[0053] The repairing module includes a module constructed by using diffusion models (DM), and the repairing module can also be constructed by using other neural networks, which is not limited herein. The repairing module is used for repairing the image to be processed, for example, the repairing module can repair a damaged area in the image to be processed.

[0054] In the present disclosure, the colored image features and the intermediate image features are fused and then input into the repairing module for image repairing, which can make the repairing module better repair the damaged area of the image to be processed based on the colored image features.

[0055] Referring to FIG. 3, the first fused features are input into the repairing module for image repairing, and the repaired image features are output.

[0056] S205, input the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0057] In the embodiments of the present disclosure, the resolution enhancement module includes a feature extraction layer (such as a convolutional network), a nonlinear mapping layer (such as a multi-layer perceptron, MLP), and a network reconstruction layer (such as a deconvolution layer). The repaired image features are input into the feature extraction layer and the nonlinear mapping layer for image resolution enhancement, and then target image features are obtained. The target image features are input into the network reconstruction layer for reconstruction, and then the target image is obtained.

[0058] Referring to FIG. 3, the repaired image features are input into the resolution enhancement module, and the target image is output.

[0059] In the embodiments of the present disclosure, the image processing model combines a feature extraction module, a coloring module, a repairing module and a resolution enhancement module, and the intermediate image feature image obtained after the feature extraction module performs feature extraction on the to-be-processed image can be applied to the subsequent coloring module and repairing module, and finally the obtained repaired image feature is input into the resolution enhancement module for resolution enhancement, thereby efficiently and uniformly implementing coloring, repairing and resolution enhancement of the to-be-processed image, and sharing feature information in the implementation process, thereby improving the image processing efficiency and the quality of the target image.

[0060] FIG. 4 is a step flowchart of another image processing method provided by an exemplary embodiment of the present disclosure. As shown in FIG. 4, the image processing method specifically includes the following steps:

[0061] S401, obtaining a to-be-processed image and a reference image of the to-be-processed image.

[0062] The content similarity between the reference image and the to-be-processed image is greater than a preset threshold. The content similarity refers to the similarity between the content included in the reference image and the content included in the to-be-processed image. For example, the to-be-processed image includes a landmark building, and the reference image also includes the landmark building. In addition, the preset threshold is, for example, 80%.

[0063] In the embodiments of the present disclosure, image coloring is a technology of adding colors to grayscale images or videos, and plays an important role in restoring old photos and old movies. However, image coloring essentially has uncertainty, because an object can have multiple acceptable colors, and when freely coloring without explicit color guidance, the coloring model sometimes generates colors that do not conform to the actual or user expectations. Therefore, appropriate color guidance is needed to help the coloring module generate more realistic or user-expected colors. Image repairing and image resolution enhancement also face such challenges, and without proper reference images as guidance, the added or reconstructed content can be inconsistent with the actual situation or expectations. The reference image of the present disclosure can improve the quality of the target image.

[0064] Referring to FIG. 5, another image processing model 50 is shown, which includes a feature extraction module, a reference image retrieval module, a coloring module, a repairing module and a resolution enhancement module. The feature extraction module includes a convolution unit and a cross-attention unit, and the convolution unit includes multiple convolution layers, such as convolution layer a1, convolution layer a2 and convolution layer a3 in FIG. 5. The coloring module includes a conversion layer and multiple deconvolution layers, such as deconvolution layer b1, deconvolution layer b2 and deconvolution layer b3 in FIG. 5.

[0065] In addition, the most similar image to the to-be-processed image can be obtained by searching the image library, as the reference image.

[0066] In the embodiments of the present disclosure, referring to FIG. 5, after obtaining the to-be-processed image, the to-be-processed image is input into the reference image retrieval module for retrieval of similar images, to obtain a reference image.

[0067] S402, input the to-be-processed image and the reference image into the feature extraction module for feature extraction, to obtain an intermediate image feature.

[0068] In the embodiments of the present disclosure, referring to FIG. 5, the feature extraction module includes a convolution unit and a cross-attention unit. The to-be-processed image and the reference image are input into the feature extraction module for feature extraction, to obtain an intermediate image feature. The process includes: inputting the to-be-processed image into the convolution unit for feature extraction, to obtain a first image feature; inputting the reference image into the convolution unit for feature extraction, to obtain a second image feature; and inputting the first image feature and the second image feature into the cross-attention unit for attention calculation, to obtain the intermediate image feature.

[0069] Referring to FIG. 5, the processing process of the convolution unit on the to-be-processed image is: inputting the to-be-processed image into a convolution layer a1 to output a first image feature A11, inputting the first image feature A11 into a convolution layer a2 to output a first image feature A12, and inputting the first image feature A12 into a convolution layer a3 to output a first image feature A13. The processing process of the convolution unit on the reference image is: inputting the reference image into the convolution layer a1 to output a second image feature A21, inputting the second image feature A21 into the convolution layer a2 to output a second image feature A22, and inputting the second image feature A22 into the convolution layer a3 to output a second image feature A23.

[0070] In the embodiments of the present disclosure, referring to FIG. 5, the processing process of the convolution unit on the to-be-processed image is: inputting the to-be-processed image into a convolution layer a1 to output a first image feature A11, inputting the first image feature A11 into a convolution layer a2 to output a first image feature A12, and inputting the first image feature A12 into a convolution layer a3 to output a first image feature A13. The processing process of the convolution unit on the reference image is: inputting the reference image into the convolution layer a1 to output a second image feature A21, inputting the second image feature A21 into the convolution layer a2 to output a second image feature A22, and inputting the second image feature A22 into the convolution layer a3 to output a second image feature A23.

[0071] In one embodiment, the convolution unit includes a plurality of convolution layers, each convolution layer outputs a corresponding first image feature, and each convolution layer outputs a corresponding second image feature. The first image feature and the second image feature are input into the cross-attention unit for attention calculation, to obtain the intermediate image feature. The process includes: inputting the first image feature and the second image feature output by the last convolution layer into the cross-attention unit for attention calculation, to obtain the intermediate image feature.

[0072] Further, the first image feature (such as the first image feature A13) and the second image feature (such as the second image feature A23) output by the last convolution layer (such as the convolution layer a3 in FIG. 5) are input into the cross-attention unit for attention calculation, to obtain the intermediate image feature.

[0073] Referring to FIG. 6, a schematic diagram of a cross-attention unit is shown, which includes a plurality of down-sampling layers and a cross-attention layer. The plurality of down-sampling layers, such as down-sampling layer c1, down-sampling layer c2, and down-sampling layer c3 in FIG. 6

[0074] In the embodiments of the present disclosure, the first image feature and the second image feature are input into the cross-attention unit for attention calculation to obtain an intermediate image feature, including: inputting the first image feature into the plurality of down-sampling layers for down-sampling to obtain one first down-sampled image feature corresponding to the output of each down-sampling layer; inputting the second image feature into the plurality of down-sampling layers for down-sampling to obtain one second down-sampled image feature corresponding to the output of each down-sampling layer; for the first down-sampled image feature and the second down-sampled image feature obtained by the same layer down-sampling layer, inputting the first down-sampled image feature and the second down-sampled image feature into the cross-attention layer for attention calculation to obtain a reference feature for the same layer down-sampling layer; fusing the reference feature and the first down-sampled image feature to obtain a second fusion feature; and splicing a plurality of second fusion features corresponding to the plurality of down-sampling layers to obtain the intermediate image feature.

[0075] As shown in FIG. 6, the first image feature A13 is input into the down-sampling layer c1 for down-sampling to obtain the first down-sampled image feature C11, the first down-sampled image feature C11 is input into the down-sampling layer c2 for down-sampling to obtain the first down-sampled image feature C12, and the first down-sampled image feature C12 is input into the down-sampling layer c3 for down-sampling to obtain the first down-sampled image feature C13. The second image feature A23 is input into the down-sampling layer c1 for down-sampling to obtain the second down-sampled image feature C21, the second down-sampled image feature C21 is input into the down-sampling layer c2 for down-sampling to obtain the second down-sampled image feature C22, and the second down-sampled image feature C22 is input into the down-sampling layer c3 for down-sampling to obtain the second down-sampled image feature C23. The cross-attention layer is used to perform attention calculation on the first down-sampled image feature C11 and the second down-sampled image feature C21 to obtain the reference feature D1. The cross-attention layer is used to perform attention calculation on the first down-sampled image feature C12 and the second down-sampled image feature C22 to obtain the reference feature D2. The cross-attention layer is used to perform attention calculation on the first down-sampled image feature C13 and the second down-sampled image feature C23 to obtain the reference feature D3. Then, the first down-sampled image feature C11 is fused with the reference feature D1 to obtain the second fusion feature E1. The first down-sampled image feature C12 is fused with the reference feature D2 to obtain the second fusion feature E2. The first down-sampled image feature C13 is fused with the reference feature D3 to obtain the second fusion feature E3. Finally, the second fusion feature E1, the second fusion feature E2, and the second fusion feature E3 are spliced to obtain the intermediate image feature.

[0076] In the embodiments of the present disclosure, in order to enhance the modeling capability of the image processing model in the coloring and repairing tasks for the semantic connection between the to-be-processed image and the reference image, and increase the explainability of the image processing model, the present disclosure provides a cross-attention unit. The structure of the cross-attention unit is shown in FIG. 6. The input of the cross-attention unit includes the first image feature of the to-be-processed image and the second image feature of the reference image, and the output is the intermediate image feature. In the present disclosure, the cross-attention mechanism is used to promote the information interaction between the to-be-processed image and the reference image. In addition, the cross-attention unit is used for subsequent coloring and repairing of the to-be-processed image, and the features of the reference image can be input into the coloring and repairing process.

[0077] In addition, in the cross-attention layer, a convolutional network can be used for convolution operation to query the features Q input (Query features), key features K ref (Key features) and value features V ref (Value features) from the first down-sampled image features and the second down-sampled image features, and then Q input and K ref An association matrix is calculated by matrix multiplication and a softmax (normalized exponential) function, and the association matrix is multiplied by V ref to obtain the reference features.

[0078] In the embodiments of the present disclosure, the cross-attention unit with a pyramid structure is used to aggregate the features of the to-be-processed image and the reference image. The pyramid structure means that the features will be processed at different resolution levels, which helps to capture the image details from coarse to fine. The cross-attention mechanism can make the network focus on the most relevant feature parts in the to-be-processed image and the reference image, and effectively combine the feature information of the two.

[0079] In an embodiment, the first image feature and the second image feature can also be directly input into the cross-attention layer shown in FIG. 6 to obtain the reference features, and then the first image feature and the reference features are fused to obtain the intermediate image features.

[0080] In the embodiments of the present disclosure, a pyramid-shaped attention layer hierarchy is constructed by adopting the strategy of applying the cross-attention layer in multiple scales shown in FIG. 6. As shown in FIG. 6, the first image feature and the second image feature are processed by downsampling to obtain feature representations of different scales (the first downsampled image feature and the second downsampled image feature), and the first downsampled image feature and the second downsampled image feature at each scale are processed by the cross-attention layer to obtain the feature output (the reference feature) of the scale. Finally, the reference feature is fused with the first downsampled image feature of the corresponding scale to obtain the second fused feature, and the second fused features of different scales are spliced together to obtain the multi-scale feature output (the intermediate image feature).

[0081] S403, input the intermediate image feature into the conversion layer for color conversion processing to obtain the converted image feature.

[0082] The coloring module includes a conversion layer and a plurality of deconvolution layers. The conversion layer adopts a nonlinear mapping function, that is, the intermediate image feature is mapped by the nonlinear mapping function to obtain the converted image feature after coloring processing.

[0083] S404, input the converted image feature into the plurality of deconvolution layers to obtain the coloring image feature output by the deconvolution layer.

[0084] The deconvolution layer corresponds to the coloring image feature one by one. Referring to FIG. 5, the deconvolution layer b1 outputs the coloring image feature B1, the deconvolution layer b2 outputs the coloring image feature B3, and the deconvolution layer b3 outputs the coloring image feature B3.

[0085] In the embodiments of the present disclosure, the coloring image feature corresponds to the image to be processed after coloring.

[0086] In addition, the coloring module can be trained separately. The training process of the coloring module is as follows: obtaining a first sample image and a reference sample image with color and content similarity greater than a threshold, the first sample image being a color image, converting the first sample image into a grayscale image, inputting the grayscale image and the reference sample image into the feature extraction module to obtain a sample image feature, then inputting the sample image feature into the coloring module to obtain a sample coloring image feature, then processing the sample coloring image feature to obtain an output coloring image, calculating the loss value of the output coloring image and the first sample image, and adjusting the model parameters of the feature extraction module and the coloring module according to the loss value.

[0087] S405, fuse the first image feature output by the corresponding convolution layer, the coloring image feature output by the deconvolution layer, and the intermediate image feature to obtain the first fused feature.

[0088] The number of the first fusion features is the same as the number of the plurality of convolution layers, and the dimensions of the first image features and the colored image features that are fused are the same.

[0089] Referring to FIG. 5, if the image to be processed is an N*M image, after the convolution layer a1, the first image feature A11 of N*M is obtained, after the convolution layer a2, the first image feature A12 of is obtained, and after the convolution layer a3, the first image feature A13 of is obtained. Further, the dimension of the colored image feature B1 is After the deconvolution of the deconvolution layer b2, the colored image feature B2 of is obtained, and after the deconvolution of the deconvolution layer b3, the colored image feature B3 of N*M is obtained.

[0090] Further, the first image feature A11, the intermediate image feature, and the colored image feature B3 are fused to obtain the first fusion feature F1. The first image feature A12, the intermediate image feature, and the colored image feature B2 are fused to obtain the first fusion feature F2. The first image feature A13, the intermediate image feature, and the colored image feature B1 are fused to obtain the first fusion feature F3.

[0091] In the embodiment of the present disclosure, the first image feature of the image to be processed, the colored image feature, and the intermediate image feature obtained through the cross-attention unit are fused to generate a final feature representation (first fusion feature) that integrates all useful information, thereby providing rich guidance for subsequent image inpainting and image resolution enhancement.

[0092] S406, input the first fusion feature into the inpainting module to perform image inpainting to obtain an inpainted image feature.

[0093] The specific implementation process of this step is referred to S204, which will not be described here.

[0094] In the embodiment of the present disclosure, the plurality of first fusion features are input into the inpainting module to perform image inpainting to obtain an inpainted image feature.

[0095] Further, the inpainting module can be trained separately. The training process is as follows: a second sample image is obtained, and the second sample image is a complete and undamaged image. The second sample image is damaged to obtain a damaged sample image. The damaged image is input into the inpainting module, and the second sample image is used as a label to train the inpainting module.

[0096] Further, since the repair module shares feature information with the coloring module, it can generate more natural and harmonious content while ensuring the rationality of the color. This means that when repairing the damaged part of the image, not only can the structure and texture be accurately filled, but also the color of the repaired area can be ensured to be consistent with the overall image.

[0097] S407, the repair image feature is subjected to resolution enhancement processing through the feature extraction layer and the nonlinear mapping layer to obtain the target image feature.

[0098] The resolution enhancement module includes a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer. Referring to FIG. 5, after the repair image feature is subjected to feature extraction through the feature extraction layer, it is input into the nonlinear mapping layer to obtain the target image feature.

[0099] Further, the repair image feature is input into the resolution enhancement module for image resolution enhancement processing to obtain the target image, including: determining the repair image according to the repair image feature; segmenting the repair image according to the preset size to obtain a plurality of segmented images; for each pixel in the repair image, determining the pixel coordinates of the pixel in the repair image, and determining the pixel features of other pixels in the segmented image where the pixel is located as the associated pixel features of the pixel; inputting the preset size, the pixel coordinates of each pixel in the repair image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image.

[0100] It can be understood that the repair image feature corresponds to the processed image (repair image) that has been colored and repaired. For example, referring to FIG. 7, the repair image is determined according to the repair image feature, the pixel coordinates of each pixel in the repair image are determined, such as the pixel coordinates of pixel P in the repair image in FIG. 7 being (x, y), and then the preset size is obtained, such as the size of the repair image being N (pixels) * M (pixels), and the preset size being Then the repair image is segmented into 9 segmented images (such as k1 to k9 in FIG. 7) according to the preset size, and further, the pixel features of other pixels in the segmented image where the pixel P is located are determined, which can be extracted in the repair image feature.

[0101] Further, the preset size, the pixel coordinates of each pixel in the repair image, and the associated pixel features are input into the resolution enhancement module for image resolution enhancement processing to obtain the target image, including: obtaining a resolution magnification; inputting the resolution magnification, the preset size, the pixel coordinates of each pixel in the repair image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image, wherein the resolution corresponding to the target image is the product of the resolution of the processed image and the resolution magnification.

[0102] The resolution magnification is set by the user according to requirements, and the cloud server acquires the resolution magnification sent by the terminal device. In the embodiment of the present disclosure, the resolution magnification can be any magnification, for example, the resolution magnification is 1.5 times, and the resolution of the output target image is 1.5 times the resolution of the to-be-processed image. The embodiment of the present disclosure can set any resolution magnification as required to achieve magnification of any ratio.

[0103] In addition, the related art uses a preset magnification (such as 2 times and 4 times) to simulate the magnification effect of the to-be-processed image, which does not conform to the continuous perception characteristics of vision in the real world, and in actual application, if the magnification does not match the requirements, unnecessary waste of computing resources can also be caused.

[0104] Further, in the resolution enhancement module, the features of each pixel in the repaired image are jointly affected by three factors: the pixel features of the pixel points near the pixel, the pixel coordinates of the pixel, and the size of the segmented image. Since these pixel coordinates and the size of the segmented image are continuously changing, the resolution enhancement module can therefore associate the resolution of the output image with these continuous variables to achieve continuous resolution output.

[0105] In the embodiment of the present disclosure, the resolution enhancement module can be trained separately. The training process is as follows: a target sample image is acquired; the target sample image is down-sampled to obtain a plurality of training sample images, the resolution of the target sample image being the product of the resolution of the training sample images and the corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; and the resolution enhancement module is trained using the plurality of training sample images, the continuous plurality of magnifications, and the target sample image.

[0106] wherein, the continuous magnifications are, for example, 1.1 times, 1.2 times, 1.3 times, 1.4 times, …, or 1.05 times, 1.1 times, 1.15 times, 1.2 times, … The continuous magnifications of the present disclosure can be in various forms, which are not limited.

[0107] Specifically, for each training sample image, the training sample image is segmented according to a preset size to obtain a plurality of segmented sample images.

[0108] For each sample pixel in the training sample image, the pixel coordinates of the sample pixel in the training sample image are determined, and the pixel features of other sample pixels in the segmented sample image where the sample pixel is located are determined as the associated pixel features of the sample pixel.

[0109] For each training sample image, the pixel coordinates of each sample pixel in the repaired image, the associated pixel features of the sample pixels, and the corresponding magnification factor are determined as inputs of the resolution enhancement module, and the target sample image is used as a label to train the resolution enhancement module.

[0110] Exemplarily, the target sample image has a high resolution, such as a resolution of R. The target sample image is down-sampled to obtain a plurality of training sample images with lower resolutions, such as R / 1.2, R / 1.4, R / 1.6, R / 1.8, R / 2, and R / 2.2 images. Then, the low-resolution training sample images are used as inputs of the resolution enhancement module, and the target sample image is used as a label to train the resolution enhancement module. Further, the low-resolution training sample images are segmented according to the preset size, the pixel coordinates are determined, and the pixel features (i.e., convolutional features) of the sample pixels in the segmented images are determined. Then, the preset size, the pixel coordinates, the pixel features, and the magnification factor are input into the resolution enhancement module for processing to obtain a predicted image. The model parameters of the resolution enhancement module are adjusted by calculating the loss value of the predicted image and the target sample image. The resolution enhancement module trained by using training sample images with different resolutions can perform resolution magnification by any factor on a to-be-processed image.

[0111] In the embodiments of the present disclosure, the continuous magnification factor training method is adopted, and images with target resolutions can be output in a continuous spatial range according to requirements, and any additional down-sampling step is omitted.

[0112] Further, the resolution enhancement module can support continuous magnification factor of image resolution enhancement. This means that the user can select any resolution magnification factor according to the requirements, and the resolution enhancement module will correspondingly enhance the resolution of the image, and is no longer limited to the preset resolution magnification factor (such as 4 times and 2 times). The resolution magnification method provided by the present disclosure can maintain the details and clarity of the image, and is particularly useful for repairing old images and low-resolution images.

[0113] S408, the target image features and the first fusion features are fused to obtain third fusion features.

[0114] The first fusion features fused with the target image features have the same dimension as the target image features. For example, referring to FIG. 5, the dimension of the first fusion features F1 is N*M, and the dimension of the target image features is also N*M. The first fusion features are fused with the target image features and the first fusion features F1 to obtain the third fusion features.

[0115] In the embodiments of the present disclosure, the target image is obtained by processing the first fused feature and the target image feature after fusion, which can improve the quality of the target image.

[0116] S409, processing the third fused feature through the network reconstruction layer to obtain the target image.

[0117] The network reconstruction layer adopts a deconvolution network, and the deconvolution network can be used to deconvolve the third fused feature to obtain the target image.

[0118] In the embodiments of the present disclosure, the feature extraction module, the reference image retrieval module, the coloring module, the inpainting module and the resolution enhancement module can also be uniformly trained. Specifically, the training process is as follows: a fourth sample image and a label image of the fourth sample image are obtained. The fourth sample image is a low-resolution, damaged gray image. The label image of the fourth sample image is a high-resolution, undamaged color image. The fourth sample image is taken as the input of the image processing model, and the flowchart in FIG. 5 is referred to, to output a predicted image, determine a loss value of the predicted image and the label image, and adjust the model parameters of the image processing model according to the loss value.

[0119] In summary, the present disclosure can learn and optimize image coloring, image inpainting and resolution enhancement processing as a unified task. Such an end-to-end approach can better understand the internal relationship between different tasks, and cooperatively process multiple aspects of the image to be processed, thereby improving the overall image processing effect compared to the serial processing of these tasks. In addition, in the existing serial processing, errors in the previous processing steps can accumulate and affect subsequent steps, while the end-to-end approach can reduce this problem to some extent and enhance the overall effect.

[0120] Further, the present disclosure combines retrieval technology to obtain reference images similar to the content of the image to be processed, which can provide additional guidance information for image coloring, image inpainting and resolution enhancement. Based on the reference images, the present disclosure can more accurately infer the color, texture and other features of the region to be processed, thereby improving the stability of image processing and making the generated content more natural and harmonious in vision.

[0121] In addition, the present disclosure also has a pyramid structure cross-attention unit, which is designed to effectively capture and integrate information between the reference image and the image to be processed. Through multi-scale capture of different details of the image, the image processing model can not only focus on local features, but also consider global context when processing, thereby better understanding the relationship between the reference image and the image to be processed. At the same time, this cross-attention mechanism makes the decision-making process of the model more interpretable, which is convenient for understanding why the image processing model makes a specific processing decision.

[0122] Finally, the resolution enhancement module provided by the present disclosure can realize image magnification of any magnification. Unlike the conventional resolution enhancement module which can only process a fixed magnification, the resolution enhancement module provided by the present disclosure can magnify the image in continuous space, which provides flexibility for different needs of magnification processing. In addition, the design of the resolution enhancement module provided by the present disclosure also considers the calculation efficiency, so that it can reduce the consumption of computing resources while ensuring the effect in actual application.

[0123] The present disclosure also provides an image processing method applied to a cloud server, the image processing method comprising: receiving a to-be-processed image sent by a terminal device, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into a repair module for image repair to obtain repaired image features; inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image, and sending the target image to the terminal device.

[0124] The specific implementation process of this embodiment is referred to the above-mentioned embodiments, which will not be described here.

[0125] In the embodiments of the present disclosure, in addition to providing an image processing method, an image processing device is also provided, as shown in FIG. 8, the image processing device 80 comprises an extraction module 81, a processing module 82, a fusion module 83, an image repair module 84 and an enhancement module 85, wherein:

[0126] The extraction module 81 is configured to obtain a to-be-processed image, and input the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features.

[0127] The processing module 82 is configured to input the intermediate image features into a coloring module for coloring processing to obtain colored image features.

[0128] The fusion module 83 is configured to fuse the colored image features and the intermediate image features to obtain first fused features.

[0129] The image repair module 84 is configured to input the first fused features into a repair module for image repair to obtain repaired image features.

[0130] The enhancement module 85 is configured to input the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0131] In an embodiment, the extraction module 81 is specifically configured to: acquire a reference image of the to-be-processed image, the similarity of the reference image to the to-be-processed image being greater than a preset threshold; and input the to-be-processed image and the reference image into a feature extraction module to perform feature extraction, to obtain an intermediate image feature.

[0132] In an embodiment, the feature extraction module includes a convolution unit and a cross-attention unit, and the extraction module 81 is specifically configured to, when inputting the to-be-processed image and the reference image into the feature extraction module to perform feature extraction, to obtain the intermediate image feature: input the to-be-processed image into the convolution unit to perform feature extraction, to obtain a first image feature; input the reference image into the convolution unit to perform feature extraction, to obtain a second image feature; and input the first image feature and the second image feature into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature.

[0133] In an embodiment, the cross-attention unit includes a plurality of down-sampling layers and a cross-attention layer, and the extraction module 81 is specifically configured to, when inputting the first image feature and the second image feature into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature: input the first image feature into the plurality of down-sampling layers to perform down-sampling, to obtain, for each down-sampling layer, one first down-sampled image feature output by the down-sampling layer; input the second image feature into the plurality of down-sampling layers to perform down-sampling, to obtain, for each down-sampling layer, one second down-sampled image feature output by the down-sampling layer; for the first down-sampled image feature and the second down-sampled image feature obtained by the same layer down-sampling layer, input the first down-sampled image feature and the second down-sampled image feature into the cross-attention layer to perform attention calculation, to obtain a reference feature for the same layer down-sampling layer; fuse the reference feature and the first down-sampled image feature, to obtain a second fused feature; and splice a plurality of second fused features corresponding to the plurality of down-sampling layers, to obtain the intermediate image feature.

[0134] In an embodiment, the convolution unit includes a plurality of convolution layers, each convolution layer outputting one first image feature and one second image feature, and the extraction module 81 is specifically configured to, when inputting the first image feature and the second image feature into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature: input the first image feature and the second image feature output by the last convolution layer into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature.

[0135] In an embodiment, the coloring module includes a conversion layer and a plurality of deconvolution layers, the number of the plurality of deconvolution layers being the same as the number of the plurality of convolution layers, and the processing module 82 is specifically configured to: input the intermediate image feature into the conversion layer to perform color conversion processing, to obtain a converted image feature; and input the converted image feature into the plurality of deconvolution layers, to obtain a coloring image feature output by the deconvolution layer, wherein the deconvolution layer and the coloring image feature are in one-to-one correspondence.

[0136] In an embodiment, the fusion module 83 is specifically configured to: fuse the first image features corresponding to the convolutional layer outputs, the colored image features corresponding to the deconvolutional layer outputs, and the intermediate image features to obtain first fusion features, wherein the number of the first fusion features is the same as the number of the plurality of convolutional layers, and the dimensions of the first image features and the colored image features that are fused are the same.

[0137] In an embodiment, the enhancement module 85 is specifically configured to: determine the pixel coordinates of the pixels in the repair image features and the pixel features in the segmentation image of the preset size; and input the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain a target image.

[0138] In an embodiment, when the enhancement module 85 inputs the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain a target image, the enhancement module 85 is specifically configured to: obtain a resolution magnification; and input the resolution magnification, the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain the target image, wherein the resolution corresponding to the target image is the product of the resolution of the image to be processed and the resolution magnification.

[0139] In an embodiment, the system further includes a training module (not shown), which is configured to train the resolution enhancement module in the following manner: obtaining a target sample image; performing resolution reduction processing on the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being the product of the resolution of the training sample images and a corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; and training the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnifications, and the target sample image.

[0140] In an embodiment, the resolution enhancement module includes a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer, and the enhancement module 85 is specifically configured to: perform resolution enhancement processing on the repair image features through the feature extraction layer and the nonlinear mapping layer to obtain target image features; perform feature fusion on the target image features and the first fusion features to obtain third fusion features; and perform processing on the third fusion features through the network reconstruction layer to obtain a target image.

[0141] The specific implementation process is described above in the embodiments, and will not be described here again.

[0142] In addition, in some of the processes described in this specification and illustrated in the accompanying drawings, multiple operations can be performed in an order different from the order in which the operations are presented herein. It will be appreciated that the operations can be performed in an order different than the order in which they are presented herein or in parallel, and that the order of the operations should not be construed as a limitation unless such a limitation is expressly presented in this specification. In addition, the processes can include more or fewer operations than those presented in this specification, and the operations can be performed in serial or in parallel. It should be noted that the descriptions herein of "first", "second", and the like, are used to distinguish different messages, devices, modules, and the like, and do not represent a chronological order or a limitation on the types of "first" and "second".

[0143] FIG. 9 is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device 90 is configured to execute the image processing method and the image processing method described above. As shown in FIG. 9, the electronic device includes a memory 94 and a processor 95.

[0144] The memory 94 is configured to store computer programs and can be configured to store various other data to support operations on the electronic device. The memory 94 can be an Object Storage Service (OSS).

[0145] The memory 94 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk.

[0146] The processor 95 is coupled to the memory 94 and is configured to execute the computer programs in the memory 94 to: acquire a to-be-processed image, input the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; input the intermediate image features into a coloring module for coloring processing to obtain colored image features; fuse the colored image features and the intermediate image features to obtain first fused features; input the first fused features into a repair module for image inpainting to obtain inpainted image features; and input the inpainted image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0147] In an embodiment, when the processor 95 inputs the to-be-processed image into the feature extraction module for feature extraction to obtain the intermediate image features, the processor 95 is specifically configured to: acquire a reference image of the to-be-processed image, the content similarity between the reference image and the to-be-processed image being greater than a preset threshold; and input the to-be-processed image and the reference image into the feature extraction module for feature extraction to obtain the intermediate image features.

[0148] In an embodiment, the feature extraction module comprises a convolution unit and a cross-attention unit, and the processor 95 is specifically configured to: input the to-be-processed image into the convolution unit to perform feature extraction, to obtain a first image feature; input the reference image into the convolution unit to perform feature extraction, to obtain a second image feature; and input the first image feature and the second image feature into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature.

[0149] In an embodiment, the cross-attention unit comprises a plurality of down-sampling layers and a cross-attention layer, and the processor 85 is specifically configured to: input the first image feature into the plurality of down-sampling layers to perform down-sampling, to obtain a first down-sampled image feature corresponding to an output of each down-sampling layer; input the second image feature into the plurality of down-sampling layers to perform down-sampling, to obtain a second down-sampled image feature corresponding to an output of each down-sampling layer; input the first down-sampled image feature and the second down-sampled image feature obtained for the same layer down-sampling layer into the cross-attention layer to perform attention calculation, to obtain a reference feature for the same layer down-sampling layer; fuse the reference feature and the first down-sampled image feature, to obtain a second fusion feature; and splice a plurality of second fusion features corresponding to the plurality of down-sampling layers, to obtain the intermediate image feature.

[0150] In an embodiment, the convolution unit comprises a plurality of convolution layers, each convolution layer outputs a corresponding first image feature and a corresponding second image feature, and the processor 95 is specifically configured to: input the first image feature and the second image feature output by the last convolution layer into the cross-attention unit to perform attention calculation, to obtain the intermediate image feature.

[0151] In an embodiment, the coloring module comprises a conversion layer and a plurality of deconvolution layers, the number of the plurality of deconvolution layers is the same as the number of the plurality of convolution layers, and the processor 95 is specifically configured to: input the intermediate image feature into the conversion layer to perform color conversion processing, to obtain a converted image feature; and input the converted image feature into the plurality of deconvolution layers, to obtain a coloring image feature output by the deconvolution layer, where the deconvolution layer corresponds to the coloring image feature in a one-to-one manner.

[0152] In an embodiment, the processor 95 is specifically configured to fuse the first image features corresponding to the convolutional layer outputs, the colored image features corresponding to the deconvolutional layer outputs and the intermediate image features to obtain the first fused features when fusing the colored image features and the intermediate image features to obtain the first fused features, wherein the number of the first fused features is the same as the number of the plurality of convolutional layers, and the dimensions of the first image features and the colored image features that are fused are the same.

[0153] In an embodiment, the processor 95 is specifically configured to determine a repair image according to the repair image features, split the repair image according to a preset size to obtain a plurality of split images, determine, for each pixel in the repair image, a pixel coordinate of the pixel in the repair image and pixel features of other pixels in the split image in which the pixel is located as associated pixel features of the pixel, and input the preset size, the pixel coordinates of the pixels in the repair image and the associated pixel features into the resolution enhancement module to perform the image resolution enhancement processing to obtain the target image when inputting the repair image features into the resolution enhancement module to perform the image resolution enhancement processing to obtain the target image.

[0154] In an embodiment, the processor 95 is specifically configured to obtain a resolution magnification, and input the resolution magnification, the preset size, the pixel coordinates of the pixels in the repair image and the associated pixel features into the resolution enhancement module to perform the image resolution enhancement processing to obtain the target image when inputting the resolution magnification, the preset size, the pixel coordinates of the pixels in the repair image and the associated pixel features into the resolution enhancement module to perform the image resolution enhancement processing to obtain the target image, wherein the resolution corresponding to the target image is the product of the resolution of the to-be-processed image and the resolution magnification.

[0155] In an embodiment, the processor 95 is further configured to train the resolution enhancement module in the following manner: obtain a target sample image; perform resolution reduction processing on the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being the product of the resolution of the training sample image and a corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; and train the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnifications and the target sample image.

[0156] In an embodiment, the resolution enhancement module comprises a feature extraction layer, a nonlinear mapping layer and a network reconstruction layer, and the processor 95 is specifically configured to perform resolution enhancement processing on the repair image features through the feature extraction layer and the nonlinear mapping layer to obtain target image features, perform feature fusion on the target image features and the first fused features to obtain third fused features, and perform processing on the third fused features through the network reconstruction layer to obtain the target image when inputting the repair image features into the resolution enhancement module to perform the image resolution enhancement processing to obtain the target image.

[0157] Further, as shown in FIG. 9, the electronic device also includes a firewall 91, a load balancer 92, a communication component 96, a power component 93, and other components. Only some of the components are shown in FIG. 9, and the electronic device can include many more components not shown in FIG. 9.

[0158] Accordingly, the embodiments of the present disclosure also provide a computer program product storing a computer program, which causes a processor to implement the steps in the above-described method when the computer program / instructions are executed by the processor.

[0159] Accordingly, the embodiments of the present disclosure also provide a computer program product, which includes computer program / instructions, which causes a processor to implement the steps in the above-described method when the computer program / instructions are executed by the processor.

[0160] The communication component in FIG. 9 above is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, and other mobile communication networks, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related texts from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0161] The power component in FIG. 9 above provides power to various components of the device where the power component is located. The power component can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power component is located.

[0162] Those skilled in the art will understand that embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Accordingly, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0163] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0164] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0165] These computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0166] In one typical configuration, the computing device includes one or more processors, such as a processor (CPU and / or GPU), an input / output interface, a network interface, and a memory.

[0167] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer-readable media.

[0168] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can store text by any method or technology. The text can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store text that can be accessed by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0169] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0170] The above is only an embodiment of the present disclosure and is not intended to limit the present disclosure. The present disclosure can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present disclosure shall be included in the scope of claims of the present disclosure.

Claims

1. An image processing method, wherein, The method comprises: acquiring a to-be-processed image, and inputting the to-be-processed image into a feature extraction module to perform feature extraction, to obtain intermediate image features; inputting the intermediate image features into a coloring module to perform coloring processing, to obtain colored image features; fusing the colored image features and the intermediate image features, to obtain first fused features; inputting the first fused features into a repair module to perform image repair, to obtain repaired image features; inputting the repaired image features into a resolution enhancement module to perform image resolution enhancement processing, to obtain a target image.

2. The image processing method of claim 1, wherein, The method comprises: acquiring a reference image of the to-be-processed image, wherein a content similarity between the reference image and the to-be-processed image is greater than a preset threshold; inputting the to-be-processed image and the reference image into the feature extraction module to perform feature extraction, to obtain the intermediate image features.

3. The image processing method of claim 2, wherein, The feature extraction module comprises a convolution unit and a cross-attention unit, and the method comprises: inputting the to-be-processed image into the convolution unit to perform feature extraction, to obtain first image features; inputting the reference image into the convolution unit to perform feature extraction, to obtain second image features; inputting the first image features and the second image features into the cross-attention unit to perform attention calculation, to obtain the intermediate image features.

4. The image processing method of claim 3, wherein, The cross-attention unit comprises a plurality of down-sampling layers and a cross-attention layer, and the method comprises: inputting the first image features into the plurality of down-sampling layers to perform down-sampling, to obtain one first down-sampled image feature corresponding to the output of each down-sampling layer; inputting the second image features into the plurality of down-sampling layers to perform down-sampling, to obtain one second down-sampled image feature corresponding to the output of each down-sampling layer; for the first down-sampled image features and the second down-sampled image features obtained for the same layer down-sampling layer, inputting the first down-sampled image features and the second down-sampled image features into the cross-attention layer to perform attention calculation, to obtain a reference feature for the same layer down-sampling layer; fusing the reference feature and the first down-sampled image features, to obtain second fused features; splicing a plurality of second fused features corresponding to the plurality of down-sampling layers, to obtain the intermediate image features.

5. The image processing method according to claim 3 or 4, wherein The convolution unit comprises a plurality of convolution layers, each convolution layer outputs one first image feature and one second image feature, and the method comprises: inputting the first image features and the second image features output by the last convolution layer into the cross-attention unit to perform attention calculation, to obtain the intermediate image features.

6. The image processing method of claim 5, wherein, The coloring module comprises a conversion layer and a plurality of deconvolution layers, the number of the plurality of deconvolution layers being the same as the number of the plurality of convolution layers, the inputting of the intermediate image feature into the coloring module for coloring processing to obtain a colored image feature comprising: color conversion processing of the intermediate image feature in the conversion layer to obtain a converted image feature; inputting of the converted image feature into the plurality of deconvolution layers to obtain a deconvolution layer output colored image feature, wherein the deconvolution layer corresponds to the colored image feature one by one.

7. The image processing method of claim 6, wherein, The fusion of the colored image feature and the intermediate image feature to obtain a first fusion feature comprising: fusion of the first image feature output by the corresponding convolution layer, the colored image feature output by the deconvolution layer and the intermediate image feature to obtain the first fusion feature, wherein the number of the first fusion feature is the same as the number of the plurality of convolution layers, and the dimensions of the first image feature and the colored image feature for fusion are the same.

8. The image processing method of claim 1, wherein, The inputting of the repair image feature into the resolution enhancement module for image resolution enhancement processing to obtain a target image comprising: determination of a repair image according to the repair image feature; segmentation of the repair image according to a preset size to obtain a plurality of segmented images; for each pixel in the repair image, determination of a pixel coordinate of the pixel in the repair image and determination of pixel features of other pixels in the segmented image in which the pixel is located as associated pixel features of the pixel; inputting of the preset size, the pixel coordinates of the pixels in the repair image and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image.

9. The image processing method of claim 8, wherein, The inputting of the pixel coordinates, the preset size and the pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image comprising: acquisition of a resolution magnification; inputting of the resolution magnification, the preset size, the pixel coordinates of the pixels in the repair image and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image, wherein a resolution corresponding to the target image is a product of a resolution of the to-be-processed image and the resolution magnification.

10. The image processing method of claim 9, wherein, The resolution enhancement module is trained in the following manner: acquisition of a target sample image; resolution reduction processing of the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being a product of the resolution of the training sample image and a corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; training of the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnifications and the target sample image.

11. The image processing method according to any one of claims 1 to 4, wherein, The resolution enhancement module comprises a feature extraction layer, a nonlinear mapping layer and a network reconstruction layer, the inputting of the repair image feature into the resolution enhancement module for image resolution enhancement processing to obtain a target image comprising: The repair image feature is subjected to resolution enhancement processing through the feature extraction layer and the nonlinear mapping layer to obtain a target image feature; The target image feature and the first fusion feature are subjected to feature fusion to obtain a third fusion feature; The third fusion feature is subjected to processing through the network reconstruction layer to obtain the target image.

12. An image processing method, wherein, The image processing method is applied to a cloud server and includes: receiving a to-be-processed image sent by a terminal device and inputting the to-be-processed image into a feature extraction module to extract features to obtain an intermediate image feature; inputting the intermediate image feature into a coloring module to perform coloring processing to obtain a colored image feature; fusing the colored image feature and the intermediate image feature to obtain a first fusion feature; inputting the first fusion feature into a repair module to perform image repair to obtain a repair image feature; inputting the repair image feature into a resolution enhancement module to perform image resolution enhancement processing to obtain a target image and sending the target image to the terminal device.

13. An image processing apparatus, comprising: The image processing method includes: a feature extraction module configured to obtain a to-be-processed image and input the to-be-processed image into the feature extraction module to extract features to obtain an intermediate image feature; a processing module configured to input the intermediate image feature into a coloring module to perform coloring processing to obtain a colored image feature; a fusion module configured to fuse the colored image feature and the intermediate image feature to obtain a first fusion feature; an image repair module configured to input the first fusion feature into a repair module to perform image repair to obtain a repair image feature; a resolution enhancement module configured to input the repair image feature into the resolution enhancement module to perform image resolution enhancement processing to obtain a target image.

14. An electronic device, comprising: The image processing method includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method according to any one of claims 1 to 12 when executing the computer program.

15. A computer program, wherein, The computer program includes computer instructions executable by a processor to implement the image processing method according to any one of claims 1 to 12.

Images