Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining a to-be-processed image, inputting the to-be-processed image into a feature extraction module for feature extraction, and obtaining an intermediate image feature; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the coloring image feature and the intermediate image feature to obtain a first fusion feature; inputting the first fusion feature into a restoration module for image restoration to obtain a restored image feature; and inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image. The coloring image features output by the coloring module are used for processing of the restoration module and the resolution enhancement module, feature information sharing among the coloring module, the restoration module and the resolution enhancement module is achieved, and the image quality of the obtained target image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method and device and electronic equipment. BACKGROUND

[0002] At present, in order to convert low-resolution old images into high-definition high-resolution color images, independent steps are adopted for coloring, repairing and resolution enhancement. Among them, image coloring can automatically color black and white photos; image repairing can identify and correct damaged parts in the image, such as scratches, spots, fading and blank areas; and super-resolution technology is used to improve the resolution of the image, making the image look clearer and more delicate.

[0003] However, using independent steps for coloring, repairing and resolution enhancement of images leads to the problem that the image obtained after image processing has low quality. SUMMARY

[0004] The present application provides an image processing method, device and electronic equipment, which solves the problem that the image obtained after image processing has low quality.

[0005] The first aspect of the embodiment of the present application provides an image processing method, comprising: acquiring a to-be-processed image, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into a repairing module for image repairing to obtain repaired image features; and inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0006] The second aspect of the embodiment of the present application provides an image processing method applied to a cloud server, comprising:

[0007] receiving a to-be-processed image sent by a terminal device, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features;

[0008] inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features;

[0009] fusing the colored image features and the intermediate image features to obtain first fused features;

[0010] inputting the first fused features into a repairing module for image repairing to obtain repaired image features;

[0011] inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image;

[0012] transmitting the target image to the terminal device.

[0013] The third aspect of the embodiment of the present application provides an image processing device, comprising:

[0014] The extraction module is configured to acquire the to-be-processed image and input the to-be-processed image into the feature extraction module to perform feature extraction, so as to obtain intermediate image features.

[0015] The processing module is configured to input the intermediate image features into the coloring module to perform coloring processing, so as to obtain colored image features.

[0016] The fusion module is configured to fuse the colored image features and the intermediate image features, so as to obtain first fusion features.

[0017] The image repairing module is configured to input the first fusion features into the repairing module to perform image repairing, so as to obtain repaired image features.

[0018] The enhancement module is configured to input the repaired image features into the resolution enhancement module to perform image resolution enhancement processing, so as to obtain the target image.

[0019] The fourth aspect of the embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the image processing method of the first aspect and / or the image processing method of the second aspect are implemented.

[0020] The fifth aspect of the embodiment of the present application provides a computer program product, and the computer program product stores a computer program, and when the computer program is executed by a processor, the processor implements the image processing method of the first aspect and / or the image processing method of the second aspect.

[0021] The embodiment of the present application is applied to the scene of improving image quality, and the present application comprises: acquiring a to-be-processed image, inputting the to-be-processed image into a feature extraction module to perform feature extraction, so as to obtain intermediate image features; inputting the intermediate image features into a coloring module to perform coloring processing, so as to obtain colored image features; fusing the colored image features and the intermediate image features, so as to obtain first fusion features; inputting the first fusion features into a repairing module to perform image repairing, so as to obtain repaired image features; and inputting the repaired image features into a resolution enhancement module to perform image resolution enhancement processing, so as to obtain a target image. The colored image features output by the coloring module are used for processing of the repairing module and the resolution enhancement module, feature information sharing among the coloring module, the repairing module and the resolution enhancement module is implemented, that is, image repairing is performed on the basis of the image being colored, resolution enhancement is performed on the basis of the image being colored and repaired, and the subsequent image processing steps consider the features obtained in the previous steps, thereby improving the image quality of the obtained target image. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0023] Figure 1 An application scenario diagram provided for an exemplary embodiment of the application;

[0024] Figure 2 A step flowchart of an image processing method provided for an exemplary embodiment of the application;

[0025] Figure 3 A schematic diagram of an image processing model provided for an exemplary embodiment of the application;

[0026] Figure 4 A step flowchart of another image processing method provided for an exemplary embodiment of the application;

[0027] Figure 5 A schematic diagram of another image processing model provided for an exemplary embodiment of the application;

[0028] Figure 6 A schematic diagram of a cross-attention unit provided for an exemplary embodiment of the application;

[0029] Figure 7 A schematic diagram of a repaired image provided for an exemplary embodiment of the application;

[0030] Figure 8 A structural block diagram of an image processing apparatus provided for an exemplary embodiment of the application;

[0031] Figure 9 A structural schematic diagram of an electronic device provided for an exemplary embodiment of the application. DETAILED DESCRIPTION

[0032] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions of the application will be described clearly and completely below with reference to the embodiments of the application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0033] With the rapid development of artificial intelligence technology, the field of image inpainting has undergone tremendous changes. In traditional image inpainting methods, the processing process mostly requires manual operation by image processing experts. This process not only consumes time and effort, but the final inpainting effect is largely dependent on the skills and experience of the experts. Currently, with the use of artificial intelligence technology, image inpainting has become more automated and intelligent.

[0034] Further, in the related art, the resolution enhancement, inpainting and coloring of images are usually performed sequentially as independent steps, i.e., the image is respectively subjected to resolution enhancement, inpainting and coloring, and then merged to obtain the final image. This processing method has its limitations. On the one hand, due to the lack of sharing of feature information between steps, a poor result may be obtained.

[0035] Based on the above problems, the present application provides an image processing method, which comprises the following steps: obtaining a to-be-processed image, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into an inpainting module for image inpainting to obtain inpainted image features; and inputting the inpainted image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image. In this way, the feature information can be shared among the modules during the entire processing process, and the quality of the final image, such as the visual effect, can be enhanced by coordinating the steps.

[0036] In this embodiment, the execution device of the image processing method is not limited. Alternatively, the image processing method can implement the overall image processing method by means of a cloud computing system. For example, the image processing method can be applied to a cloud server to run various models by means of the advantages of resources on the cloud; relative to the application to the cloud, the image processing method can also be applied to a server device such as a conventional server, a cloud server or a server array.

[0037] In addition, with reference to Figure 1 , an application scenario of the present application is provided. The application scenario includes a terminal and a cloud server, wherein the cloud server is deployed with an image processing model, the terminal sends a to-be-processed image with poor quality to the cloud server, the cloud server inputs the to-be-processed image into the image processing model for image processing, and outputs a target image with high quality, and then the cloud server sends the target image to the terminal.

[0038] Figure 1 This is only one example application scenario of the present application, and the present application can also be applied in other related scenarios, which are not limited herein.

[0039] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0040] Figure 2 A step flow chart of an image processing method is provided for an exemplary embodiment of the present application. As shown in the step flow chart of the image processing method, the method specifically includes the following steps: Figure 2

[0041] S201, obtaining a to-be-processed image, and inputting the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features.

[0042] Referring to Figure 3 , an image processing model 30 is shown, which includes a feature extraction module, a coloring module, a repairing module, and a resolution enhancement module. The image processing model is pre-trained.

[0043] In the embodiment of the present application, the to-be-processed image can be sent by a terminal to a server, and the to-be-processed image includes at least one of the following features: black and white, low resolution, and damaged area. The damaged area includes scratches, spots, discoloration, or blank areas.

[0044] Further, the to-be-processed image is inputted into Figure 3 the feature extraction module for feature extraction to obtain intermediate image features of the to-be-processed image. As shown in Figure 5 , the feature extraction module includes a convolution unit and a cross-attention unit; the convolution unit includes multiple convolution layers.

[0045] S202, inputting the intermediate image features into the coloring module for coloring processing to obtain colored image features.

[0046] Referring to Figure 3 , the image features output by the coloring module are colored image features, which refer to colored image features after coloring the to-be-processed image. The colored image features can obtain the colored image after being imaged.

[0047] S203, fusing the colored image features and the intermediate image features to obtain first fused features.

[0048] Wherein, fusing the colored image features and the intermediate image features can make the first fused features have both the features of the to-be-processed image and the features of the colored image.

[0049] Referring to Figure 3 , the colored image features and the intermediate image features are fused to obtain the first fused features.

[0050] ​Further, in order to further ensure the coordination of colors, the server fuses the colored image features after coloring processing with the original gray image features (intermediate image features of the to-be-processed image), and such feature fusion ensures smooth transition and reasonable matching of image colors in the repairing process.

[0051] S204, input the first fused features into a repairing module for image repairing to obtain repaired image features.

[0052] The repairing module includes a module constructed by using diffusion models (DM), and the repairing module can also be constructed by using other neural networks, which are not limited herein. The repairing module is used for repairing the to-be-processed image, for example, the repairing module can repair the damaged area in the to-be-processed image.

[0053] In this application, the colored image features and the intermediate image features are fused and then input into the repairing module for image repairing, which can make the repairing module better repair the damaged area of the to-be-processed image based on the colored image features.

[0054] With reference to Figure 3 , the first fused features are input into the repairing module for image repairing, and repaired image features are output.

[0055] S205, input the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0056] In the embodiments of this application, the resolution enhancement module includes a feature extraction layer (such as a convolutional network), a nonlinear mapping layer (such as a multilayer perceptron, MLP), and a network reconstruction layer (such as a deconvolution layer). The repaired image features are input into the feature extraction layer and the nonlinear mapping layer for image resolution enhancement, and then target image features are obtained. The target image features are input into the network reconstruction layer for reconstruction, and then a target image is obtained.

[0057] With reference to Figure 3 , the repaired image features are input into the resolution enhancement module, and a target image is output.

[0058] In the embodiments of this application, the image processing model combines the feature extraction module, the coloring module, the repairing module, and the resolution enhancement module. The intermediate image features obtained by the feature extraction module after feature extraction of the to-be-processed image can be applied to the subsequent coloring module and repairing module. Finally, the repaired image features are input into the resolution enhancement module for resolution enhancement, so that the coloring, repairing, and resolution enhancement of the to-be-processed image can be efficiently and uniformly implemented, and the feature information is shared during the implementation process, thereby improving the image processing efficiency and the quality of the target image.

[0059] Figure 4 Another step flow chart of an image processing method is provided for the exemplary embodiments of the present application. As shown in the image processing method, the steps specifically include the following steps: Figure 4

[0060] S401, obtaining a to-be-processed image and a reference image of the to-be-processed image.

[0061] The content similarity between the reference image and the to-be-processed image is greater than a preset threshold. The content similarity refers to the similarity between the content included in the reference image and the content included in the to-be-processed image. For example, the to-be-processed image includes a landmark building, and the reference image also includes the landmark building. In addition, the preset threshold is, for example, 80%.

[0062] In the embodiments of the present application, image coloring is a technology of adding colors to grayscale images or videos, which plays an important role in restoring old photos and old movies. However, image coloring essentially has uncertainty because an object can have multiple acceptable colors, and when coloring freely without explicit color guidance, the coloring model sometimes generates colors that do not conform to the actual or user expectations. Therefore, appropriate color guidance is needed to help the coloring module generate more realistic or user-expected colors. Image inpainting and image resolution enhancement also face such challenges. Without proper reference images as guidance, the added or reconstructed content can be inconsistent with the actual situation or expectations. The reference image of the present application can improve the quality of the target image.

[0063] Referring to Figure 5 , another image processing model 50 is shown, which includes a feature extraction module, a reference image retrieval module, a coloring module, an inpainting module, and a resolution enhancement module. The feature extraction module includes a convolution unit and a cross-attention unit. The convolution unit includes multiple convolution layers, such as convolution layer a1, convolution layer a2, and convolution layer a3 in Figure 5 . The coloring module includes a conversion layer and multiple deconvolution layers, such as deconvolution layer b1, deconvolution layer b2, and deconvolution layer b3 in Figure 5 .

[0064] In addition, the most similar image to the to-be-processed image can be obtained by searching the image library as the reference image.

[0065] In the embodiments of the present application, referring to Figure 5 , after obtaining the to-be-processed image, the to-be-processed image is input into the reference image retrieval module for similar image retrieval to obtain the reference image.

[0066] S402, inputting the to-be-processed image and the reference image into the feature extraction module for feature extraction to obtain intermediate image features.

[0067] ​wherein, referring to Figure 5 The feature extraction module comprises a convolution unit and a cross-attention unit, and the feature extraction module is used for extracting features of the to-be-processed image and the reference image to obtain intermediate image features, comprising: inputting the to-be-processed image into the convolution unit to extract features to obtain first image features; inputting the reference image into the convolution unit to extract features to obtain second image features; and inputting the first image features and the second image features into the cross-attention unit to perform attention calculation to obtain the intermediate image features.

[0068] Referring to Figure 5 The processing process of the convolution unit on the to-be-processed image is: inputting the to-be-processed image into the convolution layer a1 to output the first image features A11, inputting the first image features A11 into the convolution layer a2 to output the first image features A12, and inputting the first image features A12 into the convolution layer a3 to output the first image features A13. The processing process of the convolution unit on the reference image is: inputting the reference image into the convolution layer a1 to output the second image features A21, inputting the second image features A21 into the convolution layer a2 to output the second image features A22, and inputting the second image features A22 into the convolution layer a3 to output the second image features A23.

[0069] The input of the convolution unit is a low-resolution gray image (to-be-processed image), and the to-be-processed image is limited in spatial resolution and color information. In order to perform the next step of processing, one or more convolution layers are needed to increase the feature representation of the gray image. The convolution layer captures the local features of the gray image, such as edges, corners, etc., and obtains higher-level features by increasing the network depth.

[0070] In an optional embodiment, the convolution unit comprises: a plurality of convolution layers, each convolution layer outputs a corresponding first image feature, and each convolution layer outputs a corresponding second image feature; and the first image features and the second image features are input into the cross-attention unit to perform attention calculation to obtain the intermediate image features, comprising: inputting the first image features and the second image features output by the last convolution layer into the cross-attention unit to perform attention calculation to obtain the intermediate image features.

[0071] Further, the first image features (such as the first image features A13) and the second image features (such as the second image features A23) output by the last convolution layer (such as the convolution layer a3 in Figure 5 ) are input into the cross-attention unit to perform attention calculation to obtain the intermediate image features.

[0072] Referring to Figure 6 , a schematic image of a cross-attention unit is shown, and the cross-attention unit comprises a plurality of down-sampling layers and a cross-attention layer. The plurality of down-sampling layers, such as Figure 6 the down-sampling layer c1, the down-sampling layer c2, and the down-sampling layer c3 in

[0073] In the embodiment of the present application, the first image feature and the second image feature are input into the cross attention unit for attention calculation to obtain the intermediate image feature, including: inputting the first image feature into multiple down-sampling layers for down-sampling to obtain one first down-sampled image feature corresponding to the output of each down-sampling layer; inputting the second image feature into multiple down-sampling layers for down-sampling to obtain one second down-sampled image feature corresponding to the output of each down-sampling layer; for the first down-sampled image feature and the second down-sampled image feature obtained by the same layer down-sampling layer, inputting the first down-sampled image feature and the second down-sampled image feature into the cross attention layer for attention calculation to obtain the reference feature for the same layer down-sampling layer; fusing the reference feature and the first down-sampled image feature to obtain the second fusion feature; splicing multiple second fusion features corresponding to multiple down-sampling layers to obtain the intermediate image feature.

[0074] As shown in Figure 6 , the first image feature A13 is input into the down-sampling layer c1 for down-sampling to obtain the first down-sampled image feature C11, the first down-sampled image feature C11 is input into the down-sampling layer c2 for down-sampling to obtain the first down-sampled image feature C12, and the first down-sampled image feature C12 is input into the down-sampling layer c3 for down-sampling to obtain the first down-sampled image feature C13. The second image feature A23 is input into the down-sampling layer c1 for down-sampling to obtain the second down-sampled image feature C21, the second down-sampled image feature C21 is input into the down-sampling layer c2 for down-sampling to obtain the second down-sampled image feature C22, and the second down-sampled image feature C22 is input into the down-sampling layer c3 for down-sampling to obtain the second down-sampled image feature C23. The cross attention layer is used to perform attention calculation on the first down-sampled image feature C11 and the second down-sampled image feature C21 to obtain the reference feature D1. The cross attention layer is used to perform attention calculation on the first down-sampled image feature C12 and the second down-sampled image feature C22 to obtain the reference feature D2. The cross attention layer is used to perform attention calculation on the first down-sampled image feature C13 and the second down-sampled image feature C23 to obtain the reference feature D3. Then, the first down-sampled image feature C11 is fused with the reference feature D1 to obtain the second fusion feature E1. The first down-sampled image feature C12 is fused with the reference feature D2 to obtain the second fusion feature E2. The first down-sampled image feature C13 is fused with the reference feature D3 to obtain the second fusion feature E3. Finally, the second fusion feature E1, the second fusion feature E2 and the second fusion feature E3 are spliced to obtain the intermediate image feature.

[0075] In the embodiments of the present application, in order to enhance the modeling capability of the image processing model in the coloring and repairing tasks for the semantic connection between the to-be-processed image and the reference image, and increase the explainability of the image processing model, the present application provides a cross-attention unit. The structure of the specific cross-attention unit is as shown in Figure 6 The input of the cross-attention unit includes the first image feature of the to-be-processed image and the second image feature of the reference image, and the output is the intermediate image feature. In the present application, the cross-attention mechanism is used to promote the information interaction between the to-be-processed image and the reference image. In addition, the cross-attention unit is used for subsequent coloring and repairing of the to-be-processed image, and the features of the reference image can be input into the coloring and repairing process.

[0076] In addition, in the cross-attention layer, a convolutional network can be used for convolution operation to query the feature Q input (Query feature), the key feature K ref (Key feature) and the value feature V ref (Value feature) from the first down-sampled image feature and the second down-sampled image feature, and then Q input and K ref are multiplied by a correlation matrix calculated by matrix multiplication and a softmax (normalization exponential) function, and the reference feature is obtained by multiplying the correlation matrix and V ref .

[0077] In the embodiments of the present application, the cross-attention unit with a pyramid structure is used to aggregate the features of the to-be-processed image and the reference image. The pyramid structure means that the features will be processed at different resolution levels, which helps to capture the image details from coarse to fine. The cross-attention mechanism can make the network focus on the most relevant feature part in the to-be-processed image and the reference image, and effectively combine the feature information of the two.

[0078] In an embodiment, the first image feature and the second image feature can also be directly input Figure 6 into the cross-attention layer as shown in the figure to obtain the reference feature, and then the first image feature and the reference feature are fused to obtain the intermediate image feature.

[0079] In the embodiments of the present application, the strategy of applying the cross-attention layer at multiple scales is used to construct a pyramid-shaped attention layer structure as shown in Figure 6 Figure 6 ​As shown, the first image feature and the second image feature are processed by downsampling to obtain feature representations of different scales (first downsampled image features and second downsampled image features), and each of the first downsampled image features and the second downsampled image features at each scale is processed by a cross-attention layer to obtain a feature output (reference feature) at the scale. Finally, the reference feature is fused with the first downsampled image feature at the corresponding scale to obtain a second fused feature, and the second fused features at different scales are spliced together to obtain a multi-scale feature output (intermediate image feature).

[0080] S403, input the intermediate image feature into the conversion layer for color conversion processing to obtain a converted image feature.

[0081] The coloring module includes a conversion layer and a plurality of deconvolution layers. The conversion layer uses a nonlinear mapping function, i.e., the intermediate image feature is mapped by a nonlinear mapping function to obtain a converted image feature after coloring processing.

[0082] S404, input the converted image feature into the plurality of deconvolution layers to obtain a coloring image feature output by the deconvolution layer.

[0083] The deconvolution layer corresponds to the coloring image feature one-to-one. Referring to Figure 5 The deconvolution layer b1 outputs a coloring image feature B1, the deconvolution layer b2 outputs a coloring image feature B3, and the deconvolution layer b3 outputs a coloring image feature B3.

[0084] In the embodiment of the present application, the coloring image feature corresponds to a colored image to be processed.

[0085] In addition, the coloring module can be trained separately. The training process of the coloring module is as follows: a first sample image and a reference sample image with a color and content similarity greater than a threshold are obtained, the first sample image is a color image, the first sample image is converted into a grayscale image, the grayscale image and the reference sample image are input into the feature extraction module to obtain a sample image feature, and then the sample image feature is input into the coloring module to obtain a sample coloring image feature. Then process the sample coloring image feature to obtain an output coloring image, calculate the loss value of the output coloring image and the first sample image, and adjust the model parameters of the feature extraction module and the coloring module according to the loss value.

[0086] S405, fuse the first image feature output by the corresponding convolution layer, the coloring image feature output by the deconvolution layer, and the intermediate image feature to obtain a first fused feature.

[0087] The number of first fused features is the same as the number of convolution layers, and the dimensions of the first image features and the coloring image features to be fused are the same.

[0088] Referring toFigure 5 If the image to be processed is an N*M image, after the convolution layer a1, the first image feature A11 of N*M is obtained, after the convolution layer a2, the first image feature A12 of N*M is obtained, and after the convolution layer a3, the first image feature A13 of N*M is obtained. Further, the dimension of the colored image feature B1 is After the deconvolution of the deconvolution layer b2, the colored image feature B2 of N*M is obtained, and after the deconvolution of the deconvolution layer b3, the colored image feature B3 of N*M is obtained.

[0089] Further, the first image feature A11, the intermediate image feature, and the colored image feature B3 are fused to obtain the first fusion feature F1. The first image feature A12, the intermediate image feature, and the colored image feature B2 are fused to obtain the first fusion feature F2. The first image feature A13, the intermediate image feature, and the colored image feature B1 are fused to obtain the first fusion feature F3.

[0090] In the embodiments of the present application, the first image feature of the image to be processed, the colored image feature, and the intermediate image feature obtained through the cross-attention unit are fused to generate a final feature representation (first fusion feature) that integrates all useful information, providing rich guidance for subsequent image inpainting and image resolution enhancement.

[0091] S406, input the first fusion feature into the inpainting module for image inpainting to obtain an inpainted image feature.

[0092] The specific implementation process of this step is referred to S204, which will not be described here.

[0093] In the embodiments of the present application, the first fusion feature is input into the inpainting module for image inpainting to obtain an inpainted image feature.

[0094] Further, the inpainting module can be trained separately. The training process is to obtain a second sample image, which is a complete and undamaged image. The second sample image is damaged to obtain a damaged sample image. The damaged image is input into the inpainting module, and the second sample image is used as a label to train the inpainting module.

[0095] In addition, since the inpainting module shares feature information with the coloring module, it can generate more natural and harmonious content while ensuring the rationality of the color. This means that when repairing the damaged part of the image, not only the structure and texture can be accurately filled, but also the color of the repaired area can be ensured to be consistent with the overall image.

[0096] ​​S407, the repaired image feature is subjected to resolution enhancement processing by the feature extraction layer and the nonlinear mapping layer to obtain a target image feature.

[0097] The resolution enhancement module includes a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer. Referring to Figure 5 , the repaired image feature is subjected to feature extraction by the feature extraction layer and is input into the nonlinear mapping layer to obtain a target image feature.

[0098] Further, the repaired image feature is input into the resolution enhancement module for image resolution enhancement processing to obtain a target image, including: determining a repaired image according to the repaired image feature; segmenting the repaired image according to a preset size to obtain a plurality of segmented images; for each pixel in the repaired image, determining the pixel coordinates of the pixel in the repaired image, and determining the pixel features of other pixels in the segmented image where the pixel is located as the associated pixel features of the pixel; inputting the preset size, the pixel coordinates of each pixel in the repaired image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image.

[0099] It can be understood that the repaired image feature corresponds to a processed image (repaired image) that has been colored and repaired. For example, referring to Figure 7 , the repaired image is determined according to the repaired image feature, and the pixel coordinates of each pixel in the repaired image are determined, such as Figure 7 , the pixel coordinates of the pixel P in the repaired image are (x, y), and then a preset size is obtained, such as the size of the repaired image is N (pixels) * M (pixels), and the preset size is Then the repaired image is segmented into 9 segmented images (such as k1 to k9 in Figure 7 ) according to the preset size, and further, the pixel features of other pixels in the segmented image where the pixel P is located are determined, which can be extracted in the repaired image feature.

[0100] Further, the preset size, the pixel coordinates of each pixel in the repaired image, and the associated pixel features are input into the resolution enhancement module for image resolution enhancement processing to obtain a target image, including: obtaining a resolution magnification; inputting the resolution magnification, the preset size, the pixel coordinates of each pixel in the repaired image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image, wherein the resolution corresponding to the target image is the product of the resolution of the processed image and the resolution magnification.

[0101] The resolution magnification is set by the user according to requirements, and the cloud server acquires the resolution magnification sent by the terminal device. In the embodiment of the present application, the resolution magnification can be any magnification, for example, the resolution magnification is 1.5 times, and the resolution of the output target image is 1.5 times the resolution of the image to be processed. The resolution magnification can be set to any value according to requirements in the embodiment of the present application, so as to realize magnification of any ratio.

[0102] In addition, the related art uses preset magnifications (such as 2 times and 4 times) to simulate the magnification effect of the image to be processed. This method does not conform to the continuous perception characteristics of vision in the real world, and in actual application, if the magnification does not match the requirements, it can also cause unnecessary waste of computing resources.

[0103] Further, in the resolution enhancement module, the characteristics of each pixel in the repaired image are jointly affected by three factors: the pixel characteristics of the pixel points near the pixel, the pixel coordinates of the pixel, and the size of the segmented image. Since these pixel coordinates and the size of the segmented image are continuously changing, the resolution enhancement module can therefore associate the resolution of the output image with these continuous variables, realizing continuous resolution output.

[0104] In the embodiment of the present application, the resolution enhancement module can be trained separately. The training process is as follows: obtaining a target sample image; performing down-resolution processing on the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being the product of the resolution of the training sample images and the corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; and training the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnifications, and the target sample image.

[0105] wherein, the continuous magnifications are, for example, 1.1 times, 1.2 times, 1.3 times, 1.4 times, …, or 1.05 times, 1.1 times, 1.15 times, 1.2 times, … The continuous magnifications of the present application can be in various forms, which are not limited.

[0106] Specifically, for each training sample image, the training sample image is segmented according to a preset size to obtain a plurality of segmented sample images.

[0107] For each sample pixel in the training sample image, the pixel coordinates of the sample pixel in the training sample image are determined, and the pixel characteristics of other sample pixels in the segmented sample image where the sample pixel is located are determined as the associated pixel characteristics of the sample pixel.

[0108] For each training sample image, the pixel coordinates of each sample pixel in the repaired image, the associated pixel features of the sample pixels, and the corresponding magnification factor are determined as the input of the resolution enhancement module, and the target sample image is used as the label to train the resolution enhancement module.

[0109] For example, the resolution of the target sample image is high, such as R. The target sample image is down-sampled to obtain a plurality of low-resolution training sample images, such as R / 1.2 image, R / 1.4 image, R / 1.6 image, R / 1.8 image, R / 2 image, and R / 2.2 image. Then, the low-resolution training sample images are used as the input of the resolution enhancement module, and the target sample image is used as the label to train the resolution enhancement module. Further, the low-resolution training sample images are segmented according to the preset size, the pixel coordinates are determined, and the pixel features (i.e., convolutional features) of the sample pixels in the segmented images are determined. Then, the preset size, the pixel coordinates, the pixel features, and the magnification factor are input into the resolution enhancement module for processing to obtain a predicted image. The model parameters of the resolution enhancement module are adjusted by calculating the loss value of the predicted image and the target sample image. The resolution enhancement module trained by using training sample images with different resolutions can perform resolution magnification of any multiple.

[0110] In the embodiments of the present application, the continuous magnification training method is adopted, which can output images with target resolution in a continuous spatial range according to the requirements, and any additional down-sampling step is omitted.

[0111] Further, the resolution enhancement module can support continuous multiple image resolution enhancement for any resolution enhancement requirement. This means that the user can select any resolution magnification according to the requirements, and the resolution enhancement module will correspondingly enhance the resolution of the image, and is no longer limited to the preset resolution magnification (such as 4 times and 2 times). The resolution magnification method provided by the present application can maintain the details and clarity of the image, and is particularly useful for repairing old images and low-resolution images.

[0112] S408, the target image features and the first fusion features are fused to obtain third fusion features.

[0113] The first fusion features fused with the target image features have the same dimension as the target image features, for example, referring to Figure 5 The dimension of the first fusion features F1 is N*M, and the dimension of the target image features is also N*M. The first fusion features are fused with the target image features and the first fusion features F1 to obtain third fusion features.

[0114] In the embodiments of the present application, the target image is obtained by processing the first fused feature and the target image feature, which can improve the quality of the target image.

[0115] S409, processing the third fused feature through the network reconstruction layer to obtain the target image.

[0116] The network reconstruction layer adopts a deconvolution network, and the deconvolution network can be used to deconvolve the third fused feature to obtain the target image.

[0117] In the embodiments of the present application, the feature extraction module, the reference image retrieval module, the coloring module, the inpainting module and the resolution enhancement module can also be uniformly trained. Specifically, the training process is as follows: a fourth sample image and a label image of the fourth sample image are obtained. The fourth sample image is a low-resolution, damaged gray image. The label image of the fourth sample image is a high-resolution, undamaged color image. The fourth sample image is taken as the input of the image processing model, and the prediction image is output according to the process of the reference image retrieval module, the coloring module, the inpainting module and the resolution enhancement module. The loss value of the prediction image and the label image is determined, and the model parameters of the image processing model are adjusted according to the loss value. Figure 5

[0118] In summary, the present application can learn and optimize the image coloring, image inpainting and resolution enhancement processing as a unified task. Such an end-to-end method can better understand the internal relationship between different tasks, cooperatively process multiple aspects of the image to be processed, and thus improve the overall image processing effect compared with the serial processing method. In addition, in the existing serial processing, the errors of the previous processing steps may accumulate and affect the subsequent steps, while the end-to-end method can reduce such problems to some extent and enhance the overall effect.

[0119] Further, the present application combines retrieval technology to obtain reference images similar to the content of the image to be processed through retrieval. These reference images can provide additional guidance information for image coloring, image inpainting and resolution enhancement. Based on the reference images, the present application can more accurately infer the color, texture and other features of the region to be processed, thereby improving the stability of image processing and making the generated content more natural and harmonious in vision.

[0120] ​In addition, the application also internally has a cross-attention unit in a pyramid structure, which is designed to effectively capture and integrate information between the reference image and the image to be processed, capture different details of the image in a multi-scale manner, so that the image processing model can not only pay attention to local features, but also consider the global context when processing, thereby better understanding the correlation between the reference image and the image to be processed. At the same time, this cross-attention mechanism makes the decision-making process of the model more interpretable, making it easier to understand why the image processing model makes a particular processing decision.

[0121] Finally, the resolution enhancement module provided by the application can realize image enlargement with any magnification. Unlike the traditional resolution enhancement module which can only process a fixed magnification, the resolution enhancement module provided by the application can enlarge the image in continuous space, providing flexibility for different needs of enlargement processing. In addition, the design of the resolution enhancement module provided by the application also considers the computational efficiency, so that it can reduce the consumption of computing resources while ensuring the effect in actual application.

[0122] The application also provides an image processing method applied to a cloud server, which comprises the following steps: receiving an image to be processed sent by a terminal device, inputting the image to be processed into a feature extraction module for feature extraction to obtain intermediate image features; inputting the intermediate image features into a coloring module for coloring processing to obtain colored image features; fusing the colored image features and the intermediate image features to obtain first fused features; inputting the first fused features into a repair module for image repair to obtain repaired image features; inputting the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image, and sending the target image to the terminal device.

[0123] The specific implementation process of this embodiment is referred to the above-mentioned embodiments, which will not be described here.

[0124] In the embodiments of the application, in addition to providing an image processing method, an image processing device is also provided, as shown in Figure 8 The image processing device 80 comprises an extraction module 81, a processing module 82, a fusion module 83, an image repair module 84 and an enhancement module 85, wherein:

[0125] The extraction module 81 is configured to acquire an image to be processed, and input the image to be processed into a feature extraction module for feature extraction to obtain intermediate image features.

[0126] The processing module 82 is configured to input the intermediate image features into a coloring module for coloring processing to obtain colored image features.

[0127] The fusion module 83 is configured to fuse the colored image features and the intermediate image features to obtain first fused features.

[0128] the image restoration module 84 is configured to input the first fused feature into the image restoration module for image restoration to obtain a restored image feature;

[0129] the enhancement module 85 is configured to input the restored image feature into the image resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0130] In an optional embodiment, the extraction module 81 is specifically configured to: acquire a reference image of the image to be processed, the similarity of the reference image to the content of the image to be processed being greater than a preset threshold; and input the image to be processed and the reference image into the feature extraction module for feature extraction to obtain an intermediate image feature.

[0131] In an optional embodiment, the feature extraction module includes a convolution unit and a cross-attention unit, and when the extraction module 81 inputs the image to be processed and the reference image into the feature extraction module for feature extraction to obtain the intermediate image feature, the extraction module 81 is specifically configured to: input the image to be processed into the convolution unit for feature extraction to obtain a first image feature; input the reference image into the convolution unit for feature extraction to obtain a second image feature; and input the first image feature and the second image feature into the cross-attention unit for attention calculation to obtain the intermediate image feature.

[0132] In an optional embodiment, the cross-attention unit includes a plurality of down-sampling layers and a cross-attention layer, and when the extraction module 81 inputs the first image feature and the second image feature into the cross-attention unit for attention calculation to obtain the intermediate image feature, the extraction module 81 is specifically configured to: input the first image feature into the plurality of down-sampling layers for down-sampling to obtain, for each down-sampling layer, one first down-sampled image feature output by the down-sampling layer; input the second image feature into the plurality of down-sampling layers for down-sampling to obtain, for each down-sampling layer, one second down-sampled image feature output by the down-sampling layer; for the first down-sampled image feature and the second down-sampled image feature obtained by the same layer down-sampling layer, input the first down-sampled image feature and the second down-sampled image feature into the cross-attention layer for attention calculation to obtain a reference feature for the same layer down-sampling layer; fuse the reference feature and the first down-sampled image feature to obtain a second fused feature; and splice a plurality of second fused features corresponding to the plurality of down-sampling layers to obtain the intermediate image feature.

[0133] In an optional embodiment, the convolution unit includes a plurality of convolution layers, each convolution layer outputting one first image feature and one second image feature, and when the extraction module 81 inputs the first image feature and the second image feature into the cross-attention unit for attention calculation to obtain the intermediate image feature, the extraction module 81 is specifically configured to: input the first image feature and the second image feature output by the last convolution layer into the cross-attention unit for attention calculation to obtain the intermediate image feature.

[0134] In an optional embodiment, the coloring module comprises a conversion layer and a plurality of deconvolution layers, the number of the plurality of deconvolution layers is the same as the number of the plurality of convolution layers, and the processing module 82 is specifically configured to: input the intermediate image features into the conversion layer for color conversion processing to obtain converted image features; and input the converted image features into the plurality of deconvolution layers to obtain coloring image features output by the deconvolution layers, wherein the deconvolution layers correspond one-to-one to the coloring image features.

[0135] In an optional embodiment, the fusion module 83 is specifically configured to: fuse the first image features output by the corresponding convolution layers, the coloring image features output by the deconvolution layers, and the intermediate image features to obtain first fusion features, wherein the number of the first fusion features is the same as the number of the plurality of convolution layers, and the dimensions of the first image features and the coloring image features that are fused are the same.

[0136] In an optional embodiment, the enhancement module 85 is specifically configured to: determine pixel coordinates of pixels corresponding to the repair image features in the segmented image of the preset size; and input the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0137] In an optional embodiment, when the enhancement module 85 inputs the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image, the enhancement module 85 is specifically configured to: obtain a resolution magnification; and input the resolution magnification, the preset size, the pixel coordinates of the pixels in the repair image, and the associated pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image, wherein the resolution corresponding to the target image is the product of the resolution of the image to be processed and the resolution magnification.

[0138] In an optional embodiment, the system further comprises a training module (not shown), which is configured to train the resolution enhancement module in the following manner: obtaining a target sample image; performing down-resolution processing on the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being the product of the resolution of the training sample images and a corresponding magnification, wherein different training sample images correspond to different magnifications, and the plurality of magnifications of the plurality of training sample images are continuous; and training the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnifications, and the target sample image.

[0139] In an optional embodiment, the resolution enhancement module includes a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer, and the enhancement module 85 is specifically configured to: perform resolution enhancement processing on the repaired image features through the feature extraction layer and the nonlinear mapping layer to obtain target image features; perform feature fusion on the target image features and the first fused features to obtain third fused features; and perform processing on the third fused features through the network reconstruction layer to obtain a target image.

[0140] The implementation process is specifically described in the above embodiments, which will not be repeated here.

[0141] In addition, some of the processes described in the above embodiments and the accompanying drawings include a plurality of operations that occur in a specific order, but it should be clear that these operations can be executed in the order in which they appear in this document or in parallel, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. "First" and "second" are not of different types.

[0142] Figure 9 A structural schematic diagram of an electronic device is provided for the exemplary embodiments of the present application. The electronic device 90 is configured to run the image processing method and the image processing method described above. As shown in the figure, Figure 9 The electronic device includes a memory 94 and a processor 95.

[0143] The memory 94 is configured to store computer programs and can be configured to store various data to support operations on the electronic device. The memory 94 can be an object storage service (OSS).

[0144] The memory 94 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0145] The processor 95, coupled with the memory 94, is configured to execute a computer program in the memory 94, so as to: acquire a to-be-processed image, input the to-be-processed image into a feature extraction module for feature extraction to obtain intermediate image features; input the intermediate image features into a coloring module for coloring processing to obtain colored image features; fuse the colored image features and the intermediate image features to obtain first fused features; input the first fused features into a repairing module for image repairing to obtain repaired image features; and input the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain a target image.

[0146] In an optional embodiment, when the processor 95 inputs the to-be-processed image into the feature extraction module for feature extraction to obtain the intermediate image features, the processor 95 is specifically configured to: acquire a reference image of the to-be-processed image, the content similarity between the reference image and the to-be-processed image being greater than a preset threshold; and input the to-be-processed image and the reference image into the feature extraction module for feature extraction to obtain the intermediate image features.

[0147] In an optional embodiment, the feature extraction module includes a convolution unit and a cross-attention unit, and when the processor 95 inputs the to-be-processed image and the reference image into the feature extraction module for feature extraction to obtain the intermediate image features, the processor 95 is specifically configured to: input the to-be-processed image into the convolution unit for feature extraction to obtain first image features; input the reference image into the convolution unit for feature extraction to obtain second image features; and input the first image features and the second image features into the cross-attention unit for attention calculation to obtain the intermediate image features.

[0148] In an optional embodiment, the cross-attention unit includes a plurality of down-sampling layers and a cross-attention layer, and when the processor 85 inputs the first image features and the second image features into the cross-attention unit for attention calculation to obtain the intermediate image features, the processor 85 is specifically configured to: input the first image features into the plurality of down-sampling layers for down-sampling to obtain one first down-sampled image feature corresponding to the output of each down-sampling layer; input the second image features into the plurality of down-sampling layers for down-sampling to obtain one second down-sampled image feature corresponding to the output of each down-sampling layer; for the first down-sampled image features and the second down-sampled image features obtained by the same layer down-sampling layer, input the first down-sampled image features and the second down-sampled image features into the cross-attention layer for attention calculation to obtain a reference feature for the same layer down-sampling layer; fuse the reference feature and the first down-sampled image features to obtain second fused features; and splice a plurality of second fused features corresponding to the plurality of down-sampling layers to obtain the intermediate image features.

[0149] In an optional embodiment, the convolution unit comprises a plurality of convolution layers, each convolution layer outputs a corresponding first image feature, and each convolution layer outputs a corresponding second image feature; when the processor 95 inputs the first image features and the second image features into the cross-attention unit to perform attention calculation to obtain intermediate image features, the processor 95 is specifically configured to: input the first image features and the second image features output by the last convolution layer into the cross-attention unit to perform attention calculation, to obtain the intermediate image features.

[0150] In an optional embodiment, the coloring module comprises a conversion layer and a plurality of deconvolution layers, the number of the plurality of deconvolution layers is the same as the number of the plurality of convolution layers, and when the processor 95 inputs the intermediate image features into the coloring module to perform coloring processing to obtain colored image features, the processor 95 is specifically configured to: input the intermediate image features into the conversion layer to perform color conversion processing to obtain converted image features; and input the converted image features into the plurality of deconvolution layers to obtain the colored image features output by the deconvolution layers, wherein the deconvolution layers correspond one-to-one to the colored image features.

[0151] In an optional embodiment, when the processor 95 fuses the colored image features and the intermediate image features to obtain first fusion features, the processor 95 is specifically configured to: fuse the first image features output by the corresponding convolution layers, the colored image features output by the deconvolution layers, and the intermediate image features to obtain the first fusion features, wherein the number of the first fusion features is the same as the number of the plurality of convolution layers, and the dimensions of the first image features and the colored image features that are fused are the same.

[0152] In an optional embodiment, when the processor 95 inputs the repaired image features into the resolution enhancement module to perform image resolution enhancement processing to obtain a target image, the processor 95 is specifically configured to: determine a repaired image according to the repaired image features; split the repaired image according to a preset size to obtain a plurality of split images; for each pixel in the repaired image, determine a pixel coordinate of the pixel in the repaired image, and determine pixel features of other pixels in the split image in which the pixel is located as associated pixel features of the pixel; and input the preset size, the pixel coordinates of the pixels in the repaired image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain the target image.

[0153] In an optional embodiment, when the processor 95 inputs the preset size, the pixel coordinates of the pixels in the repaired image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain a target image, the processor 95 is specifically configured to: obtain a resolution magnification; and input the resolution magnification, the preset size, the pixel coordinates of the pixels in the repaired image, and the associated pixel features into the resolution enhancement module to perform image resolution enhancement processing to obtain the target image, wherein a resolution corresponding to the target image is a product of a resolution of the to-be-processed image and the resolution magnification.

[0154] In an optional embodiment, the processor 95 is further configured to train the resolution enhancement module in the following manner: obtaining a target sample image; performing down-resolution processing on the target sample image to obtain a plurality of training sample images, the resolution of the target sample image being a product of the resolution of the training sample images and a corresponding magnification factor, wherein the different training sample images correspond to different magnification factors, and the plurality of magnification factors of the plurality of training sample images are continuous; and training the resolution enhancement module using the plurality of training sample images, the continuous plurality of magnification factors, and the target sample image.

[0155] In an optional embodiment, the resolution enhancement module comprises a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer, and when the processor 95 inputs the repaired image feature into the resolution enhancement module to perform image resolution enhancement processing to obtain the target image, the processor 95 is specifically configured to: perform resolution enhancement processing on the repaired image feature through the feature extraction layer and the nonlinear mapping layer to obtain a target image feature; perform feature fusion on the target image feature and the first fused feature to obtain a third fused feature; and perform processing on the third fused feature through the network reconstruction layer to obtain the target image.

[0156] Further, as shown in Figure 9 the electronic device further comprises a firewall 91, a load balancer 92, a communication component 96, a power supply component 93, and other components. Figure 9 only some components are shown schematically, and it does not mean that the electronic device only includes Figure 9 the components shown in the figure.

[0157] Correspondingly, the embodiments of the present application also provide a computer program product storing a computer program, when the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the above-mentioned method.

[0158] Correspondingly, the embodiments of the present application also provide a computer program product, comprising a computer program / instruction, when the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the above-mentioned method.

[0159] The above Figure 9The communication component in the electronic device 100 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices. The electronic device 100 can access a wireless network based on a communication standard, such as WiFi, a 2G, 3G, 4G / LTE, 5G, or the like cellular communication network, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0160] The power component in the electronic device 100 provides power to various components of the electronic device 100. The power component can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 100. Figure 9

[0161] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one example, the present application can be implemented in software and / or firmware. In one embodiment, the present application can be implemented in hardware using a combination of logic gates, logic circuits, look up tables, or other hardware components. In one embodiment, the present application can be implemented by machine code stored in a computer-readable medium.

[0162] The present application is described in relation to flowcharts and / or block diagrams that illustrate the architecture, functionality, and operation of embodiments of methods, apparatus (systems) and computer program products according to the present application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagrams in which functions are described can also be implemented as computer readable code on a computer readable medium. The computer readable medium is a computer program product. The computer readable medium can be a computer storage medium. The computer storage medium can be any medium that can be read by a computer. Examples of computer storage media include magnetic disks, magnetic tapes, CD-ROMs, DVDs, Blu-Ray Discs®, RAM, ROM, EEPROM, and Figure 1 The flowchart and / or block diagrams in which functions are described can also be implemented as computer readable code on a computer readable medium. The computer readable medium is a computer program product. The computer readable medium can be a computer storage medium. The computer storage medium can be any medium that can be read by a computer. Examples of computer storage media include magnetic disks, magnetic tapes, CD-ROMs, DVDs, Blu-Ray Discs®, RAM, ROM, EEPROM, and

[0163] These computer program instructions can also be stored in a computer readable medium that can direct a computer, a programmable data processing apparatus, or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagrams in which functions are described can also be implemented as computer readable code on a computer readable medium. The computer readable medium is a computer program product. The computer readable medium can be a computer storage medium. The computer storage medium can be any medium that can be read by a computer. Examples of computer storage media include magnetic disks, magnetic tapes, CD-ROMs, DVDs, Blu-Ray Discs®, RAM, ROM, EEPROM, and Figure 1 ​the function specified in the one or more blocks.

[0164] These computer program instructions can also be loaded onto a computer or other programmable text processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the operations specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 the steps of a function specified in the one or more blocks.

[0165] In one typical configuration, the computing device includes one or more processors, such as a central processing unit (CPU) and / or a graphics processing unit (GPU), an input / output interface, a network interface, and a memory.

[0166] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or random access memory (RAM), among others. The memory is an example of computer readable media.

[0167] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0169] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. An image processing method, characterized in that, include: The image to be processed is acquired and input into the feature extraction module for feature extraction to obtain intermediate image features; The intermediate image features are input into the coloring module for coloring processing to obtain the colorized image features; The first fused feature is obtained by fusing the colored image features and the intermediate image features; The first fused feature is input into the repair module for image repair to obtain the repaired image features; The repaired image features are input into the resolution enhancement module for image resolution enhancement processing to obtain the target image.

2. The image processing method according to claim 1, characterized in that, The step of inputting the image to be processed into the feature extraction module for feature extraction to obtain intermediate image features includes: Obtain a reference image of the image to be processed, wherein the content similarity between the reference image and the image to be processed is greater than a preset threshold; The image to be processed and the reference image are input into the feature extraction module for feature extraction to obtain the intermediate image features.

3. The image processing method according to claim 2, characterized in that, The feature extraction module includes a convolutional unit and a cross-attention unit. The step of inputting the image to be processed and the reference image into the feature extraction module for feature extraction to obtain the intermediate image features includes: The image to be processed is input into the convolutional unit for feature extraction to obtain the first image features; The reference image is input into the convolutional unit for feature extraction to obtain the second image features; The first image feature and the second image feature are input into the cross-attention unit for attention calculation to obtain the intermediate image feature.

4. The image processing method according to claim 3, characterized in that, The cross-attention unit includes: multiple downsampling layers and cross-attention layers; the step of inputting the first image features and the second image features into the cross-attention unit for attention calculation to obtain the intermediate image features includes: The first image feature is input into the plurality of downsampling layers for downsampling, to obtain a first downsampled image feature output by each downsampling layer; The second image feature is input into the plurality of downsampling layers for downsampling, to obtain a second downsampled image feature output by each downsampling layer; For the first and second downsampled image features obtained from the same downsampled layer, the first and second downsampled image features are input into the cross attention layer for attention calculation to obtain reference features for the same downsampled layer. The reference feature and the first downsampled image feature are fused to obtain the second fused feature; The intermediate image features are obtained by concatenating the multiple second fusion features corresponding to the multiple downsampling layers.

5. The image processing method according to claim 3 or 4, characterized in that, The convolutional unit includes multiple convolutional layers, each outputting a corresponding first image feature and a corresponding second image feature; the step of inputting the first image feature and the second image feature into the cross-attention unit for attention calculation to obtain the intermediate image feature includes: The first and second image features output from the last convolutional layer are input into the cross-attention unit for attention calculation to obtain the intermediate image features.

6. The image processing method according to claim 5, characterized in that, The coloring module includes a transformation layer and multiple deconvolutional layers, wherein the number of deconvolutional layers is the same as the number of convolutional layers. The step of inputting the intermediate image features into the coloring module for coloring processing to obtain colored image features includes: The intermediate image features are input into the conversion layer for color conversion processing to obtain the converted image features; The transformed image features are input into the plurality of deconvolution layers to obtain the colorized image features output by the deconvolution layers, wherein the deconvolution layers correspond one-to-one with the colorized image features.

7. The image processing method according to claim 6, characterized in that, The process of fusing the colored image features and the intermediate image features to obtain the first fused feature includes: The first fused feature is obtained by fusing the first image feature output from the corresponding convolutional layer, the colorized image feature output from the deconvolutional layer, and the intermediate image feature. The number of the first fused features is the same as the number of the multiple convolutional layers, and the dimensions of the first image feature and the colorized image feature being fused are the same.

8. The image processing method according to claim 1, characterized in that, The step of inputting the repaired image features into the resolution enhancement module for image resolution enhancement processing to obtain the target image includes: The repaired image is determined based on the features of the repaired image; The repaired image is segmented according to a preset size to obtain multiple segmented images; For each pixel in the repaired image, determine the pixel coordinates of the pixel in the repaired image, and determine the pixel features of other pixels in the segmented image where the pixel is located as the associated pixel features of the pixel; The preset size, the pixel coordinates of each pixel in the repaired image, and the associated pixel features are input into the resolution enhancement module to perform image resolution enhancement processing, thereby obtaining the target image.

9. The image processing method according to claim 8, characterized in that, The step of inputting the pixel coordinates, the preset size, and the pixel features into the resolution enhancement module for image resolution enhancement processing to obtain the target image includes: Get the resolution magnification factor; The resolution magnification factor, the preset size, the pixel coordinates of each pixel in the repaired image, and the associated pixel features are input into the resolution enhancement module for image resolution enhancement processing to obtain the target image. The resolution of the target image is the product of the resolution of the image to be processed and the resolution magnification factor.

10. The image processing method according to claim 9, characterized in that, The resolution enhancement module is trained in the following manner: Acquire the target sample image; The target sample image is down-resolution processed to obtain multiple training sample images. The resolution of the target sample image is the product of the resolution of the training sample image and the corresponding magnification factor. Different training sample images correspond to different magnification factors, and the multiple magnification factors of the multiple training sample images are consecutive. The resolution enhancement module is trained using the multiple training sample images, multiple consecutive magnifications, and the target sample image.

11. The image processing method according to any one of claims 1 to 4, characterized in that, The resolution enhancement module includes a feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer. The step of inputting the restored image features into the resolution enhancement module for image resolution enhancement processing to obtain the target image includes: The repaired image features are processed through the feature extraction layer and the nonlinear mapping layer to obtain the target image features; The target image features and the first fusion feature are fused to obtain the third fusion feature; The third fusion feature is processed through the network reconstruction layer to obtain the target image.

12. An image processing method, characterized in that, Applied to cloud servers, the image processing method includes: The system receives an image to be processed sent by a terminal device and inputs the image to be processed into a feature extraction module for feature extraction to obtain intermediate image features. The intermediate image features are input into the coloring module for coloring processing to obtain the colorized image features; The first fused feature is obtained by fusing the colored image features and the intermediate image features; The first fused feature is input into the repair module for image repair to obtain the repaired image features; The repaired image features are input into the resolution enhancement module for image resolution enhancement processing to obtain the target image, and the target image is sent to the terminal device.

13. An image processing apparatus, characterized in that, include: The extraction module is used to acquire the image to be processed and input the image to be processed into the feature extraction module for feature extraction to obtain intermediate image features; The processing module is used to input the intermediate image features into the coloring module for coloring processing to obtain the colorized image features; The fusion module is used to fuse the colorized image features and the intermediate image features to obtain a first fused feature; The image restoration module is used to input the first fused features into the restoration module for image restoration to obtain restored image features; An enhancement module is used to input the repaired image features into a resolution enhancement module for image resolution enhancement processing to obtain the target image.

14. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the image processing method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, The computer program product stores a computer program that, when executed by a processor, causes the processor to implement the image processing method as described in any one of claims 1 to 12.

Images