Image processing method, system, equipment and medium

Abstract

An image processing method, system, device and medium, comprising: acquiring an image to be processed, and inputting the image into a preset super-resolution processing model for super-resolution processing to obtain a super-resolution processing result; the super-resolution processing at least comprises the following steps: acquiring a to-be-processed image and inputting the to-be-processed image into an image encoder to obtain a latent code corresponding to the to-be-processed image; obtaining a description text corresponding to the to-be-processed image and inputting the description text into a text encoder to obtain text features; inputting the to-be-processed image into the edge extractor to obtain edge features; inputting the to-be-processed image into the semantic extractor to obtain semantic features; inputting the latent code into a UNet de-noising device, and carrying out iterative diffusion de-noising to obtain a de-noised latent code; and fusing the denoised latent code and the edge feature, and inputting the fused image into an image decoder to output a super-resolution processing result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, system, device and medium. Background Art

[0002] As an important way to preserve text, the clarity of text images is crucial for the accurate dissemination of information. However, over time, text images often suffer from multiple complex degradations, causing the image to become blurred, affecting the readability of the text and the efficiency of information transmission.

[0003] To solve this problem, people usually use manual image editing software (such as Photoshop, PS for short) to adjust and clarify the text image to improve its readability. Although this method can improve the image quality to a certain extent, it is inefficient and depends on the operator's technical level, making it difficult to apply on a large scale.

[0004] In addition, existing image processing methods have problems with clarity and realism, and are unable to meet users' demands for high-quality images. Summary of the invention

[0005] The purpose of the present invention is to provide an image processing method, system, device and medium, aiming to solve the technical problems of poor clarity and realism in existing image processing methods.

[0006] To achieve the above object, the present invention provides an image processing method, comprising:

[0007] Obtain an image to be processed, and input it into a preset super-resolution processing model for super-resolution processing to obtain a super-resolution processing result; the image to be processed contains text, and the preset super-resolution processing model includes an image encoder, a UNet denoiser, a text encoder, an edge extractor, a semantic extractor and an image decoder;

[0008] The super-resolution processing includes at least the following steps: obtaining an image to be processed and inputting it into an image encoder to obtain a latent code corresponding to the image to be processed;

[0009] Obtain the description text corresponding to the image to be processed and input it into the text encoder to obtain text features;

[0010] Input the image to be processed into the edge extractor to obtain edge features;

[0011] Input the image to be processed into the semantic extractor to obtain semantic features;

[0012] The text features, edge features, and semantic features are fused with the UNet denoiser through the cross-attention mechanism to guide the denoising process;

[0013] The latent code is input into the UNet denoiser and iterative diffusion denoising is performed to obtain the denoised latent code;

[0014] The denoised latent code and edge features are fused and input into the image decoder to output the super-resolution processing result.

[0015] Optionally, the preset super-resolution processing model is obtained through pre-training, and the pre-training at least includes the following steps:

[0016] Acquire image data including text and perform preprocessing to obtain paired data including target image data, original image data, and text description corresponding to the target image data;

[0017] Construct the loss function of the network, which includes noise loss, L1 loss, LPIPS loss and edge loss;

[0018] Perform end-to-end joint training on the network structure and use the constructed loss function to calculate the corresponding loss value;

[0019] The network parameters are adjusted through iterative optimization to obtain a super-resolution processing model.

[0020] Optionally, the preprocessing includes at least the following steps:

[0021] Using image data including text as target image data;

[0022] The target image data is downsampled and used as the original image data;

[0023] Get the text description corresponding to the target image data based on BLIP.

[0024] Optionally, the noise loss is used to calculate the L2 loss between the UNet denoiser output noise and the original preset noise;

[0025] L1 loss is used to calculate the L1 loss of original image data and model output data;

[0026] LPIPS loss uses the pre-trained VGG network to calculate the L2 loss between the features corresponding to the target image data and the features corresponding to the model output data;

[0027] Edge loss is used to calculate the L2 loss between the edge features corresponding to the target image data and the edge features corresponding to the model output data.

[0028] Optionally, the text includes at least one of common text, numbers, and special symbols in different countries or regions.

[0029] Optionally, the feature fusion of the i-th intermediate layer of UNet is specifically as follows: weighting the edge features and the semantic features respectively through corresponding attention mechanisms;

[0030] Calculate the sum of the output of the attention mechanism for text features, the output of the weighted attention mechanism for edge features, and the output of the weighted attention mechanism for semantic features.

[0031] Optionally, the image encoder is constructed using the encoder structure of Stable Diffusion, the text encoder is constructed using the BLIP model, the edge extractor is constructed using the Canny edge extraction operator, and the semantic extractor is constructed using the TransOCR model.

[0032] Corresponding to the image processing method, the present invention provides an image processing system, which includes:

[0033] The image encoder is used to obtain the image to be processed, output the latent code corresponding to the image to be processed and input it into the UNet denoiser. The image to be processed contains text.

[0034] A text encoder is used to obtain the description text corresponding to the image to be processed and output text features;

[0035] An edge extractor is used to obtain an image to be processed and output edge features corresponding to the image to be processed;

[0036] A semantic extractor, used to obtain an image to be processed and output semantic features corresponding to the image to be processed;

[0037] UNet denoiser is used to obtain latent codes, perform iterative diffusion denoising, and output denoised latent codes. Text features, edge features, and semantic features are fused with the UNet denoiser through a cross-attention mechanism to guide the denoising process.

[0038] Image decoder, used to fuse denoising latent code and edge features and output super-resolution processing results.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides an image processing device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the image processing method as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an image processing program is stored, and when the image processing program is executed by a processor, the steps of the image processing method described above are implemented.

[0041] The beneficial effects of the present invention are:

[0042] (1) Compared with the prior art, the present invention effectively improves the clarity and realism of text images by combining an image encoder, a UNet denoiser, a text encoder, an edge extractor, a semantic extractor and an image decoder; compared with manual image editing methods, the present method realizes automated processing, improves processing efficiency, reduces dependence on the operator's technical level, and makes large-scale application possible; in addition, compared with traditional image processing methods, CNN-based deep learning methods and GAN-based methods, the present method better retains the detailed information of text images through the training of diffusion models, solves the problems of limited clarity improvement and insufficient realism in the prior art, and meets the user's demand for high-quality images; in addition, by introducing a cross-attention mechanism, text features, edge features and semantic features can be more effectively fused with the UNet denoiser; the model's understanding of the structure and content of text in the image is improved, thereby better maintaining the clarity and readability of the text during the denoising process;

[0043] (2) Compared with the prior art, the present invention further improves the training effect of the model by designing a comprehensive loss function including noise loss, L1 loss, LPIPS loss and edge loss; the model can simultaneously focus on noise reduction, image detail preservation, feature space consistency and edge information accuracy during training, thereby improving the overall quality of the image after super-resolution processing;

[0044] (3) Compared with the prior art, the present invention provides rich training data for the model through preprocessing steps, including acquisition of target image data, generation of original image data and acquisition of text description; enhances the adaptability and generalization ability of the model to text images under different degradation conditions, and improves the accuracy and robustness of super-resolution processing;

[0045] (4) Compared with the prior art, the present invention defines the calculation methods of noise loss, L1 loss, LPIPS loss and edge loss, making the loss calculation in the model training process more accurate and comprehensive; it helps the model to better learn how to reduce noise, maintain image details, maintain feature space consistency and the accuracy of edge information, and further improve the quality of super-resolution images;

[0046] (5) Compared with the prior art, the present invention improves the model's feature extraction capability and image generation quality by using advanced models and algorithms such as Stable Diffusion, BLIP, Canny edge extraction operator and TransOCR to build a network structure; enables the model to more accurately capture and restore the details of text images, and improves the effect of super-resolution processing;

[0047] (6) Compared with the prior art, the present invention constructs a complete image processing system, which effectively improves the clarity and realism of text images and realizes the full-process automated processing from image acquisition to super-resolution output; it improves processing efficiency and reduces operational complexity, making the application of super-resolution technology more extensive and convenient.

[0048] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0050] In the figure:

[0051] Figure 1 The figure is a flow chart of an embodiment of an image processing method of the present invention.

[0052] Figure 2 A framework diagram of an image processing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product or device that includes a series of steps or units.

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0058] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0059] See also Figure 1 As shown, an image processing method provided by an embodiment of the present invention includes:

[0060] Obtain an image to be processed, and input it into a preset super-resolution processing model for super-resolution processing to obtain a super-resolution processing result; the image to be processed contains text, and the preset super-resolution processing model includes an image encoder, a UNet denoiser, a text encoder, an edge extractor, a semantic extractor and an image decoder;

[0061] The super-resolution processing includes at least the following steps: obtaining an image to be processed and inputting it into an image encoder to obtain a latent code corresponding to the image to be processed;

[0062] Obtain the description text corresponding to the image to be processed and input it into the text encoder to obtain text features;

[0063] Input the image to be processed into the edge extractor to obtain edge features;

[0064] Input the image to be processed into the semantic extractor to obtain semantic features;

[0065] The text features, edge features, and semantic features are fused with the UNet denoiser through the cross-attention mechanism to guide the denoising process;

[0066] The latent code is input into the UNet denoiser and iterative diffusion denoising is performed to obtain the denoised latent code;

[0067] The denoised latent code and edge features are fused and input into the image decoder to output the super-resolution processing result.

[0068] It is understandable that the characters include at least one of the common characters, numbers, and special symbols of different countries or regions.

[0069] The present invention effectively improves the clarity and realism of text images by combining an image encoder, a UNet denoiser, a text encoder, an edge extractor, a semantic extractor and an image decoder; compared with the manual image editing method, the present method realizes automated processing, improves processing efficiency, reduces dependence on the operator's technical level, and makes large-scale application possible; in addition, compared with traditional image processing methods, deep learning methods based on CNN (Convolutional Neural Network) and methods based on GAN (Generative Adversarial Network), the present method better retains the detailed information of text images through the training of diffusion models, solves the problems of limited clarity improvement and insufficient realism in the prior art, and meets the user's demand for high-quality images. In addition, by introducing a cross-attention mechanism, text features, edge features and semantic features can be more effectively fused with the UNet denoiser; the model's understanding of the text structure and content in the image is improved, thereby better maintaining the clarity and readability of the text during the denoising process.

[0070] In this embodiment, the preset super-resolution processing model is obtained through pre-training, and the pre-training includes at least the following steps:

[0071] Acquire image data including text and perform preprocessing to obtain paired data including target image data, original image data, and text description corresponding to the target image data;

[0072] Construct the loss function of the network, which includes noise loss, L1 loss, LPIPS (Learned Perceptual Image Patch Similarity) loss and edge loss;

[0073] Perform end-to-end joint training on the network structure and use the constructed loss function to calculate the corresponding loss value;

[0074] The network parameters are adjusted through iterative optimization to obtain a super-resolution processing model.

[0075] The present invention can better train the learning ability of the network model by iteratively optimizing and adjusting the network parameters, and generate a super-resolution processing result (i.e., a high-resolution text image) with a better super-resolution effect.

[0076] The present invention further improves the training effect of the model by designing a comprehensive loss function including noise loss, L1 loss, LPIPS loss and edge loss; the model can simultaneously focus on noise reduction, image detail preservation, feature space consistency and edge information accuracy during training, thereby improving the overall quality of the image after super-resolution processing.

[0077] In this embodiment, the preprocessing includes at least:

[0078] Using image data including text as target image data;

[0079] After downsampling the target image data, the formula can be expressed as the original image data:

[0080] X LR = Downsample(X HR )

[0081] Among them, X LR It is the low-resolution image data obtained after the target image data is downsampled. Downsample represents the downsampling operation;

[0082] The text description corresponding to the target image data is obtained based on the BLIP (Bootstrapping Language-Image Pre-training) model.

[0083] Furthermore, the original image data and the target image data can also be randomly flipped, translated, rotated, scaled and other geometric transformations to improve the availability of image data during network training.

[0084] The present invention provides rich training data for the model through preprocessing steps, including the acquisition of target image data, the generation of original image data and the acquisition of text description; enhances the adaptability and generalization ability of the model to text images under different degradation conditions, and improves the accuracy and robustness of super-resolution processing.

[0085] In this embodiment, the noise loss is used to calculate the L2 loss between the output noise of the UNet denoiser and the original preset noise;

[0086] L1 loss is used to calculate the L1 loss of original image data and model output data;

[0087] LPIPS loss uses the pre-trained VGG (Visual Geometry Group) network to calculate the L2 loss between the features corresponding to the target image data and the features corresponding to the model output data;

[0088] Edge loss is used to calculate the L2 loss between the edge features corresponding to the target image data and the edge features corresponding to the model output data.

[0089] In this embodiment, the noise loss L noise The calculation formula is as follows:

[0090]

[0091] Among them, t represents the time step, ∈ represents the preset noise, ∈ θ (·) represents the noise predicted by the UNet denoiser, Z t is the latent code at the current step length t, X LR is the image to be processed, C text is the text feature, X canny is the edge feature of the image to be processed, X OCR is the semantic feature of the image to be processed;

[0092] The calculation formula of L1 loss is as follows:

[0093]

[0094] Among them, X HR is the original image data, Output data for the model;

[0095] LPIPS loss L LPIPS The calculation formula is as follows:

[0096]

[0097] Among them, Φ represents the feature extraction operation of the VGG network, Φ(X HR ) is the corresponding feature of the target image data, Output data corresponding to the model features;

[0098] Edge loss L canny The calculation formula is as follows:

[0099]

[0100] Among them, Canny(X HR ) is the edge feature corresponding to the target image data, The output data of the model corresponds to the edge features.

[0101] Therefore, the overall loss function L of the network can be expressed as:

[0102] L=L noise +L1+L LPIPS +L canny .

[0103] The present invention defines the calculation methods of noise loss, L1 loss, LPIPS loss and edge loss, so that the loss calculation in the model training process is more accurate and comprehensive; it helps the model to better learn how to reduce noise, maintain image details, maintain feature space consistency and the accuracy of edge information, and further improve the quality of super-resolution images.

[0104] In this embodiment, the feature fusion of the i-th intermediate layer of UNet is specifically as follows: weighting the edge features and semantic features respectively through corresponding attention mechanisms;

[0105] Calculate the sum of the output of the attention mechanism for text features, the output of the weighted attention mechanism for edge features, and the output of the weighted attention mechanism for semantic features.

[0106] The present invention introduces a cross-attention mechanism, so that text features, edge features and semantic features can be more effectively integrated with the UNet denoiser; the model's understanding of the structure and content of text in the image is improved, thereby better maintaining the clarity and readability of the text during the denoising process.

[0107] In this embodiment, the image encoder is constructed using the Stable Diffusion encoder structure, the text encoder is constructed using the BLIP model, the edge extractor is constructed using the edge extraction operator, and the semantic extractor is constructed using the TransOCR (Transformer-based Optical Character Recognition) model.

[0108] Specifically, the image encoder is constructed using the Stable Diffusion encoder structure, which converts the image to be processed X LR Input the image encoder to obtain the latent code Z corresponding to the image to be processed LR , the formula can be expressed as:

[0109] Z LR =E(X LR )

[0110] Among them, X LR represents the image to be processed, Z LR represents the latent code extracted by the image encoder E. It can be understood that the resolution of the image to be processed is lower than that of the super-resolution processing result.

[0111] The text encoder is built using BLIP. BLIP is a powerful text-image correspondence model that can encode the input text into text features, correspond to image features, and control the generation of images. Its formula can be expressed as:

[0112] C text =CLIP(text)

[0113] Among them, text represents the description text corresponding to the image to be processed, C text Represents the text features obtained through BLIP. The text features are fused with the UNet denoiser through the cross-attention mechanism to guide the denoising process.

[0114] The edge extractor is constructed using the Canny edge extraction operator. The edge extractor is the Canny edge extractor, which is used to control the edge structure of the generated text image and enhance the sharpness of the text image. In particular, the edge is obtained through the Canny operator, and its formula can be specifically expressed as:

[0115] X cannt =Canny(X LR )

[0116] Where X canny Represents the edge features corresponding to the image to be processed (i.e., edge contour image). The edge features are fused with the UNet denoiser through the cross-attention mechanism to guide the denoising process.

[0117] The semantic extractor is constructed using the TransOCR model. The semantic extractor is an OCR (Optical Character Recognition) feature extractor, which is used to extract the OCR semantic features of text images and improve the recognizability of generated text images. Its formula can be expressed as:

[0118] X OCR =OCR(X LR )

[0119] Where X OCR Represents the semantic features of the image to be processed obtained by the semantic extractor. The OCR semantic features are fused with the UNet denoiser through the cross attention mechanism to guide the denoising process.

[0120] The UNet denoiser is constructed using UNet. UNet has a multi-layer and multi-scale network structure and is a natural denoiser, which facilitates the diffusion and iteration of noise in the latent space. In particular, the initialization noise and the latent code of the image to be processed are feature fused to retain the feature information of the image to be processed. The formula can be expressed as:

[0121] Z=Concat(Z LR ,Z noise )

[0122] Where Z noiserepresents the noise of the initialization setting, Concat represents the channel-level stacking operation, and Z represents the latent code obtained by the Concat operation. Based on the above text encoder, edge extractor and semantic extractor, the formula for diffusion iteration in the UNet latent space can be expressed as:

[0123] Z t-1 =Z t -∈ θ (Z t ,t,X LR ,C text ,X canny ,X OCR )

[0124] Among them, ∈ θ (·) represents the noise predicted by the UNet denoiser, Z t is the latent code at the current step length t, Z t-1 is the latent code at the previous step length t-1. After the iteration is completed, the latent code Z0 at the step length 0 can be obtained.

[0125] In particular, the module branches of the above-mentioned text encoder, edge extractor and semantic extractor serve as auxiliary branches of the UNet denoiser in the diffusion model. Through the cross-attention mechanism, the text features of different modalities, Canny edges, OCR semantic features and the low-resolution latent features of the UNet intermediate layer are interacted and further integrated to improve the final text super-resolution effect. The feature fusion of the i-th intermediate layer of UNet is specifically as follows: weighting the edge features and semantic features through the corresponding attention mechanisms respectively; calculating the sum of the output of the attention mechanism of the text features, the output of the weighted attention mechanism of the edge features, and the output of the weighted attention mechanism of the semantic features. Therefore, the cross-attention feature fusion formula of the i-th intermediate layer of UNet can be specifically expressed as:

[0126] Z (i) =

[0127] Atten text (Q,K text ,V text )+λ*Atten Canny (Q,K Canny ,V Canny )+

[0128] θ*Atten OCR (Q,K oCR ,V OCR );

[0129] Among them, Z (i) Atten represents the latent features obtained by fusion of the i-th intermediate layer of UNet. text (Q,K text ,Vtext ) is the attention mechanism for text features, Atten Canny (Q,K Canny ,V Canny ) is the attention mechanism for edge features, λ is the weight parameter of the attention mechanism for edge features, Atten OCR (Q,K OCR ,V OCR ) is the attention mechanism for semantic features, and θ is the weight parameter of the attention mechanism for semantic features.

[0130] In this embodiment,

[0131] and Represents the weight matrix for text features in the criss-cross attention layer.

[0132] Specifically,

[0133] and Represents the weight matrix for edge features in the criss-cross attention layer.

[0134] and Special and Represents the weight matrix for semantic features in the criss-cross attention layer.

[0135] In the feature fusion formula of the above cross-attention mechanism, it can be seen that Atten text , Atten Canny and Atten OCR The three cross attentions share Q, thereby increasing the reuse of latent features, improving processing efficiency while ensuring the interaction of multiple latent features.

[0136] Preferably, λ=0.5, θ=0.1.

[0137] The image decoder is constructed using the Stable Diffusion decoder, which can convert the edge feature X canny The denoised Z0 is input into the image decoder to obtain the final super-resolution processing result (i.e., high-resolution text image), which can be expressed as follows:

[0138]

[0139] in The super-resolution processing result obtained by decoding the image decoder D.

[0140] The present invention constructs a network structure by using advanced models and algorithms such as Stable Diffusion, BLIP, edge extraction operators and TransOCR, thereby improving the model's feature extraction capability and image generation quality; enabling the model to more accurately capture and restore the details of text images, thereby improving the effect of super-resolution processing.

[0141] like Figure 2 As shown, the present invention also provides an image processing system, which includes:

[0142] The image encoder is used to obtain the image to be processed, output the latent code corresponding to the image to be processed and input it into the UNet denoiser. The image to be processed contains text.

[0143] The text encoder is used to obtain the description text corresponding to the image to be processed and output text features to perform text control on the UNet denoiser;

[0144] Canny edge extractor, used to obtain the image to be processed and output the edge features corresponding to the image to be processed so as to control the edge structure of the UNet denoiser;

[0145] The semantic extractor is used to obtain the image to be processed and output the semantic features corresponding to the image to be processed so as to perform semantic control on the UNet denoiser;

[0146] UNet denoiser is used to obtain latent codes, perform iterative diffusion denoising, and output denoised latent codes. Text features, edge features, and semantic features are fused with the UNet denoiser through a cross-attention mechanism to guide the denoising process.

[0147] Image decoder, used to fuse denoising latent code and edge features and output super-resolution processing results.

[0148] The present invention constructs a complete image processing system, which effectively improves the clarity and realism of text images, realizes the full-process automated processing from image acquisition to super-resolution output; improves processing efficiency, reduces operational complexity, and makes the application of super-resolution technology more extensive and convenient.

[0149] An embodiment of the present invention further provides an image processing device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement Figure 1 The image processing method shown.

[0150] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0151] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer readable storage medium may be a read-only memory, a magnetic disk or an optical disk.

[0152] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, device embodiments, and storage medium embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0154] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. An image processing method, characterized in that: Obtain an image to be processed, and input it into a preset super-resolution processing model for super-resolution processing to obtain a super-resolution processing result; the image to be processed contains text, and the preset super-resolution processing model includes an image encoder, a UNet denoiser, a text encoder, an edge extractor, a semantic extractor and an image decoder; The super-resolution processing includes at least the following steps: obtaining an image to be processed and inputting it into an image encoder to obtain a latent code corresponding to the image to be processed; Obtain the description text corresponding to the image to be processed and input it into the text encoder to obtain text features; Input the image to be processed into the edge extractor to obtain edge features; Input the image to be processed into the semantic extractor to obtain semantic features; The text features, edge features, and semantic features are fused with the UNet denoiser through the cross-attention mechanism to guide the denoising process; The latent code is input into the UNet denoiser and iterative diffusion denoising is performed to obtain the denoised latent code; The denoised latent code and edge features are fused and input into the image decoder to output the super-resolution processing result.

2. The image processing method according to claim 1, characterized in that: The preset super-resolution processing model is obtained through pre-training, and the pre-training includes at least the following steps: Acquire image data including text and perform preprocessing to obtain paired data including target image data, original image data, and text description corresponding to the target image data; Construct the loss function of the network, which includes noise loss, L1 loss, LPIPS loss and edge loss; Perform end-to-end joint training on the network structure and use the constructed loss function to calculate the corresponding loss value; The network parameters are adjusted through iterative optimization to obtain a super-resolution processing model.

3. The image processing method according to claim 2, characterized in that: Preprocessing includes at least the following steps: Using image data including text as target image data; The target image data is downsampled and used as the original image data; Get the text description corresponding to the target image data based on BLIP.

4. The image processing method according to claim 2, characterized in that: The noise loss is used to calculate the L2 loss between the output noise of the UNet denoiser and the original preset noise; L1 loss is used to calculate the L1 loss of original image data and model output data; LPIPS loss uses the pre-trained VGG network to calculate the L2 loss between the features corresponding to the target image data and the features corresponding to the model output data; Edge loss is used to calculate the L2 loss between the edge features corresponding to the target image data and the edge features corresponding to the model output data.

5. The image processing method according to claim 1, characterized in that: The characters include at least one of the common characters, numbers, and special symbols of different countries or regions.

6. The image processing method according to claim 1, characterized in that: The feature fusion of the i-th intermediate layer of UNet is as follows: the edge features and semantic features are weighted by the corresponding attention mechanisms respectively; Calculate the sum of the output of the attention mechanism for text features, the output of the weighted attention mechanism for edge features, and the output of the weighted attention mechanism for semantic features.

7. The image processing method according to claim 1, characterized in that: The image encoder is built using the encoder structure of StableDiffusion, the text encoder is built using the BLIP model, the edge extractor is built using the Canny edge extraction operator, and the semantic extractor is built using the TransOCR model.

8. An image processing system, characterized in that: include: The image encoder is used to obtain the image to be processed, output the latent code corresponding to the image to be processed and input it into the UNet denoiser. The image to be processed contains text. A text encoder is used to obtain the description text corresponding to the image to be processed and output text features; An edge extractor is used to obtain an image to be processed and output edge features corresponding to the image to be processed; A semantic extractor, used to obtain an image to be processed and output semantic features corresponding to the image to be processed; UNet denoiser is used to obtain latent codes, perform iterative diffusion denoising, and output denoised latent codes. Text features, edge features, and semantic features are fused with the UNet denoiser through a cross-attention mechanism to guide the denoising process. Image decoder, used to fuse denoising latent code and edge features and output super-resolution processing results.

9. An image processing device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image processing program, and when the image processing program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images