Image restoration method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and discloses an image restoration method, device and equipment, a storage medium and a computer program product.The method comprises the steps that in response to an image restoration request, a to-be-restored image is input into an image restoration model, and the image restoration model is used for processing multiple image restoration tasks; the image restoration model comprises a cue word generator and a restoration network, performing preliminary processing on the to-be-restored image through the cue word generator to obtain a dynamic task cue word corresponding to the image restoration task, and performing image restoration on the to-be-restored image through the restoration network based on the dynamic task cue word to obtain a restored image. Therefore, the universality of image restoration processing can be improved, the cross-task cooperation capability of image restoration processing can be improved, and the real-time application requirement can be met.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, device, storage medium and computer program product. Background Art

[0002] At present, document image restoration is an important research direction in the field of image processing technology and multimodal large models, and is widely used in document digitization, archive protection, document recognition, smart office and other scenarios. Existing document image restoration methods usually set up corresponding independent models for each image restoration task, which has the defects of lack of versatility, poor cross-task collaboration and inability to meet real-time application requirements. Summary of the invention

[0003] The main purpose of the present application is to provide an image restoration method, apparatus, device, storage medium and computer program product, aiming to solve the technical problems that the existing document image restoration methods usually set up a corresponding independent model for each image restoration task, resulting in a lack of universality, poor cross-task collaboration and inability to meet real-time application requirements.

[0004] To achieve the above object, the present application provides an image restoration method, which comprises:

[0005] In response to an image restoration request, inputting the image to be restored into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network;

[0006] Preliminary processing is performed on the image to be repaired by the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task;

[0007] Based on the dynamic task prompt word, the image to be repaired is repaired through the restoration network to obtain a repaired image.

[0008] Optionally, the preliminarily processing the image to be repaired by the prompt word generator to obtain a dynamic task prompt word corresponding to the image repair task includes:

[0009] Extracting a priori features from the image to be repaired, wherein the a priori features are additional information dynamically extracted from the image to be repaired;

[0010] Based on the prior features, dynamic task prompt words corresponding to the image restoration task are generated, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

[0011] Optionally, the image restoration task includes a dewarping task, and the prior features include a document mask; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, includes:

[0012] If the image restoration task is the dewarping task, obtaining a document mask from the image to be restored by using a document segmentation model;

[0013] A dynamic task prompt word corresponding to the dewarping task is generated according to the document mask and the coordinate information of each pixel in the image to be repaired.

[0014] Optionally, the image restoration task includes a shadow removal task, and the prior features include a document background; extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes:

[0015] If the image restoration task is a shadow removal task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated;

[0016] Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background;

[0017] A dynamic task prompt word corresponding to the shadow removal task is generated based on the document background.

[0018] Optionally, the image restoration task includes an appearance enhancement task, and the prior features include a difference between the image to be restored and a document background; extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes:

[0019] If the image restoration task is an appearance enhancement task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated;

[0020] Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background;

[0021] Calculating the difference between the image to be repaired and the document background;

[0022] A dynamic task prompt word corresponding to the image restoration task is generated based on the difference between the image to be restored and the document background.

[0023] Optionally, the image restoration task includes a deblurring task, and the prior features include a gradient map of the image to be restored; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, includes:

[0024] If the image restoration task is a deblurring task, calculating a gradient map of the image to be restored;

[0025] A dynamic task prompt word corresponding to the deblurring task is generated based on the gradient map of the image to be repaired.

[0026] Optionally, the image restoration task includes a binarization task, and the prior features include an initial binarization result and a threshold map; extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes:

[0027] If the image restoration task is a binarization task, an initial binarization result and a threshold map are generated by a binarization algorithm;

[0028] Calculating a gradient map of the image to be repaired;

[0029] A dynamic task prompt word corresponding to the binarization task is generated based on the initial binarization result, the threshold map and the gradient mapping map.

[0030] Optionally, performing image restoration on the image to be restored by using the restoration network based on the dynamic task prompt word to obtain a restored image includes:

[0031] Fusing the dynamic task prompt word with the image to be repaired to obtain a fusion feature;

[0032] The fused features are input into a restoration network, and the image to be restored is restored through the restoration network to obtain a restored image.

[0033] Optionally, inputting the fused features into a restoration network, performing image restoration on the image to be restored through the restoration network, and obtaining a restored image includes:

[0034] The fused features are input into the recovery network, and the dynamic task prompt words are used as the task execution prompts of the recovery network:

[0035] Based on the task execution prompt, the image to be repaired is repaired through the restoration network to obtain a repaired image.

[0036] Optionally, before inputting the image to be repaired into the image repair model in response to the image repair request, the method further includes:

[0037] Obtain data sets corresponding to multiple image restoration tasks;

[0038] The initial model is trained based on the data set to obtain an image restoration model.

[0039] Optionally, the training of the initial model based on the data set to obtain the image restoration model includes:

[0040] Training the initial model based on the data set to obtain a trained model;

[0041] Evaluate the performance of the trained model based on the evaluation indicators corresponding to the multiple image restoration tasks to obtain an evaluation result;

[0042] The trained model is adjusted according to the evaluation result to obtain an image restoration model.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes an image restoration device, which includes:

[0044] A model input module, used for inputting the image to be repaired into the image repair model in response to the image repair request, wherein the image repair model is used to process various image repair tasks, and the image repair model includes a prompt word generator and a restoration network;

[0045] A prompt word generation module, used to perform preliminary processing on the image to be repaired through the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task;

[0046] The image restoration module is used to perform image restoration on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image.

[0047] Optionally, the prompt word generation module is also used to extract prior features from the image to be repaired, wherein the prior features are additional information dynamically extracted from the image to be repaired; and generate dynamic task prompt words corresponding to the image restoration task based on the prior features, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

[0048] Optionally, the image restoration task includes a dewarping task, and the prior feature includes a document mask; the prompt word generation module is also used to obtain the document mask from the image to be restored through a document segmentation model if the image restoration task is the dewarping task; and generate a dynamic task prompt word corresponding to the dewarping task based on the document mask and the coordinate information of each pixel in the image to be restored.

[0049] Optionally, the image restoration task includes a shadow removal task, and the prior features include a document background; the prompt word generation module is further used to, if the image restoration task is a shadow removal task, eliminate the text content in the image to be restored through an expansion operation to obtain an image after text removal; eliminate artifacts of the image after text removal through a median filter to obtain the document background; and generate a dynamic task prompt word corresponding to the shadow removal task based on the document background.

[0050] Optionally, the image restoration task includes an appearance enhancement task, and the prior features include the difference between the image to be restored and the document background; the prompt word generation module is also used to, if the image restoration task is an appearance enhancement task, eliminate the text content in the image to be restored by an expansion operation to obtain a text-eliminated image; eliminate artifacts of the text-eliminated image by a median filter to obtain the document background; calculate the difference between the image to be restored and the document background; and generate a dynamic task prompt word corresponding to the image restoration task based on the difference between the image to be restored and the document background.

[0051] Optionally, the image restoration task includes a deblurring task, and the prior features include a gradient map of the image to be restored; the prompt word generation module is further used to calculate the gradient map of the image to be restored if the image restoration task is a deblurring task; and generate a dynamic task prompt word corresponding to the deblurring task based on the gradient map of the image to be restored.

[0052] In addition, to achieve the above-mentioned purpose, the present application also proposes an image restoration device, which includes a memory, a processor, and an image restoration program stored in the memory and executable on the processor, wherein the image restoration program is configured to implement the image restoration method described above.

[0053] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which an image repair program is stored, and when the image repair program is executed by a processor, the image repair method as described above is implemented.

[0054] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes an image repair program, and when the image repair program is executed by a processor, it implements the image repair method described above.

[0055] One or more technical solutions proposed in this application have at least the following technical effects:

[0056] In the present application, it is disclosed that in response to an image restoration request, an image to be restored is input into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network. The prompt word generator performs preliminary processing on the image to be restored to obtain dynamic task prompt words corresponding to the image restoration task, and based on the dynamic task prompt words, the image to be restored is restored through the restoration network to obtain a restored image; since the present application restores the image to be restored through the image restoration model used to process a variety of image restoration tasks, the versatility of the image restoration processing can be improved; and the present application also generates dynamic task prompt words corresponding to the image restoration task to guide the restoration network to restore the image to be restored, thereby improving the cross-task collaboration capability of the image restoration processing to meet the needs of real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 This is a flow chart of the first embodiment of the image restoration method of the present application;

[0060] Figure 2 A schematic diagram of an image restoration model according to an embodiment of the image restoration method of the present application;

[0061] Figure 3 This is a flow chart of the second embodiment of the image restoration method of the present application;

[0062] Figure 4 This is a flow chart of the third embodiment of the image restoration method of the present application;

[0063] Figure 5 This is a flowchart of a fourth embodiment of the image restoration method of the present application;

[0064] Figure 6 This is a schematic diagram of the module structure of the image restoration device according to an embodiment of the present application;

[0065] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the image restoration method in the embodiment of the present application.

[0066] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0068] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0069] At present, document image restoration is an important research direction in the field of image processing technology and multimodal large models, and is widely used in document digitization, archive protection, document recognition, smart office and other scenarios. With the rapid development of deep learning and artificial intelligence technology, deep learning-based image restoration methods have made significant progress. These methods train neural networks to restore complete and clear images from damaged or missing document images to improve the readability and accuracy of documents. The main applications of current document image restoration include: Text restoration: Restoring missing text information caused by water stains, blurring, damage and other problems. Image restoration: Reconstructing illustrations, charts and other content in documents. Layout restoration: Repairing layout damage caused by folding, tearing or photocopying problems in documents.

[0070] However, existing document image restoration methods usually show the following three limitations in practical applications: (1) Method singleness: Most existing methods handle different restoration tasks independently and are optimized for specific tasks, such as only handling text restoration or image restoration. They lack versatility and cannot adapt to multi-task requirements, resulting in complex systems. (2) Poor cross-task collaboration: In documents, elements such as text, images, and layouts are interdependent. Existing methods are difficult to efficiently collaborate between different tasks, resulting in inconsistent restoration effects. (3) Difficulty in balancing model efficiency and performance: Existing multi-task methods require the design of multiple models separately, resulting in high computing resource requirements and inability to meet real-time application requirements.

[0071] Therefore, in order to overcome the above-mentioned defects, the present application provides a solution, which includes: in response to an image restoration request, inputting an image to be restored into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network, performing preliminary processing on the image to be restored by the prompt word generator to obtain dynamic task prompt words corresponding to the image restoration task, and performing image restoration on the image to be restored based on the dynamic task prompt words through the restoration network to obtain a restored image; since the present application restores the image to be restored by an image restoration model used to process a variety of image restoration tasks, the versatility of the image restoration processing can be improved; and the present application also guides the restoration network to restore the image to be restored by generating dynamic task prompt words corresponding to the image restoration task, thereby improving the cross-task collaboration capability of the image restoration processing and meeting the real-time application requirements.

[0072] It should be noted that the execution subject of this embodiment can be an image restoration device with data processing, network communication and program running functions, such as a computer, or other electronic devices that can achieve the same or similar functions, and this embodiment does not limit this.

[0073] Based on this, the present application embodiment provides an image restoration method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the image restoration method of the present application.

[0074] In a first embodiment, the image restoration method includes:

[0075] Step S10: In response to an image restoration request, inputting the image to be restored into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network.

[0076] It should be understood that the application scenario of the embodiment of the present application may be image restoration of document images. Of course, it may also be other image restoration scenarios, and the embodiment of the present application is not limited to this.

[0077] It should be noted that an image restoration request may refer to an operation instruction issued by a user or a system to restore or improve a specific image. The image to be restored may refer to the original image that needs to be restored, and the image to be restored may contain various defects, such as distortion, shadow, blur, poor quality, etc. Various image restoration tasks include but are not limited to at least one of a dedistortion task, a deshadowing task, an appearance enhancement task, a deblurring task, and a binarization task. A prompt word generator may refer to a component in an image restoration model, which is responsible for generating a dynamic task prompt word (DTPrompt) according to the image to be restored and the specified image restoration task. The dynamic task prompt word is used to guide the subsequent restoration process. The restoration network may refer to another key component in the image restoration model, which is responsible for receiving the dynamic task prompt word generated by the prompt word generator and the image to be restored, performing image restoration processing, and finally outputting the restored image.

[0078] In a specific implementation, when a user or system issues an image restoration request, the image restoration request is received and parsed, the image to be restored is obtained, and the image to be restored is input into the image restoration model.

[0079] Step S20: Preliminary processing is performed on the image to be repaired by the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task.

[0080] It is understandable that in order to provide clear task guidance for the subsequent image restoration process, the restoration network can perform the restoration task more accurately. In this embodiment, the prompt word generator performs preliminary processing on the image to be restored to obtain the dynamic task prompt word corresponding to the image restoration task. The dynamic task prompt word may refer to a prompt information dynamically generated according to the image to be restored and the restoration task, which is used to guide the restoration network to perform a specific restoration task during the image restoration process.

[0081] In a specific implementation, within the image restoration model, a prompt word generator can generate dynamic task prompt words based on the image to be restored and the image restoration task specified by the user (such as dewarping, deshadowing, etc.).

[0082] Step S30: performing image restoration on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image.

[0083] It should be understood that, based on the dynamic task prompt word, the image to be repaired is repaired by the restoration network, and the repaired image can be obtained by the restoration network receiving the dynamic task prompt word generated by the prompt word generator, and then performing image repair processing according to the prompt word and the image to be repaired. During the processing, the restoration network will use its own learned knowledge and prior features to repair and optimize the image. Of course, in addition to being used as a prompt for task execution, the dynamic task prompt word in this embodiment can also be used as supplementary information to enhance the overall performance of the network and improve the accuracy and consistency of the repair effect.

[0084] For ease of understanding, refer to Figure 2 This invention is provided for illustration, but is not intended to limit the present application. Figure 2 This is a schematic diagram of an image restoration model according to an embodiment of the image restoration method of the present application. Figure 2 In the embodiment, the image repair model can be an end-to-end model of UniDocFix (Unified Document Fixing). The UniDocFix model takes a document image as input and generates a clear document image after being repaired by different tasks. In the specific implementation, the document image to be repaired (i.e., the image to be repaired) is denoted as I s The specific steps of the image restoration method include: first, s The input is sent to the DTPropt generator (i.e., prompt word generator). The generator extracts the corresponding features from the input image according to the specific task to generate DTPropt. Next, DTPropt is passed to the recovery network, which serves as both a task guide for the recovery network and a prompt word generator from I when performing a specific task. s The auxiliary information derived from is used to improve the restoration effect and overall performance. Finally, the restoration network outputs the restored image.

[0085] In summary, this application proposes a universal model UniDocFix (Unified Document Fixing), which combines five document image repair tasks, including dewarping, deshadowing, appearance enhancement, deblurring, and binarization. Its main technical points are as follows:

[0086] 1. A new visual prompt method, Dynamic Task Prompt (DTPrompt), is proposed. DTPrompt is used to instruct UniDocFix to perform different repair tasks. For each task, DTPrompt extracts and includes different prior features, which are additional information dynamically extracted from the input image, thereby effectively guiding the restoration network to perform specific tasks.

[0087] 2. In addition to being a prompt for task execution, DTPrompt can also serve as supplementary information to enhance the overall performance of the network and improve the accuracy and consistency of repair results.

[0088] 3. DTPropt is more flexible than traditional visual prompting methods. It shows higher adaptability when processing high-resolution and variable-resolution inputs. It can be seamlessly integrated into the model to meet the needs of different document image restoration tasks. It is not limited by input resolution and has stronger flexibility and scalability.

[0089] This embodiment uses an image restoration model that is used to process a variety of image restoration tasks to restore the image to be restored, thereby improving the versatility of the image restoration process. In addition, this application also generates dynamic task prompt words corresponding to the image restoration task to guide the restoration network to restore the image to be restored, thereby improving the cross-task collaboration capability of the image restoration process and meeting real-time application requirements.

[0090] Reference Figure 3 , Figure 3 This is a flow chart of the second embodiment of the image restoration method of the present application, based on the above Figure 1 The first embodiment shown is a second embodiment of the image restoration method of the present application.

[0091] In the second embodiment, the step S20 includes:

[0092] Step S201: extracting a priori features from the image to be restored, wherein the a priori features are additional information dynamically extracted from the image to be restored.

[0093] It should be understood that in order to improve the accuracy of the dynamic task prompt words, in this embodiment, the prior features are first extracted from the image to be repaired, and then the dynamic task prompt words corresponding to the image repair task are generated based on the prior features. Among them, the prior features can refer to additional information dynamically extracted from the image to be repaired, which helps the model to better understand the image content and provide useful guidance in the repair process. For example, the prior features can be a document mask, a document background, the difference between the image to be repaired and the document background, a gradient map of the image to be repaired, and an initial binarization result and a threshold map.

[0094] In a specific implementation, a specific algorithm or model in the prompt word generator is used to perform preliminary processing and analysis on the image to be repaired, and to extract useful prior features therefrom.

[0095] Step S202: Generate dynamic task prompt words corresponding to the image restoration task based on the prior features, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

[0096] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s The input document image is first processed by the DTPrompt generator, which extracts prior features from the input image and generates a DTPrompt for a specific task, which helps guide the recovery network to perform different tasks and improve the overall performance. s ∈R h×w×3 ,For different tasks, the way to construct DTPrompt is also different. The calculation formula is as follows:

[0097] DTPrompt=G(I s ,task)∈R h×w×3

[0098] In the formula, G represents the DTPrompt generator, which is responsible for generating the output from I according to the specified task. s Extract prior features.

[0099] This embodiment first extracts prior features from the image to be repaired, and then generates dynamic task prompt words corresponding to the image restoration task based on the prior features, thereby improving the accuracy of the dynamic task prompt words, which in turn helps guide the restoration network to perform different tasks and improve the overall performance.

[0100] Furthermore, the image restoration task includes a dewarping task, the prior features include a document mask, and the prior features are extracted from the image to be restored, and dynamic task prompt words corresponding to the image restoration task are generated based on the prior features, including: if the image restoration task is the dewarping task, obtaining the document mask from the image to be restored through a document segmentation model; and generating dynamic task prompt words corresponding to the dewarping task based on the document mask and the coordinate information of each pixel in the image to be restored.

[0101] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s , the document mask can be used to enhance the model’s understanding of document boundaries and reduce the learning difficulty by decoupling the edge removal and content correction processes. The DTPrompt generator uses the document mask as a priori feature for the dewarping task, which is denoted as P m (I s )∈R h×w, specifically, it is a binary image obtained by processing with the document segmentation model, where the document area is 1 and the background area is 0. In addition, the reverse mapping prediction in the dewarping task is essentially a problem related to the coordinate position. DTPrompt also introduces the x-coordinate and y-coordinate as additional prior features, denoted by P cx ∈R h×w and P cy ∈R h×w .P cx and P cy Represents the coordinate value of the pixel at (i, j), that is, P cx (i,j)=i,P cy (i, j) = j, which can better perceive the position information. The overall DTPrompt of the dewarping task is obtained by concatenating the above-mentioned prior features along the channel dimension. The overall DTPrompt of the dewarping task is expressed as:

[0102] G(I s ,"dewarp")=[P m (I s ),P cx ,P cy ]

[0103] Furthermore, the image restoration task includes a shadow removal task, and the prior features include a document background; extracting the prior features from the image to be restored and generating dynamic task prompt words corresponding to the image restoration task based on the prior features includes: if the image restoration task is a shadow removal task, eliminating the text content in the image to be restored by a dilation operation to obtain an image after text removal; eliminating artifacts of the image after text removal by a median filter to obtain a document background; and generating dynamic task prompt words corresponding to the shadow removal task based on the document background.

[0104] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s For the shadow removal task, the document background with shadows is considered as a priori feature to improve performance. In order to obtain the background from the document image, a dilation operation is first used to eliminate the text content in the document, and then a median filter is used to smooth the artifacts introduced by incomplete elimination. The whole process is represented by P bg (I s )∈R h×w×3 ,Therefore, the DTPrompt of the shadow removal task is expressed as:

[0105] G(I s ,"deshadow")=P bg (I s )

[0106] Furthermore, the image restoration task includes an appearance enhancement task, and the prior features include the difference between the image to be restored and the document background; the prior features are extracted from the image to be restored, and dynamic task prompt words corresponding to the image restoration task are generated based on the prior features, including: if the image restoration task is an appearance enhancement task, the text content in the image to be restored is eliminated by a dilation operation to obtain a text-eliminated image; artifacts of the text-eliminated image are eliminated by a median filter to obtain a document background; the difference between the image to be restored and the document background is calculated; and the dynamic task prompt words corresponding to the image restoration task are generated based on the difference between the image to be restored and the document background.

[0107] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s Related appearance enhancement methods usually use background light, shadow map or white balance kernel as prior features to promote appearance enhancement based on inherent image concepts. However, it is challenging to accurately obtain these prior features, and usually requires an additional model for training and prediction. For simplicity, UniDocFix innovatively uses the original image and document background P bg (I s ) diff (I s ) is used as the prior feature of this task and as the initial enhancement guide of the model. The specific formula is as follows:

[0108] P diff (I s )=255-abs(I s -P bg (I s ))

[0109] G(I s ,"appearance")=P diff (I s )

[0110] Furthermore, the image restoration task includes a deblurring task, and the prior features include a gradient map of the image to be restored; extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes: if the image restoration task is a deblurring task, calculating the gradient map of the image to be restored; and generating a dynamic task prompt word corresponding to the deblurring task based on the gradient map of the image to be restored.

[0111] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s , gradient distribution is a priori feature widely used in traditional deblurring methods, and is usually used as regularization information to constrain the solution space of the optimization function. Therefore, UniDocFix uses the gradient map P of the input image g (I s )∈R h×w As an additional input, the purpose is to let the model implicitly learn the gradient prior information, rather than using the gradient distribution prior to constrain the output solution space. The DTPrompt of the deblurring task is expressed as:

[0112] G(I s ,"deblur")=[P g (I s ),P g (I s ),P g (I s )]

[0113] Furthermore, the image restoration task includes a binarization task, and the prior features include an initial binarization result and a threshold map; the extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes: if the image restoration task is a binarization task, generating an initial binarization result and a threshold map through a binarization algorithm; calculating a gradient mapping map of the image to be restored; and generating a dynamic task prompt word corresponding to the binarization task based on the initial binarization result, the threshold map and the gradient mapping map.

[0114] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s In the binarization task, integrating prior features is widely used as a supplement to performance enhancement. Therefore, DTPrompt uses the Sauvola binarization algorithm to generate the initial binarization result P b (I s )∈R h×w and threshold map P t (I s )∈R h×w As a priori features. In addition, the gradient map P g (I s )∈R h×w As an additional prior feature, DTPrompt for the binarization task is expressed as:

[0115] G(I s ,"binarize")=[P b (Is ),P t (I s ),P g (I s )]

[0116] Reference Figure 4 , Figure 4 This is a flow chart of the third embodiment of the image restoration method of the present application. Based on the above embodiments, the third embodiment of the image restoration method of the present application is proposed.

[0117] In the third embodiment, the step S30 includes:

[0118] Step S301: Fusing the dynamic task prompt word with the image to be repaired to obtain a fusion feature.

[0119] It should be understood that in order to improve the efficiency of image restoration, in this embodiment, the dynamic task prompt word is directly fused with the image to be restored and input into the restoration network, and the image to be restored is restored by the restoration network to obtain the restored image.

[0120] Step S302: inputting the fused features into a restoration network, performing image restoration on the image to be restored through the restoration network, and obtaining a restored image.

[0121] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the image to be repaired is I s , this application adopts a direct fusion method to seamlessly integrate DTPrompt into the restoration network. Specifically, this application integrates DTPrompt and the input image feature I along the channel dimension s Connect and construct a new input for the recovery network with a dimension of R h×w×6 . Due to the simple structure of DTPropt, the choice of recovery network is highly flexible. This application uses the Restormer model as the recovery network and builds a document repair network called UniDocFix without any modification. The network can support high-resolution input images with a maximum size of 1600×1600 pixels and adapt to input requirements with variable resolution. At the same time, other recovery networks can also be used interchangeably because DTPropt does not require a specific module in the network.

[0122] Furthermore, in order to improve the image restoration effect, in this embodiment, after the dynamic task prompt words are fused with the image to be restored and input into the restoration network, the dynamic task prompt words can also be used as the task execution prompt of the restoration network to guide the restoration network to perform image restoration on the image to be restored. The step S302 includes: inputting the fused features into the restoration network, and using the dynamic task prompt words as the task execution prompt of the restoration network: performing image restoration on the image to be restored through the restoration network based on the task execution prompt to obtain a restored image.

[0123] In this embodiment, the dynamic task prompt word is fused with the image to be repaired and input into the restoration network, and the image to be repaired is repaired by the restoration network to obtain a repaired image, thereby improving the efficiency of image repair.

[0124] Reference Figure 5 , Figure 5 This is a flow chart of the fourth embodiment of the image restoration method of the present application. Based on the above embodiments, the fourth embodiment of the image restoration method of the present application is proposed.

[0125] In the fourth embodiment, before step S10, the method further includes:

[0126] Step S01: Obtain data sets corresponding to multiple image restoration tasks.

[0127] It should be understood that in order to improve the reliability of the image restoration model, in this embodiment, data sets corresponding to multiple image restoration tasks are obtained in advance to train the initial model to obtain the image restoration model.

[0128] Step S02: training the initial model based on the data set to obtain an image restoration model.

[0129] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, in order to achieve de-distortion, de-shadowing, appearance enhancement, de-blurring and binarization of document images, multiple high-quality data sets are selected for training and testing to verify the performance and robustness of the algorithm.

[0130] 1. De-distortion task:

[0131] In order to solve the dewarping problem of document images, this application uses the Doc3D dataset and the DIR300 benchmark for training and testing. Doc3D is a synthetic dataset containing about 100,000 samples, each of which includes a geometrically distorted document image and its corresponding inverse mapping. DIR300 is a benchmark dataset in real scenes, containing 300 geometrically distorted document images and their corresponding planar benchmark images.

[0132] 2. Shadow removal task:

[0133] For shadow processing in document images, the training set of this application consists of 14,200 synthetic images in the FTSRD dataset and 4,371 real images in the RDD training set. In the testing phase, the Jung dataset (containing 87 images), the Kligler dataset (containing 300 images) and the OSR dataset (containing 237 images) were selected as evaluation datasets to verify the performance of the model in different scenarios.

[0134] 3. Appearance enhancement task:

[0135] The training data for this task includes 90,000 synthetic images from the Doc3DShade dataset and 450 real images from the RealDAE training set. The test data includes 150 images from the RealDAE test set and 130 images from the DocUNet dataset.

[0136] 4. Deblurring task:

[0137] This task uses the Text Deblur Dataset (TDD), which contains 66,000 training samples. 40,000 samples are randomly selected for model training, and 1,600 test samples of the TDD dataset are used as the test set to evaluate the performance of the model in the document image deblurring task.

[0138] 5. Binarization task:

[0139] In order to achieve efficient binarization of document images, this application uses the DIBCO'18 dataset as the test set and the (H)-DIBCO dataset as the training data. In addition, the Noisy Office dataset is also combined to further enhance the generalization ability and robustness of the model.

[0140] Through the comprehensive utilization of the above-mentioned multi-source datasets, this application demonstrates excellent image restoration and enhancement capabilities in multi-task scenarios, and is suitable for various complex document image processing needs.

[0141] Furthermore, in order to comprehensively measure the model performance, the step S02 includes: training the initial model based on the data set to obtain a trained model; evaluating the performance of the trained model based on the evaluation indicators corresponding to the multiple image restoration tasks to obtain an evaluation result; adjusting the trained model according to the evaluation result to obtain an image restoration model.

[0142] For ease of understanding, the following examples are provided, but are not intended to limit the present application. As an example, the present application designs diverse and accurate evaluation indicators for different tasks to comprehensively measure model performance:

[0143] 1. Shadow removal, appearance enhancement and deblurring tasks:

[0144] The performance evaluation of these tasks uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as the main indicators. PSNR is used to measure the similarity between the restored image and the reference image, while SSIM provides a more detailed evaluation by analyzing brightness, contrast and structural information.

[0145] 2. De-distortion task:

[0146] The dewarping task combines the following three indicators for comprehensive evaluation:

[0147] (1) Multi-scale structural similarity (MS-SSIM): Based on traditional SSIM, it considers image features at multiple scales to evaluate image quality more comprehensively.

[0148] (2) Local distortion (LD): The dewarping performance is evaluated by calculating the offset between the dewarped result and the planar reference image.

[0149] (3) Alignment Distortion (AD): As an improved version of LD, AD provides a more accurate evaluation by excluding the offset noise in low-texture regions and alleviating the impact of global transformation.

[0150] 3. Binarization task:

[0151] For the binarization task, the following three metrics are used:

[0152] (1) Peak signal-to-noise ratio (PSNR): used to measure the overall similarity between the binarization result and the reference image.

[0153] (2) F-measure (FM): Comprehensively evaluates the precision and recall of the binarization results, reflecting the overall performance.

[0154] (3) Pseudo F-measure (pFM): An enhanced indicator specifically used to measure the detailed performance of binarization quality.

[0155] Through the above evaluation system, this application can verify its performance superiority in different tasks in a scientific and quantitative manner, thereby meeting the actual application needs in multi-task scenarios.

[0156] In the training process of this application, the model training was performed on 8 GPUs, with a total training step number of 100,000 steps and a global batch size of 128. During the training process, the AdamW optimizer with weight decay was used, and the decay value was 5×10 -4 The learning rate schedule uses cosine learning rate decay, and the maximum learning rate is set to 2×10-4 .

[0157] Before unified training, in order to ensure the stability of model training, 50,000 steps of pre-training were first performed on the dewarping task to initialize the model parameters. This pre-training stage is crucial for unified training because the dewarping task is significantly different from other tasks: dewarping mainly involves coordinate regression, while other tasks focus on image content regression.

[0158] In the unified training stage, the sampling weight of each task is set to 0.2, including dewarping, deshading, appearance enhancement, deblurring and binarization tasks. Except for the binarization task which uses the standard cross entropy loss, the other tasks use L1 loss as the supervisory signal. In order to improve the training efficiency and enhance the robustness of the model, the training images of deshading, appearance enhancement, deblurring and binarization tasks are randomly cropped into 256×256 image blocks, while in the dewarping task, the input image is adjusted to a fixed size of 256×256.

[0159] Through this training strategy, the present application can simultaneously optimize the performance of different tasks in multi-task scenarios and improve the generalization ability of the model.

[0160] The UniDocFix (unified document repair method) technical solution proposed in this application brings the following significant beneficial effects:

[0161] 1. Unified processing of multiple tasks: By designing a unified repair method suitable for multi-task scenarios, UniDocFix can handle multiple problems in documents (such as dewarping, deshading, appearance enhancement, deblurring, and binarization) at the same time, significantly improving the efficiency and accuracy of the document repair process and avoiding the complex coordination between multiple independent models.

[0162] 2. Efficient cross-task collaboration capability: The DTPrompt (dynamic task prompt) method is used to flexibly extract and fuse task-specific prior features to achieve efficient collaboration between different repair tasks, improve the consistency and accuracy of the repair effect, and avoid the fluctuation of repair quality caused by insufficient collaboration between tasks in traditional methods.

[0163] 3. Enhanced model performance and flexibility: DTPrompt not only serves as a prompt for task execution, but also as supplementary information to improve model performance. It is more flexible than traditional visual prompting methods and can seamlessly adapt to inputs of different resolutions, especially supporting high-resolution document images up to 1600×1600 pixels.

[0164] 4. Save computing resources and improve real-time performance: By processing multiple tasks in the same network and using a simplified DTPrompt method, UniDocFix can significantly reduce the demand for computing resources. Compared with traditional multi-model methods, it reduces redundant calculations and can better meet the needs of real-time applications.

[0165] 5. Applicable to a variety of document repair tasks: This application can be widely used in document digitization, archive protection, document recognition, smart office and other fields. It is suitable for different types of document image repair tasks and has strong versatility and adaptability.

[0166] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image restoration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0167] This application also provides an image restoration device, please refer to Figure 6 , the image restoration device comprises:

[0168] A model input module 10, configured to input the image to be repaired into an image repair model in response to an image repair request, wherein the image repair model is configured to process a variety of image repair tasks, and the image repair model includes a prompt word generator and a restoration network;

[0169] A prompt word generating module 20, configured to perform preliminary processing on the image to be repaired through the prompt word generator to obtain a dynamic task prompt word corresponding to the image repair task;

[0170] The image restoration module 30 is used to perform image restoration on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image.

[0171] The image restoration device provided by the present application adopts the image restoration method in the above-mentioned embodiment, which can solve the technical problems that the existing document image restoration method usually sets a corresponding independent model for each image restoration task for processing, thereby having the defects of lack of versatility, poor cross-task collaboration ability, and inability to meet real-time application requirements. Compared with the prior art, the beneficial effects of the image restoration device provided by the present application are the same as the beneficial effects of the image restoration method provided by the above-mentioned embodiment, and the other technical features in the image restoration device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0172] The present application provides an image restoration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image restoration method in the above-mentioned embodiment 1.

[0173] Reference below Figure 7 , which shows a schematic diagram of the structure of an image restoration device suitable for implementing an embodiment of the present application. The image restoration device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The image restoration device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0174] like Figure 7 As shown, the image repair device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to RAM (Random Access Memory) 1004. In RAM 1004, various programs and data required for the operation of the image repair device are also stored. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image restoration device to communicate wirelessly or wired with other devices to exchange data. Although the image restoration device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0175] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0176] The image restoration device provided by the present application adopts the image restoration method in the above embodiment, which can solve the technical problems that the existing document image restoration method usually sets a corresponding independent model for each image restoration task for processing, thereby having the defects of lack of versatility, poor cross-task collaboration ability and inability to meet real-time application requirements. Compared with the prior art, the beneficial effects of the image restoration device provided by the present application are the same as the beneficial effects of the image restoration method provided by the above embodiment, and the other technical features in the image restoration device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0177] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0178] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0179] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image restoration method in the above-mentioned embodiment.

[0180] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory) or flash memory, optical fiber, CD-ROM (CD-Read Only Memory, portable compact disk read-only memory), optical storage device, magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0181] The computer-readable storage medium may be included in the image restoration device; or may exist independently without being assembled into the image restoration device.

[0182] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image restoration device, the image restoration device executes the image restoration method.

[0183] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or it can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0184] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0186] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image restoration method, and can solve the technical problems that the existing document image restoration method usually sets a corresponding independent model for each image restoration task for processing, thereby having the defects of lack of versatility, poor cross-task collaboration, and inability to meet real-time application requirements. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the image restoration method provided by the above-mentioned embodiment, and will not be repeated here.

[0187] The present application also provides a computer program product, including a computer program, which implements the above-mentioned image restoration method when executed by a processor.

[0188] The computer program product provided by the present application can solve the technical problems that the existing document image restoration methods usually set up corresponding independent models for each image restoration task, resulting in lack of versatility, poor cross-task collaboration, and inability to meet real-time application requirements. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the image restoration method provided by the above-mentioned embodiment, and will not be elaborated here.

[0189] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

[0190] The present application discloses A1, an image restoration method, the image restoration method comprising:

[0191] In response to an image restoration request, inputting the image to be restored into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network;

[0192] Preliminary processing is performed on the image to be repaired by the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task;

[0193] Based on the dynamic task prompt word, the image to be repaired is repaired through the restoration network to obtain a repaired image.

[0194] A2. The image restoration method as described in A1, wherein the prompt word generator performs preliminary processing on the image to be restored to obtain a dynamic task prompt word corresponding to the image restoration task, including:

[0195] Extracting a priori features from the image to be repaired, wherein the a priori features are additional information dynamically extracted from the image to be repaired;

[0196] Based on the prior features, dynamic task prompt words corresponding to the image restoration task are generated, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

[0197] A3. The image restoration method as described in A2, wherein the image restoration task includes a dedistortion task, and the prior features include a document mask; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, comprises:

[0198] If the image restoration task is the dewarping task, obtaining a document mask from the image to be restored by using a document segmentation model;

[0199] A dynamic task prompt word corresponding to the dewarping task is generated according to the document mask and the coordinate information of each pixel in the image to be repaired.

[0200] A4. The image restoration method as described in A2, wherein the image restoration task includes a shadow removal task, and the prior features include a document background; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, comprises:

[0201] If the image restoration task is a shadow removal task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated;

[0202] Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background;

[0203] A dynamic task prompt word corresponding to the shadow removal task is generated based on the document background.

[0204] A5. The image restoration method as described in A2, wherein the image restoration task includes an appearance enhancement task, and the prior feature includes a difference between the image to be restored and a document background; extracting the prior feature from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior feature comprises:

[0205] If the image restoration task is an appearance enhancement task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated;

[0206] Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background;

[0207] Calculating the difference between the image to be repaired and the document background;

[0208] A dynamic task prompt word corresponding to the image restoration task is generated based on the difference between the image to be restored and the document background.

[0209] A6. The image restoration method as described in A2, wherein the image restoration task includes a deblurring task, and the prior features include a gradient map of the image to be restored; and the extracting of the prior features from the image to be restored and generating a dynamic task prompt corresponding to the image restoration task based on the prior features comprises:

[0210] If the image restoration task is a deblurring task, calculating a gradient map of the image to be restored;

[0211] A dynamic task prompt word corresponding to the deblurring task is generated based on the gradient map of the image to be repaired.

[0212] A7. The image restoration method as described in A2, wherein the image restoration task includes a binarization task, and the prior features include an initial binarization result and a threshold map; the step of extracting the prior features from the image to be restored and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features includes:

[0213] If the image restoration task is a binarization task, an initial binarization result and a threshold map are generated by a binarization algorithm;

[0214] Calculating a gradient map of the image to be repaired;

[0215] A dynamic task prompt word corresponding to the binarization task is generated based on the initial binarization result, the threshold map and the gradient mapping map.

[0216] A8. The image restoration method as described in any one of A1 to A7, wherein the image restoration is performed on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image, comprising:

[0217] Fusing the dynamic task prompt word with the image to be repaired to obtain a fusion feature;

[0218] The fused features are input into a restoration network, and the image to be restored is restored through the restoration network to obtain a restored image.

[0219] A9. The image restoration method as described in A8, wherein the fusion feature is input into a restoration network, and the image to be restored is restored by the restoration network to obtain a restored image, comprising:

[0220] The fused features are input into the recovery network, and the dynamic task prompt words are used as the task execution prompts of the recovery network:

[0221] Based on the task execution prompt, the image to be repaired is repaired through the restoration network to obtain a repaired image.

[0222] A10. The image restoration method as described in any one of A1 to A7, wherein before inputting the image to be restored into the image restoration model in response to the image restoration request, the method further comprises:

[0223] Obtain data sets corresponding to multiple image restoration tasks;

[0224] The initial model is trained based on the data set to obtain an image restoration model.

[0225] A11. The image restoration method as described in A10, wherein the initial model is trained based on the data set to obtain the image restoration model, comprising:

[0226] Training the initial model based on the data set to obtain a trained model;

[0227] Evaluate the performance of the trained model based on the evaluation indicators corresponding to the multiple image restoration tasks to obtain an evaluation result;

[0228] The trained model is adjusted according to the evaluation result to obtain an image restoration model.

[0229] The present application also discloses B12, an image restoration device, the image restoration device comprising:

[0230] A model input module, used for inputting the image to be repaired into the image repair model in response to the image repair request, wherein the image repair model is used to process various image repair tasks, and the image repair model includes a prompt word generator and a restoration network;

[0231] A prompt word generation module, used to perform preliminary processing on the image to be repaired through the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task;

[0232] The image restoration module is used to perform image restoration on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image.

[0233] B13. In the image restoration device as described in B12, the prompt word generation module is also used to extract prior features from the image to be restored, wherein the prior features are additional information dynamically extracted from the image to be restored; and generate dynamic task prompt words corresponding to the image restoration task based on the prior features, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

[0234] B14. In the image restoration device as described in B13, the image restoration task includes a dewarping task, and the prior feature includes a document mask; the prompt word generation module is also used to obtain the document mask from the image to be restored through a document segmentation model if the image restoration task is the dewarping task; and generate a dynamic task prompt word corresponding to the dewarping task based on the document mask and the coordinate information of each pixel in the image to be restored.

[0235] B15. An image restoration device as described in B13, wherein the image restoration task includes a shadow removal task, and the prior features include a document background; the prompt word generation module is further used to, if the image restoration task is a shadow removal task, eliminate text content in the image to be restored by an expansion operation to obtain a text-eliminated image; eliminate artifacts of the text-eliminated image by a median filter to obtain a document background; and generate a dynamic task prompt word corresponding to the shadow removal task based on the document background.

[0236] B16. An image restoration device as described in B13, wherein the image restoration task includes an appearance enhancement task, and the prior features include the difference between the image to be restored and the document background; the prompt word generation module is further used to, if the image restoration task is an appearance enhancement task, eliminate the text content in the image to be restored by a dilation operation to obtain a text-eliminated image; eliminate artifacts of the text-eliminated image by a median filter to obtain the document background; calculate the difference between the image to be restored and the document background; and generate a dynamic task prompt word corresponding to the image restoration task based on the difference between the image to be restored and the document background.

[0237] B17. In the image restoration device as described in B13, the image restoration task includes a deblurring task, and the prior features include a gradient map of the image to be restored; the prompt word generation module is further used to calculate the gradient map of the image to be restored if the image restoration task is a deblurring task; and generate a dynamic task prompt word corresponding to the deblurring task based on the gradient map of the image to be restored.

[0238] The present application also discloses C18, an image restoration device, which includes: a memory, a processor, and an image restoration program stored in the memory and executable on the processor, wherein the image restoration program implements the image restoration method described above when executed by the processor.

[0239] The present application also discloses D19, a storage medium, on which an image repair program is stored, and when the image repair program is executed by a processor, the image repair method as described above is implemented.

[0240] The present application also discloses E20, a computer program product, which includes an image restoration program, and when the image restoration program is executed by a processor, the image restoration method as described above is implemented.

Claims

1. An image restoration method, characterized in that: The image restoration method comprises: In response to an image restoration request, inputting the image to be restored into an image restoration model, wherein the image restoration model is used to process a variety of image restoration tasks, and the image restoration model includes a prompt word generator and a restoration network; Preliminary processing is performed on the image to be repaired by the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task; Based on the dynamic task prompt word, the image to be repaired is repaired through the restoration network to obtain a repaired image.

2. The image restoration method according to claim 1, characterized in that: The step of performing preliminary processing on the image to be repaired by the prompt word generator to obtain a dynamic task prompt word corresponding to the image repair task includes: Extracting a priori features from the image to be repaired, wherein the a priori features are additional information dynamically extracted from the image to be repaired; Based on the prior features, dynamic task prompt words corresponding to the image restoration task are generated, wherein the dynamic task prompt words are used to guide the restoration network to perform different tasks during the image restoration process.

3. The image restoration method according to claim 2, characterized in that: The image restoration task includes a dedistortion task, and the prior features include a document mask; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, includes: If the image restoration task is the dewarping task, obtaining a document mask from the image to be restored by using a document segmentation model; A dynamic task prompt word corresponding to the dewarping task is generated according to the document mask and the coordinate information of each pixel in the image to be repaired.

4. The image restoration method according to claim 2, characterized in that: The image restoration task includes a shadow removal task, and the prior features include a document background; extracting the prior features from the image to be restored, and generating a dynamic task prompt word corresponding to the image restoration task based on the prior features, includes: If the image restoration task is a shadow removal task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated; Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background; A dynamic task prompt word corresponding to the shadow removal task is generated based on the document background.

5. The image restoration method according to claim 2, characterized in that: The image restoration task includes an appearance enhancement task, the prior features include the difference between the image to be restored and the document background; the extracting the prior features from the image to be restored and generating the dynamic task prompt words corresponding to the image restoration task based on the prior features include: If the image restoration task is an appearance enhancement task, the text content in the image to be restored is eliminated by a dilation operation to obtain an image after the text is eliminated; Eliminating artifacts of the image after text elimination by using a median filter to obtain a document background; Calculating the difference between the image to be repaired and the document background; A dynamic task prompt word corresponding to the image restoration task is generated based on the difference between the image to be restored and the document background.

6. The image restoration method according to claim 2, characterized in that: The image restoration task includes a deblurring task, and the prior feature includes a gradient map of the image to be restored; The extracting a priori features from the image to be repaired and generating a dynamic task prompt word corresponding to the image repair task based on the priori features includes: If the image restoration task is a deblurring task, calculating a gradient map of the image to be restored; A dynamic task prompt word corresponding to the deblurring task is generated based on the gradient map of the image to be repaired.

7. An image restoration device, characterized in that: The image restoration device comprises: A model input module, used for inputting the image to be repaired into the image repair model in response to the image repair request, wherein the image repair model is used to process various image repair tasks, and the image repair model includes a prompt word generator and a restoration network; A prompt word generation module, used to perform preliminary processing on the image to be repaired through the prompt word generator to obtain dynamic task prompt words corresponding to the image repair task; The image restoration module is used to perform image restoration on the image to be restored through the restoration network based on the dynamic task prompt word to obtain a restored image.

8. An image restoration device, characterized in that: The image restoration device comprises: a memory, a processor, and an image restoration program stored in the memory and executable on the processor, wherein the image restoration program implements the image restoration method according to any one of claims 1 to 6 when executed by the processor.

9. A storage medium, characterized in that: An image repair program is stored on the storage medium, and when the image repair program is executed by the processor, the image repair method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises an image restoration program, and when the image restoration program is executed by a processor, the image restoration method according to any one of claims 1 to 6 is implemented.

Images