A multi-task document image enhancement method and system based on low-rank adaptation

Abstract

The application discloses a kind of multi-task document image enhancement method and system based on low rank adaptation, comprising: obtaining document image enhancement instruction, input document image enhancement instruction analysis model to obtain instruction hidden space embedding feature;Instruction hidden space embedding feature is input into task classification model, obtain the task related degree vector of instruction and preset task;Dynamic fusion low rank adapter weight to trunk model, obtain document image enhancement model;Get document image, standardize block processing is carried out to document image, and obtain input document image block sequence;Input document image block sequence and instruction hidden space embedding feature are input into the document image enhancement model after adaptation task, and obtain enhanced output result;According to original cutting order splicing, obtain final document image enhancement result.Intelligent processing of multi-task document image enhancement is realized, and the system is driven by natural language instruction to complete the specified type of image enhancement task, which improves the convenience and operation efficiency of human-computer interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document processing, and in particular to a multi-task document image enhancement method and system based on low-rank adaptation. BACKGROUND

[0002] In actual application scenarios, document images often suffer from various degradation factors, mainly including two typical problems: one is the physical damage of the document itself, including but not limited to watermark coverage, ink penetration, paper creases and other inherent defects; the other is the secondary degradation introduced in the digital acquisition process, such as motion blur caused by scanner noise, camera jitter, and image blur caused by inaccurate focusing. These degradation problems not only significantly reduce the visual readability of the document, but also cause serious performance bottlenecks in subsequent key processing links (such as optical character recognition, document structure analysis, multi-modal document understanding, etc.). The core goal of the document image enhancement task is to restore the degraded document image and improve its visual quality, thereby supporting more accurate text extraction and information understanding.

[0003] Most existing methods use multiple multi-task independent models, only sharing a part of the main model part to perform multi-task enhancement on the image, which easily leads to waste of storage and computing resources. For example, patent CN202411516628.7 "Multi-task image processing method, system and device" combines image enhancement decoder, image super-resolution decoder and other multi-task decoders with a general encoder to obtain a multi-task network model, which introduces the complexity of coexistence of multiple large-parameter task decoders.

[0004] In addition, there are methods that use multi-modal large models, such as patent CN202310843671.3 "Document image processing method and system based on instructions" uses instructions and image inputs to a multi-modal large model to directly process and obtain output. This approach can achieve good results, but the resource overhead and training investment of multi-modal large models are large. SUMMARY

[0005] In view of the above defects of the prior art, the present application aims to provide a multi-task document image enhancement method and system based on low-rank adaptation, which realizes intelligent processing of multi-task document image enhancement, and can drive the system to complete the specified type of image enhancement task through natural language instructions, greatly improving the convenience and operation efficiency of human-computer interaction.

[0006] The technical problem solved by the present application adopts the following technical solutions:

[0007] In the first aspect, a multi-task document image enhancement method based on low-rank adaptation includes the following steps:

[0008] Step S1, obtaining a document image enhancement instruction, inputting a document image enhancement instruction analysis model to obtain a corresponding instruction hidden space embedding feature, and a task relevance degree vector for representing the relevance of the instruction to a preset task;

[0009] Step S2, according to the task relevance degree vector, dynamically fusing a low-rank adapter weight to the backbone model using a task routing module to obtain a document image enhancement model after task adaptation;

[0010] Step S3, obtaining a document image, performing block processing on the document image through a block processing module to obtain an input document image block sequence of the document image enhancement model;

[0011] Step S4, inputting the input document image block sequence and the instruction hidden space embedding feature into the document image enhancement model after task adaptation to obtain an enhanced output result;

[0012] Step S5, the document image block sequence enhancement output result is spliced according to the original cropping order through the block processing module to obtain a final document image enhancement result.

[0013] Further, in step S1, the training process of the document image enhancement instruction analysis model includes:

[0014] Obtaining instruction data corresponding to different document image enhancement tasks, pre-processing the instruction data to obtain pre-processed document image enhancement task instruction data;

[0015] Inputting the pre-processed document image enhancement task instruction data into a language model to obtain corresponding instruction hidden space embedding features, and inputting the instruction hidden space embedding features into a deep learning neural network for classification task training; after training, the language model part of the document image enhancement instruction analysis model is used to extract the corresponding instruction hidden space embedding features, and the task classification network part is used to obtain the task relevance degree vector representing the relevance of the instruction to the preset task.

[0016] Further, the preset task includes binarization, low-light enhancement, noise reduction, deblurring, shadow removal, seal removal, watermark removal, and distortion correction tasks.

[0017] Further, in step S2, the process of dynamically fusing a low-rank adapter weight by the task routing module includes:

[0018] In the model inference stage, according to the task relevance vector of the instruction, the task weight distribution is calculated using the task classification unit according to different task relevance, and one or more corresponding low-rank adapters in the low-rank adapter expert model library are selected for parameter fusion with the document image enhancement backbone model. The parameters of each low-rank adapter are multiplied by the corresponding weight, so that the fused model is adapted to the task demand corresponding to the instruction.

[0019] Further, in step S3, the input document image block sequence is composed of a plurality of input document images cropped into preset size image blocks; when the size of the remaining boundary region of the image is insufficient for cropping, the image boundary position is filled and then cropped, including filling pure black pixels or pure white pixels.

[0020] Further, in step S4, the construction process of the document image enhancement model after task adaptation includes:

[0021] The multi-task joint training framework is used to optimize the parameters of the multi-task document image enhancement backbone model, so that it has basic enhancement capability;

[0022] Each preset task is independently configured with a low-rank adapter, and the parameters of the multi-task document image enhancement backbone model are kept untrainable during the model updating process in the fine-tuning stage, and only the adapter parameters are optimized through back propagation;

[0023] In the inference stage, the task relevance vector output by the document image enhancement instruction analysis model is parsed to dynamically combine the parameters of multiple low-rank adapters with the multi-task document image enhancement backbone model for fusion, thereby obtaining a document image enhancement model after task adaptation.

[0024] Further, in step S5, the document image block sequence enhancement output result splicing process includes: reconstructing the topology structure of the image block using spatial position coding, establishing the geometric correspondence relationship of the original document, and strictly aligning the output document image block with the original input to restore the output image corresponding to the input document image.

[0025] In a second aspect, a multi-task document image enhancement system based on low-rank adaptation includes:

[0026] An instruction analysis module is configured to receive and analyze a document image enhancement instruction in natural language form, and generate an instruction semantic representation vector through a document image enhancement instruction analysis model;

[0027] A task routing module is configured to select a low-rank adapter expert corresponding to a task, and includes a low-rank adapter expert library of preset tasks, a task weight distribution is calculated by inputting a task classification model according to an instruction semantic representation vector, and a low-rank adapter expert parameter is dynamically selected for fusion with a multi-task document image enhancement backbone model;

[0028] The block processing module is configured to perform adaptive cropping and blocking operation on the document image, crop and combine the document image according to the spatial position coding, and ensure that the divided image blocks are consistent with the preset size by using boundary padding and cropping techniques.

[0029] The document image enhancement module is configured to obtain an enhanced document image block output sequence according to the document image block sequence and the corresponding document image enhancement instruction semantic representation vector as input.

[0030] In a third aspect, a terminal device includes a memory, a processor, and a program of a low-rank adaptation based multi-task document image enhancement method stored in the memory and executable on the processor. When the processor executes the program of the low-rank adaptation based multi-task document image enhancement method, the steps of any one of the low-rank adaptation based multi-task document image enhancement method are implemented.

[0031] In a fourth aspect, a computer readable storage medium stores a program of a low-rank adaptation based multi-task document image enhancement method. When the processor executes the program of the low-rank adaptation based multi-task document image enhancement method, the steps of any one of the low-rank adaptation based multi-task document image enhancement method are implemented.

[0032] The present application realizes a parameter efficient multi-task document image enhancement system, which makes the processing of the document image more convenient in actual use. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of an embodiment of the present application.

[0034] Figure 2 is a principle block diagram of a low-rank adaptation based multi-task document image enhancement system of an embodiment of the present application.

[0035] Figure 3 is a flowchart of a low-rank adaptation based multi-task document image enhancement system of an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0037] With reference to Figure 1 The present embodiment provides a multi-task document image enhancement method based on low-rank adaptation, comprising the following steps:

[0038] Step S1, obtaining a document image enhancement instruction, inputting a document image enhancement instruction analysis model to obtain corresponding instruction hidden space embedding features, and a task relevance degree vector for representing the relevance of the instruction and a preset task.

[0039] In the embodiments of the present application, the training of the document image enhancement instruction analysis model comprises the following steps: obtaining instruction data corresponding to different document image enhancement tasks, pre-processing the instruction data to remove illegal symbols and cut off the tail segment of excessively long text to obtain pre-processed document image enhancement task instruction data; inputting the pre-processed document image enhancement task instruction data into a pre-trained language model to obtain corresponding instruction hidden space embedding features, and inputting the instruction hidden space embedding features into a deep learning neural network for classification task training. After training, the language model part has the ability to extract semantic features corresponding to the instruction and the preset document image enhancement task, and obtains a document image enhancement instruction analysis model. The language model part of the document image enhancement instruction analysis model is used to extract corresponding instruction hidden space embedding features, and the task classification network part is used to obtain a task relevance degree vector representing the relevance of the instruction and a preset task. The preset task includes tasks such as binarization, low-light enhancement, noise reduction, deblurring, shadow removal, seal removal, watermark removal, and distortion correction.

[0040] Step S2, according to the task relevance degree vector, using a task routing module to dynamically fuse low-rank adapter weights to a backbone model to obtain a document image enhancement model after task adaptation.

[0041] In the embodiment of the present application, the training of the task classification model is obtained by the training of the document image enhancement instruction parsing model in step S1. Instruction data corresponding to different document image enhancement tasks is obtained, and after pre-processing, the instruction data is input into a pre-trained language model to obtain corresponding instruction hidden space embedding features, and the instruction hidden space embedding features are input into a deep learning neural network for classification task training. After training, the deep learning neural network classification model connected after the language model part is used as the task classification model. The output result of the task classification model is subjected to a function to obtain a task relevance vector of the instruction and the preset task. The task relevance vector of the instruction and the preset task can represent the relevance of the instruction and the task, and the closer to 1, the higher the relevance.

[0042] In step S3, a document image is obtained, and the document image is subjected to block processing by a block processing module to obtain an input document image block sequence of the document image enhancement model.

[0043] In the embodiment of the present application, for the low-rank adaptation expert model, first, a document image enhancement backbone model is pre-trained using a multi-task learning method in a pre-training stage. Subsequently, a plurality of low-rank adaptation expert models are introduced in a fine-tuning stage, which correspond one-to-one to the document image enhancement tasks. During the training in the fine-tuning stage, the parameters of the document image enhancement backbone model are frozen, and only the low-rank adaptation expert model of each task is subjected to gradient back propagation to update the model parameters. In actual use, the parameters of each low-rank adaptation expert model are weighted and added according to the weight of the task relevance vector to obtain the expert model parameters most suitable for the instruction task, and then the expert model parameters are fused into the multi-task document image enhancement backbone model to obtain the document image enhancement model after task adaptation.

[0044] In step S4, the input document image block sequence and the instruction hidden space embedding features are input into the document image enhancement model after task adaptation to obtain an enhanced output result.

[0045] In the embodiment of the present application, the input document image block sequence is composed of image blocks of a preset size cropped from an input document image. A document image with a higher resolution is input, and the document image is cropped into a plurality of square blocks with the same preset resolution. When the size of the remaining boundary region of the original document image is insufficient for cropping, the image boundary position is filled. When filling, pure black pixels or pure white pixels are randomly filled for different document images. After filling, the cropping requirement is met, and the cropping continues.

[0046] In step S5, the document image block sequence enhancement output result is subjected to a block processing module and spliced in the original cropping order to obtain a final document image enhancement result.

[0047] ​In the embodiment of the present application, each image block is normalized to adapt its pixel value range to the input requirements of the deep learning model. The image block sequence is first input into the encoder to extract multi-scale features through multiple convolutional blocks, generating a high-dimensional feature map. The instruction latent space embedding feature interacts with the visual feature through the cross-attention mechanism, enabling the model to dynamically adjust the enhancement strategy according to the instruction. The fused feature is input into the decoder fused with the expert model, and the spatial resolution is gradually restored through upsampling and skip connection to generate the enhanced image block.

[0048] In the embodiment of the present application, the document image block sequence enhancement output result splicing process includes reconstructing the image block topology using spatial position encoding, establishing the geometric correspondence of the original document, and strictly aligning the output document image block with the original input to restore the output image corresponding to the input document image.

[0049] The present application also provides a multi-task document image enhancement system based on low-rank adaptation, as shown in Figure 2 , which comprises:

[0050] The instruction analysis module S100 uses a pre-trained language model to initialize the instruction understanding unit constructed, which is used to receive and analyze document image enhancement instructions in natural language form, and generates instruction semantic representation vectors through a document image enhancement instruction analysis model. The instruction semantic encoder and the feature extraction network convert the user input text instruction, such as "remove the watermark on the picture", into a high-dimensional semantic representation vector.

[0051] The task routing module S200 is used to specify the selection of the low-rank adapter expert corresponding to the task. It is composed of a task classifier and a dynamic parameter fusion unit. The task classifier uses a deep learning neural network architecture, inputs the instruction semantic representation vector, and outputs the probability distribution of each preset task. The low-rank adapter expert library contains an adapter group configured independently for multiple tasks. According to the instruction semantic representation vector, the task classification model is input to calculate the task weight distribution and dynamically select the low-rank adapter expert parameters, which are fused with the multi-task document image enhancement backbone model;

[0052] The block processing module S300 is used to perform adaptive cropping and blocking operations on the document image. The document image is cropped and combined according to the spatial position encoding, and boundary padding and cropping techniques are used to ensure that the divided image blocks are consistent with the preset size.

[0053] The document image enhancement module S400 is used for taking the document image block sequence and the corresponding document image enhancement instruction semantic representation vector as input, each image block is subjected to normalization processing, and the pixel value range thereof is adapted to the input requirement of the deep learning model. The image block sequence is first input into an encoder, multi-scale features are extracted through a plurality of convolutional blocks, and a high-dimensional feature map is generated. The instruction hidden space embedding feature is interacted with the visual feature through a cross-attention mechanism, so that the model can dynamically adjust the enhancement strategy according to the instruction. The fused feature is input into a decoder fused with an expert model, and the spatial resolution is gradually restored through upsampling and skip connection, to generate enhanced image blocks, and an enhanced document image block output sequence is obtained.

[0054] In the embodiment of the present application, a flowchart of a multi-task document image enhancement system based on low-rank adaptation is as shown in Figure 3 Firstly, the system receives a natural language enhancement instruction input by a user, an instruction parsing module performs semantic analysis on the instruction by using a pre-trained language model to generate an instruction embedding feature. A task routing module analyzes the instruction embedding feature to obtain the weight of the task correlation degree vector, and dynamically loads the corresponding low-rank adapter parameter group according to the score, and the parameter fusion unit is executed after the weighted combination and fusion with the document image enhancement backbone model. After the input document image is cropped into a square image block sequence of a preset size by a block processing module, the instruction embedding feature obtained by parsing the instruction by the instruction parsing module is input into the document image enhancement module. The output enhanced document image block sequence is input into the block processing module, spliced in the original cropping order, to obtain a document image enhancement result corresponding to the size of the document image, and finally output as an enhanced document image.

[0055] The multi-task document image enhancement system based on low-rank adaptation of the embodiment of the present application can be deployed on various platforms or devices, supports a plurality of document image enhancement tasks, and can enhance the document image according to the instruction in a real scene.

[0056] The embodiment of the present application further provides a computer readable storage medium, which comprises a computer program, when the computer program runs on an electronic device, so that the electronic device executes the multi-task document image enhancement method based on low-rank adaptation as any one of the above.

[0057] The embodiment of the present application further provides an apparatus comprising a processor and a memory connected thereto, wherein the processor is configured to execute a computer program stored in the memory to execute the multi-task document image enhancement method based on low-rank adaptation as any one of the above.

[0058] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate way in one or more embodiments or examples.

[0059] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-task document image enhancement method based on low-rank adaptation, characterized in that, The method comprises the following steps: Step S1, obtaining a document image enhancement instruction, inputting a document image enhancement instruction analysis model to obtain a corresponding instruction hidden space embedding feature, and a task relevance degree vector for representing the relevance of the instruction to a preset task; Step S2, dynamically fusing a low-rank adapter weight to a backbone model using a task routing module according to the task relevance degree vector to obtain a document image enhancement model after task adaptation; Step S3, obtaining a document image, performing block processing on the document image using a block processing module to obtain an input document image block sequence of the document image enhancement model; Step S4, inputting the input document image block sequence and the instruction hidden space embedding feature into the document image enhancement model after task adaptation to obtain an enhanced output result; Step S5, the document image block sequence enhancement output result is spliced in the original cropping order by the block processing module to obtain a final document image enhancement result; In step S2, the process of dynamically fusing the low-rank adapter weight by the task routing module comprises: In the model inference stage, according to the task relevance degree vector of the task to which the instruction belongs, the task classification unit calculates the task weight distribution according to different task relevance degrees, and selects one or more corresponding low-quality adapters in the low-rank adapter expert model library to perform parameter fusion with the document image enhancement backbone model. Each low-rank adapter parameter is multiplied by the corresponding weight, so that the fused model is adapted to the task demand corresponding to the instruction; In step S4, the construction process of the document image enhancement model after task adaptation comprises: Optimize the parameters of the multi-task document image enhancement backbone model based on a multi-task joint training framework to make it have basic enhancement capability; Each preset task is independently configured with a low-rank adapter. In the model updating process in the fine-tuning stage, the parameters of the multi-task document image enhancement backbone model are kept untrainable, and only the adapter parameters are optimized through back propagation; In the inference stage, the task relevance degree vector output by the document image enhancement instruction analysis model is used to dynamically combine the parameters of multiple low-rank adapters with the multi-task document image enhancement backbone model to obtain the document image enhancement model after task adaptation.

2. The multi-task document image enhancement method based on low-rank adaptation according to claim 1, characterized in that, In step S1, the training process of the document enhancement instruction analysis model comprises: Obtain instruction data corresponding to different document image enhancement tasks, preprocess the instruction data to obtain preprocessed document image enhancement task instruction data; Input the preprocessed document image enhancement task instruction data into a language model to obtain corresponding instruction hidden space embedding features, and input the instruction hidden space embedding features into a deep learning neural network for classification task training; after training, the language model part of the document image enhancement instruction analysis model is used to extract the corresponding instruction hidden space embedding features, and the task classification network part is used to obtain the task relevance degree vector representing the relevance of the instruction to the preset task.

3. The method of claim 2, wherein, The preset tasks include binarization, low-light enhancement, noise reduction, deblurring, shadow removal, seal removal, watermark removal, and distortion correction tasks.

4. The method of claim 1, wherein, In step S3, the input document image block sequence is composed of image blocks of a preset size cropped from the input document image; when the size of the remaining boundary region of the image is insufficient for cropping, the image boundary position is padded and then cropped, including padding with pure black pixels or pure white pixels.

5. The method of claim 1, wherein, In step S5, the document image block sequence enhancement output result splicing process includes: reconstructing the image block topology with spatial position coding, establishing the geometric correspondence of the original document, and strictly aligning the output document picture block with the original input to restore the output image corresponding to the input document image.

6. A multi-task document image enhancement system based on low-rank adaptation, characterized in that, Comprise: The instruction analysis module is used for receiving and analyzing the document image enhancement instruction in natural language form, and generating an instruction semantic representation vector through a document image enhancement instruction analysis model; The task routing module is used for selecting a low-rank adapter expert corresponding to a task, contains a low-rank adapter expert library of preset tasks, inputs a task classification model according to the instruction semantic representation vector, calculates a task weight distribution, dynamically selects low-rank adapter expert parameters, and fuses them with a multi-task document image enhancement main model; The process of dynamically fusing low-rank adapter weights by the task routing module includes: In the model inference stage, according to the task relevance vector of the instruction, the task weight distribution is calculated using the task classification unit according to different task relevance, and one or more corresponding low-quality adapters in the low-rank adapter expert model library are selected, which are fused with the document image enhancement main model; each low-rank adapter parameter is multiplied by the corresponding weight, so that the fused model is adapted to the task demand corresponding to the instruction; The block processing module is used for performing adaptive cropping and blocking operations on the document image, cropping and combining the document image according to the spatial position coding, and using boundary padding and cropping techniques to ensure that the divided image blocks are consistent with the preset size; The document image enhancement module is used for obtaining an enhanced document image block output sequence according to the document image block sequence and the corresponding document image enhancement instruction semantic representation vector as the input of the document image enhancement model after the adaptation task. The construction process of the document image enhancement model after the adaptation task includes: Optimize the multi-task document image enhancement main model parameters based on the multi-task joint training framework to make it have basic enhancement capability; Each preset task is independently configured with a low-rank adapter, and the multi-task document image enhancement main model parameters are kept untrainable during the model updating process in the fine-tuning stage, and only the adapter parameters are optimized through back propagation; In the inference stage, the task relevance vector output by the document image enhancement instruction analysis model is used to dynamically combine multiple low-rank adapter parameters with the multi-task document image enhancement main model to obtain the document image enhancement model after the adaptation task.

7. A terminal device, characterized by comprising: The terminal device includes a memory, a processor, and a program of a multi-task document image enhancement method based on low-rank adaptation stored in the memory and executable on the processor. When the processor executes the program of the multi-task document image enhancement method based on low-rank adaptation, the steps of the multi-task document image enhancement method based on low-rank adaptation as claimed in any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program of a low-rank adaptation based multi-task document image enhancement method. When the program of the low-rank adaptation based multi-task document image enhancement method is executed by a processor, the steps of the low-rank adaptation based multi-task document image enhancement method in any one of claims 1-5 are implemented.

Images