Ancient book repairing method and system based on generative large language model
Through the generative large language model, the image enhancement and feature processing of ancient books is solved, and the problems of low repair efficiency and poor deep learning performance are achieved, efficient and accurate digitization and text recognition of ancient books are achieved, and the utilization rate of ancient books is improved.
Patent Information
- Application Number
- CN202411467419.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing ancient book restoration technology is inefficient, deep learning algorithms have poor performance in ancient book restoration tasks, and the existing digital ancient book has low resolution and mostly black and white images, making it difficult to easily search.
Ancient books repair method based on a generative large language model is adopted to enhance images through deep convolutional neural networks, and visual converters encode and decode defective image features of ancient books, and combine state space mapping and image text recognition to generate complete ancient books.
It significantly improves the accuracy and efficiency of ancient books restoration, is robust, and the algorithm performance is improved by about 26%, and the time efficiency is more than 10 times higher than that of artificial methods.
Smart Images

Figure CN118982834A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ancient book restoration, and in particular to an ancient book restoration method and system based on a generative large language model. Background Art
[0002] For a long time, the protection of ancient books has mainly adopted the following protection methods: (1) original protection method, that is, protecting ancient books as "cultural relics"; (2) regenerative protection method, photocopying and image preservation of ancient books, so that ancient books can exist in the form of paper or microfilm; (3) converting the content of ancient books into digital text. At present, the conversion of ancient book content into digital text mainly relies on manual input by experts, which is extremely time-consuming.
[0003] In recent years, deep learning has carried out a lot of pioneering work in the digitization of ancient books, and has accumulated relatively mature technologies and experience in OCR (optical character recognition), AI (artificial intelligence) sentence reading, entity recognition, etc. Take OCR application as an example. When scanning paper ancient books with electronic devices, the content on the ancient books will be automatically transcribed into the computer and the corresponding digital documents will be generated. The efficiency can be increased by up to 10 million times compared with manual input. OCR technology is widely used in image text recognition, subtitle translation, and various card tickets and industry document recognition in commercial business. These technologies can gradually migrate to the direction of intelligent digitization of ancient books. Technological progress has brought two aspects to the intelligent restoration and arrangement of ancient books: one is the intelligent transformation of ancient book texts, and the other is the intelligent use of ancient books. Converting the content of paper ancient books into digital texts is only the first step in the restoration and arrangement of ancient books. On this basis, another problem to be solved is how to organize and classify the massive obscure content of ancient books to form interactive, touchable, and visual digital humanities works for people to consult and use. Otherwise, the ancient books entered into the computer will continue to "sleep". Based on artificial intelligence technology, my country has established a number of ancient book sorting automation and visualization platforms. For example, the "Song and Yuan Xue'an Knowledge Graph Visualization System" processed and analyzed the 2.4 million-word "Song and Yuan Xue'an", and extracted the characters, time, place, works, etc. involved in more than 2,000 Song and Yuan Neo-Confucian scholars and nearly 100 academic schools to construct a knowledge graph. However, the intelligence level of many platforms is still low. For example, when keywords are entered, the searched contents are isolated and disordered. A smarter ancient book sorting and utilization platform should evolve from version 1.0 to version 2.0. For example, content retrieval should be "based on categories", and the retrieved contents should be related to each other and organically classified by artificial intelligence.
[0004] To sum up, the shortcomings of current ancient book restoration technology are mainly as follows: (1) The level of ancient book restoration technology is limited, and the methods of ancient book restoration need to rely on professional technicians and equipment. The existing methods of ancient book restoration involve technologies such as cultural relic restoration, anti-corrosion and anti-insect. The current technical level is relatively backward and needs to rely heavily on hardware equipment such as manual labor, which cannot effectively repair and protect ancient books.
[0005] (2) The current restoration methods of ancient books limit the digital protection and utilization of ancient books. At present, the protection of ancient books mainly adopts the original protection method, that is, the ancient books are protected as "cultural relics". Many existing digitized ancient books are converted from microfilm, with low resolution and mostly black and white images. Due to the particularity of ancient books, ancient book restoration and identification technology faces some challenges, such as the diversity and variants of ancient characters, and the high similarity of glyphs.
[0006] (3) Many of the existing digitized ancient books are converted from microfilm, with low resolution and mostly black and white images. Even if all ancient books are photocopied and published by digital means, it is difficult for readers to use them conveniently. Since photocopied ancient books do not have punctuation marks, they are very inconvenient to read. In addition, this is not conducive to searching the content of ancient books. If you want to check a certain content, you have to read the original text page by page, which makes it difficult to quickly retrieve knowledge. Therefore, in order to improve the utilization rate of traditional ancient books, the content of ancient books must be converted into digital text. In the past, this conversion mainly relied on manual entry by experts, which was extremely time-consuming.
[0007] (4) Deep learning methods have been proven to be effective in image restoration. However, it is very difficult to directly use these existing neural network algorithms for restoration in ancient book restoration tasks, and the detection performance will be greatly reduced. This is because these methods were originally designed based on the characteristics of language tasks, such as machine translation. The lengths of the input sequence and the output sequence are the same and fixed, which allows the network to easily learn the correspondence between the two text modalities. As a result, the detection performance is reduced. Summary of the invention
[0008] The present invention aims to at least solve one of the technical problems in the prior art, namely, the low efficiency of the manual input method and the poor performance of the ancient book restoration method using the deep learning algorithm.
[0009] To this end, the first aspect of the present invention provides an ancient book restoration method based on a generative large language model.
[0010] A second aspect of the present invention provides an ancient book restoration system based on a generative large language model.
[0011] The present invention provides an ancient book restoration method based on a generative large language model, comprising: S1. Image enhancement is performed using a zero-reference depth curve estimation method based on deep learning; wherein a deep convolutional neural network is trained to estimate a dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is an ancient book defect image, and the output is a dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range adjustment curve is used to adjust the dynamic range of the input image pixel by pixel to obtain an enhanced image; S2, encoding pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using a plurality of pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image; S3, generating features of defective image blocks to be repaired according to the large language model; wherein, using a plurality of pixel decoders included in the visual converter neural network model to generate features of defective image blocks to be repaired according to pixel feature vectors of the defective image of the ancient book, the generated features of the defective image of the ancient book are obtained; S4, performing state space mapping on the generated features of the ancient book defect image to restore the features of the defect image; S5. Perform image text recognition on the defective image of the ancient book and the repaired defective image, and output the complete ancient book text.
[0012] The ancient book repair method based on the generative large language model according to the above technical solution of the present invention may also have the following additional technical features: In the above technical solution, step S1 includes: S11, automatically mapping the ancient book defect image to its enhanced version using a light enhancement curve, wherein the light enhancement curve is expressed as:
[0013] in, represents the light enhancement curve; Represents pixel coordinates; Indicates curve input; Represents a given input An enhanced version of It is a trainable curve parameter used to adjust the curve size and control the exposure level of the image; S12, using a zero-reference depth curve estimation method to learn the best fitting curve mapping parameters between the input ancient book defect image and its corresponding standard illumination image; and outputting a set of pixel-by-pixel curve parameters of a high-order curve corresponding to the ancient book defect image according to the input ancient book defect image, including:
[0014] in, A pixel-by-pixel curve representing a higher-order curve, represents the hyperbolic tangent activation function, represents the rectified linear unit activation function, represents the convolutional layer; S13. Use spatial consistency loss, exposure control loss, color constancy loss and lighting smoothness loss as loss functions to evaluate the quality of the enhanced image.
[0015] In the above technical solution, the design principles of the light enhancement curve include: Make each pixel of the enhanced image within the normalized interval; Make the illumination enhancement curve a monotonic curve to maintain the difference between adjacent pixels; The illumination enhancement curve learning task is made differentiable during gradient back-propagation.
[0016] In the above technical solution, step S2 includes: Initialize the pixel encoder, where the pixel encoder is a bidirectional encoding structure based on the visual converter; each pixel encoder includes a multi-head attention network and a feedforward neural network; Iterate through each encoder in turn to get the attention score; Create a multi-head attention matrix, which includes query matrix, key matrix and value matrix; Calculate the attention score based on the multi-head attention matrix; Normalize the attention score to obtain a normalized attention score; The normalized attention score is input into the feedforward neural network, and the log semantic vector, i.e., the pixel feature vector of the ancient book defect image, is output through the feedforward neural network.
[0017] In the above technical solution, step S3 includes: Initialize the pixel decoder, where the pixel decoder is a bidirectional encoding structure based on the visual transformer; each decoder includes an encoder-decoder attention network and a feedforward neural network; Traverse each decoder in turn to obtain the encoder-decoder attention score; Create an encoder-decoder attention matrix, which includes a query matrix, a key matrix, and a value matrix; According to the encoder-decoder attention matrix, calculate the encoder-decoder attention score; Normalize the encoder-decoder attention score to obtain the normalized encoder-decoder attention score; The normalized encoder-decoder attention is input into the feedforward neural network, and the generated features of the ancient book defect image are output through the feedforward neural network.
[0018] In the above technical solution, step S4 includes: S41, using a global state space mapping method to capture the long-range dependency of the image through the state space equation; wherein, for a given generation feature of an ancient book defect image, it is fed into two parallel first branch networks and second branch networks to obtain a first output result and a second output result respectively, and the first output result and the second output result are connected and then passed through a linear layer network to obtain the final output result of the global state space mapping, including:
[0019]
[0020]
[0021] in, A first output result representing the output of the first branch network; A second output result representing the output of the second branch network; Represents the normalization function LayerNorm; Represents two-dimensional pixel operations; represents the linear layer activation function; Represents a deep convolutional network; represents a linear layer network; Representing the generated features of the input ancient book defect image; represents the final output result of the global state space mapping, represents the vector connection function; The expression of the global state space mapping method is:
[0022] in, () represents the global state space mapping; S42, using the local state space mapping method to restore the defective image of the ancient book; wherein, for the final output result of the global state space mapping of the input, the normalization function LayerNorm is used for residual linking, and the global state space mapping method proposed in S41 is used to capture the long-term spatial dependency; after obtaining the long-term dependency, a convolution operation is performed, and the output result of the convolution is sent to the channel attention to enhance the semantics of the interaction between channels, and the output result is sent to the convolution layer again to help the network perform gradient propagation. The expression of the obtained output result is:
[0023] in, represents the output result of step S42; Represents the channel attention mechanism; Represents global features; represents the convolutional layer; Represents local features; S43, using a loss function to evaluate the quality of the restored ancient book image; the loss function includes:
[0024]
[0025] in, express norm, represents the hyperparameters of the loss function, Represents the loss function value.
[0026] In the above technical solution, the algorithm of the first branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create convolutional layers to feed the expanded input into a deep convolutional network; Use SiLU function to activate convolutional network; Create a two-dimensional repair function 2D-SSM, and send the linear layer output input to the 2D-SSM function; Perform LayerNorm residual connection on the output result of the 2D-SSM function to obtain the first output result; The algorithm of the second branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create a linear layer and send the expanded input to a linear layer network; Use SiLU function to activate the linear layer network; Get the second output result.
[0027] In the above technical solution, step S5 includes: S51, using the pre-trained language model to build an ancient book text recognizer; wherein, obtaining the restored ancient book image vector , a fully connected neural network is used to build a text recognition task layer on the last layer of the pre-trained language model DETR to obtain a text recognizer; S52, training and testing the text recognizer; using the trained text recognizer to perform image text recognition and output complete ancient text.
[0028] In the above technical solution, the training and testing of the text recognizer includes: S521, obtaining a weight vector and a bias term by training a fully connected neural network; S522, for the input vector of the first layer of the fully connected neural network Pooling; S523, according to the prediction results of the text recognition task layer, the cross entropy loss function is used to calculate the text recognition detection loss estimate, and the text recognition is performed on the ancient book image vector; wherein, the formulas of the forward propagation function of the fully connected network and the loss function of the fully connected network used in the calculation process are as follows:
[0029]
[0030] in, Represents the forward propagation function of the fully connected network; represents the network activation function; Represents an ancient book image vector; represents the weight vector; represents the deviation term; represents the loss function of the fully connected network; Indicates the number of samples; Indicates the sample number; Indicates The label of each sample, the positive value is 1 and the negative value is 0; Indicates The probability that a sample is predicted to be positive.
[0031] The present invention also provides an ancient book restoration system based on a generative large language model, comprising: The ancient book defect area detection unit adopts a zero-reference depth curve estimation method based on deep learning to perform image enhancement; wherein a deep convolutional neural network is trained to estimate the dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is the ancient book defect image, and the output is the dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range adjustment curve is used to adjust the dynamic range of the input image pixel by pixel to obtain an enhanced image; The ancient book defect image encoding unit encodes the pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using a plurality of pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image; The ancient book defect image generation unit generates the defect image block features to be repaired according to the large language model; wherein the defect image block features to be repaired are generated according to the pixel feature vector of the ancient book defect image using a plurality of pixel decoders included in the visual converter neural network model, thereby obtaining the generated features of the ancient book defect image; An ancient book defect image restoration unit performs state space mapping on the generated features of the ancient book defect image to restore the features of the defect image; The ancient book text recognition unit performs image text recognition on the defective images of ancient books and the repaired defective images, and outputs the complete ancient book text.
[0032] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are: The ancient book restoration method provided by the present invention can effectively improve the accuracy and efficiency of ancient book restoration, has good restoration effect, and is robust. According to the performance evaluation results, compared with the traditional intelligent restoration method, the algorithm performance of the present invention is improved by about 26%, and compared with the manual restoration method, the time efficiency is improved by more than 10 times.
[0033] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 It is a flow chart of an ancient book repair method based on a generative large language model according to an embodiment of the present invention; Figure 2 It is a structural block diagram of an ancient book restoration system based on a generative large language model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0037] Refer to the following Figure 1 and Figure 2 To describe the ancient book repair method and system based on the generative large language model provided according to some embodiments of the present invention.
[0038] Some embodiments of the present application provide an ancient book repair method based on a generative large language model.
[0039] like Figure 1 As shown, the first embodiment of the present invention proposes an ancient book repair method based on a generative large language model, including the following steps S1-S5.
[0040] S1. Image enhancement is performed using a zero-reference depth curve estimation method based on deep learning; wherein a deep convolutional neural network is trained to estimate a dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is an ancient book defect image, and the output is a dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range adjustment curve is used to adjust the dynamic range of the input image pixel by pixel to obtain an enhanced image.
[0041] Specifically, step S1 proposes an intuitive and simple nonlinear curve method to achieve ancient book image enhancement. Specifically, a zero-reference deep curve estimation method based on deep learning is used for low-light image enhancement, which can cope with diverse lighting conditions, including uneven and poor lighting cases. The main steps are: first, a lightweight deep network is trained to estimate the dynamic range adjustment curve of pixels and high-order images; the curve no longer performs image-to-image mapping, but the task is reformulated as an image-specific curve estimation problem. In particular, the method uses low-light images (ancient book defect images) as input and produces high-order curves as its output. These curves are then used to adjust the dynamic range of the input pixel by pixel to obtain an enhanced image. In this way, the range of the enhanced image is maintained to preserve the contrast of adjacent pixels. Importantly, it is differentiable, so the adjustable parameters of the curve can be learned through a deep convolutional neural network. The proposed deep network is lightweight and can be iteratively applied to approximate high-order curves for more robust and accurate dynamic range adjustment.
[0042] In some embodiments, step S1 includes steps S11 - S13 .
[0043] S11. Automatically map the ancient book defect image to its enhanced version using the illumination enhancement curve, wherein the adaptive curve parameter is the only dependent item in the input image, and the illumination enhancement curve is expressed as:
[0044] in, represents the light enhancement curve; Represents pixel coordinates; Indicates curve input; Represents a given input An enhanced version of It is a trainable curve parameter used to adjust the curve size and can control the exposure level of the image; in a specific embodiment, the illumination enhancement curve (LE curve) is applied to the three RGB channels respectively instead of just the illumination channel; the three-channel adjustment can better preserve the inherent color and reduce the risk of over-saturation.
[0045] In a specific embodiment, the design principles of the illumination enhancement curve include: Make each pixel of the enhanced image in the normalized range To avoid overflow and truncation leading to information loss; Make the illumination enhancement curve a monotonic curve to maintain the difference (i.e. contrast) between adjacent pixels; The illumination enhancement curve learning task is made differentiable during gradient back-propagation.
[0046] The illumination enhancement curve designed based on the above can increase or decrease the dynamic range of the input image. This ability is not only beneficial for enhancing low-light areas, but also for removing overexposure artifacts. Iterative application of the illumination enhancement curve formula can achieve more general adjustments to cope with challenging low-light conditions. The iterative formula of the illumination enhancement curve is as follows:
[0047] in, represents the curve iteration function, is the number of iterations used to control the curvature; in one embodiment, when The performance is optimal when the initial value of is set to 8. Indicates The curve training parameters at the iteration.
[0048] Curve iteration can adjust the image within a wider dynamic range, but it still adjusts the image globally and cannot enhance the local area. To solve this problem, It is expressed as a pixel-by-pixel parameter, that is, each pixel of a given input image has a corresponding curve, where the best fit is Parameters to adjust the dynamic range of the curve. Then, The curve iteration formula is rewritten as follows:
[0049] in, It is a parameter map with the same size as the given image; it assumes that pixels in the local area have the same intensity (and therefore the same adjustment curve), so neighboring pixels in the output still maintain a monotonic relationship.
[0050] S12. Use a zero-reference depth curve estimation method to learn the best fitting curve mapping parameters between the input ancient book defect image and its corresponding standard illumination image; and output a set of pixel-by-pixel curve parameters of a high-order curve corresponding to the ancient book defect image based on the input ancient book defect image.
[0051] In a specific embodiment, a seven-layer convolutional neural network with symmetrical connections is created, where each layer of the network consists of 32 layers of size 3 3 and a convolution kernel with a stride of 1. Then create a ReLU function to activate the network, discard the downsampling and batch normalization layers to reduce the error of adjacent pixels. Finally, use the Tanh activation function after the image passes through the convolution layer to generate 24 parameter mappings for 8 iterations (n=8), where three curve parameters map three image channels. The output is the pixel-by-pixel curve parameters of the high-order curve. The expression is as follows:
[0052] in, A pixel-by-pixel curve representing a higher-order curve, represents the hyperbolic tangent activation function, represents the rectified linear unit activation function, Represents a convolutional layer.
[0053] S13. Use spatial consistency loss, exposure control loss, color constancy loss and lighting smoothness loss as loss functions to evaluate the quality of the enhanced image.
[0054] Specifically, the calculation method is as follows:
[0055] in, is the final loss of the module, The spatial consistency is lost. To control the loss of exposure, is the loss of color constancy, is the weight of color constancy loss, is the lighting smoothness loss, The weight of the lighting smoothness loss. S2. Encode the pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using multiple pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image.
[0056] In some embodiments, step S2 includes: Initialize the pixel encoder, where the pixel encoder is a bidirectional encoding structure based on the visual converter, specifically including 12 encoders; each pixel encoder includes a multi-head attention network and a feedforward neural network; Iterate through each encoder in turn to get the attention score; Create a multi-head attention matrix, which includes query matrix, key matrix and value matrix; Calculate the attention score based on the multi-head attention matrix; Normalize the attention score to obtain a normalized attention score; The normalized attention score is input into the feedforward neural network, and the log semantic vector, i.e., the pixel feature vector of the ancient book defect image, is output through the feedforward neural network.
[0057] In a specific embodiment, step S2 includes the following steps S21-S23.
[0058] S21. Input the defect image of the ancient book into a multi-head attention network of multiple pixel encoders for attention calculation to obtain a pixel multi-head attention vector.
[0059] First, the input image goes through an embedding learning process to obtain three embedding vectors, a combination of a token embedding vector, a segment embedding vector, and a position embedding vector, expressed as:
[0060] in, is the image embedding vector, is the token embedding vector, is the position embedding vector, is the segment embedding vector.
[0061] Furthermore, a multi-head attention matrix is created to calculate the attention score of the pixel multi-head attention vector, where the pixel encoder is composed of a multi-head attention network and a feedforward neural network; the corresponding calculation formula is:
[0062]
[0063] in, represents pixel multi-head attention, represents the pixel attention connection function, Indicates Pixel attention, represents the vector size, represents the weight vector, represents the query matrix, represents the normalization function, represents the key matrix, represents the value matrix, represents the transpose of a matrix, express Hyperparameters of .
[0064] S22, normalize the attention score to obtain a normalized attention score, and the corresponding formula is:
[0065] Among them, Θ represents the attention vector after residual connection, θ represents pixel attention, and Sublayer(•) represents residual connection.
[0066] S23, input the attention vector θ after residual connection into the feed-forward neural network FFN for linear transformation to obtain the pixel feature vector of the ancient book defect image. The corresponding formula is:
[0067] in, represents the activation function of the neuron, 0 represents the gradient of the activation function, represents the weight of the first layer, represents the weight of the second layer, represents the parameters to be learned in the first layer, Represents the parameters to be learned in the second layer.
[0068] S3. Generate features of defective image blocks to be repaired based on the large language model; wherein, multiple pixel decoders included in the visual converter neural network model are used to generate features of defective image blocks to be repaired based on pixel feature vectors of the ancient book defective image, so as to obtain generated features of the ancient book defective image.
[0069] In some embodiments, step S3 includes: Initialize the pixel decoder, where the pixel decoder is a bidirectional decoding structure based on the visual converter, including 12 decoders; each decoder includes an encoder-decoder attention network and a feedforward neural network; Traverse each decoder in turn to obtain the encoder-decoder attention score; Create an encoder-decoder attention matrix, which includes a query matrix, a key matrix, and a value matrix; According to the encoder-decoder attention matrix, calculate the encoder-decoder attention score; Normalize the encoder-decoder attention score to obtain the normalized encoder-decoder attention score; The normalized encoder-decoder attention is input into the feedforward neural network, and the generated features of the ancient book defect image are output through the feedforward neural network.
[0070] In a specific embodiment, step S3 includes the following steps S31-S33.
[0071] S31, inputting the pixel feature vector of the ancient book defect image obtained in step S2 into the encoder-decoder attention network of multiple pixel decoders to perform encoder-decoder attention calculation, obtaining the pixel encoder-decoder attention vector, creating an encoder-decoder attention matrix to calculate the attention score of the pixel encoder-decoder attention vector, wherein the pixel decoder is composed of an encoder-decoder attention network and a feedforward neural network; the corresponding formula is:
[0072]
[0073] in, represents pixel encoding-decoding attention, represents the pixel encoding-decoding connection function, Indicates pixel encoder-decoder attention, represents the vector size, represents the weight vector, represents the query matrix, represents the normalization function, represents the key matrix, represents the value matrix, Represents the transpose of a matrix.
[0074] S32, normalize the attention score to obtain the normalized encoding-decoding attention score, and the corresponding formula is:
[0075] in, represents the encoding-decoding vector after residual connection, represents the pixel encoding-decoding attention vector, represents a residual connection.
[0076] S33, the encoding-decoding attention vector after residual connection Input into the feed-forward neural network FFN for linear transformation to obtain the pixel feature vector of the ancient book image. The corresponding formula is:
[0077] in, represents the activation function of the neuron, 0 represents the gradient of the activation function, represents the weight of the first layer, represents the weight of the second layer, represents the parameters to be learned in the first layer, Represents the parameters to be learned in the second layer.
[0078] S4, performing state space mapping on the generation features of the ancient book defect image to restore the features of the defect image; specifically, performing state space mapping on the generation features of the ancient book defect image generated in step S3 to enhance the quality of the generated image and achieve the restoration effect of the ancient book image.
[0079] In some embodiments, step S4 includes steps S41 - S43 .
[0080] S41. In order to maintain learning efficiency, existing restoration networks based on visual transformer neural networks usually divide the input into small blocks or use shifted window attention. However, this will reduce the interactive semantic information at the entire image level. In step S41, a global state space mapping method is used to capture the long-range dependencies of the image through the state space equation; wherein, for a given generation feature of an ancient book defect image, it enters two parallel first branch networks and second branch networks to obtain the first output result and the second output result respectively.
[0081] Specifically, the algorithm of the first branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create convolutional layers to feed the expanded input into a deep convolutional network; Use SiLU function to activate convolutional network; Create a two-dimensional repair function 2D-SSM, and send the output input of the linear layer to the 2D-SSM function; Perform LayerNorm residual connection on the output result of the 2D-SSM function to obtain the first output result; The algorithm of the second branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create a linear layer and send the expanded input to a linear layer network; Use SiLU function to activate the linear layer network; Get the second output result.
[0082] Subsequently, the first output result and the second output result are connected and passed through a linear layer network to obtain a final output result of the global state space mapping.
[0083] The expression of the global state space mapping method is as follows:
[0084] The calculation methods include:
[0085]
[0086]
[0087] in, A first output result representing the output of the first branch network; A second output result representing the output of the second branch network; Represents the normalization function LayerNorm; Represents two-dimensional pixel operations; represents the linear layer activation function; Represents a deep convolutional network; represents a linear layer network; Representing the generated features of the input ancient book defect image; represents the final output result of the global state space mapping, represents the vector connection function; S42. In the past, the block design in the restoration network based on the converter neural network mainly followed the process of "normalization → attention mechanism → normalization → multi-layer perceptron". However, although the attention mechanism can model the global dependency of pixels, it cannot restore the image very well. In step S42, the local state space mapping method is used to restore the defective image of the ancient book; wherein, for the final output result of the input global state space mapping, the normalization function LayerNorm is used for residual linking, and the global state space mapping method proposed in S41 is used to capture the long-term spatial dependency; the algorithm steps include: First, the final output result of the input global state space mapping is sent to the normalization function LayerNorm for residual link input, and then passes through a convolutional layer for local feature extraction; Using the global state space mapping method proposed in step S41 to obtain spatial long-term dependencies; Create channel attention and feed spatial long-term dependencies into channel attention to enhance the semantics of channel interaction; The output result is sent to the convolution layer again to help the network to propagate gradients; Finally, the final output of the module is obtained .
[0088] Specifically, LayerNorm is first used for residual connection, and the formula is as follows:
[0089] in, Represents global features, Indicates the output result of spatial long-term dependency. Represents the LayerNorm function.
[0090] Then, it passes through a convolutional layer for local feature extraction. The formula is as follows:
[0091] The global state space mapping method proposed in step S41 is used to obtain the spatial long-term dependency, and the formula is as follows:
[0092] in, Represents the hyperparameters of the local state-space mapping method.
[0093] Furthermore, since the flat feature map is processed as a one-dimensional labeled sequence, the number of neighborhood pixels in the sequence is mainly affected by the flattening strategy. For example, when the four-way expansion strategy is adopted, the anchor point has only four nearest neighbor pixels available, that is, some spatially close pixels in the two-dimensional feature map are far away from each other in the one-dimensional labeled sequence. This excessive distance will lead to the problem of local pixel forgetting. Therefore, a convolution operation is performed after obtaining the spatial long-term dependency to help restore the neighborhood similarity. Specifically, LayerNorm is first used to normalize the features, and then a convolutional layer is used to compensate for the local characteristics. In order to maintain efficiency, the convolutional layer adopts a bottleneck structure. The formula is as follows:
[0094] Furthermore, the output of the convolutional layer usually introduces a large number of hidden state layers to remember the long dependencies of pixels. We found that the output results have channel redundancy problems. Therefore, the output results of the convolutional layer are sent to the channel attention to enhance the semantics of channel interaction. This avoids channel redundancy. In addition, the output results are sent to the convolutional layer again to help the network perform gradient propagation; finally, the final output result of the module is obtained. The formula is as follows:
[0095] in, represents the output result of step S42; Represents the channel attention mechanism; Represents global features; represents the convolutional layer; Represents local features; S43, using a loss function to evaluate the quality of the restored ancient book image; the loss function includes:
[0096]
[0097] in, express norm, represents the hyperparameters of the loss function, Represents the loss function value.
[0098] S5. Perform image text recognition on the defective image of the ancient book and the repaired defective image, and output the complete ancient book text.
[0099] In some embodiments, step S5 includes S51 - S52 .
[0100] S51, using the pre-trained language model to build an ancient book text recognizer; wherein, obtaining the restored ancient book image vector , a fully connected neural network is used to build a text recognition task layer on the last layer of the pre-trained language model DETR to obtain a text recognizer; the formula of the text recognizer is:
[0101] in, is the final ancient book image vector, is the learned pre-training phase vector, is the learned fully connected layer vector.
[0102] S52, training and testing the text recognizer; using the trained text recognizer to perform image text recognition and output complete ancient text.
[0103] In some embodiments, step S52 includes: S521, obtaining a weight vector and a bias term by training a fully connected neural network; S522, for the input vector of the first layer of the fully connected neural network Pooling; specifically, the number of input neurons in the first layer of the fully connected neural network is 1024, and the number of output neurons is 2; S523, according to the prediction results of the text recognition task layer, the cross entropy loss function is used to calculate the text recognition detection loss estimate, and the text recognition is performed on the ancient book image vector; wherein, the formulas of the forward propagation function of the fully connected network and the loss function of the fully connected network used in the calculation process are as follows:
[0104]
[0105] in, Represents the forward propagation function of the fully connected network; represents the network activation function; Represents an ancient book image vector; represents the weight vector; represents the deviation term; represents the loss function of the fully connected network; Indicates the number of samples; Indicates the sample number; Indicates The label of each sample, the positive value is 1 and the negative value is 0; Indicates The probability that a sample is predicted to be positive.
[0106] Some other embodiments of the present invention provide an ancient book restoration system based on a generative large language model, which is applied to the ancient book restoration method proposed in any of the above embodiments, such as Figure 2 As shown, the ancient book restoration system includes: an ancient book defect area detection unit, an ancient book defect image encoding unit, an ancient book defect image generation unit, an ancient book defect image restoration unit and an ancient book text recognition unit.
[0107] The ancient book defect area detection unit adopts a zero-reference depth curve estimation method based on deep learning to perform image enhancement; wherein a deep convolutional neural network is trained to estimate the dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is the ancient book defect image, and the output is the dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range of the input image is adjusted pixel by pixel using the dynamic range adjustment curve to obtain an enhanced image; The ancient book defect image encoding unit encodes the pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using a plurality of pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image; The ancient book defect image generation unit generates the defect image block features to be repaired according to the large language model; wherein the defect image block features to be repaired are generated according to the pixel feature vector of the ancient book defect image using a plurality of pixel decoders included in the visual converter neural network model, thereby obtaining the generated features of the ancient book defect image; An ancient book defect image restoration unit performs state space mapping on the generated features of the ancient book defect image to restore the features of the defect image; The ancient book text recognition unit performs image text recognition on the defective images of ancient books and the repaired defective images, and outputs the complete ancient book text.
[0108] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0109] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for restoring ancient books based on a generative large language model, characterized in that: include: S1. Using a zero-reference depth curve estimation method based on deep learning to perform image enhancement; wherein, a deep convolutional neural network is trained to estimate a dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is an ancient book defect image, and the output is a dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range adjustment curve is used to adjust the dynamic range of the input image pixel by pixel to obtain an enhanced image; S2, encoding pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using a plurality of pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image; S3, generating features of defective image blocks to be repaired according to the large language model; wherein, using a plurality of pixel decoders included in the visual converter neural network model to generate features of defective image blocks to be repaired according to pixel feature vectors of the defective image of the ancient book, the generated features of the defective image of the ancient book are obtained; S4, performing state space mapping on the generated features of the ancient book defect image to restore the features of the defect image; S5. Perform image text recognition on the defective image of the ancient book and the repaired defective image, and output the complete ancient book text.
2. The ancient book restoration method based on the generative large language model according to claim 1 is characterized in that: Step S1 includes: S11, automatically mapping the ancient book defect image to its enhanced version using a light enhancement curve, wherein the light enhancement curve is expressed as: in, represents the light enhancement curve; Represents pixel coordinates; Indicates curve input; Represents a given input An enhanced version of It is a trainable curve parameter used to adjust the curve size and control the exposure level of the image; S12, using a zero-reference depth curve estimation method to learn the best fitting curve mapping parameters between the input ancient book defect image and its corresponding standard illumination image; and outputting a set of pixel-by-pixel curve parameters of a high-order curve corresponding to the ancient book defect image according to the input ancient book defect image, including: in, A pixel-by-pixel curve representing a higher-order curve, represents the hyperbolic tangent activation function, represents the rectified linear unit activation function, represents the convolutional layer; S13. Use spatial consistency loss, exposure control loss, color constancy loss and lighting smoothness loss as loss functions to evaluate the quality of the enhanced image.
3. The ancient book restoration method based on the generative large language model according to claim 2 is characterized in that: The design principles of the illumination enhancement curve include: Make each pixel of the enhanced image within the normalized interval; Make the illumination enhancement curve a monotonic curve to maintain the difference between adjacent pixels; The illumination enhancement curve learning task is made differentiable during gradient back-propagation.
4. The ancient book restoration method based on a generative large language model according to claim 1 is characterized in that: Step S2 includes: Initialize the pixel encoder, where the pixel encoder is a bidirectional encoding structure based on the visual converter; each pixel encoder includes a multi-head attention network and a feedforward neural network; Iterate through each encoder in turn to get the attention score; Create a multi-head attention matrix, which includes query matrix, key matrix and value matrix; Calculate the attention score based on the multi-head attention matrix; Normalize the attention score to obtain a normalized attention score; The normalized attention score is input into the feedforward neural network, and the log semantic vector, i.e., the pixel feature vector of the ancient book defect image, is output through the feedforward neural network.
5. The ancient book restoration method based on a generative large language model according to claim 1 is characterized in that: Step S3 includes: Initialize the pixel decoder, where the pixel decoder is a bidirectional encoding structure based on the visual transformer; each decoder includes an encoder-decoder attention network and a feedforward neural network; Traverse each decoder in turn to obtain the encoder-decoder attention score; Create an encoder-decoder attention matrix, which includes a query matrix, a key matrix, and a value matrix; According to the encoder-decoder attention matrix, calculate the encoder-decoder attention score; Normalize the encoder-decoder attention score to obtain the normalized encoder-decoder attention score; The normalized encoder-decoder attention is input into the feedforward neural network, and the generated features of the ancient book defect image are output through the feedforward neural network.
6. The ancient book restoration method based on a generative large language model according to claim 1 is characterized in that: Step S4 includes: S41, using a global state space mapping method to capture the long-range dependency of the image through the state space equation; wherein, for a given generation feature of an ancient book defect image, it is fed into two parallel first branch networks and second branch networks to obtain a first output result and a second output result respectively, and the first output result and the second output result are connected and then passed through a linear layer network to obtain the final output result of the global state space mapping, including: in, A first output result representing the output of the first branch network; A second output result representing the output of the second branch network; Represents the normalization function LayerNorm; Represents two-dimensional pixel operations; represents the linear layer activation function; Represents a deep convolutional network; represents a linear layer network; Representing the generated features of the input ancient book defect image; represents the final output result of the global state space mapping, represents the vector connection function; The expression of the global state space mapping method is: in, represents the global state space mapping; S42, using the local state space mapping method to restore the defective image of the ancient book; wherein, for the final output result of the global state space mapping of the input, the normalization function LayerNorm is used for residual linking, and the global state space mapping method proposed in S41 is used to capture the long-term spatial dependency; after obtaining the long-term dependency, a convolution operation is performed, and the output result of the convolution is sent to the channel attention to enhance the semantics of the interaction between channels, and the output result is sent to the convolution layer again to help the network perform gradient propagation. The expression of the obtained output result is: in, represents the output result of step S42; Represents the channel attention mechanism; Represents global features; represents the convolutional layer; Represents local features; S43, using a loss function to evaluate the quality of the restored ancient book image; the loss function includes: in, express norm, represents the hyperparameters of the loss function, Represents the loss function value.
7. The ancient book restoration method based on the generative large language model according to claim 6 is characterized in that: The algorithm of the first branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create convolutional layers to feed the expanded input into a deep convolutional network; Use SiLU function to activate convolutional network; Create a two-dimensional repair function 2D-SSM, and send the output input of the linear layer to the 2D-SSM function; Perform LayerNorm residual connection on the output result of the 2D-SSM function to obtain the first output result; The algorithm of the second branch network includes: The generated features of the given ancient book defect image The feature channel is expanded to A linear layer of represents the convolution channel, Indicates the predefined channel expansion factor; Create a linear layer and send the expanded input to a linear layer network; Use SiLU function to activate the linear layer network; Get the second output result.
8. The ancient book restoration method based on a generative large language model according to claim 1 is characterized in that: Step S5 includes: S51, using the pre-trained language model to build an ancient book text recognizer; wherein, obtaining the restored ancient book image vector , a fully connected neural network is used to build a text recognition task layer on the last layer of the pre-trained language model DETR to obtain a text recognizer; S52, training and testing the text recognizer; using the trained text recognizer to perform image text recognition and output complete ancient text.
9. The ancient book restoration method based on the generative large language model according to claim 8 is characterized in that: The training and testing of the text recognizer includes: S521, obtaining a weight vector and a bias term by training a fully connected neural network; S522, for the input vector of the first layer of the fully connected neural network Perform pooling; S523, according to the prediction results of the text recognition task layer, the cross entropy loss function is used to calculate the text recognition detection loss estimate, and the text recognition is performed on the ancient book image vector; wherein, the formulas of the forward propagation function of the fully connected network and the loss function of the fully connected network used in the calculation process are as follows: in, Represents the forward propagation function of the fully connected network; represents the network activation function; Represents an ancient book image vector; represents the weight vector; represents the deviation term; represents the loss function of the fully connected network; Indicates the number of samples; Indicates the sample number; Indicates The label of each sample, the positive value is 1 and the negative value is 0; Indicates The probability that a sample is predicted to be positive.
10. An ancient book restoration system based on a generative large language model, characterized in that: include: The ancient book defect area detection unit adopts a zero-reference depth curve estimation method based on deep learning to perform image enhancement; wherein a deep convolutional neural network is trained to estimate the dynamic range adjustment curve of pixels and images, the input of the deep convolutional neural network is the ancient book defect image, and the output is the dynamic range adjustment curve, the adjustable parameters of the dynamic range adjustment curve are learned by the deep convolutional neural network, and the dynamic range of the input image is adjusted pixel by pixel using the dynamic range adjustment curve to obtain an enhanced image; The ancient book defect image encoding unit encodes the pixel features of the ancient book defect image according to the large language model; wherein the ancient book defect image is encoded using a plurality of pixel encoders included in the visual converter neural network model to obtain a pixel feature vector of the ancient book defect image; The ancient book defect image generation unit generates the defect image block features to be repaired according to the large language model; wherein the defect image block features to be repaired are generated according to the pixel feature vector of the ancient book defect image using a plurality of pixel decoders included in the visual converter neural network model, thereby obtaining the generated features of the ancient book defect image; An ancient book defect image restoration unit performs state space mapping on the generated features of the ancient book defect image to restore the features of the defect image; The ancient book text recognition unit performs image text recognition on the defective images of ancient books and the repaired defective images, and outputs the complete ancient book text.
Citation Information
Patent Citations
Character recognition method and device and electronic equipment
CN110084172A
Image restoration method and device and computer readable medium
CN118014903A
Image processing method and apparatus, portrait super-resolution reconstruction method and apparatus, and portrait super-resolution reconstruction model training method and apparatus, electronic device, and storage medium
WO2022057837A1
Cited By
Ancient book intelligent identification and restoration method and device based on machine learning
CN119888234A
Ancient book repair system based on adaptive damage modeling and diffusion inversion, construction method and application thereof
CN122347527A
