A data enhanced text image super-resolution reconstruction method and device

Abstract

The embodiment of the application provides a data-enhanced text image super-resolution reconstruction method and device. Through weight mask mixing technology and data generation based on a generative adversarial network, the distribution space of training data is significantly expanded, the generalization ability and reconstruction quality of the model are improved, and the method has important application prospects in the field of text image super-resolution reconstruction. Through CutBlur technology, local area smoothing mixing is realized, then high-quality text image samples are generated by using a generative adversarial network, and finally the generalization ability and reconstruction performance of the model are significantly improved through the synergistic effect of the two methods.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image technology, and in particular to a data-enhanced text image super-resolution reconstruction method and apparatus. Background Technology

[0002] Text image super-resolution reconstruction (TISRR), an important branch of computer vision, aims to recover high-quality, clearly legible text content from low-resolution (LR) text images. With the development of deep learning technology, super-resolution reconstruction methods based on deep neural networks have made significant progress. However, these methods typically require a large number of low-resolution-high-resolution (HR) image pairs as training data to learn complex nonlinear mapping relationships.

[0003] In text image super-resolution reconstruction tasks, obtaining large-scale, high-quality text image datasets faces challenges such as high collection difficulty and high annotation costs. Publicly available text image datasets are often limited in size and limited in scene, making it difficult to cover the diverse text degradation conditions in the real world, such as blurring, noise, and distortion. This limitation of datasets leads to overfitting problems in trained super-resolution reconstruction models, resulting in insufficient generalization ability when faced with unseen text images.

[0004] Data augmentation, as an effective strategy to alleviate the problem of data scarcity, has been widely used in various computer vision tasks. Traditional data augmentation methods include geometric transformation, color adjustment, noise injection, etc., but these methods are too simple and difficult to effectively increase the diversity and complexity of the dataset [4]. In recent years, hybrid data augmentation methods such as CutMix have generated new training samples by combining local regions of different images. However, such methods often produce unnatural pixel value abrupt changes at the stitching boundaries, affecting the model's learning of continuous textures.

[0005] On the other hand, generative data augmentation based on Generative Adversarial Networks (GANs) can create entirely new training samples, but the quality and stability of their generation still need improvement. This is especially true in the text-image domain, where generated images need to maintain the structural integrity and semantic consistency of the text, placing higher demands on generative models.

[0006] Based on this, the present invention proposes a data augmentation method that combines hybrid and generative approaches, specifically optimized for the characteristics of text image super-resolution reconstruction tasks. Summary of the Invention

[0007] In view of the above problems, a text image super-resolution reconstruction technique is proposed to overcome or at least partially solve these problems. Addressing the issue of insufficient training dataset size and diversity in text image super-resolution reconstruction tasks, this invention proposes a method and apparatus combining hybrid data augmentation and generative data augmentation. This method significantly expands the distribution space of the training data through weighted mask mixing techniques and data generation based on generative adversarial networks, improving the model's generalization ability and reconstruction quality, and has significant application prospects in the field of text image super-resolution reconstruction.

[0008] A data-enhanced text image super-resolution reconstruction method, the method comprising:

[0009] From the original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. ;

[0010] The ESRGAN generator is pre-trained using a hybrid augmented dataset, and the generator and discriminator are trained alternately. The generator is then optimized using a loss function to obtain a well-trained generator.

[0011] The trained generator will be used to perform super-resolution reconstruction on the original low-resolution images to obtain new training image data. According to the original dataset Hybrid augmented dataset and new training image data Enhanced image data is obtained by combining image data using a hierarchical sampling strategy based on the hyperparameter α. .

[0012] Furthermore, from the original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. Previously, this also included the original dataset. The dataset is divided into training set, validation set, and test set in a ratio of 8:1:1.

[0013] Furthermore, from the original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. ;include:

[0014] The proportion of randomly selected mask areas conforms to a uniform distribution. A rectangular mask area;

[0015] Calculate adaptive smoothing parameters ,in An enhanced image is created using a smooth weight mask; where Based on the smoothing parameter, The area of ​​the mask region. The total area of ​​the image;

[0016] The enhanced images were evaluated using the Learned Perceptual Image Patch Similarity (LPIPS) metric. Low-quality samples with LPIPS scores below a threshold of 0.3 were removed, resulting in a hybrid augmentation dataset. The formula for calculating LPIPS is as follows:

[0017]

[0018] Furthermore, the randomly selected mask area ratio conforms to a uniform distribution. The rectangular mask region includes:

[0019] Two pairs of low-resolution and high-resolution images were randomly selected from the training dataset. ), denote low-resolution images as The corresponding high-definition image is Wherein, the reconstruction magnification is represented by s;

[0020] Randomly select a rectangular area in the image As a base mask, the distance variation function is calculated. The calculation formula is:

[0021] .

[0022] Furthermore, the calculation of adaptive smoothing parameters ,in Enhanced images are created using smooth weight masks; including:

[0023] Construct a bidirectional sigmoid weight mask This ensures that the weight is close to 1 in the central region inside the mask, and close to 0 in the region outside the mask far from the boundary.

[0024] Near the mask boundaries, the weights exhibit a smooth S-shaped transition, and the width of the transition band can be adjusted by parameters. Control, using this smooth weight mask, creates the enhanced image, where the weight mask is calculated using the following formula:

[0025]

[0026] The enhanced image is generated using a smooth weight mask, calculated as follows:

[0027]

[0028]

[0029] in, Represents element-wise multiplication. This indicates a downsampling operation. Indicates an upsampling operation. and These represent the pixel values ​​of the low-resolution and high-resolution images, respectively.

[0030] Furthermore, the step of sequentially pre-training the ESRGAN generator using a hybrid augmented dataset, alternately training the generator and discriminator, and optimizing it using a loss function to obtain a trained generator includes:

[0031] The learning rate was set to 0.0002, the number of training epochs was 50, and the ESRGAN generator was pre-trained using a hybrid augmentation dataset to obtain generated images;

[0032] Parameters are set using a loss function, including the characteristic modulation coefficients. , Using the Adam optimizer, the generator learning rate was set to 0.001, the discriminator learning rate to 0.004, and the total number of training rounds to 200. Based on the set parameters, the generated image was used as input data, and adversarial training was performed in an alternating training mode of training the generator once and the discriminator once, to obtain the trained generator.

[0033] Furthermore, the learning rate is set to 0.0002, the training epochs are 50, and the ESRGAN generator is pre-trained using a hybrid augmentation dataset to obtain generated images; this includes the following processing through a feature extraction front-end and an information processing backbone module:

[0034] In the front-end processing stage, initial features are acquired and feature dimensions are expanded through single-layer convolution operations. Specifically, shallow features... The visual features, obtained through the first convolutional layer, are shown in the following formula:

[0035] in, Refers to the initial convolution operation unit used to process low-resolution input images, extracting the resulting representation features. It is then fed into the network backbone for deep feature mining;

[0036] The core backbone network is constructed using cascaded residual feature aggregation units to enhance feature extraction capabilities and information transmission efficiency. It consists of N cascaded residual computation units and feature modulation coefficients; the computation formula is as follows:

[0037]

[0038] in, The mapping function representing the i-th residual calculation unit, This represents the information flow input to the i-th feature fusion module. This refers to the output characteristics generated by this module;

[0039] By reconstructing the extracted deep features First, the image passes through a convolutional layer, then undergoes resolution upscaling to obtain the generated image, as shown in the following formula:

[0040]

[0041] in, This represents a super-resolution reconstructed image. Indicates an upsampling operation. These are the convolutional layers after the main body and the convolutional layers after upsampling, respectively.

[0042] Furthermore, the parameter setting via the loss function includes:

[0043] Loss function of generator Combined with combat losses Content loss and perceived loss The calculation formula is:

[0044]

[0045] Among the losses in combat Content loss To ensure that the generated image is consistent with the real high-resolution image at the pixel level, the calculation formula is as follows: Perceived loss Based on the pre-trained VGG network, its calculation formula is as follows: , This represents the feature extraction function of the pre-trained VGG network.

[0046] Furthermore, the trained generator will be used to perform super-resolution reconstruction on the original low-resolution image to obtain new training image data. According to the original dataset Hybrid augmented dataset and new training image data By combining image data using a hierarchical sampling strategy based on the hyperparameter α, enhanced image data that can be dynamically adjusted according to the training results can be obtained. ,include:

[0047] Using a trained generator and a reconstruction formula, the original image is reconstructed. Super-resolution reconstruction is performed on low-resolution images to generate new training image data. :

[0048]

[0049] Based on the hyperparameter α, which can be dynamically adjusted according to the training results, a hierarchical sampling strategy is used for new training image data. Original image and the image after blending enhancement The three types of data are combined to obtain enhanced image data. The formula for combining image data through hierarchical sampling is:

[0050]

[0051] in, This is a hyperparameter.

[0052] A data-enhanced text-image super-resolution reconstruction apparatus, the apparatus comprising:

[0053] Hybrid data preprocessing module for processing raw datasets The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. ;

[0054] The generative model training module is used to pre-train the ESRGAN generator using a hybrid augmented dataset, alternately train the generator and discriminator, and optimize the generator through a loss function to obtain a trained generator.

[0055] The dataset fusion module is used to perform super-resolution reconstruction on the original low-resolution images using a trained generator, thereby obtaining new training image data. According to the original dataset Hybrid augmented dataset and new training image data Enhanced image data is obtained by combining image data using a hierarchical sampling strategy based on the hyperparameter α. .

[0056] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements a data-enhanced text image super-resolution reconstruction method.

[0057] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a data-enhanced text image super-resolution reconstruction method.

[0058] A computer program product includes a computer program that, when executed by a processor, implements a data-enhanced text-image super-resolution reconstruction method.

[0059] The embodiments of this invention have the following advantages: By employing weighted mask mixing technology and generative adversarial network-based data generation, the distribution space of training data is significantly expanded, improving the model's generalization ability and reconstruction quality, and showing significant application prospects in the field of text image super-resolution reconstruction. CutBlur technology is used to achieve smooth blending of local regions, and then generative adversarial networks are used to generate high-quality text image samples. Ultimately, the synergistic effect of these two methods significantly improves the model's generalization ability and reconstruction performance. Attached Figure Description

[0060] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the steps of a data-enhanced text image super-resolution reconstruction method provided in some embodiments of the present invention;

[0062] Figure 2 This is a flowchart of a combined hybrid and generative text image data augmentation process for super-resolution reconstruction of text images, provided by some embodiments of the present invention.

[0063] Figure 3 This is a schematic diagram of the overall network structure of the generator of a data-enhanced text image super-resolution reconstruction method and apparatus provided in some embodiments of the present invention. Detailed Implementation

[0064] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0065] The core concept of this invention lies in proposing a data augmentation method that combines hybrid and generative approaches, based on the characteristics of text image super-resolution reconstruction tasks. This method first achieves smooth blending of local regions using CutBlur technology, then utilizes generative adversarial networks to generate high-quality text image samples, and finally, the synergistic effect of the two methods significantly improves the model's generalization ability and reconstruction performance.

[0066] Reference Figure 1 and Figure 2 The illustration shows a method for data-enhanced super-resolution reconstruction of text images according to some embodiments of the present invention, which may specifically include the following steps:

[0067] S1, from the original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. ;

[0068] S2. The ESRGAN generator is pre-trained using the hybrid augmented dataset, and the generator and discriminator are trained alternately. The generator is then optimized using the loss function to obtain the trained generator.

[0069] S3. The trained generator will be used to perform super-resolution reconstruction on the original low-resolution image to obtain new training image data. According to the original dataset Hybrid augmented dataset and new training image data Enhanced image data is obtained by combining image data using a hierarchical sampling strategy based on the hyperparameter α. .

[0070] First, the CutBlur technique is used to achieve smooth blending of local regions. Then, a generative adversarial network is used to generate high-quality text image samples. Finally, the synergistic effect of the two methods significantly improves the model's generalization ability and reconstruction performance.

[0071] This invention organically combines improved hybrid data augmentation and generative data augmentation to form a synergistic augmentation strategy. This strategy is based on the following core theories:

[0072] The principle of complementarity: Hybrid data augmentation primarily increases data diversity by reorganizing local regions of existing data, while generative data augmentation creates entirely new data samples. The two are naturally complementary in terms of coverage of the data distribution space. Hybrid methods maintain the authenticity of the original data and the consistency of local features, while generative methods expand the boundaries of the data distribution and introduce new feature combination patterns.

[0073] Progressive learning mechanism: First, the initial training set is expanded through hybrid data augmentation to provide a richer training foundation for the generative model. Then, the trained generative model is used to further expand the data distribution, forming a progressive data augmentation method.

[0074] In some embodiments of the present invention, step S1 involves retrieving data from the original dataset. The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. Previously, this also included the original dataset. The dataset is divided into training set, validation set, and test set in a ratio of 8:1:1.

[0075] The above step S1 starts from the original dataset. The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. ;include:

[0076] The proportion of randomly selected mask areas conforms to a uniform distribution. A rectangular mask area;

[0077] Calculate adaptive smoothing parameters ,in An enhanced image is created using a smooth weight mask; where Based on the smoothing parameter, The area of ​​the mask region. This represents the total area of ​​the image. This allows the smoothing intensity to be adaptively adjusted based on the size of the mask region.

[0078] The enhanced images were evaluated using the Learned Perceptual Image Patch Similarity (LPIPS) metric. Low-quality samples with LPIPS scores below a threshold of 0.3 were removed, resulting in a hybrid augmentation dataset. The formula for calculating LPIPS is as follows:

[0079]

[0080] The above step S2 involves pre-training the ESRGAN generator using a hybrid augmentation dataset, alternating between training the generator and discriminator, and optimizing the generator using a loss function to obtain a trained generator. This includes setting the learning rate to 0.0002, training for 50 epochs, pre-training the ESRGAN generator using a hybrid augmentation dataset, and obtaining the generated image.

[0081] Parameters are set using a loss function, including the characteristic modulation coefficients. , Using the Adam optimizer, the generator learning rate was set to 0.001, the discriminator learning rate to 0.004, and the total number of training rounds to 200. Based on the set parameters, the generated image was used as input data, and adversarial training was performed in an alternating training mode of training the generator once and the discriminator once, to obtain the trained generator.

[0082] In step S3 above, the trained generator will be used to perform super-resolution reconstruction on the original low-resolution image to obtain new training image data. According to the original dataset Hybrid augmented dataset and new training image data By combining image data using a hierarchical sampling strategy based on the hyperparameter α, enhanced image data that can be dynamically adjusted according to the training results can be obtained. ,include:

[0083] Using a trained generator and a reconstruction formula, the original image is reconstructed. Super-resolution reconstruction is performed on low-resolution images to generate new training image data. : ;

[0084] Based on the hyperparameter α, which can be dynamically adjusted according to the training results, a hierarchical sampling strategy is used for new training image data. Original image and the image after blending enhancement The three types of data are combined to obtain enhanced image data. The formula for combining image data through hierarchical sampling is:

[0085] ;in, This is a hyperparameter.

[0086] For example, a data-augmented text image super-resolution reconstruction method employs a three-stage training strategy by combining hybrid and generative data augmentation methods. The relevant algorithm pseudocode is shown in Table 1:

[0087] 1. Phase One: Hybrid Data Preprocessing; First, the dataset is split: the original dataset is split into... The images were divided into training, validation, and test sets in an 8:1:1 ratio. Then, CutBlur enhancement was performed, applying an improved CutBlur method to each image pair in the training set to generate enhanced samples. The specific process was as follows: First, a rectangular mask region was randomly selected, with the mask area ratio conforming to a uniform distribution. Then calculate the adaptive smoothing parameters. ,in Finally, enhanced image pairs are created using a smooth weight mask. To perform quality screening on the enhanced images, this invention uses the Learned Perceptual Image Patch Similarity (LPIPS) metric to evaluate the quality of the enhanced images, removing low-quality samples with LPIPS scores below a threshold of 0.3. The LPIPS calculation formula is as follows:

[0088]

[0089] 2. Phase Two: Generative Model Training; The first phase is the pre-training phase, using a hybrid augmented dataset. The ESRGAN generator was pre-trained for 50 epochs with a learning rate of 0.0002. Then came the adversarial training phase, where the generator and discriminator were trained alternately, with the generator trained once and the discriminator trained once. Loss function weight settings: , The Adam optimizer is used, and the generator learning rate is... The discriminator learning rate is 0.004, and the total number of training rounds is 200.

[0090] 3. Phase Three: Dataset Fusion; Using the trained generator, perform super-resolution reconstruction on the original low-resolution images to generate new training data pairs: Then, a hierarchical sampling strategy is used to combine the original images. Image after blending enhancement Images generated using hybrid image enhancement training Three types of data: ;in, These are hyperparameters that can be manually set initially and dynamically adjusted based on training results.

[0091] In some embodiments of the present invention, the proportion of randomly selected mask area conforms to a uniform distribution. The rectangular mask region includes: two pairs of low-resolution and high-resolution images randomly selected from the training dataset ( ), denote low-resolution images as The corresponding high-definition image is Wherein, the reconstruction magnification is represented by s;

[0092] Randomly select a rectangular area in the image As a base mask, the distance variation function is calculated. The calculation formula is: .

[0093] The calculation of adaptive smoothing parameters ,in Enhanced images are created using smooth weight masks; including:

[0094] Construct a bidirectional sigmoid weight mask This ensures that the weight is close to 1 in the central region inside the mask, and close to 0 in the region outside the mask far from the boundary.

[0095] Near the mask boundaries, the weights exhibit a smooth S-shaped transition, and the width of the transition band can be adjusted by parameters. Control, using this smooth weight mask, creates the generated enhanced image (new training sample), where the weight mask is calculated using the following formula: Enhanced images (new training samples) are generated using a smooth weight mask, calculated as follows:

[0096] ;

[0097] ;in, Represents element-wise multiplication. This indicates a downsampling operation. Indicates an upsampling operation. and These represent the pixel values ​​of the low-resolution and high-resolution images, respectively.

[0098] The learning rate was set to 0.0002, and the training epochs were 50. The ESRGAN generator was pre-trained using a hybrid augmentation dataset to obtain generated images. The process includes three functional modules: a feature extraction front-end, an information processing backbone, and an image reconstruction back-end. The following processing steps are performed:

[0099] In the front-end processing stage, initial features are acquired and feature dimensions are expanded through single-layer convolution operations. Specifically, shallow features... The visual features, obtained through the first convolutional layer, are shown in the following formula: ;in, Refers to the initial convolution operation unit used to process low-resolution input images, extracting the resulting representation features. It is then fed into the network backbone for deep feature mining;

[0100] The core backbone network is constructed using cascaded residual feature aggregation units to enhance feature extraction capabilities and information transmission efficiency. It consists of N cascaded residual computation units and feature modulation coefficients; the computation formula is as follows: ;in, The mapping function representing the i-th residual calculation unit, This represents the information flow input to the i-th feature fusion module. This refers to the output characteristics generated by this module;

[0101] By reconstructing the extracted deep features First, the image passes through a convolutional layer, then undergoes resolution upscaling to obtain the generated image, as shown in the following formula: ;in, This represents a super-resolution reconstructed image. Indicates an upsampling operation. These are the convolutional layers after the main body and the convolutional layers after upsampling, respectively.

[0102] Parameters are set using a loss function, including the generator's loss function. Combined with combat losses Content loss and perceived loss The calculation formula is:

[0103] ;

[0104] Among them, combating losses Content loss To ensure that the generated image is consistent with the real high-resolution image at the pixel level, the calculation formula is as follows: Perceived loss Based on the pre-trained VGG network, its calculation formula is as follows: , This represents the feature extraction function of the pre-trained VGG network.

[0105] It should be noted that, unless otherwise specified, some parameters appearing in the formulas of this application are generally general parameters. That is, the same parameter symbols listed in the above formulas or subsequent formulas should be understood as the same parameter. The same parameter appearing in each formula can be referred to each other, unless the parameter has other parameter descriptions, in which case the other descriptions shall prevail.

[0106] For example, based on the characteristics of text image super-resolution reconstruction tasks, this invention proposes a data augmentation method combining hybrid and generative approaches. This method first achieves smooth blending of local regions using CutBlur technology, then utilizes generative adversarial networks to generate high-quality text image samples, and finally, the synergistic effect of the two methods significantly improves the model's generalization ability and reconstruction performance. The overall process is as follows: Figure 2 As shown.

[0107] By improving the hybrid data augmentation method of CutBlur, specifically addressing the pixel value abrupt change problem at the image stitching point caused by the traditional CutMix method, this invention adopts the hybrid data augmentation method of CutBlur from the literature and designs a weight mask for the TISRR task to achieve a natural transition in local regions. The specific steps are as follows:

[0108] First, two pairs of low-resolution and high-resolution images are randomly selected from the training dataset. ), denote low-resolution images as The corresponding high-definition image is The reconstruction magnification is represented by s, and then a rectangular region is randomly selected from the image. As the base mask, the distance variation function is first calculated. The calculation formula is: Then calculate the adaptive smoothing parameters. The calculation formula is: ;in, Based on the smoothing parameter, The area of ​​the mask region. This represents the total area of ​​the image. This allows the smoothing intensity to be adaptively adjusted based on the size of the mask region.

[0109] Finally, a bidirectional sigmoid weight mask is constructed. This design ensures that the weights are close to 1 in the central region inside the mask, and close to 0 in the outer region far from the boundary. Secondly, near the mask boundary, the weights exhibit a smooth S-shaped transition, and the width of the transition band can be controlled by parameters. Control. The formula for calculating the weight mask is: ;

[0110] New training samples are generated using this smooth weight mask, calculated as follows:

[0111] ;

[0112] ;in, Represents element-wise multiplication. This indicates a downsampling operation. Indicates an upsampling operation. and These represent the pixel values ​​of the low-resolution and high-resolution images, respectively.

[0113] Specifically, to further expand the diversity of datasets, this invention designs a generative data augmentation method based on generative adversarial networks (GANs). This method employs an improved ESRGAN architecture as the generator, generating high-quality super-resolution text images through adversarial training.

[0114] The overall network structure of the generator is as follows: Figure 3 As shown, the overall model structure is derived from ESRGAN. This generative network architecture comprises three functional modules: a feature extraction front-end, an information processing backbone, and an image reconstruction back-end. In the front-end processing stage, initial features are acquired and feature dimensions are expanded through single-layer convolutional operations. The core backbone network is responsible for learning complex feature representations. This part is designed flexibly, allowing for the selection of different network structures to maximize feature utilization efficiency based on specific needs. In this invention, cascaded residual feature aggregation units are used to construct the backbone network to enhance feature extraction capabilities and information transmission efficiency. The generator's loss function incorporates adversarial loss. Content loss and perceived loss The calculation formula is:

[0115]

[0116] Among the losses in combat This loss function is designed to ensure the generated super-resolution image appears visually realistic. Content loss ensures pixel-level consistency between the generated image and the real high-resolution image; its calculation formula is as follows: The perceptual loss, based on a pre-trained VGG network, constrains the quality of the generated image from the perspective of high-level semantic features. Its calculation formula is as follows: , This represents the feature extraction function of the pre-trained VGG network.

[0117] Detailed description of the generator network model process: Shallow features The visual features, obtained through the first convolutional layer, are shown in the following formula:

[0118] ,in, Refers to the initial convolution operation unit used to process low-resolution input images, extracting the resulting representation features. The data is then fed into the network backbone for deep feature mining. The backbone architecture consists of N cascaded residual computation units and feature modulation coefficients. composition,

[0119] The calculation formula is: ;

[0120] in, The mapping function representing the i-th residual calculation unit, This represents the information flow input to the i-th feature fusion module. This represents the output characteristic generated by the module. The input to the residual unit comes from... and The element-wise accumulation results, combined with a global residual connection design, effectively reduce training complexity and improve the stability of the optimization process. Finally, the extracted deep features are reconstructed. First, the image passes through a convolutional layer, then undergoes resolution upscaling to generate the final image.

[0121] As shown in the formula below: ;

[0122] in, This represents a super-resolution reconstructed image. Indicates an upsampling operation. These are convolutional layers following the main body and convolutional layers after upsampling, respectively. The image restoration terminal consists of three key components: front-end feature integration convolution, center resolution enhancement module, and back-end detail optimization convolution. Among them, the resolution enhancement component, as the core unit of the reconstruction process, is responsible for transforming the low-dimensional feature space to the target output scale, achieving accurate magnification and detail restoration of visual content.

[0123] In a specific example, the performance test of this invention is based on the TextZoom dataset, which contains 17,367 pairs of training images and 4,373 pairs of test images. The experiment uses SRCNN and TSRN as benchmark super-resolution reconstruction networks, PSNR and SSIM as image quality evaluation metrics, and the accuracy of three text recognizers—ASTER, MORAN, and CRNN—as text recognition performance metrics.

[0124] A higher PSNR value indicates a smaller difference between two images, signifying higher quality. Given a noisy image... and an image of size m×n ,

[0125] The specific formulas for calculating PSNR and MSE are as follows:

[0126] ; ;

[0127] in, L is the maximum possible value of a pixel in the image. For RGB images, L is typically 255. Let's assume there are N pixels in the image. and These represent the grayscale or color value of the i-th pixel in the original image and the enhanced image, respectively.

[0128] The Structural Similarity Index (SSIM) is a metric that compares image quality by measuring the degree of similarity between two images, rather than simply calculating the error between them. The SSIM metric considers image brightness, contrast, and structural information during its calculation.

[0129] The SSIM calculation formula is shown below: ;

[0130] in, and These refer to the average pixel values ​​of the original image and the enhanced image, respectively. The pixel covariance between the original image and the enhanced image is a constant. and Used for stability. The SSIM metric takes values ​​between -1 and 1.

[0131] Recognition accuracy (ACC) refers to the proportion of correct predictions made by a given classification model for a given sample set. In recent years, research on text image super-resolution reconstruction has typically used open-source models such as CRNN, ASTER, and MORAN to recognize the restored SR images and calculate the recognition accuracy.

[0132] The formula for calculating ACC is as follows: ;

[0133] in, The number of correctly identified samples, The total number of samples in the test set.

[0134] like Figure 3 As shown, in the generator of the above embodiment, a relatively blurry and low-resolution original image is used as input data for the shallow feature convolutional layer. This image is then processed sequentially through the shallow feature layer, the main body layer, and the reconstruction layer (i.e., the above generator network architecture includes three functional modules: a feature extraction front-end, an information processing backbone, and an image reconstruction back-end). The processed image is then output by the convolutional layer of the reconstruction layer. A comparison between the input and output images shows that the processed image is superior to the low-resolution original image in both clarity and resolution.

[0135] The experimental results are shown in Table 2. Using the improved hybrid data augmentation alone can improve the PSNR of SRCNN by 0.07 dB and the text recognition accuracy by 0.9%. Combining hybrid and generative data augmentation methods on SRCNN achieves a PSNR improvement of 0.09 dB and a recognition accuracy improvement of 0.4%. Furthermore, on the more complex TSRN network, the combined data augmentation method improves PSNR by 0.06 dB and recognition accuracy by 0.8%. The ablation experiments verify the complementarity of hybrid and generative augmentation, demonstrating that the combination of the two methods produces a synergistic effect. In conclusion, the data augmentation method proposed in this invention can effectively improve the performance of text image super-resolution reconstruction models.

[0136] Table 1. Pseudocode of Data Augmentation Algorithms

[0137]

[0138] Table 2. Statistics of Generative Data Augmentation Experiment Results

[0139]

[0140] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0141] Some embodiments of the present invention also provide an apparatus for data-enhanced text image super-resolution reconstruction, used to implement the above-mentioned data-enhanced text image super-resolution reconstruction method, which may specifically include the following modules:

[0142] The hybrid data preprocessing module is used to obtain the training dataset as training samples from the original dataset, and to process the samples using the improved CutBlur algorithm to obtain the hybrid augmented dataset.

[0143] The generative model training module is used to pre-train the ESRGAN generator using a hybrid augmented dataset, alternately train the generator and discriminator, and optimize the generator through a loss function to obtain a trained generator.

[0144] The dataset fusion module is used to perform super-resolution reconstruction on the original low-resolution image using the trained generator to obtain new training image data. Based on the original dataset, the hybrid augmentation dataset, and the new training image data, the module combines the image data using a hierarchical sampling strategy based on the hyperparameter α to obtain augmented image data.

[0145] This invention addresses the problem of insufficient training dataset size and diversity in text image super-resolution reconstruction tasks by proposing a method and apparatus that combines hybrid data augmentation and generative data augmentation. Through weighted masking techniques and data generation based on generative adversarial networks, the distribution space of training data is significantly expanded, improving the model's generalization ability and reconstruction quality, demonstrating significant application potential in the field of text image super-resolution reconstruction.

[0146] Some embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0147] Some embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the method described above.

[0148] Some embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0149] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0156] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0157] The above provides a detailed description of the data-enhanced text image super-resolution reconstruction method and apparatus. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data-enhanced text image super-resolution reconstruction method, characterized in that, The method includes: From the original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. This includes: randomly selecting mask area ratios that conform to a uniform distribution. Rectangular mask region; calculate adaptive smoothing parameters. ,in An enhanced image is created using a smooth weight mask; where Based on the smoothing parameter, The area of ​​the mask region. The total area of ​​the image is used as the basis for quality assessment of the augmented image using a learned perceptual patch similarity metric. Low-quality samples with LPIPS scores below a threshold of 0.3 are removed to obtain the hybrid augmented dataset. The formula for calculating LPIPS is as follows: The ESRGAN generator is pre-trained using a hybrid augmented dataset, and the generator and discriminator are trained alternately. The generator is then optimized using a loss function to obtain a well-trained generator. The trained generator will be used to perform super-resolution reconstruction on the original low-resolution images to obtain new training image data. According to the original dataset Hybrid augmented dataset and new training image data Based on the hyperparameter α, a hierarchical sampling strategy is used to combine image data to obtain enhanced image data. .

2. The method according to claim 1, characterized in that, The original dataset The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. Previously, this also included the original dataset. The dataset is divided into training set, validation set, and test set in a ratio of 8:1:

1.

3. The method according to claim 1, characterized in that, The randomly selected mask area ratio conforms to a uniform distribution. The rectangular mask region includes: Two pairs of low-resolution and high-resolution images were randomly selected from the training dataset. ), denote low-resolution images as The corresponding high-definition image is Wherein, the reconstruction magnification is represented by s; Randomly select a rectangular area in the image As a base mask, the distance variation function is calculated. The calculation formula is: 。 4. The method according to claim 1, characterized in that, The calculation of adaptive smoothing parameters ,in Enhanced images are created using smooth weight masks; include: Construct a bidirectional sigmoid weight mask This ensures that the weight is close to 1 in the central region inside the mask, and close to 0 in the region outside the mask far from the boundary. Near the mask boundaries, the weights exhibit a smooth S-shaped transition, and the width of the transition band can be adjusted by parameters. Control, using this smooth weight mask, creates the generated enhanced image, where, The formula for calculating the weight mask is: , Enhanced images are generated using smooth weight masks. The calculation formula is: , , in, Represents element-wise multiplication. This indicates a downsampling operation. Indicates an upsampling operation. and These represent the pixel values ​​of the low-resolution and high-resolution images, respectively.

5. The method according to claim 1, characterized in that, The process involves sequentially pre-training the ESRGAN generator using a hybrid augmented dataset, alternately training the generator and discriminator, and optimizing the generator using a loss function to obtain a well-trained generator; including: The learning rate was set to 0.0002, the number of training epochs was 50, and the ESRGAN generator was pre-trained using a hybrid augmentation dataset to obtain generated images; Parameters are set using a loss function, including the characteristic modulation coefficients. , Using the Adam optimizer, the generator learning rate was set to 0.001, the discriminator learning rate to 0.004, and the total number of training rounds to 200. Based on the set parameters, the generated image was used as input data, and adversarial training was performed in an alternating training mode of training the generator once and the discriminator once, to obtain the trained generator.

6. The method according to claim 5, characterized in that, The learning rate was set to 0.0002, the number of training rounds was 50, and the ESRGAN generator was pre-trained using a hybrid augmentation dataset to obtain generated images; This includes the following processing through the feature extraction front-end and the main information processing module: In the front-end processing stage, initial features are acquired and feature dimensions are expanded through single-layer convolution operations. Specifically, shallow features... Obtained through the first convolutional layer, it contains visual features. As shown in the formula below: , in, Refers to the initial convolution operation unit used to process low-resolution input images, extracting the resulting representation features. It is then fed into the network backbone for deep feature mining; The core backbone network is constructed using cascaded residual feature aggregation units, consisting of N cascaded residual calculation units and feature modulation coefficients. The calculation formula is as follows: , in, The mapping function representing the i-th residual calculation unit, This represents the information flow input to the i-th feature fusion module. This refers to the output characteristics generated by this module; By reconstructing the extracted deep features First, the image passes through a convolutional layer, then undergoes resolution upscaling to obtain the generated image, as shown in the following formula: ； in, This indicates a super-resolution reconstructed image. Indicates an upsampling operation. These are the convolutional layers after the main body and the convolutional layers after upsampling, respectively.

7. The method according to claim 5, characterized in that, The parameter setting via the loss function includes: Loss function of generator Combined with combat losses Content loss and perceived loss ,in, The calculation formula is: ; Among the losses in combat Content loss To ensure that the generated image is consistent with the real high-resolution image at the pixel level, the calculation formula is as follows: Perceived loss Based on the pre-trained VGG network, its calculation formula is as follows: , This represents the feature extraction function of the pre-trained VGG network.

8. The method according to claim 1, characterized in that, The process involves using a trained generator to perform super-resolution reconstruction on the original low-resolution image, obtaining new training image data. According to the original dataset Hybrid augmented dataset and new training image data By combining image data using a hierarchical sampling strategy based on the hyperparameter α, enhanced image data that can be dynamically adjusted according to the training results can be obtained. ,include: Using a trained generator and a reconstruction formula, the original image is reconstructed. Super-resolution reconstruction is performed on low-resolution images to generate new training image data. : ; Based on the hyperparameter α, which can be dynamically adjusted according to the training results, a hierarchical sampling strategy is used for new training image data. Original image and the image after blending enhancement The three types of data are combined to obtain enhanced image data. The formula for combining image data through hierarchical sampling is: ,in, This is a hyperparameter.

9. A data-enhanced text-image super-resolution reconstruction apparatus, characterized in that, The device includes: Hybrid data preprocessing module for processing raw datasets The training dataset is obtained as training samples, and the improved CutBlur algorithm is used to process the samples to obtain a hybrid augmented dataset. This includes: randomly selecting mask area ratios that conform to a uniform distribution. Rectangular mask region; calculate adaptive smoothing parameters. ,in An enhanced image is created using a smooth weight mask; where Based on the smoothing parameter, The area of ​​the mask region. The total area of ​​the image is used as the basis for quality assessment of the augmented image using a learned perceptual patch similarity metric. Low-quality samples with LPIPS scores below a threshold of 0.3 are removed to obtain the hybrid augmented dataset. The formula for calculating LPIPS is as follows: The generative model training module is used to pre-train the ESRGAN generator using a hybrid augmented dataset, alternately train the generator and discriminator, and optimize the generator through a loss function to obtain a trained generator. The dataset fusion module is used to perform super-resolution reconstruction on the original low-resolution images using a trained generator, thereby obtaining new training image data. According to the original dataset Hybrid augmented dataset and new training image data Based on the hyperparameter α, a hierarchical sampling strategy is used to combine image data to obtain enhanced image data. .

Images