Model training method and device, image super-resolution method, and storage medium

Abstract

The embodiment of the application provides a model training method and device, an image super-resolution method and a storage medium, and belongs to the technical field of image processing. The model training method comprises the following steps: obtaining a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than that of the low-quality image; inputting the low-quality image into an image super-resolution network model to obtain a super-resolution image; inputting the super-resolution image and the target image into a text detection network model respectively to obtain a super-resolution image feature and a target image feature corresponding to the text content; calculating a loss value of the image super-resolution network model based on the super-resolution image feature and the target image feature, and iteratively training the image super-resolution network model according to the loss value. The application can better preserve and enhance text details, improve the text readability of a screen content image, and avoid distortion of the screen content image after super-resolution processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training method and apparatus, an image super-resolution method, and a storage medium. Background Technology

[0002] Image super-resolution (ISR) technology based on deep learning can reconstruct high-resolution (HR) images from low-resolution (LR) images, thereby improving the detail and sharpness of the images by increasing their resolution.

[0003] Traditional image super-resolution methods are primarily designed for raw natural scenes. These super-resolution models are trained on natural images and lack sufficient understanding of the characteristics of screen content images, failing to capture textual information within them. When using traditional image super-resolution models to reconstruct screen content images, text details cannot be preserved or enhanced, leading to image distortion. Summary of the Invention

[0004] This application provides a model training method and apparatus, an image super-resolution method, and a storage medium, which can solve the problem that "traditional image super-resolution methods cannot preserve and enhance text details when super-reconstructing screen content images, thus leading to image distortion." To achieve the above objectives, the technical solutions provided by this application are as follows:

[0005] In a first aspect, embodiments of this application provide a model training method, including: acquiring a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than the resolution of the low-quality image;

[0006] The low-quality image is input into the image super-resolution network model to obtain the super-resolution image;

[0007] The super-resolution image and the target image are respectively input into the text detection network model to obtain the super-resolution image features and the target image features corresponding to the text content;

[0008] Based on the super-resolution image features and the target image features, the loss value of the image super-resolution network model is calculated, and the image super-resolution network model is iteratively trained according to the loss value.

[0009] Secondly, embodiments of this application provide an image super-resolution method, including:

[0010] The screen content image to be processed is input into the trained image super-resolution network model to obtain the super-resolution image output by the image super-resolution network model; wherein, the image super-resolution network model is trained according to the model training method described in the first aspect.

[0011] Thirdly, embodiments of this application provide a model training apparatus, including:

[0012] An image acquisition module is used to acquire a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than the resolution of the low-quality image;

[0013] The super-resolution module is used to input the low-quality image into the image super-resolution network model to obtain a super-resolution image;

[0014] The super-resolution image and the target image are respectively input into the text detection network model to obtain the super-resolution image features and the target image features corresponding to the text content;

[0015] The iterative training module is used to calculate the loss value of the image super-resolution network model based on the super-resolution image features and the target image features, and to perform iterative training on the image super-resolution network model based on the loss value.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method as described in the first aspect.

[0017] In the embodiments provided in this application, the training method for the image super-resolution network model has been improved. This training method extracts text features from the super-resolution image generated by the image super-resolution network model and the high-quality target image using a text detection network model, and then calculates the loss value between these text features. Subsequently, based on this loss value, the image super-resolution network model is iteratively trained, guiding it to generate a super-resolution image that is perceptually closer to the target image. Therefore, the image super-resolution network model provided in this application can better preserve and enhance text details during super-resolution processing of screen content images, improving the text readability of the screen content images and avoiding distortion of the screen content images after super-resolution processing.

[0018] Furthermore, the target image can be obtained by bolding the font in the original ground truth image. It's understandable that bolding text makes it more eye-catching and improves readability. Therefore, using the bolded screen content image as the target image to train the super-resolution model can effectively avoid the phenomenon of font thinning after super-resolution, while also better suppressing text edge artifacts.

[0019] Furthermore, low-quality images can be obtained by downsampling and compressing the original ground truth image. This process simulates low-resolution images in real-world applications and the image quality loss caused by compression, making the trained model more adaptable to super-resolution scenarios involving compressed screen content images.

[0020] Furthermore, the FAST text detection network model extracts shallow features such as edges and textures as well as deep features such as semantic information from images. This fully integrates information from different levels, making the extracted features more comprehensive, rich, and representative. This allows the image super-resolution network model to have better adaptability and robustness when facing complex and ever-changing real-world data, enabling it to better improve the perceived quality of text.

[0021] In practice, other loss functions, such as L1 Loss and GAN Loss, can be combined to iteratively train the image super-resolution network model. L1 Loss effectively avoids gradient explosion and improves the model's generalization ability; GAN Loss captures and enhances textures and edges in the image, thereby enhancing the realism and detail of the generated super-resolution image. Thus, combining multiple loss functions can further improve the model's performance and visual effects.

[0022] Furthermore, during the training of image super-resolution network models, stochastic gradient descent algorithm can be used to train the model, thereby accelerating the training process and improving model training efficiency.

[0023] The model can also be evaluated periodically, and the training strategy can be adjusted based on the evaluation results. This allows for timely identification of problems that arise during model training, enabling adjustments to the training strategy and further improving model performance. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram illustrating the principle of a model training method provided in an embodiment of this application;

[0027] Figure 3A schematic diagram illustrating the principle of another model training method provided in this application embodiment;

[0028] Figure 4 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation

[0029] Image quality can be degraded during the image acquisition, storage, and transmission stages. For example, poor performance of the image acquisition equipment directly affects the quality of the acquired image, potentially resulting in blurriness and noise. Furthermore, image compression can be used to save transmission bandwidth and storage space. This process significantly reduces image file size, thus saving storage space and accelerating data transmission, but it may lead to the loss or blurring of some image details.

[0030] Screen content images (SCIs) are frequently used in scenarios such as multi-screen interaction, online meetings, and online education. As an important carrier of information, SCIs often contain a large amount of text. This text carries the key information of the image, and its clarity and readability directly affect the effectiveness of information transmission; it may be difficult to read due to low image resolution or other reasons.

[0031] For example, if the resolution of a screen content image is lower than that of the smart interactive screen, the screenshot image will appear pixelated or blurry when displayed on the smart interactive screen, resulting in poor text readability of the screen content image.

[0032] For example, due to limited transmission bandwidth and storage space, as well as the high demand for low-latency communication, screen content images are usually transmitted at a low bit rate, which causes the screen content images transmitted to the receiver to be damaged due to compression, which also makes the text readability of the screen content images poor.

[0033] Image super-resolution technology can reconstruct a high-resolution image from a low-resolution image, improving its detail and clarity by increasing the image's resolution. Therefore, theoretically, by performing super-resolution reconstruction on screen content images, a high-resolution image of the screen content can be obtained, thereby improving the readability of the text within it.

[0034] However, traditional image super-resolution methods are primarily designed for raw natural scenes and lack sufficient understanding of the characteristics of screen content images, failing to capture textual information within them. For example, traditional image super-resolution network models can be pre-trained on real natural images using the ImageNet visualization database. This pre-trained model focuses on general image feature extraction, but its ability to capture textual details and semantic information is relatively weak. There are significant statistical differences between natural images and screen content images; natural images tend to follow a Gaussian distribution, while the distribution of screen content images varies greatly and lacks a clear pattern. Consequently, when applying this pre-trained image super-resolution network model to screen content image super-resolution tasks, it fails to preserve or enhance textual details and may even exacerbate or produce unexpected image distortion.

[0035] Based on this, this application proposes a model training method. This method extracts text features from the super-resolution image generated by the image super-resolution network model and the high-quality target image through a text detection network model, and then calculates the loss value between these text features. Subsequently, the image super-resolution network model is iteratively trained based on this loss value, guiding it to generate a super-resolution image that is perceptually closer to the target image. Therefore, the image super-resolution network model provided in this application can better preserve and enhance text details during super-resolution processing of screen content images, improve the readability of the text in the screen content images, and avoid distortion of the screen content images after super-resolution processing.

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0037] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0038] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] In the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] Figure 1 The model training method disclosed in this application, which can be applied to any electronic device, includes the following steps:

[0041] S102, acquire the target image and the low-quality image.

[0042] The target image and the low-quality image contain text content, and the target image has a higher resolution than the low-quality image.

[0043] When training the model, it is necessary to use pairs of low-quality images (LQ images) and high-quality images (HQ images), that is, to use pairs of low-resolution images and high-resolution images as input data and output targets for training the super-resolution network model.

[0044] In one embodiment, a large number of high-quality ground truth images (GT images) can be prepared before training the model, such as 30,000 4K ultra-high-definition ground truth images, and these images can be added to the ground truth dataset. This ground truth dataset can include image samples with sharp edges, such as screen content images. It can also include image samples with homogeneous regions, such as natural images. Sharp edges can include text edges, object edges, color change edges, etc., and homogeneous regions can include regions with continuous identical textures or continuous smooth regions. Subsequently, the training dataset for the model is constructed based on the ground truth dataset, ensuring the diversity and representativeness of the training data, thereby enabling the model to learn a wider range of features during training and improving its generalization ability.

[0045] It is worth mentioning that the number of ground truth images in the ground truth dataset can be set according to the actual application scenario and business needs, and this application does not impose any restrictions on it.

[0046] In practice, the data collectors can subjectively judge whether the ground truth image samples have sharp edges, and collect ground truth image samples with sharp edges. For example, screenshots of PPT documents with different styles and languages, screenshots of Word documents, and screenshots of the display device desktop, file content, and web page content can all be collected manually.

[0047] In another embodiment, to improve the efficiency of ground truth image acquisition, ground truth image samples with sharp edges can also be automatically acquired by a computer device. For example, the computer device can automatically read various types of files and take screenshots, and use edge detection algorithms to identify ground truth image samples with sharp edges.

[0048] In one embodiment, a low-quality image can be obtained by downsampling and compressing the aforementioned ground truth image.

[0049] In practice, a low-quality image can be obtained by downsampling and compressing the original ground truth image before training the model. For example, downsampling the ground truth image by a factor of 2 and compressing it using H.264 or HEVC (High Efficiency Video Coding, or H.265) encoding will reduce the width and height of the resulting low-quality image to half that of the original ground truth image. Downsampling simulates low-resolution images in real-world applications, while compression simulates the image quality loss caused by compression, making the trained model more adaptable to super-resolution scenarios involving compressed screen content images.

[0050] It is worth mentioning that the downsampling factor and compression encoding method can be set according to the actual application scenario and business needs, and this application does not impose any restrictions on them.

[0051] S104: Input the low-quality image into the image super-resolution network model to obtain the super-resolution image (SR image).

[0052] In implementation, the image super-resolution network model can be any deep learning-based image super-resolution network model, such as SRCNN (Super-Resolution Convolutional Neural Network) or SRGAN (Super-Resolution Generative Adversarial Network), and this application does not impose any restrictions on it.

[0053] S106, Input the super-resolution image and the target image into the text detection network model respectively to obtain the super-resolution image features and target image features corresponding to the text content.

[0054] In implementation, both the target image and the low-quality image contain text content. Accordingly, in step S104, the super-resolution image obtained by inputting the low-quality image into the image super-resolution network model also contains this text content. Subsequently, based on the super-resolution image containing the text content and the target image, the text detection network model can output super-resolution image features and target image features corresponding to the text content, respectively.

[0055] See Figure 2 In one embodiment, the target image can be any of the ground truth images mentioned above, and the low-quality image can be obtained by downsampling and compressing the ground truth image. Therefore, the low-quality image and the corresponding ground truth image serve as the input data and output target for training the super-resolution network model.

[0056] In this embodiment, the super-resolution image obtained in step S104 and the corresponding ground truth image are input into the text detection network model to obtain super-resolution image features corresponding to the super-resolution image and target image features corresponding to the ground truth image. Then, during step S108, the image super-resolution network model can be iteratively trained based on the loss value between the super-resolution image features and the target image features, guiding the image super-resolution network model to generate a super-resolution image that is perceptually closer to the ground truth image.

[0057] See Figure 3 In another embodiment, the target image can be a pseudo-ground truth image (Pseudo GT image) obtained by bolding the font of any of the aforementioned ground truth images. The low-quality image can be obtained by downsampling and compressing the ground truth image. Therefore, the low-quality image and the corresponding pseudo-ground truth image serve as the input data and output target for training the super-resolution network model.

[0058] In this embodiment, a pseudo-ground image can be obtained by bolding the font of the original ground truth image before training the model. The super-resolution image obtained in step S104 and the corresponding pseudo-ground image are then input into the text detection network model to obtain super-resolution image features corresponding to the super-resolution image and target image features corresponding to the pseudo-ground image. Furthermore, during step S108, the image super-resolution network model can be iteratively trained based on the loss value between the super-resolution image features and the target image features, guiding the image super-resolution network model to generate a super-resolution image that is perceptually closer to the pseudo-ground image.

[0059] It is worth mentioning that a deep learning-based text bolding network model, morphological methods, or sharpening techniques can be used to bold the text in the original ground truth image. Of course, other methods can also be used to bold the text in the original ground truth image; this application does not limit the method of bolding text in the ground truth image.

[0060] It's understandable that bolding text makes it more eye-catching and improves readability. Furthermore, using the bolded screen content image as the target image to train the super-resolution model can effectively avoid the phenomenon of thinner fonts after super-resolution, while also better suppressing text edge artifacts.

[0061] S108: Based on the super-resolution image features and the target image features, calculate the loss value of the image super-resolution network model, and iteratively train the image super-resolution network model according to the loss value.

[0062] In implementation, a pre-set batch size, such as 64, can be used to train the image super-resolution network model iteratively multiple times. Each iteration calculates the model loss through forward propagation and updates the model parameters through backpropagation until the model reaches a pre-set stopping condition.

[0063] For example, when the target image is a ground truth image, 64 ground truth images and corresponding low-quality images can be obtained in each training iteration. Then, steps S104-S108 are executed sequentially to guide the image super-resolution network model to generate super-resolution images that are perceptually closer to the ground truth images. This allows the trained image super-resolution network model to better preserve and enhance the text details of the screen content image, improve the text readability of the screen content image, and avoid distortion of the screen content image after super-resolution processing.

[0064] For example, when the target image is a pseudo-ground value image, 64 pseudo-ground value images and corresponding low-quality images can be obtained in each training iteration. Then, steps S104-S108 are executed sequentially to guide the image super-resolution network model to generate super-resolution images that are perceptually closer to the pseudo-ground value images. This allows the trained image super-resolution network model to effectively avoid the phenomenon of thinner fonts after super-resolution and to better suppress text edge artifacts.

[0065] It is worth mentioning that the pre-set stopping condition can be minimizing the model loss value, or the model loss stabilizing, meaning the loss of the image super-resolution network model basically no longer decreases, or reaching a preset number of iterations. Of course, other conditions can also be used to determine when the image super-resolution network model has completed training, and this application does not impose any restrictions on this.

[0066] In one embodiment, the process of calculating the loss value of the image super-resolution network model based on super-resolution image features and target image features may specifically include: obtaining super-resolution image features and target image features of multiple preset network layers, and calculating the difference value between the super-resolution image features and target image features of each preset network layer using the square of the L2 norm; dividing the difference value by the feature map size of the corresponding preset network layer to obtain the average difference value; and summing the average difference values ​​corresponding to multiple preset network layers to obtain the loss value.

[0067] For example, loss functions used to calculate the loss value of an image super-resolution network model include:

[0068]

[0069] For ease of description, the loss function provided in this embodiment can be called Screen Content Image Perceptual Loss (SCIs Perceptual Loss, or SCIs Perceptual Loss for short).

[0070] Where N represents the number of preset network layers in the text detection network model, including shallow feature layers and deep feature layers; x represents the super-resolution image, and φ... i (x) represents the feature map extracted from the super-resolution image at the i-th layer of the preset network; y represents the target image, φ i (y) represents the feature map extracted from the target image at the i-th layer of the preset network; C i H i W i These represent the number of channels, height, and width of the feature map at layer i, respectively. The square of the L2 norm can be used to measure the distance between two feature maps. In other words, it measures the difference between the super-resolution image generated by the super-resolution network model and the target image at the perceptual level by calculating the difference between the feature maps.

[0071] In one embodiment, the text detection network model can be Faster Arbitrarily-Shaped TextDetector with Minimalist Kernel Representation, abbreviated as FAST network. It can mainly consist of a backbone, a neck, and a head. The backbone of the FAST text detection network is divided into four stages: stage1, stage2, stage3, and stage4. The output layers of each of stages1, 2, 3, 4, and the head can be selected as preset network layers for feature extraction, in which case the number of preset network layers N is 5. The preset network layers encompass both shallow and deep networks. Shallow networks can capture some basic, local feature information of the data, such as simple but crucial visual elements like edges and textures in images. These features are extremely important for constructing the basic framework of complex features. Deep networks, on the other hand, can mine more abstract, semantic features from the data, such as identifying specific object categories and scene types in image recognition tasks. By incorporating feature extraction from both shallow and deep networks, information from different levels can be fully integrated, resulting in more comprehensive, richer, and more representative image features. The SCIs perceptual loss built upon the FAST network can guide the image super-resolution network model to generate results that are perceptually closer to the target image. Furthermore, the image super-resolution network model exhibits better adaptability and robustness when facing complex and varied real-world data, enabling it to significantly improve the perceptual quality of text.

[0072] In another embodiment, a loss function can be constructed based on a pre-trained convolutional neural network VGG (Very Deep Convolutional Networks for Large-scale Image Recognition). VGG is pre-trained on the real natural image dataset ImageNet and mainly focuses on general image feature extraction, but its ability to capture details and semantic information such as text is relatively weak.

[0073] Table 1 shows the quantitative comparison of the performance of image super-resolution network models trained using SCIs perceptual loss and VGG perceptual loss. The metrics include PSNR (Peak Signal to Noise Ratio), NIQE (NaturalnessImageQuality Evaluator), HyperIQA (HyperIQA), MUSIQA (Multi-scale Image Quality Transformer), CLIP-IQA (Contrastive Language Image Pretraining-Image Quality Assessment), and LIQE (Language-Image Quality Evaluator).

[0074] Table 1: Quantitative comparison results of training using SCIs-perceptual loss and training using VGG-perceptual loss.

[0075]

[0076] Based on the quantitative comparison results of the various evaluation indicators shown in Table 1, it can be seen that when applying the image super-resolution network model to the super-resolution scenario of screen content images, the SCIs perceptual loss proposed in this application has the following advantages compared with the VGG-based perceptual loss:

[0077] First, the model training method provided in this application leverages the advantages of text-related features. Specifically, the original VGG perceptual loss mainly focuses on general image feature extraction, and its ability to capture detailed and semantic information such as text is relatively weak. In contrast, this application uses the FAST network to construct the perceptual loss, which can better extract text features. Because the FAST network is specifically designed for text detection tasks, it can more sensitively capture information such as text structure, font, and font size in screen content images, thereby better preserving and enhancing these text details during the super-resolution process.

[0078] Secondly, the model training method provided in this application is more aligned with the semantics of screen content images. Specifically, the semantics of screen content images differ significantly from those of natural scene images. Natural scene images focus more on the recognition of objects and scenes, while the semantic focus of screen content images is on interface elements, text, and icons. The FAST network can better adapt to the semantic environment of screen content images, making the perceptual loss more aligned with the semantic needs of screen content images when measuring image differences. In image super-resolution tasks, it can guide the generated high-resolution images to be semantically more consistent with the actual requirements of screen content, such as ensuring the clarity and readability of text and the accuracy of interface elements.

[0079] Furthermore, the model training method provided in this application can improve the specificity of the super-resolution effect. VGG perceptual loss, when processing screen content images for super-resolution, may overemphasize some general image features unrelated to the screen content, resulting in ineffective optimization of key parts such as text and interface elements in the super-resolution image. This application uses a perceptual loss constructed from a FAST network specifically for text detection, which can more effectively improve the super-resolution quality of important parts in screen content images. For example, it performs better for text parts in screen content images, utilizing text features extracted by the FAST network to more finely recover details such as strokes and edges, making the text clearer and sharper after super-resolution processing. VGG perceptual loss, on the other hand, may blur or incorrectly process these text details.

[0080] It is worth mentioning that this application can also use other text detection network models to extract features from screen content images, and this application does not limit the type of text detection network model.

[0081] In one embodiment, the loss function used to calculate the loss value of the image super-resolution network model further includes the L1 loss function and / or the GAN loss function.

[0082] In practice, in addition to the SCIs perceptual loss proposed in this application, other loss functions can be combined to iteratively train the image super-resolution network model. This combination of multiple loss functions can further improve the model's performance and visual effects.

[0083] For example, image super-resolution network models can be iteratively trained using SCIs perceptual loss combined with Mean Absolute Error (MAE or L1 Loss). Combining L1 Loss effectively avoids gradient explosion and improves the model's generalization ability.

[0084] The L1 Loss can be calculated as follows:

[0085]

[0086] Among them, y i It is the target image. This refers to a super-resolution image, where the number of image samples used in the current iteration is n. For example, a combination of SCIs perceptual loss and Generative Adversarial Network Loss (GAN Loss) can be used to iteratively train the super-resolution model. Combining GAN Loss can capture and enhance texture and edges in the image, thereby enhancing the realism and detail of the generated super-resolution image.

[0087] In implementation, a Generative Adversarial Network (GAN) consists of a generator and a discriminator. The generator aims to generate data that is as realistic as possible to "fool" the discriminator, while the discriminator aims to distinguish between real data and generator-generated data as accurately as possible. Accordingly, GAN Loss mainly includes generator loss and discriminator loss. The formula for calculating GAN Loss is shown below:

[0088]

[0089] Among them, L D D(x) is the discriminator loss, D(x) is the output probability of the discriminator for the real data, D(G(z)) is the output probability of the discriminator for the data generated by the generator, and log is the natural logarithm function.

[0090] In one embodiment, step S108, which iteratively trains the image super-resolution network model based on the loss value, may include: adjusting the model parameters of the image super-resolution network model based on the stochastic gradient descent algorithm to minimize the loss value.

[0091] In practice, the Stochastic Gradient Descent (SGD) algorithm iteratively updates the model parameters to minimize the loss function. In each iteration, SGD randomly selects one or a small batch of training image samples to calculate the gradient of the loss function with respect to the model parameters, and uses this gradient to update the model parameters. This significantly reduces computation, thereby accelerating the model training process and improving training efficiency.

[0092] In one embodiment, the model training method provided in this application may further include: performing image quality assessment on the image super-resolution network model at a preset period, and adjusting the training strategy of the image super-resolution network model based on the assessment results. The training strategy includes the learning rate and / or batch size.

[0093] During the training of the image super-resolution network model, image quality assessments can be performed periodically to measure the model's performance and reconstruction effectiveness. The training strategy can then be adjusted based on the assessment results. Image quality assessment metrics can include classic metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and any one or more of the deep learning-based metrics mentioned in Table 1, such as NIQE, HyperIQA, MUSIQA, CLIP-IQA, LIQE, and Q-Align.

[0094] Adjusting the training strategy of an image super-resolution network model based on evaluation results can include adjusting the learning rate and / or batch size. This allows for timely identification of problems encountered during model training, enabling adjustments to the training strategy and further improving model performance.

[0095] It's worth noting that the learning rate is a hyperparameter used to control the step size of model parameter updates; it determines the magnitude of parameter adjustment in each update. Batch size refers to the number of samples used in a single training iteration. Generally, a smaller learning rate allows the model to converge more accurately, while a larger learning rate shortens training time. A smaller batch size helps improve the model's adaptability to new samples and avoids overfitting, while a larger batch size reduces fluctuations in parameter updates and accelerates convergence.

[0096] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0097] For example, in one embodiment of this application, a technical feature is described: using SCIs perceptual loss combined with L1 Loss to iteratively train the image super-resolution network model. In another embodiment of this application, another technical feature is described: using SCIs perceptual loss combined with GAN Loss to iteratively train the super-resolution model. Since the above two technical features are not contradictory, those skilled in the art, after reading this application specification, should be able to conceive that an implementation method that simultaneously possesses both features is also an optional implementation method, that is, simultaneously combining SCIs perceptual loss, L1 Loss, and GAN Loss to iteratively train the image super-resolution network model.

[0098] Based on the same technical concept, this application also provides an image super-resolution method, which can be applied to any computer device. The image super-resolution method includes:

[0099] The screen content image to be processed is input into the trained image super-resolution network model to obtain the super-resolution image output by the image super-resolution network model; wherein, the image super-resolution network model is trained according to the model training method provided in any of the above embodiments.

[0100] Based on the same technical concept, this application also provides a model training device, see [link to relevant documentation]. Figure 4 It can include:

[0101] The image acquisition module is used to acquire a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than that of the low-quality image;

[0102] The super-resolution module is used to input low-quality images into the image super-resolution network model to obtain super-resolution images;

[0103] The super-resolution image and the target image are respectively input into the text detection network model to obtain the super-resolution image features and the target image features;

[0104] The iterative training module is used to calculate the loss value of the image super-resolution network model based on the super-resolution image features and the target image features, and to iteratively train the image super-resolution network model based on the loss value.

[0105] Optionally, the target image can be a ground truth image or a pseudo ground truth image obtained by bolding the font of a ground truth image.

[0106] Optionally, an image acquisition module is used to: downsample and compress the ground truth image to obtain a low-quality image.

[0107] Optionally, the processing of calculating the loss value of the image super-resolution network model based on super-resolution image features and target image features may specifically include: obtaining super-resolution image features and target image features of multiple preset network layers, and calculating the difference value between the super-resolution image features and target image features of each preset network layer using the square of the L2 norm; dividing the difference value by the feature map size of the corresponding preset network layer to obtain the average difference value; and summing the average difference values ​​corresponding to multiple preset network layers to obtain the loss value.

[0108] For example, loss functions used to calculate the loss value of an image super-resolution network model include:

[0109]

[0110] Where N represents the number of preset network layers in the text detection network model, including shallow feature layers and deep feature layers; x represents the super-resolution image, and φ... i (x) represents the feature map extracted from the super-resolution image at the i-th layer of the preset network; y represents the target image, φi (y) represents the feature map extracted from the target image at the i-th layer of the preset network; C i H i W i These represent the number of channels, height, and width of the feature map at layer i, respectively. This represents the square of the L2 norm.

[0111] Optionally, the loss function used to calculate the loss value of the image super-resolution network model may also include the L1 loss function and / or the GAN loss function.

[0112] Optional, iterative training module, used for:

[0113] The model parameters of the image super-resolution network model are adjusted based on the stochastic gradient descent algorithm to minimize the loss value.

[0114] Optionally, the iterative training module is also used for:

[0115] The image super-resolution network model is subjected to image quality evaluation at a preset period, and the training strategy of the image super-resolution network model is adjusted according to the evaluation results; wherein, the training strategy includes learning rate and / or batch size.

[0116] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the model training method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device and the model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, which will not be repeated here.

[0117] Based on the same technical concept, embodiments of this application also provide a computer device, including a processor and a memory. The memory stores a computer program adapted to be loaded by the processor and executed as in any of the above embodiments, such as the model training method or image super-resolution method. The implementation principle and desired technical effects of this computer device for model training or image processing can be found in the method embodiments, and will not be repeated here.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The software product of the model training method or image super-resolution method can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including storing a number of instructions to cause a computer device to execute the various embodiments or some parts of the embodiments.

[0119] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, The model training method includes: Acquire a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than the resolution of the low-quality image; The low-quality image is input into the image super-resolution network model to obtain the super-resolution image; The super-resolution image and the target image are respectively input into the text detection network model to obtain the super-resolution image features and the target image features corresponding to the text content; Based on the super-resolution image features and the target image features, the loss value of the image super-resolution network model is calculated, and the image super-resolution network model is iteratively trained according to the loss value.

2. The model training method as described in claim 1, characterized in that, The target image is a ground truth image, or a pseudo ground truth image obtained by bolding the font of the ground truth image.

3. The model training method as described in claim 2, characterized in that, The steps for acquiring the low-quality image include: The ground truth image is downsampled and compressed to obtain the low-quality image.

4. The model training method as described in claim 1, characterized in that, The calculation of the loss value of the image super-resolution network model based on super-resolution image features and target image features includes: The super-resolution image features and target image features of multiple preset network layers are obtained, and the difference between the super-resolution image features and target image features of each preset network layer is calculated using the square of the L2 norm. Divide the difference value by the feature map size of the corresponding preset network layer to obtain the average difference value; The loss value is obtained by summing the average difference values ​​corresponding to multiple preset network layers.

5. The model training method as described in claim 4, characterized in that, The loss function used to calculate the loss value of the image super-resolution network model also includes the L1 loss function and / or the GAN loss function.

6. The model training method as described in claim 1, characterized in that, The iterative training of the image super-resolution network model based on the loss value includes: The model parameters of the image super-resolution network model are adjusted based on the stochastic gradient descent algorithm to minimize the loss value.

7. The model training method as described in claim 1, characterized in that, The model training method also includes: The image super-resolution network model is subjected to image quality evaluation at a preset period, and the training strategy of the image super-resolution network model is adjusted according to the evaluation results; wherein, the training strategy includes learning rate and / or batch size.

8. An image super-resolution method, characterized in that, The image super-resolution method includes: The screen content image to be processed is input into the trained image super-resolution network model to obtain the super-resolution image output by the image super-resolution network model; wherein, the image super-resolution network model is trained by the model training method according to any one of claims 1-7.

9. A model training device, characterized in that, include: An image acquisition module is used to acquire a target image and a low-quality image; wherein the target image and the low-quality image contain text content, and the resolution of the target image is higher than the resolution of the low-quality image; The super-resolution module is used to input the low-quality image into the image super-resolution network model to obtain a super-resolution image; The text detection module is used to input the super-resolution image and the target image into the text detection network model respectively to obtain super-resolution image features and target image features corresponding to the text content; The iterative training module is used to calculate the loss value of the image super-resolution network model based on the super-resolution image features and the target image features, and to perform iterative training on the image super-resolution network model based on the loss value.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the model training method as described in any one of claims 1 to 7.

Images