Random-resolution e-commerce detail page image super-resolution method based on Transform

Abstract

The invention belongs to the technical field of image processing, and discloses a Transform-based any-resolution e-commerce detail page image super-resolution method, which comprises the following steps of: performing feature extraction and preliminary coding on an input low-resolution e-commerce detail page image through a coder constructed by a deep neural network; carrying out further enhancement processing and global context relation modeling on the features output by the encoder by using an improved Transform module; a Transform module is added between an encoder and a decoder, a complex background and a transition area in an image are processed through the Transform module, a super-resolution model is constructed by combining the characteristics of the Transform and a convolutional neural network, supervised learning is performed by using a large-scale e-commerce image training data set, an image with any target resolution can be generated according to requirements, and the method has the advantages that the method is simple and convenient to operate, and the method is suitable for large-scale e-commerce. The flexibility and practicability of the method are enhanced, high-resolution images with richer details and clearer edges can be generated, and the visual effect of e-commerce pictures is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution. Background Art

[0002] With the rapid development of e-commerce, high-quality product images are crucial for attracting customers and increasing conversion rates. Images on e-commerce detail pages must not only clearly showcase the product's appearance and materials, but also retain rich details even after zooming in, allowing consumers to examine them more closely. However, due to limitations in camera equipment, storage, and transmission costs, the resolution of original images is often limited. Directly zooming in can easily lead to image blur, distortion, and loss of detail, severely impacting the user experience.

[0003] Traditional image super-resolution methods, such as interpolation (e.g., bilinear interpolation and bicubic interpolation), are simple and fast, but they often produce blurring and jagged effects at large magnifications, making it difficult to recover high-frequency details. In recent years, super-resolution methods based on convolutional neural networks (CNNs) have made significant progress. Deep learning models can learn the complex mapping relationship from low-resolution (LR) images to high-resolution (HR) images, significantly improving the quality of reconstructed images.

[0004] However, many existing CNN-based super-resolution methods still have some limitations:

[0005] Fixed magnification: Most models are trained and optimized for specific magnifications (such as x2, x3, x4). This lacks flexibility for output requirements of arbitrary resolutions and may require training multiple models or performing multiple upsampling operations, resulting in low efficiency.

[0006] Receptive field limitation: The convolution operation of CNN mainly focuses on local neighborhood information and has limited ability to capture long-distance dependencies and global structural information in the image, which may lead to the reconstructed image lacking texture consistency and structural integrity.

[0007] Insufficient detail recovery: When processing complex textures and fine structures, some models may still have problems with blurred details or artifacts.

[0008] The Transformer model initially achieved great success in natural language processing, thanks to its core self-attention mechanism, which effectively captures long-range dependencies in sequences. In recent years, the Transformer and its variants have also been successfully applied to computer vision tasks, including image recognition, object detection, and image generation, demonstrating powerful feature learning and global information integration capabilities. Introducing the Transformer concept to image super-resolution tasks promises to overcome the shortcomings of traditional CNNs in terms of receptive field and long-range dependency modeling.

[0009] Therefore, in response to the demand for high quality, high detail, and arbitrary resolution output of e-commerce detail page images, it is urgent to study a new super-resolution method that can effectively utilize global information, enhance detail recovery, and support arbitrary resolution output. Summary of the Invention

[0010] The purpose of the present invention is to provide a Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution, so as to address the shortcomings of the existing super-resolution methods proposed in the above background technology in terms of fixed magnification, long-distance dependency capture, and adaptability to arbitrary resolution.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution, comprising the following steps:

[0012] S1: Low-resolution image feature extraction and encoding: An encoder built using a deep neural network performs deep feature extraction and preliminary encoding on the input low-resolution (LR) e-commerce detail page image, obtaining a multi-level feature map containing image content and structure information.

[0013] S2: Feature enhancement and global relationship modeling based on the improved Transformer; the improved Transformer module is used to process the feature map output by the encoder. Through its self-attention mechanism, the Transformer module effectively captures the long-range dependencies and contextual information between different regions of the feature map, enhances and refines the features, and provides richer semantics and details for subsequent high-resolution reconstruction;

[0014] S3: Arbitrary Resolution Image Decoding and Reconstruction: A decoder module that supports arbitrary scale upsampling combines the enhanced features output by S2 with the target resolution information to reconstruct a high-resolution (HR) image of the target size. The decoder can query and synthesize pixel values ​​at corresponding locations in continuous space based on given coordinate information and contextual features, thereby achieving arbitrary resolution output.

[0015] Preferably, the encoder adopts a classic CNN architecture, a network structure optimized for image restoration tasks, removes its top classification layer, and focuses on extracting hierarchical features.

[0016] Preferably, the Transformer module can include a multi-head self-attention layer and a feedforward neural network layer, combined with residual connections and layer normalization to stabilize the training process and accelerate convergence. A windowed or hollow self-attention mechanism can be designed to adapt to the two-dimensional structure of the image and reduce computational complexity.

[0017] Preferably, the arbitrary resolution image decoding and reconstruction module can draw on the idea of ​​implicit neural representation. For example, it takes pixel coordinates as part of the input, combines them with the feature vector obtained from the Transformer module, and uses a small multi-layer perceptron (MLP) to predict the RGB value of the corresponding coordinates. The target resolution information can be input to the decoding network by adjusting the normalized range of the input coordinates or directly as a conditional input.

[0018] Preferably, the method also includes a training phase using a large-scale e-commerce image dataset containing paired low-resolution images and corresponding high-resolution images. Loss functions used during training can include L1 loss, perceptual loss, and adversarial loss to comprehensively improve the realism and visual quality of the reconstructed images.

[0019] Preferably, in order to better adapt to the characteristics of e-commerce images, multiple categories of goods can be included in the training data, and targeted data enhancement can be performed, such as random cropping, rotation, color jittering, etc., to improve the generalization ability of the model.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) Arbitrary resolution output: Through the designed decoding and reconstruction module, images of arbitrary target resolution can be generated according to demand, which greatly enhances the flexibility and practicality of the method and adapts to different display devices and application scenarios.

[0022] (2) Stronger feature representation capability: The introduction of the Transformer module, with its powerful global context modeling capability, can better capture long-distance dependencies in images and help restore more realistic and consistent textures and structures.

[0023] (3) Higher reconstruction quality: Combining the local feature extraction advantages of CNN and the global information integration capabilities of Transformer, it can generate high-resolution images with richer details, clearer edges, and fewer artifacts, significantly improving the visual effects of e-commerce images.

[0024] (4) End-to-end learning: The entire model is trained in an end-to-end manner and can automatically learn complex mappings from low-resolution to arbitrary high-resolution images, simplifying the processing flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the present invention;

[0026] Figure 2 Schematic diagram of the encoder part in the algorithm model of the present invention;

[0027] Figure 3 Schematic diagram of the transformer part in the algorithm model of the present invention;

[0028] Figure 4 This is a schematic diagram of the decoder part in the algorithm model of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] See also Figure 1-Figure 4 As shown, the present invention provides the following technical solutions:

[0031] A Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution. Figure 1 Shown, including:

[0032] S1. The encoder module extracts features and performs preliminary encoding on the input low-resolution e-commerce detail page image to obtain a deep feature representation of the image.

[0033] S2. Use an improved Transformer module to further enhance the features output by the encoder module and model the global context relationship to capture long-range dependencies between features;

[0034] S3. Reconstruct and output high-resolution e-commerce detail page images through a decoding and reconstruction module that supports any scale.

[0035] Implementation Method 1

[0036] This implementation focuses on the model training process, optimizing the model's feature extraction and detail restoration capabilities by introducing pre-training and multi-stage fine-tuning.

[0037] Step by step:

[0038] Pre-training stage: First, the general image feature representation and long-distance dependency modeling capabilities are learned using the model. For large-scale general image datasets, the encoder module mentioned in this invention (such as Figure 2 As shown), Transformer module (as Figure 3 ) and the decoder module (as Figure 4 The goal of this stage is to lay the foundation for the super-resolution task of the model's business details page images.

[0039] Initial fine-tuning: The pre-trained model is connected to the decoding and reconstruction modules and initially fine-tuned using a dataset of e-commerce detail page images containing a large number of low-resolution and corresponding high-quality high-resolution image pairs. This dataset should cover a wide range of product categories, shooting angles, and lighting conditions to enhance the model's generalization capabilities. This phase focuses on optimizing the model's end-to-end super-resolution capabilities.

[0040] Fine-tuning phase: Based on the initial fine-tuning, further fine-tuning is performed using a carefully labeled dataset of e-commerce detail page images. This dataset can contain more complex textures, fine product details, and more challenging backgrounds. The goal of this phase is to improve the model's reconstruction accuracy and visual realism when handling complex backgrounds and detailed areas, for example, through optimization using perceptual and adversarial loss functions.

[0041] Implementation method 2:

[0042] This implementation focuses on the processing and enhancement of training data, aiming to improve the model's adaptability to different image features and magnification factors through diversified data generation strategies.

[0043] Data enhancement optimization:

[0044] Dynamic Resolution and Magnification Generation: In addition to fixed-factor downsampling, dynamic, randomized target magnification or output resolution ranges can be introduced during the training data processing and augmentation steps. For example, for the same high-resolution original image, in different training iterations, randomized downsampling by any magnification factor (e.g., x2, x3.5, x4) can be used to generate low-resolution images, with the corresponding target high-resolution output set. This forces the model to learn more robust arbitrary-resolution reconstruction capabilities.

[0045] Enhanced background and foreground diversity: To address the unique characteristics of e-commerce detail page images, more complex strategies can be employed during data augmentation to synthesize training image pairs. For example, objects can be randomly cropped from high-resolution product foreground images and fused with background images from various sources (GAN-generated, real-world image libraries), while taking into account factors such as lighting and shadows. This significantly increases the diversity of training data, making the model more adaptable to the complex background and foreground conditions commonly found in e-commerce detail pages.

[0046] Image content enhancement: Introducing richer image content enhancement technologies, such as simulating compression artifacts, adding noise, and randomly adjusting brightness / contrast / saturation on original high-resolution images to simulate various degradation conditions of low-quality images in the real world, thereby enhancing the robustness of the model and its adaptability to actual application scenarios.

[0047] Implementation method three:

[0048] This implementation further improves the performance of the super-resolution model, especially detail recovery and visual quality, by introducing auxiliary tasks and integrating multiple loss functions.

[0049] Model function expansion:

[0050] Joint Optimization (Introduction of Auxiliary Tasks): In addition to the core super-resolution task, auxiliary tasks can be introduced to guide model training. For example, edge detection or texture reconstruction can be jointly optimized with the super-resolution task as auxiliary tasks. Edge detection helps the model better understand the structural information of the image and restore clear object edges; texture reconstruction helps the model better capture and generate high-frequency details in the image. The feature extraction of these auxiliary tasks can share the output of the encoder and Transformer modules.

[0051] Multi-loss function fusion and weight adjustment: Based on the loss function described in S8, a function that integrates multiple task losses is designed. For example, the total loss can be defined as: L total =λ1L pixel +λ2L perceptual +λ3L adversarial +λ4L edge +λ5L texture Among them, L edge and L texture is the loss of the auxiliary task, and λ is the corresponding weight coefficient. By dynamically adjusting these weights or optimizing hyperparameters, we can balance the importance of each task and improve the model's performance in multiple dimensions, such as pixel accuracy, visual perception quality, edge clarity, and texture detail.

[0052] Implementation method four:

[0053] This implementation focuses on the deployment efficiency and performance of the trained model in practical applications, aiming to make it more suitable for the actual needs of e-commerce platforms through model compression and acceleration technology.

[0054] Model deployment optimization:

[0055] Model pruning and quantization: Prune and quantize the trained model to reduce model parameters and computational complexity, thereby improving real-time operation efficiency on mobile or embedded devices.

[0056] Model distillation: Using knowledge distillation technology, the knowledge of a large teacher model is transferred to a small student model, maintaining or approaching the cutout performance of the original model while significantly reducing computational complexity.

[0057] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution, characterized by: The steps include: S1: The encoder module extracts features and performs preliminary encoding on the input low-resolution e-commerce detail page image to obtain a deep feature representation of the image; S2: Use the improved Transformer module to further enhance the features output by the encoder module and model the global context relationship to capture the long-range dependencies between features; S3: Through a decoding and reconstruction module that supports any scale, combined with the enhanced features output by the Transformer module and the user-specified target resolution or magnification information, reconstruct and output a high-resolution e-commerce detail page image.

2. The Transformer-based super-resolution method for e-commerce detail page images at any resolution according to claim 1, characterized in that: The encoder module of S1, the Transformer module of S2, and the decoding and reconstruction module of S3 are connected in sequence to form an end-to-end deep learning model for directly generating a target high-resolution image from a low-resolution image: A Transformer module is added between the encoder and decoder to process complex backgrounds and transition areas in the image.

3. The Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to claim 1, characterized in that: The encoder module adopts a deep convolutional neural network (CNN) architecture, a network structure optimized for image restoration tasks, removing its top classification layer to focus on extracting hierarchical features for image reconstruction.

4. The Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to claim 1 or 2, characterized in that: The Transformer module includes: An optional feature preprocessing module that converts the feature maps from the encoder module into serialized or patch-based input suitable for processing by the Transformer structure; Multiple Transformer layers, each of which contains a Multi-Head Self-Attention (MHSA) module to calculate the global correlation between input features; A Feed-Forward Network (FFN) module, which is used to perform nonlinear feature transformation on the output of the multi-head self-attention module; It also combines residual connections and layer normalization operations to stabilize the training process and accelerate model convergence.

5. The Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to claim 1 or 2, characterized in that: The arbitrary-scale decoding and reconstruction module adopts the principle of Implicit Neural Representations (INR), including: A coordinate generation and encoding module is used to generate the query coordinates of each pixel according to the resolution of the target output image, and to perform position encoding or Fourier feature encoding on these coordinates; A feature query and fusion module is used to query and fuse the enhanced features output by the Transformer module to obtain the context features of the corresponding position according to the generated query coordinates; Multiple multi-layer perceptrons (MLPs) or equivalent function approximation networks receive the fused features and encoded coordinates as input, predict and output the pixel values ​​corresponding to the query coordinates, and thus synthesize a complete target high-resolution image.

6. A Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to any one of claims 1 to 5, characterized in that: It also includes constructing a large-scale e-commerce image training dataset, which contains a large number of low-resolution images and their corresponding high-quality, high-resolution real image sample pairs for supervised learning of the super-resolution model.

7. The Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to claim 6, characterized in that: It also includes one or more of the following training data processing and enhancement steps: Perform data augmentation operations on the original high-resolution images, such as random cropping, rotation, scaling, flipping, and color jittering, to increase data diversity; Generate corresponding low-resolution images by applying a specific downsampling algorithm (such as bicubic interpolation, Gaussian blur and downsampling) to the enhanced or original high-resolution images to form the image pairs required for training; During model training, the target magnification or target output resolution range can be randomly selected or programmatically specified for each training sample or batch to train the model's adaptability to arbitrary resolution outputs. The processed and enhanced dataset is divided into training set, validation set and test set for model training, parameter tuning and performance evaluation.

8. The Transformer-based super-resolution method for e-commerce detail page images of arbitrary resolution according to claim 6 or 7, characterized in that: It also includes introducing one or more loss functions to optimize model parameters during the model training process. The loss functions include the following three: A pixel-wise loss function, such as L1 loss (sum of absolute errors) or L2 loss (mean squared error), is used to minimize the pixel-by-pixel difference between the reconstructed high-resolution image and the true high-resolution image; Perceptual loss function, which extracts deep features of the reconstructed image and the real image through a pre-trained deep neural network (such as the VGG network) and calculates the difference between these features to improve the visual realism and structural similarity of the reconstructed image; An optional adversarial loss function introduces a discriminator network to perform adversarial training with the super-resolution model (as a generator) to further improve the detail clarity and naturalness of the generated images.

9. A computer program product, characterized in that: A computer-readable storage medium storing computer program instructions is included, and when the computer program instructions are executed by a processor, a Transformer-based arbitrary resolution e-commerce detail page image super-resolution method according to any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: The invention comprises a processor, a memory and an input / output interface; the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a Transformer-based arbitrary resolution e-commerce detail page image super-resolution method according to any one of claims 1 to 8 is implemented; the input / output interface is used to receive low-resolution e-commerce detail page image data to be processed, and output high-resolution image data reconstructed by the method.

Images