Model training method, image processing method, electronic device, storage medium, and computer program product

Abstract

The application discloses a model training method, an image processing method, a device, related equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring one or more training samples, each training sample comprising a high-resolution image and a corresponding low-resolution image; for each training sample, performing the following steps: performing super-resolution processing by using a first model to obtain a super-resolution image; determining a loss value of the super-resolution processing by using the super-resolution image and the high-resolution image; and optimizing the first model by using the loss value; wherein the first model adopts a U-shaped network (UNET) architecture, the UNET architecture comprises an encoder and a decoder connected by a jump connection, and the encoder comprises one or more visual state space (VSS) modules. The model based on the UNET architecture is used for super-resolution processing, and the encoder comprising the VSS module is used in the UNET architecture, so that the super-resolution task requirement can be better met.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a model training method, an image processing method, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In related technologies, super-resolution tasks refer to the task of reconstructing a clear, high-resolution image based on a real-world low-resolution image (which can also be understood as a picture) that is affected by complex degradation factors.

[0003] Currently, neural network models (which can also be understood as super-resolution models or image super-resolution models, etc.) are commonly used to perform super-resolution tasks. Neural network models can learn the mapping relationship between high-resolution images and low-resolution images, and thus restore low-resolution images to high-resolution images based on the learned mapping relationship.

[0004] However, there is currently no effective solution to ensure that neural network models meet the requirements of super-resolution tasks. Summary of the Invention

[0005] To address the related technical issues, embodiments of this application provide a model training method, an image processing method, related equipment, a storage medium, and a computer program product.

[0006] The technical solution of this application embodiment is implemented as follows: This application provides a model training method, including: Obtain one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; For each training sample, execute: Super-resolution processing is performed using the first model to obtain a super-resolution image; The loss value of super-resolution processing is determined using super-resolution and high-resolution images; Optimize the first model using the loss value; The first model adopts a U-shaped network (UNET) architecture, which includes encoders and decoders connected by jumps. The encoder includes at least one or more vision state space (VSS) modules.

[0007] In the above scheme, the VSS module includes: The first processing module is based on State Space Model (SSM). The second processing module is based on a feed-forward network (FFN). Both the first processing module and the second processing module adopt a residual structure based on an adaptive skip layer (ASL).

[0008] In the above scheme, the first processing module includes at least a depth convolution module for extracting spatial feature information, wherein the spatial feature information is associated with receptive fields of various sizes.

[0009] In the above scheme, the first model includes multiple cascaded UNETs, ​​and the encoders of the multiple UNETs may contain the same or different number of processing layers.

[0010] In the above scheme, obtaining one or more training samples includes: For each high-resolution image, the second model is used to degrade the high-resolution image to obtain the corresponding low-resolution image; and the high-resolution image and the corresponding low-resolution image are used as a training sample.

[0011] The method in the above scheme further includes: Based on the first randomly generated sequence, determine one or more of the following: The type of fuzzy kernel associated with the second model; The interpolation method associated with the second model; The scaling factor associated with the second model; The noise pattern associated with the second model; The compression method associated with the second model.

[0012] This application also provides an image processing method, including: Using the first model, super-resolution processing is performed on the image to be processed to obtain the target image; The first model includes models trained using any of the above-mentioned model training methods.

[0013] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor. When the processor runs the computer program, it executes the steps of any of the above methods.

[0014] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0016] The model training method, image processing method, related devices, storage media, and computer program products provided in this application embodiment acquire one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; for each training sample, the following steps are performed: super-resolution processing is performed using a first model to obtain a super-resolution image; the loss value of the super-resolution processing is determined using the super-resolution image and the high-resolution image; the first model is optimized using the loss value; wherein the first model adopts a UNET architecture, the UNET architecture includes an encoder and a decoder connected by jumps, and the encoder includes at least one or more VSS modules. The solution provided in this application uses a model based on the UNET architecture (i.e., the first model mentioned above) for super-resolution processing. It can achieve multi-scale feature fusion and effectively preserve high-frequency details through skip connections, making it suitable for super-resolution tasks that are sensitive to detailed features. Furthermore, the UNET architecture employs an encoder that includes a VSS module. On the one hand, it can capture global semantics and long-distance dependencies with low computational complexity, that is, it can maintain efficient computation while processing high-resolution feature maps, achieving a balance between processing performance and efficiency. On the other hand, the VSS mechanism can adaptively focus on the features that are key to super-resolution, suppress noise and redundant information, provide cleaner global guidance information for the decoder, improve the performance of the model in super-resolution processing, and better meet the requirements of related tasks. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart of the model training method in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the process of extracting state space features from a two-dimensional SSM (2D-SSM) according to an embodiment of this application. Figure 3 This is a schematic flowchart of the image processing method according to an embodiment of this application; Figure 4 This application example illustrates the structure of an image super-resolution model based on a selected state-space model. Figure 5 This is a flowchart illustrating the process of processing feature images using the Adaptive Residual Visual State Space Block (AVSS Block) as an application example of this application. Figure 6 This is a schematic diagram illustrating the process of processing feature images using Multi-SSM (Multi-SSM) as an application example of this application. Figure 7 This is a flowchart illustrating the process of processing feature images using the UpSample module as an application example of this application. Figure 8This is a flowchart illustrating a model training method as an application example of this application; Figure 9 This is a schematic diagram of the arrangement structure of degradation operators in the degradation model of the application example of this application; Figure 10 This is a schematic diagram of the model training device structure according to an embodiment of this application; Figure 11 This is a schematic diagram of the image processing device structure according to an embodiment of this application; Figure 12 This is a schematic diagram of an electronic device structure according to an embodiment of this application; Figure 13 This is a schematic diagram of another electronic device structure according to an embodiment of this application. Detailed Implementation

[0018] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0019] Currently, neural network models that can be used to perform super-resolution tasks can generally be categorized into the following two types: Models based on convolutional neural networks can leverage the natural adaptability of convolutional operations to local image correlations to effectively reconstruct high-frequency details such as textures and edges. At the same time, these models have a small number of parameters and high inference efficiency, making them suitable for super-resolution applications with high real-time requirements, such as live streaming. Attention-based models can leverage the ability of attention mechanisms to capture long-range dependencies for image super-resolution (i.e., perform super-resolution tasks). At the same time, these models can ensure the global semantic coherence of the high-resolution images obtained after super-resolution, effectively avoiding situations where local details are clear but the overall image is incoherent.

[0020] In practical applications, when using convolutional neural network-based models to perform super-resolution tasks, the following problems may arise: With a limited receptive field, convolutional neural network-based models struggle to capture long-distance dependencies and lack the ability to model global semantic consistency. In the process of high-frequency information recovery, convolutional neural network-based models are prone to oversmoothing or artifacts, resulting in poor performance.

[0021] Correspondingly, when using an attention-based model to perform super-resolution tasks, long-distance dependencies can be directly captured by calculating the association weights between any two positions in the feature map. However, since the core of the attention mechanism is to calculate the association matrix between the query (Q, Query) key (K, Key) and value (V, Value), the computational complexity is O(n log n). ,in, This indicates the number of spatial pixels in the feature map corresponding to the correlation matrix calculation. This represents the number of channels in the feature map. As can be seen, when the feature map size is large, the computational cost increases quadratically, which leads to low efficiency in training the model and inference using the model, making it difficult to meet the needs of practical applications.

[0022] As can be seen from the above description, the two image super-resolution models mentioned above have an inherent dilemma in choosing between global receptive field and efficient computation. In other words, it is difficult to simultaneously meet the performance and efficiency requirements of super-resolution tasks.

[0023] Based on this, in various embodiments of this application, a model based on the UNET architecture is used for super-resolution processing. This model can achieve multi-scale feature fusion and effectively preserve high-frequency details through skip connections, making it suitable for super-resolution tasks sensitive to detail features. Furthermore, the UNET architecture employs an encoder that includes a VSS module. On the one hand, it can capture global semantics and long-distance dependencies with low computational complexity, meaning it can maintain high computational efficiency while processing high-resolution feature maps, achieving a balance between processing performance and efficiency. On the other hand, the VSS mechanism can adaptively focus on features key to super-resolution, suppress noise and redundant information, provide cleaner global guidance information for the decoder, improve the model's performance in super-resolution processing, and better meet the requirements of related tasks.

[0024] This application provides a model training method applied to a first electronic device, such as... Figure 1 As shown, the method includes: Step 101: Obtain one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; Step 102: For each training sample, execute: Super-resolution processing is performed using the first model to obtain a super-resolution image; The loss value of super-resolution processing is determined using super-resolution and high-resolution images; Optimize the first model using the loss value; The first model adopts the UNET architecture, and the encoder of the UNET architecture includes at least one or more visual state space (VSS) modules.

[0025] In practical applications, image data may be lost or distorted during acquisition, transmission, and storage due to physical hardware defects (such as lens defocusing, limited number of sensors, etc.), environmental interference (such as acquisition device shaking, insufficient lighting, etc.), and data processing distortion (such as data loss during transmission, storage compression, etc.), which can also be understood as image degradation. In this case, super-resolution processing (also understood as super-resolution processing) can be performed based on the first model to restore (also understood as reconstruction or conversion, etc.) the low-resolution image to a high-resolution image to meet the user's image resolution requirements. Here, the first model can also be understood as a super-resolution model, or a reconstruction model, etc., and this application embodiment does not limit it to this; the first model adopts the UNET architecture (also understood as the UNET architecture, etc.), and image processing based on the UNET architecture can take into account both multi-scale feature fusion and high-frequency detail preservation, making it suitable for image super-resolution tasks that are sensitive to detail features.

[0026] In practical applications, the UNET architecture includes an encoder and a decoder connected by skip connections. The encoder may contain multiple processing layers. In each processing layer, downsampling is used to expand the receptive field and extract features to obtain multi-scale features, which can obtain the global structure and contour information of the input image. The decoder contains multiple processing layers corresponding to the encoder. In each processing layer, the deep features extracted by the corresponding processing layer in the encoder are upsampled, and the upsampled results are fused with high-frequency features (i.e., shallow features) obtained through skip connections to gradually restore the detailed features of the image.

[0027] In each processing layer of the encoder, a patch merging module and one or more VSS modules can be configured. The patch merging module is used to downsample the image input to the processing layer. The one or more VSS modules are used to extract features from the downsampled image of the layer. Since the VSS mechanism has the ability to capture long-distance dependencies and global context, the features extracted by the VSS module can simultaneously capture the detailed features and global structure of the downsampled image of the current processing layer, achieving better performance in super-resolution tasks. Furthermore, the VSS mechanism has lower computational complexity and higher processing efficiency compared to attention-based processing schemes, better meeting the task efficiency requirements.

[0028] In practical applications, the VSS module may include a first processing module based on SSM and a second processing module based on FFN; wherein, the first processing module may also be understood as an SSM processing module, etc., and this application embodiment does not limit it in this way. The first processing module can at least realize global receptive field modeling and capture long-distance spatial dependencies based on the selective scanning mechanism of SSM; the second processing module may also be understood as an FFN processing module, etc., and this application embodiment does not limit it in this way. The second processing module can at least serve as a channel mixer and a nonlinear feature transformer to enhance the model's expressive power.

[0029] In practical applications, both the first and second processing modules can adopt residual structures (which can also be understood as adopting residual connection structures). In the first processing module, the use of residual structures can alleviate information loss during the global state scanning process, ensuring that local spatial details are not sacrificed while capturing long-range dependencies. Furthermore, the use of residual structures in the second processing module can promote gradient flow and reduce optimization difficulty, allowing the second processing module to focus on learning complex inter-channel nonlinear mappings without being limited by the gradient vanishing problem.

[0030] In practical applications, introducing ASL into the residual structure can enhance the flexibility of residual connections. Furthermore, ASL has learning capabilities (i.e., ASL is learnable) and can dynamically adjust the feature fusion ratio during the training of the first model (e.g., based on gating mechanisms). This allows for the retention of important features and the shielding of redundant features (which can also be understood as reducing noise interference between features), thereby improving the processing efficiency and performance (or robustness) of the VSS module.

[0031] Based on this, in some optional embodiments, the VSS module may include: The first processing module based on SSM; The second processing module based on FFN; Both the first processing module and the second processing module adopt an ASL-based residual structure.

[0032] In practical applications, when using the first processing module, the spatial information (which can also be understood as local context) of the image at multiple scales can be extracted first using the deep convolution module. Then, the state space features are extracted from the processing results of the deep convolution module using a two-dimensional SSM (2D-SSM). The deep convolution module can contain multiple convolution kernels of different scales to extract spatial information at multiple scales. That is, in some optional embodiments, the first processing module includes at least a deep convolution module for extracting spatial feature information, and the spatial feature information is associated with receptive fields of multiple sizes.

[0033] In practical applications, such as Figure 2 As shown, when using 2D-SSM for state space feature extraction, the two-dimensional image is first unfolded into multiple one-dimensional sequences along multiple directions (such as upper left → lower right, or upper right → lower left, etc.); then, each one-dimensional sequence is processed by the Selective State Space Sequence Model (S6) module; finally, the S6 processing results of all one-dimensional sequences are merged.

[0034] In practical applications, to further improve the performance of global feature fusion and detail refinement when using the first model for super-resolution processing, the first model can be configured to include multiple cascaded UNETs. Through staged progressive processing, refined image restoration from low resolution to high resolution can be achieved. That is, in some optional embodiments, the first model includes multiple cascaded UNETs, ​​and the encoders of the multiple UNETs may contain the same or different numbers of processing layers.

[0035] As can be seen from the above description, super-resolution processing based on the first model can achieve multi-scale feature fusion and effectively preserve high-frequency details through skip connections, making it suitable for super-resolution tasks sensitive to detailed features. Furthermore, the encoder in the UNET architecture, which includes a VSS module, can capture global semantics and long-distance dependencies with low computational complexity. In other words, it can maintain high computational efficiency while processing high-resolution feature maps, achieving a balance between processing performance and efficiency. On the other hand, the VSS mechanism can adaptively focus on the features key to super-resolution, suppress noise and redundant information, and provide cleaner global guidance information for the decoder, improving the model's performance in super-resolution processing and better meeting the requirements of related tasks.

[0036] In practical applications, the first electronic device is used at least to train the first model to improve the performance of super-resolution processing using the first model. Specifically, the first electronic device can be a general-purpose server or a computer that meets storage and processing speed requirements. This application does not limit the name of the first electronic device, as long as its function is implemented.

[0037] Before training the first model, in step 101, the first electronic device needs to acquire one or more (one or more can also be understood as at least one) training samples for training the first model. Each training sample includes at least a low-resolution image used as input to the first model and a high-resolution image used to verify the performance of the first model. That is, each training sample contains a high-resolution version and a low-resolution version of the same image. The training sample can also be understood as sample data, training data, a pair of training images, or a combination of training images, etc., and this embodiment does not limit this.

[0038] In practical applications, the specific implementation method of the first electronic device acquiring the training samples can be set according to actual needs. This application embodiment does not limit this. For example, the implementation method may include one or more of the following: Read one or more training samples that are pre-stored locally; Receive one or more training samples sent by the relevant device; Using one or more high-resolution images, a low-resolution image corresponding to each high-resolution image is obtained through degradation processing. Each high-resolution image and its corresponding low-resolution image are then used as a training sample. In other words, in some optional embodiments, the specific implementation of step 101 may include: For each high-resolution image, the second model is used to degrade the high-resolution image to obtain the corresponding low-resolution image; and the high-resolution image and the corresponding low-resolution image are used as a training sample.

[0039] In practical applications, when one or more training samples are obtained using degradation processing, the first electronic device can acquire one or more base images as high-resolution images. The specific implementation of the electronic device acquiring one or more base images can be set according to actual needs, and this embodiment does not limit this. The electronic device can preprocess each acquired base image to meet the requirements of the training samples. The preprocessing may include one or more of the following: Cropping refers to selecting a region of a specified size (which can also be understood as size, such as the number of pixels in width and height) from a base image as a high-resolution image. Data augmentation refers to the process of obtaining more high-resolution images by performing horizontal or vertical flipping on the base image to meet the data diversity requirements of the training samples.

[0040] After obtaining the high-resolution image, the first electronic device can use the second model to perform degradation processing to obtain training samples. The second model can also be understood as a degradation model, etc., and this application embodiment does not limit this interpretation.

[0041] Currently, the models used to implement degradation processing typically include the following two types: The Blind Super-Resolution Generative Adversarial Network (BSRGAN) model, based on the degradation scheme of this model, achieves image degradation through a combination of multiple random operations, specifically including: starting from a high-resolution image, generating training data consistent with the distribution of real low-resolution images by randomly selecting multiple degradation operators; The Real-Enhanced Super-Resolution Generative Adversarial Network (Real-ESRGAN) model, based on the degradation scheme of this model, enhances the complex, multi-source and nonlinear degradation features of real-world images on the basis of the degradation mode of the BSRGAN model, thereby further improving the model's adaptability to extreme or complex real-world scenes.

[0042] The BSRGAN model contains relatively simple degradation operators (which can also be understood as blurring operators), such as only including a Gaussian blur kernel. Therefore, the BSRGAN model is difficult to effectively simulate the features of real camera blurring, artifacts (such as edge distortion caused by ringing and overshoot), and other degradation processes. Furthermore, the Real-ESRGAN model contains a fixed combination order of degradation operators, namely, two superpositions of blurring, scaling, noise, and compression operations. This fixed combination order simulates degradation features that are regular and singular. In this case, when training the first model using training samples obtained from the BSRGAN or Real-ESRGAN models, the trained first model can only learn a limited number of degradation features, has poor generalization ability, and is difficult to meet the requirements of super-resolution tasks.

[0043] Based on this, various optional degradation operators can be configured for the second model, such as degradation operators associated with blur kernel type, interpolation method, scaling factor, noise mode, and compression method. Furthermore, multiple optional instances (or subclasses) can be configured for each degradation operator. Thus, when using the second model for degradation processing, by selecting the combination order of different degradation operators and choosing different instances for each degradation operator, the image degradation process caused by different factors in the real world can be realistically simulated, thereby obtaining low-resolution images containing features of different image degradation processes. In this case, using low-resolution images containing multiple image degradation process features and corresponding high-resolution images as multiple training samples can effectively ensure the diversity of training samples, meet the training requirements of the first model, and improve the generalization ability of the first model.

[0044] The instances of the degradation operators associated with the fuzzy kernel type may include one or more of the following: Isotropic Gaussian blur kernel; Anisotropic Gaussian blur kernel; Generalized isotropic Gaussian blur kernel; A generalized anisotropic Gaussian blur kernel; The platform's isotropic Gaussian blur kernel; The platform's anisotropic Gaussian blur kernel; Low-pass filter fuzzy kernel.

[0045] Instances of the degradation operators associated with the interpolation method may include one or more of the following: Region interpolation; Bilinear interpolation; Bicubic interpolation.

[0046] Instances of the degradation operators associated with the noise patterns may include one or more of the following: Gaussian noise; Poisson noise; Colored noise; Grayscale noise.

[0047] Instances of the degradation operator associated with the compression method may include one or more of the following: JPEG (Joint Photographic Experts Group) compression; Encoder compression.

[0048] In practical applications, for each high-resolution image, when the first electronic device performs degradation processing using the second model, the first electronic device can determine the order of degradation operators used by the second model and the instance of each degradation operator through random selection. This ensures the randomness of the combination of degradation operators used in multiple degradation processes, thereby improving the diversity of training samples. In other words, in some optional embodiments, the method may further include: Based on the first randomly generated sequence, determine one or more of the following: The type of fuzzy kernel associated with the second model; The interpolation method associated with the second model; The scaling factor associated with the second model; The noise pattern associated with the second model; The compression method associated with the second model.

[0049] The first sequence can also be understood as a random sequence, or a random degenerate operator sequence, etc., and this application embodiment does not limit this. The method by which the first electronic device randomly generates the first sequence can be set according to actual needs, and this application embodiment does not limit this. The first sequence can also be used to determine the order of degenerate operators used by the second model.

[0050] After obtaining training samples through the second model, in step 102, the first electronic device can use each training sample to train the first model. The specific process may include: inputting the low-resolution image in the training sample into the first model to obtain the super-resolution processing result (i.e., super-resolution image) output by the first model; then, using the super-resolution image and the high-resolution image in the training sample, combined with a pre-designed loss function to calculate the loss value; and then, using the loss value to optimize the parameters in the first model to improve the processing performance of the first model.

[0051] The loss function may include a pixel loss function and a perceptual loss function; the pixel loss function is used to calculate the pixel-level difference between the generated image (i.e., the super-resolution image) and the real high-resolution image (i.e., the high-resolution image); the perceptual loss function is used to calculate the difference between the generated image and the real high-resolution image on the intermediate layer output features of the pre-trained Visual Geometry Group (VGG) network, wherein the input of the VGG network includes the generated image and the real high-resolution image.

[0052] Accordingly, embodiments of this application also provide an image processing method applied to a second electronic device, such as... Figure 3 As shown, the method includes: Step 301: Obtain the image to be processed; Step 302: Using the first model, perform super-resolution processing on the image to be processed to obtain the target image; The first model includes models trained using any of the above-mentioned model training methods.

[0053] In practical applications, the second electronic device can be a general-purpose server or a computer that meets storage and processing speed requirements. The second electronic device can be the first electronic device or other electronic devices different from the first electronic device. The embodiments of this application do not limit the name of the second electronic device, as long as its function is realized.

[0054] In practical applications, the specific implementation of the second electronic device acquiring the image to be processed in step 301 can be set according to actual needs, and this application embodiment does not limit it.

[0055] In practical applications, when the second electronic device is the first electronic device, before step 302, the second electronic device can obtain the first model by executing any of the above training methods. Here, the specific training method of the first model has been detailed above and will not be repeated here. When the second electronic device is not the first electronic device, before step 302, the second electronic device can obtain the first model. The specific implementation method can be set according to actual needs, such as receiving the first model sent by the first electronic device. This application embodiment does not limit this.

[0056] The model training method provided in this application embodiment obtains one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; for each training sample, the following steps are performed: super-resolution processing is performed using a first model to obtain a super-resolution image; the loss value of the super-resolution processing is determined using the super-resolution image and the high-resolution image; the first model is optimized using the loss value; wherein the first model adopts a UNET architecture, the UNET architecture includes an encoder and a decoder connected by skip connections, and the encoder includes at least one or more VSS modules. The solution provided in this application uses a model based on the UNET architecture (i.e., the first model mentioned above) for super-resolution processing. It can achieve multi-scale feature fusion and effectively preserve high-frequency details through skip connections, making it suitable for super-resolution tasks that are sensitive to detailed features. Furthermore, the UNET architecture employs an encoder that includes a VSS module. On the one hand, it can capture global semantics and long-distance dependencies with low computational complexity, that is, it can maintain efficient computation while processing high-resolution feature maps, achieving a balance between processing performance and efficiency. On the other hand, the VSS mechanism can adaptively focus on the features that are key to super-resolution, suppress noise and redundant information, provide cleaner global guidance information for the decoder, improve the performance of the model in super-resolution processing, and better meet the requirements of related tasks.

[0057] The following section provides a more detailed description of this application with reference to application examples.

[0058] This application provides an example of an image super-resolution model based on a selected state-space model (hereinafter referred to as the image super-resolution model), such as... Figure 4 As shown, the input image of the image super-resolution model includes a low-resolution image with dimensions [height (H), width (W), number of channels (C)]. For red-green-blue (RGB) images, the value of C is usually 3. The image super-resolution model contains two cascaded UNETs, ​​the first UNET (i.e., Figure 4 The UNET on the left contains two processing layers, the second UNET (i.e. Figure 4 The UNET (right side) contains three processing layers. The UNET architecture includes the following modules: The Patch Embedding layer is used to segment and map images using convolutions (such as 2D convolutions). It transforms the input 2D image into a grid-like sequence of features, which can map the image to high-dimensional features while preserving the spatial structure. The feature image dimension after processing by this layer is [H / scale, W / scale, dim], where scale represents the size and stride of the convolution kernel, and dim represents the dimension of the output feature. Avss Block, such as Figure 5 As shown, the AVSS Block comprises two residual networks: Multi-SSM and FFN. The residual structures of these two networks utilize ASL to enhance the flexibility of skip connections. ASL is learnable, allowing the model to autonomously learn which features are more important during training and dynamically adjust the fusion ratio. The AVSS Block does not change the dimension of the input feature vector; that is, the output feature dimension is [H / scale, W / scale, dim]. The Multi-SSM module can acquire contextual information from different directions, enhancing the efficiency and comprehensiveness of multi-dimensional image feature capture, such as... Figure 6 As shown, in the process of processing using the Multi-SSM module, the feature image is first processed by two depthwise separable convolutions of different sizes to extract spatial information at different scales (e.g., 3x3 and 5x5). Then, a 2D-SSM layer is used to achieve a global receptive field while maintaining linear computational complexity, giving it an efficiency advantage when processing large images or high-resolution feature maps. During the 2D-SSM layer processing, the 2D features of the image are first transformed into a one-dimensional sequence along different directions using a scanning mechanism. Then, the resulting one-dimensional vectors are independently fed into the S6 module with a scanning mechanism. The S6 module is based on a state-space model, and feature extraction through this module integrates information in different directions, retaining important contextual information while filtering out irrelevant content. Finally, the four one-dimensional vectors processed by the S6 module are fused into a single 2D feature output. The Patch Merging layer is used to compress spatial dimensions by merging adjacent pixel blocks, while enhancing the feature representation of the channel dimension, balancing the computational efficiency and feature capture capability of the model; after passing through this layer, the output feature vector is [H / scale / scale, W / scale / scale, dim×scale]; UpSample module, such as Figure 7 As shown, the UpSample module mainly includes convolutional layers, interpolation (F.interpolate) operations, and concatenation operations. In the process of processing using the UpSample module, the low-resolution feature map is first restored to high resolution using interpolation operations, and then the channel dimension of the feature vector is reduced using convolutional layers to make it consistent with the feature channel dimension of the corresponding layer of the encoder. The two feature vectors are fused in the channel dimension and then convolved.

[0059] For example, as shown in Table 1, during the process of processing the input image using the first UNET, the dimension of the feature map changes continuously with the processing flow, and finally the desired super-resolution dimension is obtained.

[0060] Table 1

[0061] Based on the aforementioned image super-resolution model, this application also provides a model training method, such as... Figure 8 As shown, the specific steps include: Step 801: Data preprocessing to obtain high-resolution images; In practical applications, data preprocessing includes one or more of the following: reading image data, cropping image data, and image data enhancement (the main strategies include horizontal and vertical flipping).

[0062] Step 802: Process the high-resolution image using the degradation model to obtain the corresponding low-resolution image; In practical applications, a random sequence is pre-generated. Each high-resolution image is processed by different combinations of degradation operators when it passes through the degradation model; for example, Figure 9 As shown, the arrangement of each degradation operator in the degradation model is as follows: Blur, Resize, Noise, Compression, Resize (used to reduce the image to the size of a specified low-resolution image), and Blur.

[0063] The types of degenerate operators are as follows: Blur: Seven types of blur kernels, namely isotropic, anisotropic, generalized isotropic, generalized anisotropic, plateau isotropic, plateau anisotropic Gaussian blur kernel, and low-pass filter Sinc blur kernel. Resize: Three interpolation methods are used: Area interpolation, Bilinear interpolation, and Bicubic interpolation. Then, based on probability, the interpolation method is chosen to shrink, enlarge, or remain unchanged. Roise: Two noise modes, Gaussian noise and Poisson noise, are then superimposed with either colored noise or grayscale noise; Compression: Uses JPEG compression and encoder compression.

[0064] Step 803: Train the image super-resolution model using training samples composed of high-resolution and low-resolution images.

[0065] This application presents a degradation model that learns the mapping relationship between low resolution and high resolution. To make the degradation features diverse and irregular, a strategy of randomly selecting degradation operators is used. At the same time, to simulate more degradation types, rich blur kernels and different compression methods are introduced, aiming to better simulate the image degradation process in the real world.

[0066] Furthermore, this application demonstrates the design of a cascaded UNET structure-based selected state space model (i.e., a real image super-resolution model based on selected state space). The encoder part includes Patch Embedding, AVSS Block, and UpSample Block. The AVSS Block-based encoder has the advantage of efficiently capturing global semantics and long-range dependencies, with linear computational complexity, lower than the quadratic computational complexity of the Transformer. This allows it to maintain high computational efficiency when processing high-resolution feature maps, balancing performance and speed. Additionally, the AVSS Block's unique gating mechanism adaptively focuses on features crucial to super-resolution, suppressing noise and redundant information, providing cleaner global guidance information for the decoder. The decoder part consists of convolutional layers, upsampling layers, and residual structures. The convolutional decoder can accurately recover local details and spatial structure. Residual connections between corresponding layers of the encoder and decoder achieve complementarity between global and local features. The design of cascading two UNET-like systems achieves refined restoration from low to high resolution through a phased, progressive processing approach. It fully leverages the advantages of the UNET architecture in feature extraction and reconstruction, while multi-stage collaboration addresses the challenge of a single UNET system simultaneously achieving both global structural restoration and detailed detail enhancement in super-resolution. The design, employing an "AVSS encoder for global coverage, CNN decoder for detail enhancement, and residual connections for fusion," overcomes the limitations of single architectures in super-resolution tasks. Furthermore, through cross-architecture collaboration and innovative feature fusion, it achieves a balance between global structural consistency and sharp local details, effectively improving the subjective visual quality and objective metrics of super-resolution images while maintaining high computational efficiency, making it suitable for real-time or near-real-time super-resolution applications.

[0067] To implement the model training method of this application embodiment, this application embodiment also provides a model training device, which is installed on an electronic device, such as... Figure 10 As shown, the device includes: The first acquisition unit 1001 is used to acquire one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; Training unit 1002 is used to perform the following for each training sample: Super-resolution processing is performed using the first model to obtain a super-resolution image; The loss value of super-resolution processing is determined using super-resolution and high-resolution images; Optimize the first model using the loss value; The first model adopts the UNET architecture, which includes encoders and decoders connected by jumps, and the encoder includes at least one or more VSS modules.

[0068] In some optional embodiments, the first acquisition unit 1001 is specifically used for: For each high-resolution image, the second model is used to degrade the high-resolution image to obtain the corresponding low-resolution image; and the high-resolution image and the corresponding low-resolution image are used as a training sample.

[0069] In some optional embodiments, the first acquisition unit 1001 is further configured to: Based on the first randomly generated sequence, determine one or more of the following: The type of fuzzy kernel associated with the second model; The interpolation method associated with the second model; The scaling factor associated with the second model; The noise pattern associated with the second model; The compression method associated with the second model.

[0070] In practical applications, the first acquisition unit 1001 can be implemented by a processor in the model training device combined with a communication interface, and the training unit 1002 can be implemented by a processor in the model training device.

[0071] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above-described program units during model training. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the device can be divided into different program units to complete all or part of the processing described above. In addition, the model training device and the model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0072] To implement the image processing method of this application embodiment, this application embodiment also provides an image processing apparatus, disposed on an electronic device, such as... Figure 11 As shown, the device includes: The second acquisition unit 1101 is used to acquire the image to be processed; Processing unit 1102 is used to perform super-resolution processing on the image to be processed using the first model to obtain the target image; The first model includes a model trained using any of the above-mentioned model training methods.

[0073] In practical applications, the second acquisition unit 1101 can be implemented by a processor in the image processing device combined with a communication interface, and the processing unit 1102 can be implemented by a processor in the image processing device.

[0074] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above-described program units. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the apparatus can be divided into different program units to complete all or part of the processing described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0075] Based on the hardware implementation of the above program modules, and in order to implement the model training method of this application embodiment, this application embodiment also provides an electronic device, such as... Figure 12 As shown, the first electronic device 1200 includes: The first communication interface 1201 is capable of exchanging information with other devices; The first processor 1202 is connected to the first communication interface 1201 to enable information interaction with other devices and to execute the methods provided by one or more of the above technical solutions when running a computer program; The computer program is stored in the first memory 1203.

[0076] Specifically, the first processor 1202 is used for: One or more training samples are obtained through the first communication interface 1201, each training sample containing a high-resolution image and a corresponding low-resolution image; For each training sample, execute: Super-resolution processing is performed using the first model to obtain a super-resolution image; The loss value of super-resolution processing is determined using super-resolution and high-resolution images; Optimize the first model using the loss value; The first model adopts the UNET architecture, which includes encoders and decoders connected by jumps, and the encoder includes at least one or more VSS modules.

[0077] In some optional embodiments, the first processor 1202 is specifically used for: For each high-resolution image, the second model is used to degrade the high-resolution image to obtain the corresponding low-resolution image; and the high-resolution image and the corresponding low-resolution image are used as a training sample.

[0078] In some optional embodiments, the first processor 1202 is further configured to: Based on the first randomly generated sequence, determine one or more of the following: The type of fuzzy kernel associated with the second model; The interpolation method associated with the second model; The scaling factor associated with the second model; The noise pattern associated with the second model; The compression method associated with the second model.

[0079] It should be noted that the specific processing procedures of the first processor 1202 and the first communication interface 1201 can be understood by referring to the above method.

[0080] Of course, in practical applications, the various components in the first electronic device 1200 are coupled together via a bus system 1204. It can be understood that the bus system 1204 is used to realize communication between these components. In addition to a data bus, the bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 12 The general designated all buses as Bus System 1204.

[0081] The first memory 1203 in this embodiment is used to store various types of data to support the operation of the first electronic device 1200. Examples of such data include any computer program used to operate on the first electronic device 1200.

[0082] The methods disclosed in the embodiments of this application can be applied to the first processor 1202, or implemented by the first processor 1202. The first processor 1202 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 1202. The first processor 1202 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 1202 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 1203. The first processor 1202 reads the information in the first memory 1203 and completes the steps of the aforementioned method in combination with its hardware.

[0083] In an exemplary embodiment, the first electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0084] Based on the hardware implementation of the above program modules, and in order to implement the image processing method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 13 As shown, the second electronic device 1300 includes: The second communication interface 1301 is capable of exchanging information with other devices; The second processor 1302 is connected to the second communication interface 1301 to enable information interaction with the electronic device and to execute the methods provided by one or more technical solutions on the electronic device side when running a computer program. The computer program is stored in the second memory 1303.

[0085] Specifically, the second processor 1302 is used for: The image to be processed is obtained through the second communication interface 1301; Using the first model, super-resolution processing is performed on the image to be processed to obtain the target image; The first model includes a model trained using any of the above-mentioned model training methods.

[0086] It should be noted that the specific processing procedures of the second processor 1302 and the second communication interface 1301 can be understood by referring to the above method.

[0087] Of course, in practical applications, the various components in the second electronic device 1300 are coupled together via a bus system 1304. It can be understood that the bus system 1304 is used to achieve communication between these components. In addition to a data bus, the bus system 1304 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 13 The general designated all buses as Bus System 1304.

[0088] The second memory 1303 in this embodiment is used to store various types of data to support the operation of the second electronic device 1300. Examples of such data include any computer program used to operate on the second electronic device 1300.

[0089] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the second processor 1302. The second processor 1302 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in the form of software within the second processor 1302. The second processor 1302 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The second processor 1302 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically a second memory 1303. The second processor 1302 reads information from the second memory 1303 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0090] In an exemplary embodiment, the second electronic device 1300 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0091] It is understood that the memories (first memory 1203, second memory 1303) in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0092] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium. For example, it may include a first memory 1203 storing a computer program, which can be executed by a first processor 1202 of a first electronic device 1200 to complete the steps described in the method of the first electronic device 1200. Another example is a second memory 1303 storing a computer program, which can be executed by a second processor 1302 of a second electronic device 1300 to complete the steps described in the method of the second electronic device 1300. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0093] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 1202 of a first electronic device 1200 to complete the steps of the aforementioned transmitting end-side method, or the computer program can be executed by a second processor 1302 of a second electronic device 1300 to complete the steps of the aforementioned electronic device-side method.

[0094] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0095] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Obtain one or more training samples, each training sample containing a high-resolution image and a corresponding low-resolution image; For each training sample, execute: Super-resolution processing is performed using the first model to obtain a super-resolution image; The loss value of super-resolution processing is determined using super-resolution and high-resolution images; Optimize the first model using the loss value; The first model adopts a U-shaped network (UNET) architecture, which includes encoders and decoders connected by jumps. The encoder includes at least one or more visual state space (VSS) modules.

2. The method according to claim 1, characterized in that, The VSS module includes: The first processing module based on state-space modeling of SSM; The second processing module is based on the feedforward network FFN; Both the first processing module and the second processing module adopt a residual structure based on the Adaptive Skip Layer (ASL).

3. The method according to claim 2, characterized in that, The first processing module includes at least a depth convolution module for extracting spatial feature information, which is associated with receptive fields of various sizes.

4. The method according to claim 1, characterized in that, The first model comprises multiple cascaded UNETs, ​​and the encoders of the multiple UNETs may contain the same or different numbers of processing layers.

5. The method according to any one of claims 1 to 4, characterized in that, The acquisition of one or more training samples includes: For each high-resolution image, the second model is used to degrade the high-resolution image to obtain the corresponding low-resolution image; and the high-resolution image and the corresponding low-resolution image are used as a training sample.

6. The method according to claim 5, characterized in that, The method further includes: Based on the first randomly generated sequence, determine one or more of the following: The type of fuzzy kernel associated with the second model; The interpolation method associated with the second model; The scaling factor associated with the second model; The noise pattern associated with the second model; The compression method associated with the second model.

7. An image processing method, characterized in that, include: Using the first model, super-resolution processing is performed on the image to be processed to obtain the target image; The first model includes a model trained using the method described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 6, or performs the steps of the method according to claim 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

Images