Image super-resolution reconstruction and model training method, intelligent terminal and storage medium

Abstract

The application provides an image super-resolution reconstruction and model training method, an intelligent terminal and a storage medium, relates to the technical field of image processing, and comprises the following steps: acquiring an image to be processed; and processing the image to be processed by using an image super-resolution model to obtain a super-resolution reconstruction image corresponding to the image to be processed; wherein the image super-resolution model comprises a super-resolution backbone network, and the super-resolution backbone network comprises a low-rank adaptation expert module and a hierarchical routing subnetwork. The application is helpful to improve the super-resolution reconstruction effect of an image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image super-resolution reconstruction and model training method, a smart terminal, and a storage medium. Background Technology

[0002] Super-resolution (SR) technology has wide applications in key fields such as remote sensing, medical imaging, industrial inspection, virtual reality, and digital content restoration.

[0003] In conceiving and implementing this application, the inventors discovered at least the following problems: in different application scenarios, there are significant differences in the semantic content, statistical texture, noise characteristics, spectral characteristics and acquisition conditions of images, which leads to a significant decrease in the performance of traditional image super-resolution models in the non-training domain.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image super-resolution reconstruction and model training method, a smart terminal, and a storage medium, which helps to improve the super-resolution reconstruction effect of images.

[0006] To address the aforementioned technical problems, this application provides an image super-resolution reconstruction method, comprising:

[0007] Obtain the image to be processed;

[0008] The image super-resolution model is used to process the image to be processed to obtain the super-resolution reconstructed image corresponding to the image to be processed. The image super-resolution model includes a super-resolution backbone network, which includes a low-rank adaptation expert module and a hierarchical routing sub-network.

[0009] Optionally, the parameters of the low-rank adaptation expert module are generated by training on the training domain dataset while freezing the parameters of the super-resolution backbone network, and the parameters of the hierarchical routing sub-network are generated by training on the non-training domain dataset while freezing the parameters of the low-rank adaptation expert module.

[0010] Optionally, the training domain dataset includes a content domain difference dataset and a degradation domain difference dataset; the non-training domain dataset includes image samples from unknown domains.

[0011] Optionally, the low-rank adaptation expert module includes a linear transformation unit; the low-rank adaptation expert module is located in the linear layer of the super-resolution backbone network.

[0012] Optionally, the low-rank adaptation expert module includes convolutional units; the low-rank adaptation expert module is located in the convolutional layer of the deep feature extraction module of the super-resolution backbone network.

[0013] Optionally, the hierarchical routing subnetwork includes a weight calculation and fusion subunit, which is used to calculate the weights of each low-rank adaptation expert module based on the features of image samples in the non-training domain dataset, and to fuse the output results of each low-rank adaptation expert module based on the weights of each low-rank adaptation expert module.

[0014] Optionally, the hierarchical routing subnetwork is located at a designated layer in the super-resolution backbone network where low-rank adaptation expert modules are inserted.

[0015] This application also provides a method for training an image super-resolution model, including the following steps:

[0016] Obtain the training domain dataset and the non-training domain dataset;

[0017] With the parameters of the super-resolution backbone network to be trained frozen, the low-rank adapting expert module in the super-resolution backbone network is trained using the training domain dataset.

[0018] With the parameters of the low-rank adaptive expert module frozen after training, the hierarchical routing subnetwork in the super-resolution backbone network is trained using a non-training domain dataset to generate an image super-resolution model.

[0019] This application also provides a smart terminal, including: a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the method described above.

[0020] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0021] The image super-resolution reconstruction method provided in this application includes: acquiring an image to be processed; processing the image to be processed using an image super-resolution model to obtain a super-resolution reconstructed image corresponding to the image to be processed; wherein, the image super-resolution model includes a super-resolution backbone network, which includes a low-rank adaptation expert module and a hierarchical routing sub-network. In the above technical solution, since the image super-resolution model includes a low-rank adaptation expert module and a hierarchical routing sub-network, by independently training the low-rank adaptation expert module using the training domain dataset, efficient storage of multi-domain knowledge can be achieved, and the catastrophic forgetting caused by traditional fine-tuning methods can be avoided; by training the hierarchical routing sub-network using the non-training domain dataset, the parameters of the super-resolution backbone network and the low-rank adaptation expert module do not need to be updated, and adaptation to the unknown domain can be completed, which helps to improve the image super-resolution reconstruction effect of the image super-resolution model in the non-training domain. Therefore, using the trained image super-resolution model, high-quality super-resolution reconstructed images can be generated, thus improving the image super-resolution reconstruction effect. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0023] Figure 1 A schematic diagram of the hardware structure of a smart terminal to implement the various embodiments of this application;

[0024] Figure 2 A communication network system architecture diagram provided for an embodiment of this application;

[0025] Figure 3 This is a flowchart illustrating the image super-resolution reconstruction method according to the first embodiment;

[0026] Figure 4 This is a flowchart illustrating the image super-resolution model training method according to the second embodiment.

[0027] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0030] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, may be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of a feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., used in this application may be interpreted as inclusive, or mean any one or any combination thereof. For example, "including at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0031] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0032] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0033] It should be noted that step designations such as S10 and S20 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S20 first and then S10, etc., but these should all be within the protection scope of this application.

[0034] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0035] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0036] Smart terminals can be implemented in various forms. For example, the smart terminals described in this application may include smart terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0037] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to the embodiments of this application can also be applied to fixed-type terminals.

[0038] Please see Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal implementing various embodiments of this application. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0039] The following is combined with Figure 1 A detailed introduction to each component of the mobile terminal:

[0040] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G, and 6G.

[0041] WiFi is a short-range wireless transmission technology. Mobile terminals, through the WiFi module 102, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 102 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted as needed without changing the nature of this application.

[0042] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the mobile terminal 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, or other modes. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the mobile terminal 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0043] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0044] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0045] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0046] User input unit 107 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile terminal. Optionally, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands sent by processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Optionally, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being specifically limited here.

[0047] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal. The specific implementation is not limited here.

[0048] Interface unit 108 serves as an interface through which at least one external device can connect to mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more components within mobile terminal 100, or it may be used to transmit data between mobile terminal 100 and the external device.

[0049] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0050] The processor 110 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0051] The mobile terminal 100 may also include a power supply 111 (such as a battery) that supplies power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0052] although Figure 1 As not shown, the mobile terminal 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0053] To facilitate understanding of the embodiments of this application, the communication network system on which the mobile terminal of this application is based is described below.

[0054] Please see Figure 2 , Figure 2 This application provides a communication network system architecture diagram. The communication network system is an LTE system based on the universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and an operator's IP service 204, which are connected in sequence.

[0055] Optionally, UE201 can be the aforementioned terminal 100, which will not be described in detail here.

[0056] E-UTRAN202 includes eNodeB2021 and other eNodeB2022s. Optionally, eNodeB2021 can connect to other eNodeB2022s via backhaul (e.g., X2 interface). eNodeB2021 connects to EPC203 and can provide UE201 with access to EPC203.

[0057] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gateway) 2034, PGW (Packet Data Network Gateway) 2035, and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME2031 is the control node that handles signaling between UE201 and EPC203, providing bearer and connection management. HSS2032 is used to provide registers to manage functions such as the Home Location Register (not shown in the figure) and stores user-specific information such as service characteristics and data rates. All user data can be sent through SGW2034. PGW2035 can provide IP address allocation and other functions for UE201. PCRF2036 is the policy and charging control decision point for service data flow and IP bearer resources. It selects and provides available policy and charging control decisions for the policy and charging enforcement function unit (not shown in the figure).

[0058] IP services 204 may include the Internet, intranet, IMS (IP Multimedia Subsystem), or other IP services.

[0059] Although the above description uses the LTE system as an example, those skilled in the art should understand that this application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems (such as 5G), etc., without limitation.

[0060] Based on the above-described mobile terminal hardware structure and communication network system, various embodiments of this application are proposed.

[0061] Super-resolution (SR) technology has wide applications in key fields such as remote sensing, medical imaging, industrial inspection, virtual reality, and digital content restoration.

[0062] For example, in the field of remote sensing, satellite images and drone aerial images require high resolution to support ground feature identification and disaster monitoring; in medical imaging, high-resolution images are crucial for lesion localization and diagnosis; and in the field of digital content generation, images or screenshots generated by artificial intelligence (AI) need to be enhanced with super-resolution to meet visual requirements.

[0063] However, in different application scenarios, images exhibit significant differences in semantic content, statistical texture, noise characteristics, spectral properties, and acquisition conditions, leading to a marked decline in the performance of traditional image super-resolution models outside the training domain. For example, an image super-resolution model trained on natural images may not be able to effectively handle high-contrast edges in medical images or complex textures in remote sensing images.

[0064] In addition, in practical applications, it is often difficult to obtain a large amount of sample data from outside the training domain for model fine-tuning, and computational resources are limited.

[0065] In related technologies, the domain offset problem in super-resolution tasks is generally solved through the following three types of methods:

[0066] The pre-training-fine-tuning paradigm involves pre-training the model on a large-scale natural image dataset and then fine-tuning all parameters on the target domain dataset. While this method can improve performance in the target domain, it requires a large amount of out-of-domain data and computational resources, and the fine-tuning process can easily lead to a decline in the performance of the training domain (catastrophic forgetting).

[0067] Test-Time Adaptation (TTA) methods dynamically optimize model parameters to fit a single test image during the inference phase. While these methods do not require pre-collection of out-of-domain data, they necessitate repeated optimization for each image, resulting in high computational costs and the inability to reuse learned inter-domain knowledge.

[0068] Domain adaptation (DA) methods reduce the distributional differences between the training and target domains by aligning feature distributions or introducing domain discriminators. For example, some methods use adversarial training to force the model to learn domain-invariant features, but such methods have limited effectiveness in complex degradation patterns or cross-content domain adaptation and require the design of complex loss functions.

[0069] In summary, how to provide an image super-resolution reconstruction method that can efficiently adapt to multi-domain scenarios and significantly improve the generalization ability of unknown domains while maintaining the performance of the training domain is a technical problem that urgently needs to be solved.

[0070] To address the aforementioned technical challenges, this application provides an image super-resolution reconstruction method. This method employs a two-stage decoupled training strategy, splitting the image super-resolution model training task into two phases and introducing a low-rank adapting expert module and a hierarchical routing sub-network. In the first phase, the parameters of the super-resolution backbone network are frozen, and the low-rank adapting expert module is trained independently using the training domain dataset. This achieves efficient storage of multi-domain knowledge and avoids the catastrophic forgetting that occurs with traditional fine-tuning methods. In the second phase, the hierarchical routing sub-network is trained using a non-training domain dataset. This eliminates the need to update the parameters of the super-resolution backbone network and the low-rank adapting expert module, enabling adaptation to unknown domains and improving the image super-resolution reconstruction performance in non-training domains.

[0071] The technical solution of this application and how the technical solution of this application solves the technical problem are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0072] First Embodiment

[0073] Reference Figure 3 , Figure 3 This is a schematic flowchart of the image super-resolution reconstruction method according to the first embodiment. The above-described image super-resolution reconstruction method can be executed by a smart terminal and includes the following steps:

[0074] S10. Obtain the image to be processed.

[0075] Alternatively, the image to be processed can be a low-resolution image.

[0076] S20. Use an image super-resolution model to process the image to be processed to obtain the super-resolution reconstructed image corresponding to the image to be processed.

[0077] The image super-resolution model includes a super-resolution backbone network, which comprises a low-rank adaptation expert module and a hierarchical routing sub-network.

[0078] In some implementations, the parameters of the low-rank adaptation expert module are generated by training on the training domain dataset while freezing the parameters of the super-resolution backbone network, and the parameters of the hierarchical routing subnetwork are generated by training on the non-training domain dataset while freezing the parameters of the low-rank adaptation expert module.

[0079] In some implementations, a low-resolution image can be input into an image super-resolution model, and intermediate features can be obtained using a super-resolution backbone network. Then, the weights of each low-rank adaptation expert module are calculated and fused through a hierarchical routing sub-network to perform weighted reconstruction of the intermediate features. The fused features are then used by the upsampling decoding part of the super-resolution backbone network to generate the final high-resolution reconstructed image.

[0080] The image super-resolution reconstruction method provided in this application embodiment allows the hierarchical routing subnetwork in the image super-resolution model to dynamically calculate the fusion weights of each low-rank adaptation expert module based on the input features after inputting an unknown domain image into the trained image super-resolution model. The weighted output of the low-rank adaptation expert module is then fused with the original features of the super-resolution backbone network to generate the final high-resolution image. This method can achieve high-quality image reconstruction in both the training and non-training domains.

[0081] Second Embodiment

[0082] Based on the above embodiments of this application, a second embodiment of this application is proposed.

[0083] Reference Figure 4 , Figure 4 This is a flowchart illustrating the image super-resolution model training method according to the second embodiment. The above-described image super-resolution model training method includes the following steps:

[0084] S30. Obtain the training domain dataset and the non-training domain dataset.

[0085] The training domain dataset includes the content domain difference dataset and the degradation domain difference dataset.

[0086] Optionally, the content domain difference dataset includes image samples of different content categories collected under conditions of uniform degradation, used as training data to construct the content domain difference dataset.

[0087] For example, content domain difference datasets may include image samples such as remote sensing images, medical images, animation images, AI-generated images, and screenshots.

[0088] Optionally, the degradation domain difference dataset includes image samples of different degradation types collected under the condition of uniform content category, which are used as training data to construct the degradation domain difference dataset.

[0089] For example, a degradation domain difference dataset may include degraded images generated by a degradation process, a DIV2K unknown degradation dataset, and a RealSR dataset.

[0090] DIV2K is a widely used image super-resolution dataset that contains a large number of high-resolution images and their corresponding low-resolution images.

[0091] In the DIV2K Unknown Degradation dataset, low-resolution images are generated using unknown degradation operators, which gives the low-resolution images in the DIV2K Unknown Degradation dataset unknown degradation features.

[0092] The RealSR dataset is a real-world single-image super-resolution dataset that acquires images of the same scene at different resolutions by adjusting the focal length of a digital camera and then uses an image registration algorithm to align the images at different resolutions.

[0093] Images in the RealSR dataset exhibit realistic degradation features stemming from various factors during the camera imaging process, such as lens distortion, exposure time, and optical distortion.

[0094] In addition, the RealSR dataset contains rich texture and detail information, providing abundant training data for super-resolution reconstruction models.

[0095] Optionally, the non-training domain dataset includes image samples from unknown domains, which can be collected from data sources in the tested non-training domain, having unknown content and degradation patterns.

[0096] In this context, image samples in the unknown domain refer to image data that were not encountered by the image super-resolution model during the training phase but need to be processed during the testing or application phase. These image samples in the unknown domain can come from new scenes, new categories, or new modalities with a different distribution than the training domain dataset.

[0097] S40. With the parameters of the super-resolution backbone network to be trained frozen, the low-rank adapting expert module in the super-resolution backbone network is trained using the training domain dataset.

[0098] In some implementations, low-rank adaptation expert modules for multiple domains can be pre-built and inserted into the super-resolution backbone network to be trained.

[0099] Optionally, a super-resolution backbone network can be built based on any convolutional architecture or Transformer architecture.

[0100] In some implementations, the parameters of the low-rank adaptation expert modules for different domains can be trained using the training domain dataset while freezing the parameters of the super-resolution backbone network, so as to obtain the parameters of the low-rank adaptation expert modules corresponding to each domain.

[0101] After training is completed, the parameters of each low-rank adaptive expert module are kept fixed and not updated, while the structure and parameters of the super-resolution backbone network remain unchanged.

[0102] In this stage, by keeping the parameters of the super-resolution backbone network fixed, each low-rank adaptation expert module is trained separately using only the training domain dataset to adapt to different training domains. This allows us to retain general knowledge and improve the reconstruction performance of specific domains with only a few parameters updated.

[0103] S50. With the parameters of the low-rank adaptive expert module frozen after training, the hierarchical routing subnetwork in the super-resolution backbone network is trained using the non-training domain dataset to generate an image super-resolution model.

[0104] In some implementations, a hierarchical routing subnetwork can be constructed in the super-resolution backbone network for the trained low-rank adaptation expert module. With the parameters of the low-rank adaptation expert module and the super-resolution backbone network frozen, the hierarchical routing subnetwork can be trained using a small number of samples from the unknown target domain (from the non-training domain dataset) to achieve adaptive fusion with the low-rank adaptation expert module.

[0105] Optionally, the hierarchical routing subnetwork can be used to calculate and assign fusion weights to each low-rank adaptation expert module based on the features of the input image, and fuse the output results of each low-rank adaptation expert module to complete the super-resolution reconstruction task.

[0106] The hierarchical routing subnetwork can be deployed in the super-resolution backbone network at the hierarchical position corresponding to the low-rank adaptation expert module, so as to perform weighted fusion of the features at that position. Other hierarchies without a configured low-rank adaptation expert module will not introduce a routing subnetwork.

[0107] The hierarchical routing subnetwork is only deployed in the super-resolution backbone network at the hierarchical position corresponding to the low-rank adaptation expert module. It can output the weights of each low-rank adaptation expert module at that level according to the intermediate features of the input image, thereby achieving adaptive fusion of knowledge from different domains. In the hierarchy where no low-rank adaptation expert module is configured, the routing subnetwork is not introduced, which can reduce additional computational overhead and maintain the stability of feature propagation.

[0108] During training, a non-training domain dataset can be used as input, which contains a small number of high- and low-resolution image pairs from an unknown domain.

[0109] In this stage, with a small amount of sample data from the non-training domain (the domain not involved in training), the parameters of all low-rank adaptation expert modules are kept unchanged, and a lightweight, layer-by-layer routing subnetwork is trained. This allows for the dynamic fusion of the feature reconstruction capabilities of each low-rank adaptation expert module during inference, thereby improving the super-resolution reconstruction effect of images in the non-training domain.

[0110] Furthermore, this stage requires only a small amount of additional data and computational cost to maintain the reconstruction performance of the original training domain and improve the transfer capability in unknown non-training domains, making it suitable for both convolution-based and Transformer-based super-resolution networks.

[0111] In this embodiment, the two processing stages correspond to the intra-domain specialization of the low-rank adaptation expert modules and the cross-domain adaptation of the hierarchical routing subnetwork, respectively. This ensures that the performance of the original training domain is maintained while improving the image super-resolution reconstruction effect in the unknown domain. Specifically, when facing non-training domain data with only a few samples available, all low-rank adaptation expert modules are frozen, and a lightweight hierarchical routing subnetwork is trained. The low-rank adaptation expert modules at each layer are dynamically selected or mixed without updating the super-resolution backbone network, thus achieving domain adaptation.

[0112] The image super-resolution model training method provided in this application splits the image super-resolution model training task into two stages and introduces a low-rank adapting expert module and a hierarchical routing sub-network. In the first stage, the parameters of the super-resolution backbone network are frozen, and the low-rank adapting expert module is trained independently using the training domain dataset. This achieves efficient storage of multi-domain knowledge and avoids catastrophic forgetting caused by traditional fine-tuning methods. In the second stage, the hierarchical routing sub-network is trained using the non-training domain dataset. This eliminates the need to update the parameters of the super-resolution backbone network and the low-rank adapting expert module, enabling adaptation to the unknown domain and helping to improve the image super-resolution reconstruction effect of the image super-resolution model in the non-training domain.

[0113] Third Embodiment

[0114] Based on the above embodiments of this application, a third embodiment of this application is proposed.

[0115] In this embodiment, at least one low-rank adaptation expert module can be inserted into a designated layer of the super-resolution backbone network according to a preset insertion rule.

[0116] In some implementations, based on a pre-trained super-resolution backbone network, multiple low-rank adaptation expert modules can be combined with multiple specified layers of the super-resolution backbone network according to preset insertion rules. While keeping the original parameters of the super-resolution backbone network unchanged, each low-rank adaptation expert module adapts the content domain difference dataset and the degradation domain difference dataset in the training domain dataset and performs independent training to obtain low-rank adaptation expert modules for content domain difference and degradation domain difference.

[0117] Optionally, the low-rank adaptation expert module includes a linear transformation unit, and the low-rank adaptation expert module is located in the linear layer of the super-resolution backbone network.

[0118] Alternatively, the low-rank adaptation expert module includes convolutional units; the low-rank adaptation expert module is located in the convolutional layer of the deep feature extraction module of the super-resolution backbone network.

[0119] Optionally, the weight matrix of the low-rank adaptation expert module can be represented in a low-rank decomposition form.

[0120] In some implementations, a low-rank adaptation expert module can be inserted into a linear layer of the super-resolution backbone network; or, a low-rank adaptation expert module can be inserted into a convolutional layer of a deep feature extraction module of the super-resolution backbone network.

[0121] Optionally, for a Transformer-based super-resolution backbone network, the low-rank adaptor expert module is inserted at all linear layers; for a convolution-based super-resolution backbone network, the low-rank adaptor expert module is inserted at the core convolutional layer of the deep feature extraction module.

[0122] In some implementations, a low-rank adaptation expert module can be inserted into a selected deep feature extraction layer in the super-resolution backbone network. This low-rank adaptation expert module can be composed of residual branches formed by low-rank matrix decomposition and used to learn the domain-specific representation of the target domain features.

[0123] For super-resolution backbone networks using a Convolutional Neural Network (CNN) architecture, the low-rank adaptation expert module can adopt a low-rank decomposition form of two-dimensional convolutional kernels (Conv-LoRA). The residual branch formed by this low-rank adaptation expert module can decompose the input feature map through low-rank convolutional kernels of the content domain difference dataset and the degradation domain difference dataset, and then fuse them according to the scaling factor α to achieve domain adaptation of the convolutional weights.

[0124] For super-resolution backbone networks using Transformers, the low-rank adaptation expert module can take the form of linear-LoRA. The low-rank adaptation expert module is added to the query, key, value, and output layers of the multi-head self-attention module and the linear projection layer of the feedforward network. The projection weights are adjusted by low-rank matrix factorization to achieve domain specialization without updating the original parameters of the super-resolution backbone network.

[0125] Specifically, for multiple training sets of different specific domains, corresponding low-rank adapting expert modules are trained independently to obtain a domain-specific expert database E={E1, E2, ..., E...} K By keeping the parameters of the super-resolution backbone network unchanged during this process, each domain expert can learn and store only their own domain pattern parameters, thereby avoiding parameter interference between different domains.

[0126] The method provided in this application can efficiently model multi-domain features by inserting low-rank adapting expert modules without changing the parameters of the super-resolution backbone network. Furthermore, it enables dynamic combination and optimal interpolation of low-rank adapting expert modules by training a hierarchical routing sub-network using a small number of samples in unknown domain scenarios, thus achieving rapid domain adaptation under low data volume conditions. Compared with related techniques that rely on large-scale retraining or single-domain models, this application improves the generalization and adaptation capabilities in unknown domains while maintaining the long-term stability and specialization of low-rank adapting expert modules in each domain. Simultaneously, it requires only a small amount of additional parameters and computational overhead, and can adapt to both CNN and Transformer architectures, making it suitable for image super-resolution tasks of different types and resolutions.

[0127] Fourth embodiment

[0128] Based on any of the above embodiments of this application, a fourth embodiment of this application is proposed.

[0129] In this embodiment, the hierarchical routing subnetwork includes a weight calculation and fusion subunit. This weight calculation and fusion subunit is used to calculate the weights of each low-rank adaptation expert module based on the features of image samples in the non-training domain dataset, and to fuse the output results of each low-rank adaptation expert module based on the weights of each low-rank adaptation expert module.

[0130] Optionally, the fusion weights output by the above-mentioned hierarchical routing sub-networks satisfy the normalization constraint and are non-negative.

[0131] In some implementations, image samples from the non-training domain dataset can be used as ground truth to calculate the pixel-level reconstruction loss between the reconstructed image and the ground truth; the parameters of the hierarchical routing subnetwork can then be adjusted based on this pixel-level reconstruction loss.

[0132] For example, the loss of the hierarchical routing subnetwork can be trained using a small training set of unknown domains, which is the pixel-level reconstruction loss between the ground truth high-resolution image and the reconstructed image.

[0133] In some implementations, the hierarchical routing subnetwork is located at a designated layer in the super-resolution backbone network where low-rank adaptation expert modules are inserted.

[0134] In some implementations, a designated layer of low-rank adaptation expert modules can be inserted into the super-resolution backbone network to establish a hierarchical routing subnetwork.

[0135] The hierarchical routing subnetwork is built on all designated layers during the first-stage training of the low-rank adaptation expert module, and fine-grained control of the features of the intermediate layers of the image super-resolution model is used to improve the reconstruction effect.

[0136] In this stage, the parameters of all trained low-rank adaptation expert modules can be kept unchanged, and only the hierarchical routing subnetwork is trained. This hierarchical routing subnetwork can be deployed in the deep feature extraction layer of the super-resolution backbone network to select and combine the outputs of the low-rank adaptation expert modules from various domains based on the intermediate feature representations of the current input image.

[0137] For example, a hierarchical routing subnet can be in the first... Layer reads intermediate feature tensors The feature vector is obtained by Global Average Pooling (GAP). , The feature vector is mapped through a trainable linear mapping matrix Transform into a K-dimensional expert weight score vector The fusion coefficients are obtained by Softmax normalization. At this layer, the output residuals of each low-rank adaptation expert module are... With fusion coefficient The fused residuals are obtained by weighted summation. :

[0138] ;in, Let represent a d-dimensional real space.

[0139] Furthermore, the fusion residual With backbone network raw output Add them together to obtain the updated features of that layer. :

[0140] ;

[0141] In some implementations, the parameters of the hierarchical routing subnetwork are trained using a small number of non-training domain samples. The optimization objective is:

[0142] ;

[0143] Where L is the super-resolution reconstruction loss under task constraints. This is a high-resolution reference image outside the training domain.

[0144] The method provided in this application, by using a non-training domain dataset to train a hierarchical routing subnetwork, can make the combination of each low-rank adaptation expert module in the new domain approximate the distribution of the unknown domain as closely as possible, achieving efficient adaptation to the unknown domain under a small number of samples, and avoiding catastrophic forgetting.

[0145] Fifth Embodiment

[0146] Based on any of the above embodiments of this application, a fifth embodiment of this application is proposed.

[0147] In some implementations, the above image super-resolution model can be tested using multi-class non-natural image datasets and real-world degradation datasets.

[0148] For example, multiple representative domains, such as remote sensing, medical imaging, artificially generated images, screenshots, and ordinary natural images, can be selected as training domains and test sets, along with several unknown domain test sets of different styles and contents. Within the domain datasets, a certain number of image samples are selected for each class as both training and test sets. For non-natural image datasets, a bicubic interpolation method is uniformly used for downsampling to create low-resolution images. The low-resolution images in the real-world degradation dataset can include various degradation types. The out-of-domain datasets are used to verify the generalization performance of the image super-resolution model in unknown domains.

[0149] In some implementations, a four-fold super-resolution performance test can be performed on each dataset, and metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) can be recorded.

[0150] PSNR assesses the distortion of the processed image relative to the original image by calculating the peak signal-to-noise ratio between the original and processed images. SSIM is a metric used to measure the similarity between two images, evaluating their similarity in terms of brightness, contrast, and structure.

[0151] The method provided in this application improves the super-resolution reconstruction performance of the image super-resolution model in both the training and non-training domains by testing the model with multi-class non-natural image datasets and real-world degradation datasets.

[0152] This application also provides a smart terminal, which includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the image super-resolution reconstruction method or the steps of the image super-resolution model training method in any of the above embodiments.

[0153] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image super-resolution reconstruction method or the image super-resolution model training method in any of the above embodiments.

[0154] In the embodiments of the smart terminal and computer-readable storage medium provided in this application, all the technical features of any of the above-described method embodiments may be included. The extended and explanatory content of the specification is basically the same as that of the embodiments of the above methods, and will not be repeated here.

[0155] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform the methods described in the various possible implementations above.

[0156] This application also provides a chip, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device with the chip installed performs the methods described in the various possible implementations above.

[0157] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0158] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0159] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0160] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0161] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0162] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0163] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0165] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, storage disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0166] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image super-resolution reconstruction method, characterized in that, The method includes the following steps: Obtain the image to be processed; The image to be processed is processed using an image super-resolution model to obtain a super-resolution reconstructed image corresponding to the image to be processed; wherein, the image super-resolution model includes a super-resolution backbone network, and the super-resolution backbone network includes a low-rank adaptation expert module and a hierarchical routing sub-network.

2. The method according to claim 1, characterized in that, The method further includes: The parameters of the low-rank adaptation expert module are generated by training with the training domain dataset while freezing the parameters of the super-resolution backbone network. The parameters of the hierarchical routing sub-network are generated by training with the non-training domain dataset while freezing the parameters of the low-rank adaptation expert module.

3. The method according to claim 2, characterized in that, The training domain dataset includes a content domain difference dataset and a degradation domain difference dataset; the non-training domain dataset includes image samples from unknown domains.

4. The method according to claim 1, characterized in that, The low-rank adaptation expert module includes a linear transformation unit; the low-rank adaptation expert module is located in the linear layer of the super-resolution backbone network.

5. The method according to claim 1, characterized in that, The low-rank adaptation expert module includes convolutional units; The low-rank adaptation expert module is located in the convolutional layer of the deep feature extraction module of the super-resolution backbone network.

6. The method according to claim 2, characterized in that, The hierarchical routing subnetwork includes a weight calculation and fusion subunit, which is used to calculate the weights of each of the low-rank adaptation expert modules based on the features of the image samples in the non-training domain dataset, and to fuse the output results of each of the low-rank adaptation expert modules based on the weights of each of the low-rank adaptation expert modules.

7. The method according to claim 1 or 6, characterized in that, The hierarchical routing subnetwork is located at a designated layer in the super-resolution backbone network into which the low-rank adaptation expert module is inserted.

8. A method for training an image super-resolution model, characterized in that, The method includes the following steps: Obtain the training domain dataset and the non-training domain dataset; With the parameters of the super-resolution backbone network to be trained frozen, the low-rank adapting expert module in the super-resolution backbone network is trained using the training domain dataset. With the parameters of the low-rank adaptive expert module frozen after training, the hierarchical routing subnetwork in the super-resolution backbone network is trained using the non-training domain dataset to generate an image super-resolution model.

9. A smart terminal, characterized in that, The smart terminal includes: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

Images