Training method of image super-resolution reconstruction system, image reconstruction method and system

By using backbone network and prompt mapping generation module in the image super-resolution reconstruction system, prompt mapping functions are trained for different degradation tasks, and the problems of poor generalization ability and knowledge forgetting in the existing technology are solved, and efficient image super-resolution is achieved Rate reconstruction and cross-domain adaptability.

CN118710501BActive Publication Date: 2024-11-26XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411186704.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-26
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Existing image super-resolution technologies are difficult to achieve good generalization capabilities between different scenarios and image types, and traditional prompt generation strategies are difficult to maintain between existing knowledge and new task learning when efficiently covering the multi-dimensional details requirements of degraded tasks. Balance leads to increased difficulty in cross-task transfer learning and knowledge forgetting.

Method used

The image super-resolution reconstruction system is adopted, including a backbone network and a prompt mapping generation module. By training the prompt mapping generation module for different degradation tasks, the prompt mapping function corresponding to the degradation task is obtained, the adaptive prompt information is dynamically constructed, and the number of model training parameters is reduced. Improve model flexibility and adaptability.

Benefits of technology

It significantly improves the adaptability and computing efficiency of the model, reduces the phenomenon of knowledge forgetting, realizes continuous learning, can provide customized context information in cross-domain applications, and enhances the accuracy and stability of image super-resolution reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710501B_ABST
    Figure CN118710501B_ABST
Patent Text Reader

Abstract

The present invention provides a training method, an image reconstruction method and a system for an image super-resolution reconstruction system, wherein the training method comprises: for any degradation task: obtaining a plurality of to-be-reconstructed images corresponding to the degradation task; based on a backbone network, training a prompt mapping generation module according to each to-be-reconstructed image and a task identifier of a predetermined degradation task, obtaining a prompt mapping function corresponding to the degradation task, and then obtaining a trained image super-resolution reconstruction system, wherein the trained image super-resolution reconstruction system can dynamically construct adaptive prompt information, adaptively provide customized context information for different degradation tasks, and significantly reduce the required model training parameter amount even when keeping feature diversity unchanged, thereby improving the flexibility of the model, significantly enhancing the adaptability and computational efficiency of the model, and effectively alleviating the common knowledge forgetting phenomenon in cross-domain applications, thereby realizing continuous learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a training method for an image super-resolution reconstruction system, an image reconstruction method and a system. Background Art

[0002] Image super-resolution technology, as a key means to improve image quality, has played a vital role in many fields such as natural image analysis and medical imaging. Its fundamental purpose is to restore high-resolution details from low-resolution images to enhance visual experience and analysis accuracy. Although many methods have brought significant progress in the field of image super-resolution, most of these methods are customized for specific scenarios and it is difficult to achieve good generalization capabilities across scenarios and image types. In addition, the ever-changing data requirements force the model to continuously learn new scenarios, which poses a huge challenge to computing resources.

[0003] In recent years, cue-based continuous learning strategies have achieved certain success in tasks such as image classification. Among them, the use of multi-layer perceptrons as adaptive cue generators can effectively balance the knowledge preservation of historical tasks and the learning of new tasks. However, this method is limited in effectiveness when faced with the pixel-level accuracy requirements of super-resolution tasks or complex degradation patterns. Traditional cue generation strategies have difficulty maintaining a balance between existing knowledge and new task learning while efficiently covering the multi-dimensional detail requirements of degradation tasks. The significant differences between image data in the field of continuous image super-resolution, especially the inconsistency in low-level features such as texture and color, also increase the difficulty of cross-task transfer learning. In addition, the diversity and complexity of image degradation mechanisms, from simple interpolation to specialized degradation in medical imaging, further widens the domain gap between tasks. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a training method, an image reconstruction method and a system for an image super-resolution reconstruction system, which specifically include:

[0005] In a first aspect, the present invention provides a training method for an image super-resolution reconstruction system, which is applied to the image super-resolution reconstruction system, wherein the image super-resolution reconstruction system includes a backbone network and a prompt map generation module;

[0006] The method comprises:

[0007] For any degenerate task:

[0008] Acquire multiple images to be reconstructed corresponding to the degradation task;

[0009] Based on the backbone network, according to the task identifiers of each image to be reconstructed and the predetermined degradation task, a prompt mapping generation module is trained to obtain the prompt mapping function corresponding to the degradation task, and then a trained image super-resolution reconstruction system is obtained.

[0010] In a second aspect, the present invention provides an image reconstruction method, comprising:

[0011] Acquire an image to be reconstructed;

[0012] Extracting query features of the image to be reconstructed;

[0013] Determining the task corresponding to the image to be reconstructed according to the query feature of the image to be reconstructed and the query feature group corresponding to the plurality of pre-determined degradation tasks;

[0014] Inputting the query features of the image to be reconstructed and the tasks corresponding to the image to be reconstructed into a prompt map generation module in the image super-resolution reconstruction system trained according to the training method of any image super-resolution reconstruction system provided in the first aspect, so that the prompt map generation module obtains prompt information corresponding to the image to be reconstructed according to the query features of the image to be reconstructed and the tasks corresponding to the image to be reconstructed, and the prompt mapping functions corresponding to the degradation tasks obtained in advance through training;

[0015] The query features of the image to be reconstructed and the prompt mapping function corresponding to the image to be reconstructed are input into the backbone network in the image super-resolution reconstruction system trained according to the training method of any image super-resolution reconstruction system provided in the first aspect, so that the backbone network reconstructs the image to be reconstructed according to the query features of the image to be reconstructed and the prompt information corresponding to the image to be reconstructed.

[0016] In a third aspect, the present invention further provides an image super-resolution reconstruction system, comprising:

[0017] Image preprocessing module, task matching module, cue mapping module and backbone network;

[0018] An image preprocessing module, used to obtain the image to be reconstructed and extract query features of the image to be reconstructed;

[0019] A task matching module determines the task corresponding to the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module and the query feature group corresponding to the plurality of pre-determined degradation tasks stored;

[0020] A prompt mapping module is used to obtain prompt information corresponding to the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module, the task corresponding to the image to be reconstructed determined by the task matching module, and the prompt mapping functions corresponding to various degradation tasks obtained by pre-training;

[0021] The backbone network is used to reconstruct the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module and the prompt information corresponding to the image to be reconstructed obtained by the prompt mapping module.

[0022] Beneficial effects of the present invention:

[0023] The present invention provides a training method for an image super-resolution reconstruction system, an image reconstruction method and a system. The training method obtains a plurality of to-be-reconstructed images corresponding to any degradation task, and based on a backbone network, trains a prompt mapping generation module according to the task identifiers of the to-be-reconstructed images and predetermined degradation tasks, obtains a prompt mapping function corresponding to the degradation task, and then obtains a trained image super-resolution reconstruction system. The trained image super-resolution reconstruction system can dynamically construct adaptive prompt information, adaptively provide customized context information for different degradation tasks, and significantly reduce the required model training parameter amount even when keeping feature diversity unchanged, thereby improving the flexibility of the model, significantly enhancing the adaptability and computational efficiency of the model, and effectively alleviating the common knowledge forgetting phenomenon in cross-domain applications, thereby realizing continuous learning.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a training method for an image super-resolution reconstruction system provided by the present invention;

[0026] Figure 2 A schematic diagram of the architecture of a window-based Transformer module provided by the present invention;

[0027] Figure 3 A schematic diagram of a flow chart of an image reconstruction method provided by the present invention;

[0028] Figure 4 A schematic diagram of the architecture of an image reconstruction system provided by the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0030] Figure 1 A flow chart of a training method for an image super-resolution reconstruction system provided by the present invention is provided. The method is applied to the image super-resolution reconstruction system. The image super-resolution reconstruction system includes a backbone network and a prompt map generation module.

[0031] like Figure 1 As shown, for any degradation task, the method includes:

[0032] S101. Acquire a plurality of to-be-reconstructed images corresponding to a degradation task.

[0033] The degradation tasks correspond to different image degradation situations.

[0034] Image degradation refers to the decrease in the quality of the image acquired during the formation, recording, processing and transmission of the image due to the imperfections of the imaging system, recording equipment, transmission medium and processing method. This phenomenon manifests itself as the image becoming blurred, distorted or noisy. There are many reasons for image degradation, including motion blur caused by the movement of the target or shooting device, blur caused by long exposure, focus misalignment, blur caused by light angle, blur caused by atmospheric disturbance, etc. In addition, image degradation can also occur if the exposure time is too short and the shooting device captures too few photons. Image degradation not only affects the clarity and details of the image, but may also mask important information in the image, thereby affecting the analysis and application of the image.

[0035] S102, based on the backbone network, according to each image to be reconstructed and the task identifier of the predetermined degradation task, training the prompt mapping generation module to obtain the prompt mapping function corresponding to the degradation task, and then obtaining a trained image super-resolution reconstruction system.

[0036] This method can be used to train the prompt mapping function corresponding to each degradation task and establish the corresponding relationship between the degradation task and the prompt mapping function.

[0037] Exemplarily, before S102, query feature groups corresponding to multiple degradation tasks and task identifiers corresponding to each query feature group are first determined and stored, so as to establish a correspondence between the degradation tasks, the query feature groups, and the task identifiers. Then, after obtaining the query features of the image to be reconstructed, the query features of the image to be reconstructed are matched with the query feature groups corresponding to each degradation task. If the match is successful, it is determined that the image to be reconstructed belongs to the degradation task that successfully matches it. Further, according to the correspondence between the degradation task and the task identifier, the task identifier corresponding to the image to be reconstructed is determined.

[0038] When determining the query feature groups and task identifiers corresponding to multiple degradation tasks, the query feature groups and task identifiers corresponding to each degradation task may be determined in sequence, or the query feature groups and task identifiers of different degradation tasks may be determined simultaneously.

[0039] In a possible implementation, determining the task identifier of the degraded task includes the following steps A1-A3:

[0040] A1. Obtain multiple training images corresponding to multiple degradation tasks respectively.

[0041] A2. Extract query features of each training image respectively to obtain a query feature group corresponding to each degradation task.

[0042] Specifically, query features of multiple training images corresponding to each degradation task are extracted respectively, and the query features of all training images corresponding to each degradation task are the query feature group corresponding to the task.

[0043] A3. Cluster each task query feature group separately, and determine each cluster center as the task identifier of the corresponding degenerate task.

[0044] Optionally, a K-Means classification algorithm is used to cluster each task query feature group, and each cluster center represents a summary of a class of image features.

[0045] For example, from The first of the degradation tasks training images The query feature is expressed as ,in, represents the dimension of the feature space, The dimension is Then, the K-Means classification algorithm is used to classify the All degenerate tasks The query features of training images Divide into The task identifier is The cluster centers are composed of Indicates The total number of training images corresponding to the degradation task.

[0046] The task identification determined by this method not only effectively distinguishes the characteristics of different degraded tasks, but can also be used to accurately match the input image to be reconstructed to the corresponding task category, ensuring the efficiency and accuracy of task identification.

[0047] Furthermore, in a possible implementation, based on the backbone network, according to each image to be reconstructed and the task identifier of the predetermined degradation task, a prompt mapping generation module is trained to obtain a prompt mapping function corresponding to the degradation task, including the following steps B1-B5:

[0048] B1. In the nth round of training, the query feature of the nth image to be reconstructed is extracted, and the nth image to be reconstructed is any image to be reconstructed.

[0049] Wherein, n is a positive integer greater than or equal to 1.

[0050] Optionally, extracting the query features of the image to be reconstructed includes: extracting the query features of the image to be reconstructed through a command line illustration processor (Contrastive Language-Image Pretraining, CLIP), that is, a CLIP image editor.

[0051] Among them, the normalized output of the last layer of linear transformation of the CLIP image encoder is defined as the query feature of a single image.

[0052] B2. Constructing an nth prompt mapping function according to the query feature of the nth image to be reconstructed, the task identifier of the predetermined degradation task and the nth prompt mapping basis.

[0053] Among them, the nth prompt mapping basis is the prompt mapping basis obtained by the n-1th round of training.

[0054] Specifically, the hint mapping basis is a linear transformer, which is expressed as A two-dimensional matrix, represents the input dimension of the hint mapping basis, Represents the output dimension of the hint map basis.

[0055] In a possible implementation, the hint mapping basis is a linear transformer, and the hint mapping basis satisfies:

[0056] ,

[0057] in, Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, The dimensions are , and The three-dimensional matrix of the real number field, represents the total number of layers of the window-based Transformer module, represents the input dimension of the hint mapping basis, Represents the output dimension of the hint map basis.

[0058] In order to alleviate the problem of trainable parameter expansion, the prompt mapping basis in the above implementation can be further replaced with two low-rank matrices. Specifically, in another possible implementation, the prompt mapping basis is expressed as:

[0059] ,

[0060] in, , , Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, Indicates The first step of the degradation task The weight matrix corresponding to the task identifier is Indicates The shared matrix corresponding to each task identifier of the degenerate task, represents the intermediate variable, Much smaller than and , represents the total number of layers of the window-based Transformer module, represents the input dimension of the hint mapping basis, represents the output dimension of the hint mapping basis, The dimensions are , and The three-dimensional matrix of the real number field, The dimensions are , and The three-dimensional matrix of the real number field, Represents the index of the task identifier corresponding to each degradation task, Indicates the index of the degenerate task.

[0061] This method not only achieves a significant reduction in the number of training parameters, but also cleverly uses matrix decomposition to give the parameters stronger expressiveness. Specifically, As a specific weight matrix for each degradation task, it focuses on capturing the multi-level detailed knowledge within each degradation task, while the degradation task shares the matrix It gathers the generalization information between all clusters in a degenerate task and forms an efficient task-level knowledge framework. and output dimensions The value of can ensure that only a very small amount of training data is needed to generate the prompt mapping basis, which further promotes the lightweight of the model and improves the efficiency of learning and training.

[0062] Furthermore, in a possible implementation, the prompt mapping function is expressed as:

[0063] ,

[0064] in, Indicates The first step of the degradation task The prompt mapping function corresponding to the image, express and The inner product of Indicates The first step of the degradation task The query features corresponding to the images are Indicates The first step of the degradation task A task identifier, Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, Represents the index of the image corresponding to each degradation task, represents the index of the degenerate task, Represents the index of the task identifier corresponding to each degradation task, .

[0065] This method builds a powerful mechanism to combat catastrophic forgetting by cleverly integrating multi-granular task identification and low-rank cue mapping bases, as well as the design of a shared low-rank matrix. This design not only promotes the efficient absorption of new knowledge, but also effectively retains the learning results of previous tasks, ensuring that the model maintains comprehensive task adaptability and long-term performance stability while continuously evolving.

[0066] B3. Based on the backbone network, reconstruct the nth image to be reconstructed according to the query features of the nth image to be reconstructed and the nth prompt mapping function.

[0067] Specifically, the query feature of the nth image to be reconstructed and the nth prompt mapping function are input into the backbone network to obtain a reconstructed nth image to be reconstructed.

[0068] The SwinIR skeleton model can flexibly adapt to the super-resolution processing of various input images. The SwinIR backbone model consists of a head convolution, a feature enhancement module with long-distance skip residual connections, and an upsampling module based on pixel shuffle operations. The feature enhancement module includes six feature enhancement stages, each of which contains six window-based Transformer modules. In each Transformer module, the linear layer can extract the query, key, and value parameters involved in the self-attention mechanism.

[0069] Based on the above characteristics, the SwinIR skeleton model can be used as the backbone network.

[0070] Usually, continuous learning is based on classification tasks and cannot be directly transferred to image tasks. In order to specifically adapt to degradation tasks, the present invention adjusts the key and value variables through the prompt mapping function, so that continuous learning can be adapted to the image field.

[0071] like Figure 2As shown in FIG. 1 , in a possible implementation, the backbone network adopts the SwinIR skeleton model, and the feature enhancement module of the backbone network includes multiple feature enhancement stages, each of which includes multiple layers of window-based Transformer modules; the parameters of the linear layer output in any layer of the window-based Transformer module are expressed as:

[0072] ,

[0073] ,

[0074] in, Indicates The output of the window-based Transformer module for the The first step of the degradation task The key of the image, Indicates the input The first layer in the window-based Transformer module The first step of the degradation task The characteristic information of the image (i.e. The feature information output by the window-based Transformer module of the layer), Indicates The key weight matrix corresponding to the window-based Transformer module of the layer, Indicates The first step of the degradation task The prompt mapping function corresponding to the image, Indicates The output of the window-based Transformer module is about The first step of the degradation task The value of the image, Indicates The value weight matrix corresponding to the window-based Transformer module of the layer, Represents an index operation. represents the index of the window-based Transformer module, represents the index of the degenerate task, Indicates the index of the image corresponding to each degradation task.

[0075] As above, for Layer is a window-based Transformer module. Given the input features and the weight matrix , Represents a dimension of input features. The physical meaning corresponding to this dimension can be set according to the actual situation. Through the above method, it is possible to realize the prompt mapping function corresponding to different images to be reconstructed, adaptively adjust the key and value variables corresponding to the window-based Transformer module, and realize continuous learning.

[0076] B4. When the reconstruction effect of the nth image to be reconstructed does not meet the preset conditions, the current prompt mapping basis is updated according to the reconstruction effect of the nth image to be reconstructed to obtain the n+1th prompt mapping basis, and the n+1th round of training is performed until the reconstruction effect obtained by the xth round of training meets the preset conditions.

[0077] Here, x is a positive integer greater than or equal to n+1.

[0078] It should be noted that the reconstruction effects and preset conditions for the images to be reconstructed corresponding to different degradation tasks are not the same. For example, for an overexposed image, the reconstruction effect may correspond to the image content completion, and the preset condition may correspond to the preset image content completion degree; for a blurred image caused by low pixels, the corresponding reconstruction effect may correspond to the resolution enhancement, and the preset condition may be the preset resolution.

[0079] B5. Determine the prompt mapping function corresponding to the xth round of training as the prompt mapping function corresponding to the degradation task.

[0080] The training method of an image super-resolution reconstruction system provided by the present invention is applied to the image super-resolution reconstruction system, wherein the image super-resolution reconstruction system comprises a backbone network and a prompt mapping generation module. The method, for any degradation task, obtains a plurality of to-be-reconstructed images corresponding to the degradation task, and based on the backbone network, trains the prompt mapping generation module according to the task identifiers of each to-be-reconstructed image and a predetermined degradation task, obtains a prompt mapping function corresponding to the degradation task, and then obtains a trained image super-resolution reconstruction system. The system can continuously learn different degradation tasks, can dynamically construct adaptive prompt information, and adaptively provide customized context information for different degradation tasks. Even when the feature diversity is kept unchanged, the required model training parameter amount can be significantly reduced, the flexibility of the model is improved, the adaptability and operation efficiency of the model are significantly enhanced, and the common knowledge forgetting phenomenon in cross-domain applications is effectively alleviated, thereby realizing continuous learning.

[0081] Figure 3 A flowchart of an image reconstruction method provided by the present invention is shown in FIG. Figure 3 As shown, the method includes:

[0082] S301: Acquire an image to be reconstructed.

[0083] The image to be reconstructed is an image to be super-resolution reconstructed.

[0084] The image to be reconstructed may be an image acquired by any type of image acquisition device, for example, a photo taken by a camera, or a medical image such as a magnetic resonance imaging image, a B-ultrasound image, etc. acquired by a medical device.

[0085] S302: Extract query features of the image to be reconstructed.

[0086] Optionally, extracting the query feature of the image to be reconstructed includes: extracting the query feature of the image to be reconstructed by using a CLIP image editor.

[0087] Among them, the normalized output of the last layer of linear transformation of the CLIP image encoder is defined as the query feature of a single image.

[0088] S303 : Determine the degradation task corresponding to the image to be reconstructed according to the query feature of the image to be reconstructed and the query feature group corresponding to the predetermined plurality of degradation tasks.

[0089] The method for determining the query feature groups corresponding to the multiple degradation tasks may be specifically referred to the specific description of how to determine the task identifier of the degradation task in the above method embodiment, which will not be described in detail here.

[0090] S304. Input the query features of the image to be reconstructed and the degradation tasks corresponding to the image to be reconstructed into a prompt mapping generation module in the image super-resolution reconstruction system trained according to any image super-resolution reconstruction system training method provided by the present invention, so that the prompt mapping generation module obtains prompt information corresponding to the image to be reconstructed according to the query features of the image to be reconstructed and the tasks corresponding to the image to be reconstructed, as well as the prompt mapping functions corresponding to the degradation tasks obtained in advance through training.

[0091] It should be noted that, according to the relevant description in the embodiment corresponding to the above-mentioned training method, there is a mapping relationship between the degradation task, the query feature group, the task identifier, and the prompt mapping function. Therefore, the task identifier corresponding to the image to be reconstructed can also be determined according to the query feature of the image to be reconstructed and the task identifiers corresponding to multiple degradation tasks determined in advance; the query feature of the image to be reconstructed and the task identifier corresponding to the image to be reconstructed are input into the prompt mapping generation module in the image super-resolution reconstruction system trained according to the training method of any image super-resolution reconstruction system provided by the present invention, so that the prompt mapping generation module obtains the prompt information corresponding to the image to be reconstructed according to the query feature of the image to be reconstructed and the task identifier corresponding to the image to be reconstructed, and the prompt mapping functions corresponding to the degradation tasks obtained by pre-training.

[0092] S305. Input the query features of the image to be reconstructed and the prompt information corresponding to the image to be reconstructed into the backbone network in the image super-resolution reconstruction system trained according to the training method of any image super-resolution reconstruction system provided by the present invention, so that the backbone network reconstructs the image to be reconstructed according to the query features of the image to be reconstructed and the prompt information corresponding to the image to be reconstructed.

[0093] The present invention also provides an image super-resolution reconstruction system, see Figure 4 ,include:

[0094] Image preprocessing module 41, task matching module 42, prompt mapping module 43 and backbone network 44.

[0095] The image preprocessing module 41 is used to obtain the image to be reconstructed and extract the query features of the image to be reconstructed. Figure 4 In the figure, the image to be reconstructed is represented as , the query feature of the image to be reconstructed is expressed as .

[0096] The task matching module 42 determines the task corresponding to the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module and the query feature group corresponding to the plurality of pre-determined degradation tasks stored.

[0097] The prompt mapping module 43 is used to obtain prompt information corresponding to the image to be reconstructed based on the query features of the image to be reconstructed extracted by the image preprocessing module, the task corresponding to the image to be reconstructed determined by the task matching module, and the prompt mapping function corresponding to each degradation task obtained by pre-training.

[0098] The backbone network 44 is used to reconstruct the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module and the prompt information corresponding to the image to be reconstructed obtained by the prompt mapping module. Figure 4 In the above figure, the super-resolution reconstructed image is represented as , CONV represents the convolutional layer, and WTB represents the window-based Transformer module.

[0099] It should be noted that different modules in the image super-resolution reconstruction system based on the prompting technology can be deployed in the same electronic device or in different electronic devices. When they are deployed in different electronic devices, the electronic devices can be connected to each other by any communication method.

[0100] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the specific contents and beneficial effects and other related matters can be referred to the partial description of the method embodiment.

[0101] In order to further prove the beneficial effects of the present invention, the present invention also provides simulation data, which are as follows:

[0102] 1. Simulation conditions

[0103] This simulation experiment was conducted on the CPU Intel XEON E5-2680V4 209 CPU, NVIDIA GTX 3090 GPU and Ubuntu 16.04 operating system, using the open source pytorch1.13 from Facebook. The database used the NYU database of indoor images created by Nathan Silberman and Rob Fergus in 2012, the RealSR database of real images proposed by Cai et al., the DIV2K database open sourced by the single image super-resolution challenge NTIRE, and the IXI database consisting of images obtained from MRI scans from three different hospitals in London.

[0104] The low-resolution images in the database are generated using four different degradation modes, including bilateral interpolation, real-world degradation, complex degradation formed by random combination, and frequency domain downsampling degradation.

[0105] 2. Simulation content

[0106] In-depth simulation experiments were conducted compared with the current continuous learning prompt strategy CODA-P. CODA-P uses a set of attention-guided key-query mechanism learning units, and dynamically combines these units according to the input condition weights to form adaptive prompts. As shown in Table 1, after continuous training on four databases, the image reconstruction method based on prompt technology provided by the present invention shows a lower forgetting rate, which is reduced by 5.5%, while achieving superior performance in both average peak signal-to-noise ratio and average structural similarity, which are improved by 3.48% and 17.7% respectively.

[0107] Table 1 Experimental results of continuous training on four datasets

[0108]

[0109] The experimental results shown in Table 1 clearly show that the image reconstruction method based on hint technology provided by the present invention can flexibly cope with various image degradation modes while maintaining a low forgetting rate by virtue of its innovative adaptive hint mapping function, which strongly verifies the technical advantages of the present invention.

[0110] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0111] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A training method for an image super-resolution reconstruction system, characterized in that: Applied to an image super-resolution reconstruction system, the image super-resolution reconstruction system comprising a backbone network and a prompt map generation module; The method comprises: For any degenerate task: Acquire a plurality of to-be-reconstructed images corresponding to the degradation tasks, each of the degradation tasks corresponding to a different image degradation condition; Based on the backbone network, according to each of the to-be-reconstructed images and the predetermined task identifier of the degradation task, the prompt mapping generation module is trained to obtain a prompt mapping function corresponding to the degradation task, thereby obtaining a trained image super-resolution reconstruction system; The step of training the prompt mapping generation module based on the backbone network according to each of the to-be-reconstructed images and the predetermined task identifier of the degradation task to obtain the prompt mapping function corresponding to the degradation task includes: In the nth round of training, extracting query features of the nth image to be reconstructed, the nth image to be reconstructed is any of the images to be reconstructed, and n is a positive integer greater than or equal to 1; Constructing an nth prompt mapping function according to the query feature of the nth image to be reconstructed, the predetermined task identifier of the degradation task and the nth prompt mapping basis, wherein the nth prompt mapping basis is the prompt mapping basis obtained by the n-1th round of training; Based on the backbone network, reconstructing the nth image to be reconstructed according to the query feature of the nth image to be reconstructed and the nth prompt mapping function; When the reconstruction effect of the nth image to be reconstructed does not meet the preset condition, the current prompt mapping basis is updated according to the reconstruction effect of the nth image to be reconstructed to obtain the n+1th prompt mapping basis, and the n+1th round of training is performed until the reconstruction effect obtained by the xth round of training meets the preset condition, where x is a positive integer greater than or equal to n+1; Determine the prompt mapping function corresponding to the xth round of training as the prompt mapping function corresponding to the degradation task; The backbone network adopts a SwinIR skeleton model, and the feature enhancement module of the backbone network includes multiple feature enhancement stages, each of which includes a multi-layer window-based Transformer module; The parameters of the linear layer output in any layer of the window-based Transformer module are expressed as: , , in, Indicates The output of the window-based Transformer module for the The first step of the degradation task The key of the image, Indicates the input The first layer in the window-based Transformer module The first step of the degradation task The feature information of an image, Indicates The key weight matrix corresponding to the window-based Transformer module of the layer, Indicates The first step of the degradation task The prompt mapping function corresponding to the image, Indicates The output of the window-based Transformer module is about The first step of the degradation task The value of the image, Indicates The value weight matrix corresponding to the window-based Transformer module of the layer, Represents an index operation. represents the index of the window-based Transformer module, represents the index of the degenerate task, Represents the index of the image corresponding to each degradation task; The prompt mapping function is expressed as: , in, Indicates The first step of the degradation task The prompt mapping function corresponding to the image, express and The inner product of Indicates The first step of the degradation task The query features of an image, Indicates The first step of the degradation task A task identifier, Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, Represents the index of the image corresponding to each degradation task, represents the index of the degenerate task, Represents the index of the task identifier corresponding to each degradation task, .

2. The method according to claim 1, characterized in that Determining the task identifier of the degraded task includes: Acquire multiple training images corresponding to multiple degradation tasks respectively; Extracting query features of each of the training images respectively to obtain query feature groups corresponding to each of the degradation tasks; The query feature groups corresponding to the degradation tasks are clustered respectively, and each cluster center is determined as the task identifier of the corresponding degradation task.

3. The method according to claim 1, characterized in that: The hint mapping basis is a linear transformer, The hint mapping basis satisfies: , in, Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, The dimensions are , and The three-dimensional matrix of the real number field, represents the total number of layers of the window-based Transformer module, represents the input dimension of the hint mapping basis, Represents the output dimension of the hint map basis.

4. The method according to claim 1, characterized in that: The hint mapping basis is a linear transformer, The hint mapping basis is expressed as: , in, , , Indicates The first step of the degradation task The prompt mapping base corresponding to the task identifier, Indicates The first step of the degradation task The weight matrix corresponding to the task identifier is Indicates The shared weight matrix corresponding to each task identifier of the degradation task, represents the intermediate variable, Much smaller than and , represents the total number of layers of the window-based Transformer module, represents the input dimension of the hint mapping basis, represents the output dimension of the hint mapping basis, The dimensions are , and The three-dimensional matrix of the real number field, The dimensions are , and The three-dimensional matrix of the real number field, Represents the index of the task identifier corresponding to each degradation task, Indicates the index of the degenerate task.

5. The method according to claim 1, characterized in that The extracting the query feature of the image to be reconstructed includes: The query features of the image to be reconstructed are extracted through the CLIP image editor.

6. An image reconstruction method, characterized in that: include: Acquire an image to be reconstructed; Extracting query features of the image to be reconstructed; Determining the task corresponding to the image to be reconstructed according to the query feature of the image to be reconstructed and a query feature group corresponding to a plurality of predetermined degradation tasks; Inputting the query features of the image to be reconstructed and the tasks corresponding to the image to be reconstructed into a prompt mapping generation module in the image super-resolution reconstruction system trained according to the training method of any one of claims 1 to 5, so that the prompt mapping generation module obtains prompt information corresponding to the image to be reconstructed according to the query features of the image to be reconstructed and the tasks corresponding to the image to be reconstructed, and the prompt mapping functions corresponding to various degradation tasks obtained by pre-training; The query features of the image to be reconstructed and the prompt information corresponding to the image to be reconstructed are input into a backbone network in an image super-resolution reconstruction system trained according to the training method of any image super-resolution reconstruction system as claimed in claims 1 to 5, so that the backbone network reconstructs the image to be reconstructed according to the query features of the image to be reconstructed and the prompt information corresponding to the image to be reconstructed.

7. An image super-resolution reconstruction system, characterized in that: include: Image preprocessing module, task matching module, prompt map generation module and backbone network; The image preprocessing module is used to obtain the image to be reconstructed and extract the query features of the image to be reconstructed; The task matching module determines the task corresponding to the image to be reconstructed according to the query feature of the image to be reconstructed extracted by the image preprocessing module and the query feature group corresponding to the plurality of pre-determined degradation tasks stored; The prompt mapping generation module is used to obtain prompt information corresponding to the image to be reconstructed according to the query features of the image to be reconstructed extracted by the image preprocessing module, the task corresponding to the image to be reconstructed determined by the task matching module, and the prompt mapping function corresponding to each degradation task obtained by pre-training, wherein the task matching module is obtained according to the training method of the image super-resolution reconstruction system according to any one of claims 1 to 5; The backbone network is used to reconstruct the image to be reconstructed based on the query features of the image to be reconstructed extracted by the image preprocessing module and the prompt information corresponding to the image to be reconstructed obtained by the prompt mapping generation module. The backbone network is obtained according to the training method of the image super-resolution reconstruction system according to any one of claims 1 to 5.