Reference-based image super-resolution reconstruction method, system, equipment and medium

Abstract

The invention provides a reference-based image super-resolution reconstruction method, system and device and a medium, and belongs to the technical field of image super-resolution reconstruction, and the reference-based image super-resolution reconstruction method comprises the steps: dividing image data according to categories, dividing the divided image data of each category into a training set and a test set, constructing a reference image library; constructing an image super-resolution reconstruction model based on reference; setting a loss function of the reference-based image super-resolution reconstruction model; training the constructed image super-resolution reconstruction model based on the reference to obtain a trained image super-resolution reconstruction model based on the reference; and inputting the image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a reference-based image super-resolution reconstruction method, system, device and medium. Background Art

[0002] Image super-resolution reconstruction technology reconstructs high-resolution images from low-resolution images without upgrading hardware, significantly improving image quality. The reconstructed images can effectively improve the performance of downstream visual tasks such as classification, detection, and segmentation. These technologies fall into two categories: single-image-based methods utilize only the input image itself for reconstruction, resulting in limited detail recovery; reference-image-based methods supplement texture information with similar high-resolution images, enabling more accurate detail recovery. However, existing methods rely on a reference image of the same scene that closely matches the image to be reconstructed.

[0003] Current methods based on reference images face a significant practical challenge: obtaining reference images of the same scene is often difficult in real-world applications, especially in specialized imaging fields such as remote sensing and infrared. While these scenes lack homogenous data, they do possess a large number of historical high-resolution images of the same category. These images contain shared features, but existing research has yet to fully explore how to leverage these similar images across scenes to guide reconstruction.

[0004] To this end, the present invention provides a reference-based image super-resolution reconstruction method, system, device and medium. Summary of the Invention

[0005] The present invention provides a reference-based image super-resolution reconstruction method, system, device and medium to at least solve the problem in the prior art of how to use these cross-scene similar images to guide reconstruction, which has not been fully explored.

[0006] In a first aspect, an embodiment of the present application provides a reference-based image super-resolution reconstruction method, the method comprising: Divide the image data into categories, divide the image data of each category into training sets and test sets, and build a reference image library; Construct a reference-based image super-resolution reconstruction model; Set the loss function of the reference-based image super-resolution reconstruction model; Training the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The image data is input into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

[0007] Furthermore, the reference-based image super-resolution reconstruction model consists of two parts: category feature information guidance and super-resolution reconstruction network; Extracting category feature information of a reference image through guidance of category feature information; The extracted category feature information of the reference image and the category feature information of the low-resolution image are input into the super-resolution reconstruction network for fusion, and the category feature information of the reference image is used to guide the low-resolution image to perform super-resolution reconstruction to generate a super-resolution image.

[0008] Furthermore, extracting the category feature information of the reference image by guiding the category feature information includes: Based on the classic convolutional neural network model, the structure of the classic convolutional neural network model is set according to the number of image categories to obtain the convolutional neural network model, and the category feature information of the reference image is extracted based on the convolutional neural network model; The extracted category feature information of the reference image and the category feature information of the low-resolution image are input into the super-resolution reconstruction network for fusion, specifically including: The category feature information of the low-resolution image is spliced ​​and fused with the category feature information of the reference image in the channel dimension, and after splicing and fusion, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution.

[0009] Furthermore, the category feature information of the low-resolution image is concatenated with the category feature information of the reference image in the channel dimension, and after concatenation, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution. The expression of this process is:

[0010] Where, represents the convolution operation, is the convolution kernel size, Indicates that the feature map is spliced ​​in the channel dimension, Represents the category feature information of the reference image obtained, Represents the feature information of the acquired low-resolution image, and The size of the feature maps remains consistent.

[0011] Furthermore, the category feature information of the reference image is used to guide the low-resolution image to perform super-resolution reconstruction to generate a super-resolution image, specifically including: Through multiple convolution and upsampling operations, feature information is reconstructed and the size is adjusted to obtain a super-resolution image.

[0012] Furthermore, the loss function of the reference-based image super-resolution reconstruction model is set, specifically including: Set the classification loss guided by category feature information and the loss function of the super-resolution reconstruction network, using category labels and high-resolution images as supervision constraints respectively; Among them, the classification loss guided by category feature information uses cross entropy loss to judge the difference between the predicted label and the true label, and its expression is: ; Where, represents the class guidance loss, Indicates the categories, represents the true label, represents the predicted label; The loss function of the super-resolution reconstruction network consists of reconstruction loss, perception loss, and adversarial loss, and their expressions are defined as follows:

[0013]

[0014]

[0015] Where, represents the reconstruction loss, represents the perceived loss, Represents resistance to loss, Represents the original high-definition image, represents the super-resolution reconstructed image, For the VGG network, represents a generator, represents the discriminator; The loss function of the super-resolution reconstruction network based on the classification loss and overall loss guided by the category feature information is used to obtain the loss function of the reference-based image super-resolution reconstruction model, which is expressed as: ,in, 、 、 、 Both are hyperparameters used to balance the weights between different losses.

[0016] Furthermore, the constructed reference-based image super-resolution reconstruction model is trained, specifically including: Take a low-resolution image of known category as input; Select a reference image from a reference image library of the same category and input it into a reference-based image super-resolution reconstruction model; Test the constructed reference-based image super-resolution reconstruction model, including: Take a low-resolution image of unknown category as input; Predict the category label of the input low-resolution image through the classification network, and predict the probability based on the category label; Set thresholds; When the output probability is greater than the threshold, the category label is used to select a reference image from the reference image library of the same category; When the output probability is less than the threshold, the low-resolution image is used as the reference image to prevent misguidance; When the category of the low-resolution image is known, the reference image is selected using any of the following methods: Randomly select a reference image from the reference image library; Feature point matching is used to obtain a reference image that best matches the low-resolution image.

[0017] In a second aspect, embodiments of the present application provide a system for the reference-based image super-resolution reconstruction method described in the above aspects, the system comprising: A reference image library construction module is used to divide the image data into categories, divide the image data of each category into a training set and a test set, and build a reference image library; A model building module, used to build a reference-based image super-resolution reconstruction model; A loss function setting module is used to set the loss function of the reference-based image super-resolution reconstruction model; A model training module is used to train the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The super-resolution image output module is used to input image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

[0018] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the reference-based image super-resolution reconstruction method as described in the above aspects are implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the reference-based image super-resolution reconstruction method as described in the above aspects.

[0020] It can be seen from the above technical solutions that the present invention has the following advantages: The reference-based image super-resolution reconstruction method provided in this application breaks through the traditional same-scene limitations and develops a category-guided reference super-resolution method. By mining the common features in similar high-resolution image libraries, a more universal detail recovery model is constructed to obtain a reconstructed super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 Flowchart of the reference-based image super-resolution reconstruction method.

[0023] Figure 2 Flowchart for model training of reference-based image super-resolution reconstruction method. DETAILED DESCRIPTION

[0024] In order to make the application objectives, features, and advantages of this application more obvious and easy to understand, the technical solutions protected by this application will be clearly and completely described below using specific embodiments and drawings. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this patent, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this patent.

[0025] Various embodiments of the present disclosure will be more fully described in the following detailed description of a reference-based image super-resolution reconstruction method, system, device, and medium. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather that the present disclosure should be understood to encompass all adjustments, equivalents, and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0026] Hereinafter, the terms "include" or "may include" as used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include," "have," and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be understood as excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0027] In various embodiments of the present disclosure, the expression "or" or "at least one of A or / and B" includes any or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0028] The expressions (such as "first", "second", etc.) used in the various embodiments of the present disclosure may modify the various constituent elements in the various embodiments, but may not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used to distinguish one element from other elements. For example, a first user device and a second user device indicate different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may also be referred to as a first element.

[0029] It should be noted that when a component is described as being “connected” to another component, the first component may be directly connected to the second component, and a third component may be “connected” between the first and second components. Conversely, when a component is described as being “directly connected” to another component, it can be understood that there is no third component between the first and second components.

[0030] The term “user” used in various embodiments of the present disclosure may indicate a person who uses an electronic device, and may be a monitoring person, a test person, or an operator.

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] The embodiments of the present application provide a reference-based image super-resolution reconstruction method, system, device and medium to solve the currently urgently needed technical problem of how to use these similar images across scenes to guide reconstruction, which has not been fully explored in existing research.

[0033] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0034] Figure 1 This is a flowchart of a reference-based image super-resolution reconstruction method provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a reference-based image super-resolution reconstruction method, which specifically includes the following steps: A reference-based image super-resolution reconstruction method, the method comprising: Divide the image data into categories, divide the image data of each category into training sets and test sets, and build a reference image library; It should be noted that there are two ways to build a reference image library: One is to exclude the image to be reconstructed from the training data. Second, it is independent of the training and testing data.

[0035] There are two ways to select a reference image: One is to randomly select an image from the reference library as a reference; The second is to use the feature point matching method to obtain the image that best matches the input low-resolution image as a reference.

[0036] Construct a reference-based image super-resolution reconstruction model; Set the overall loss function of the reference-based image super-resolution reconstruction model; Training the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The image data is input into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

[0037] As an example, the reference-based image super-resolution reconstruction model consists of two parts: category feature information guidance and super-resolution reconstruction network; Extracting category feature information of a reference image through guidance of category feature information; The extracted category feature information of the reference image and the category feature information of the low-resolution image are input into the super-resolution reconstruction network for fusion, and the category feature information of the reference image is used to guide the low-resolution image to perform super-resolution reconstruction to generate a super-resolution image.

[0038] In one embodiment, extracting the category feature information of the reference image by guiding the category feature information includes: Based on the classic convolutional neural network model, the structure of the classic convolutional neural network model is designed and set according to the number of image categories and task requirements. At the same time, the attention mechanism can be introduced to enhance the feature extraction ability of the network, and convolutional neural network models of different complexities can be obtained. Based on the obtained convolutional neural network model, the category feature information of the reference image is extracted.

[0039] According to an embodiment of the present application, the method for obtaining the category feature information of the low-resolution image is as follows: Encode the input low-resolution image using one of the following methods: Convolutional neural network, Transformer block.

[0040] According to another embodiment of the present invention, the extracted category feature information of the reference image and the category feature information of the low-resolution image are input into a super-resolution reconstruction network for fusion, specifically including: The category feature information of the low-resolution image is spliced ​​and fused with the category feature information of the reference image in the channel dimension, and after splicing and fusion, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution.

[0041] It should be further explained that the category feature information of the low-resolution image is spliced ​​with the category feature information of the reference image in the channel dimension, and after splicing, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution. The expression of this process is:

[0042] Where, represents the convolution operation, is the convolution kernel size, Indicates that the feature map is spliced ​​in the channel dimension, Represents the category feature information of the reference image obtained, Represents the feature information of the acquired low-resolution image, and The size of the feature maps remains consistent.

[0043] According to an embodiment of the present application, the class feature information of the reference image is used to guide the super-resolution reconstruction of the low-resolution image to generate the super-resolution image, which specifically includes: Through multiple convolution and upsampling operations, feature information is reconstructed and the size is adjusted to obtain a super-resolution image.

[0044] According to another embodiment of the present invention, setting a loss function of a reference-based image super-resolution reconstruction model specifically includes: Set the classification loss guided by category feature information and the loss function of the super-resolution reconstruction network, using category labels and high-resolution images as supervision constraints respectively; Among them, the classification loss guided by category feature information uses cross entropy loss to judge the difference between the predicted label and the true label, and its expression is: ; Where, represents the class guidance loss, Indicates the categories, represents the true label, represents the predicted label; The loss function of the super-resolution reconstruction network consists of reconstruction loss, perception loss, and adversarial loss, and their expressions are defined as follows:

[0045]

[0046]

[0047] Where, represents the reconstruction loss, represents the perceived loss, Represents resistance to loss, Represents the original high-definition image, represents the super-resolution reconstructed image, For the VGG network, represents a generator, represents the discriminator; The loss function of the super-resolution reconstruction network based on the classification loss and overall loss guided by the category feature information is used to obtain the loss function of the reference-based image super-resolution reconstruction model, which is expressed as: ,in, 、 、 、 Both are hyperparameters used to balance the weights between different losses.

[0048] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another reference-based image super-resolution reconstruction method is provided, which trains the constructed reference-based image super-resolution reconstruction model, specifically including: Take a low-resolution image of known category as input; A reference image is selected from a reference image library of the same category and input into the reference-based image super-resolution reconstruction model. It should be noted that when constructing the reference image library, it is ensured that the low-resolution image to be reconstructed is not in its own reference image library.

[0049] Test the constructed reference-based image super-resolution reconstruction model, including: Take a low-resolution image of unknown category as input; Predict the category label of the input low-resolution image through the classification network, and predict the probability based on the category label; Set thresholds; When the output probability is greater than the threshold, the category label is used to select a reference image from the reference image library of the same category; It should be noted that when the category of the low-resolution image is known, the reference image is selected using any of the following selection methods: Randomly select a reference image from the reference image library; Use feature point matching to obtain a reference image that best matches the low-resolution image; When the output probability is less than the threshold, the low-resolution image is used as the reference image to prevent misguidance; In an exemplary embodiment, after the image data is divided into categories, the method further includes: The image data classified into the same category are unified in size and naming format, and the image data of the same category are placed in the same folder.

[0050] The present invention provides a system for a reference-based image super-resolution reconstruction method, the system comprising: A reference image library construction module is used to divide the image data into categories, divide the image data of each category into a training set and a test set, and build a reference image library; A model building module, used to build a reference-based image super-resolution reconstruction model; A loss function setting module is used to set the loss function of the reference-based image super-resolution reconstruction model; A model training module is used to train the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The super-resolution image output module is used to input image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

[0051] The reference-based image super-resolution reconstruction method provided in the embodiments of the present application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0052] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0053] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0054] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0055] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0056] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0057] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of an electronic device. The external memory card communicates with the processor through the external memory interface, enabling data storage. For example, files such as music and videos can be stored on the external memory card.

[0058] Internal memory can be used to store computer-executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. The internal memory can include a program storage area and a data storage area. The internal memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0059] The wireless communication function of electronic devices can be realized through antennas, wireless communication modules, modem processors, and baseband processors.

[0060] Wireless communication modules can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0061] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0062] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0063] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0064] A GPU is a microprocessor for image processing that connects the display screen to the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0065] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0066] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. Exemplarily, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0068] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0069] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, it will be appreciated by those skilled in the art that the technical solutions of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0070] The above-mentioned electronic device implements the reference-based image super-resolution reconstruction method of the present application, which divides the image data according to categories, divides the image data of each category into training sets and test sets, and constructs a reference image library; constructs a reference-based image super-resolution reconstruction model; sets the loss function of the reference-based image super-resolution reconstruction model; trains the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; inputs the image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image, breaks through the traditional same-scene limitation, develops a category-guided reference super-resolution method, and constructs a more universal detail recovery model by mining the common features in similar high-resolution image libraries to obtain a reconstructed super-resolution image.

[0071] The storage medium provided in the present application stores a program product capable of implementing a reference-based image super-resolution reconstruction method.

[0072] The reference-based image super-resolution reconstruction method includes: dividing image data according to categories, dividing the image data of each category into a training set and a test set, and constructing a reference image library; constructing a reference-based image super-resolution reconstruction model; setting a loss function of the reference-based image super-resolution reconstruction model; training the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; and inputting image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

[0073] Breaking through the traditional same-scene limitations, we develop a category-guided reference super-resolution method. By mining the common features in similar high-resolution image libraries, we build a more universal detail recovery model to obtain a reconstructed super-resolution image.

[0074] In some possible embodiments, the reference-based image super-resolution reconstruction method disclosed herein can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0075] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0076] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0077] For those skilled in the art, designing different forms of control circuits based on the teachings of the present invention does not require creative work. These changes, modifications, substitutions and variations to the embodiments without departing from the principles and spirit of the present invention still fall within the scope of protection of the present invention.

Claims

1. A reference-based image super-resolution reconstruction method, characterized in that: The method comprises: Divide the image data into categories, divide the image data of each category into training sets and test sets, and build a reference image library; Construct a reference-based image super-resolution reconstruction model; Set the loss function of the reference-based image super-resolution reconstruction model; Training the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The image data is input into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

2. The reference-based image super-resolution reconstruction method according to claim 1, wherein: The reference-based image super-resolution reconstruction model consists of two parts: category feature information guidance and super-resolution reconstruction network; Extracting category feature information of a reference image through guidance of category feature information; The extracted category feature information of the reference image and the category feature information of the low-resolution image are input into the super-resolution reconstruction network for fusion, and the category feature information of the reference image is used to guide the low-resolution image to perform super-resolution reconstruction to generate a super-resolution image.

3. The reference-based image super-resolution reconstruction method according to claim 2, wherein: The category feature information of the reference image is extracted by guiding the category feature information, including: Based on the classic convolutional neural network model, the structure of the classic convolutional neural network model is set according to the number of image categories to obtain the convolutional neural network model, and the category feature information of the reference image is extracted based on the convolutional neural network model; The extracted category feature information of the reference image and the category feature information of the low-resolution image are input into the super-resolution reconstruction network for fusion, specifically including: The category feature information of the low-resolution image is spliced ​​and fused with the category feature information of the reference image in the channel dimension, and after splicing and fusion, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution.

4. The reference-based image super-resolution reconstruction method according to claim 3, wherein: The category feature information of the low-resolution image is concatenated with the category feature information of the reference image in the channel dimension, and after concatenation, it is mapped to the feature space of the super-resolution reconstruction network through one-dimensional convolution. The expression of this process is: Where, represents the convolution operation, is the convolution kernel size, Indicates that the feature map is spliced ​​in the channel dimension, Represents the category feature information of the reference image obtained, Represents the feature information of the acquired low-resolution image, and The size of the feature maps remains consistent.

5. The reference-based image super-resolution reconstruction method according to claim 4, wherein: The class feature information of the reference image is used to guide the super-resolution reconstruction of the low-resolution image to generate a super-resolution image, specifically including: Through multiple convolution and upsampling operations, feature information is reconstructed and the size is adjusted to obtain a super-resolution image.

6. The reference-based image super-resolution reconstruction method according to claim 5, wherein: Set the loss function of the reference-based image super-resolution reconstruction model, including: Set the classification loss guided by category feature information and the loss function of the super-resolution reconstruction network, using category labels and high-resolution images as supervision constraints respectively; Among them, the classification loss guided by category feature information uses cross entropy loss to judge the difference between the predicted label and the true label, and its expression is: ； Where, represents the class guidance loss, Indicates the categories, represents the true label, represents the predicted label; The loss function of the super-resolution reconstruction network consists of reconstruction loss, perception loss, and adversarial loss, and their expressions are defined as follows: Where, represents the reconstruction loss, represents the perceived loss, Represents resistance to loss, Represents the original high-definition image, represents the super-resolution reconstructed image, For the VGG network, represents a generator, represents the discriminator; The loss function of the super-resolution reconstruction network based on the classification loss and overall loss guided by the category feature information is used to obtain the loss function of the reference-based image super-resolution reconstruction model, which is expressed as: ,in, 、 、 、 Both are hyperparameters used to balance the weights between different losses.

7. The reference-based image super-resolution reconstruction method according to claim 6, wherein: The constructed reference-based image super-resolution reconstruction model is trained, specifically including: Take a low-resolution image of known category as input; Select a reference image from a reference image library of the same category and input it into a reference-based image super-resolution reconstruction model; Test the constructed reference-based image super-resolution reconstruction model, including: Take a low-resolution image of unknown category as input; Predict the category label of the input low-resolution image through the classification network, and predict the probability based on the category label; Set thresholds; When the output probability is greater than the threshold, the category label is used to select a reference image from the reference image library of the same category; When the output probability is less than the threshold, the low-resolution image is used as the reference image to prevent misguidance; When the category of the low-resolution image is known, the reference image is selected using any of the following methods: Randomly select a reference image from the reference image library; Feature point matching is used to obtain a reference image that best matches the low-resolution image.

8. A system for use in the reference-based image super-resolution reconstruction method according to any one of claims 1 to 7, characterized in that: The system comprises: A reference image library construction module is used to divide the image data into categories, divide the image data of each category into a training set and a test set, and build a reference image library; A model building module, used to build a reference-based image super-resolution reconstruction model; A loss function setting module is used to set the loss function of the reference-based image super-resolution reconstruction model; A model training module is used to train the constructed reference-based image super-resolution reconstruction model to obtain a trained reference-based image super-resolution reconstruction model; The super-resolution image output module is used to input image data into the trained reference-based image super-resolution reconstruction model to obtain a super-resolution image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the reference-based image super-resolution reconstruction method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the reference-based image super-resolution reconstruction method according to any one of claims 1 to 7 are implemented.

Images