Image super-resolution method, device and equipment and storage medium thereof
Through the end-to-end mode of image feature comprehensive processing components and image super-resolution reconstruction components, dynamically adjust image super-resolution processing, solving the problem of details loss and blur in low-resolution image processing, and achieving efficient and accurate image details reconstruction.
Patent Information
- Application Number
- CN202411135647.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-20
AI Technical Summary
The prior art has problems of blur, jagging and detail loss in super-resolution processing of low-resolution images, and its generalization ability is weak, resulting in the inability to efficiently and accurately reconstruct image details.
Using the end-to-end mode of image feature comprehensive processing components and image super-resolution reconstruction components, through pre-training and adversarial training, the optimization factors and reconstruction control parameters are obtained, and the image super-resolution processing is dynamically adjusted to achieve low-resolution images. efficient and accurate detail reconstruction.
It realizes efficient and accurate detail reconstruction of low-resolution images, avoids blur and jagging, and improves image detail retention and generalization capabilities.
Smart Images
Figure CN119168861A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and is applied to image resolution optimization scenarios, and in particular to an image super-resolution method, device, equipment, and storage medium thereof. Background Art
[0002] In the field of image processing, image super-resolution technology has been widely studied and applied. Existing image super-resolution methods are mainly divided into interpolation-based methods and learning-based methods. Although these methods have achieved good results in image super-resolution processing, they still have certain limitations in intelligence and super-resolution processing.
[0003] For example, interpolation-based methods are prone to blurring and jagged edges when performing super-resolution processing and cannot retain the detail information in the original image. Learning-based methods have weak generalization capabilities and may lead to loss of details or artifacts. As a result, there is still a problem of being unable to efficiently and accurately reconstruct image details when performing super-resolution optimization on low-resolution images. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an image super-resolution method, apparatus, device and storage medium thereof to solve the problem that the prior art is unable to efficiently and accurately reconstruct image details when performing super-resolution optimization on low-resolution images.
[0005] In order to solve the above technical problems, the present application provides an image super-resolution method, which adopts the following technical solutions:
[0006] An image super-resolution method comprises the following steps:
[0007] Acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing;
[0008] Acquire a target resolution, wherein the target resolution is an expected resolution after super-resolution of the target image;
[0009] Inputting the target image and the target resolution into a pre-trained image feature comprehensive processing component to perform image feature comprehensive processing;
[0010] Receiving the image features and optimization factors output by the image feature comprehensive processing component;
[0011] Inputting the image features and the optimization factors into a pre-trained image super-resolution reconstruction component;
[0012] Filtering the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor;
[0013] Image super-resolution processing is performed based on the reconstruction control parameters, the image features and the image super-resolution reconstruction component.
[0014] Furthermore, before executing the step of inputting the target image into the pre-trained image feature comprehensive processing component, the method further includes:
[0015] Obtain N groups of training images, wherein each group of training images in the N groups of training images consists of a low-resolution image and M high-resolution images corresponding to the low-resolution image, resolutions of the M high-resolution images have gradient differences, and N and M are positive integers;
[0016] Inputting the N groups of training images into an image super-resolution model to be trained, wherein the image super-resolution model is constructed in an end-to-end mode by an image feature comprehensive processing component to be trained and an image super-resolution reconstruction component to be trained;
[0017] Extracting the image features of all the images in the N groups of training images respectively through the feature extraction layer of the image feature comprehensive processing component;
[0018] Inputting the image features into the pooling processing layer of the image feature comprehensive processing component to obtain unique global feature vectors corresponding to all images output by the pooling processing layer;
[0019] According to a preset image group number, all unique global feature vectors are grouped and sorted to obtain N groups of sorting results, wherein each group of sorting results in the N groups of sorting results is composed of a unique global feature vector of a low-resolution image and unique global feature vectors of M high-resolution images corresponding to the low-resolution image;
[0020] The unique global feature vectors in the N groups of sorting results are integrated by the feature integration layer of the image feature comprehensive processing component to obtain optimization factors when the unique global feature vectors of the low-resolution images in each group of sorting results are converted into the unique global feature vectors of the corresponding M high-resolution images;
[0021] Based on the optimization factors when the unique global feature vector of the low-resolution image in each group of sorting results is converted to the unique global feature vector of the corresponding M high-resolution images, classification and integration are performed to obtain the corresponding optimization factors when super-resolution optimization is performed on images in different low-resolution intervals to images in different high-resolution intervals, and obtain the pre-trained image feature comprehensive processing component.
[0022] Further, after executing the step of inputting the N groups of training images into the image super-resolution model to be trained, the method further includes:
[0023] Screening out low-resolution images in each group of training images from the N groups of training images, and inputting all the low-resolution images into the image super-resolution reconstruction component to be trained;
[0024] Based on the image super-resolution reconstruction component, super-resolution reconstruction of the low-resolution image is completed to obtain a reconstructed image;
[0025] Selecting M high-resolution images from each group of training images from the N groups of training images as verification images;
[0026] Inputting the reconstructed image into the image feature comprehensive processing component to obtain reconstructed image features;
[0027] According to the reconstructed image features and the image features of the M high-resolution images in each group of training images, adversarial training is performed on the image super-resolution reconstruction component until the image feature loss value meets a preset loss threshold. The training of the image super-resolution reconstruction component is completed, and a pre-trained image super-resolution reconstruction component is obtained.
[0028] Furthermore, the step of performing adversarial training on the image super-resolution reconstruction component until the image feature loss value meets a preset loss threshold, and completing the training of the image super-resolution reconstruction component, specifically includes:
[0029] Use the preset loss function:
[0030] L total =L structure +λ1L texture +λ2L edge ,
[0031] Calculate the image feature loss value for each adversarial training, where L total Represents the image feature loss value, L structure Represents the image structure feature loss value, L texture Represents the image texture feature loss value, L edge represents the image edge feature loss value, λ1 and λ2 are weight coefficients, which are used to balance the importance of image structure feature loss value, image texture feature loss value and image edge feature loss value;
[0032] If the image feature loss value of the current adversarial training does not meet the preset loss threshold, the reconstruction control parameters of the image super-resolution reconstruction component are dynamically adjusted through the back propagation algorithm, and the adversarial training is performed again;
[0033] If the image feature loss value of the current adversarial training meets the preset loss threshold, the current adversarial training is completed, the reconstruction control parameters of the image super-resolution reconstruction component when the current adversarial training is completed are recorded, the next verification image is replaced, and the adversarial training is performed again;
[0034] The training of the image super-resolution reconstruction component is completed until the image super-resolution reconstruction component completes the adversarial training based on all verification images, sorts out the reconstruction control parameters recorded when each adversarial training is completed, and outputs the reconstruction control parameter record log.
[0035] Furthermore, after executing the step of obtaining the pre-trained image super-resolution reconstruction component, the method further includes:
[0036] Based on the reconstruction control parameter record log, the reconstruction control parameters recorded when each adversarial training is completed are obtained, wherein the reconstruction control parameters include parameters of each network structure in the image super-resolution reconstruction component;
[0037] According to the corresponding relationship between the reconstructed image output by the image super-resolution reconstruction component and the verification image, a corresponding relationship between the reconstruction control parameters and the optimization factors is established.
[0038] Furthermore, the step of inputting the target image and the target resolution into the pre-trained image feature comprehensive processing component to perform image feature comprehensive processing specifically includes:
[0039] Extracting image features of the target image through the feature extraction layer of the image feature comprehensive processing component, wherein the image features include image resolution;
[0040] Identifying the resolution range corresponding to the target image before optimization through the image resolution;
[0041] Identifying, by means of the target resolution, a resolution interval corresponding to the target image after super-resolution processing;
[0042] An optimization factor corresponding to optimizing the target image to the target resolution is determined according to the resolution interval corresponding to the target image before optimization and the resolution interval corresponding to the target image after super-resolution processing.
[0043] Furthermore, the step of selecting the reconstruction control parameter of the image super-resolution reconstruction component according to the optimization factor specifically includes:
[0044] Determining the target reconstruction control parameter corresponding to the optimization factor according to the corresponding relationship between the reconstruction control parameter and the optimization factor;
[0045] Setting the target reconstruction control parameter as a reconstruction control parameter of the image super-resolution reconstruction component;
[0046] The step of performing image super-resolution processing based on the reconstruction control parameter, the image feature and the image super-resolution reconstruction component specifically includes:
[0047] Dynamically adjusting the image super-resolution reconstruction component by using the reconstruction control parameter to obtain a dynamically adjusted image super-resolution reconstruction component;
[0048] The image features are input into the dynamically adjusted image super-resolution reconstruction component, and after decoding and output, a super-resolution processed image is obtained.
[0049] In order to solve the above technical problems, the embodiment of the present application also provides an image super-resolution device, which adopts the following technical solution:
[0050] An image super-resolution device, comprising:
[0051] A target image acquisition module, used to acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing;
[0052] A target resolution acquisition module, used to acquire a target resolution, wherein the target resolution is an expected resolution after super-resolution of the target image;
[0053] An image feature comprehensive processing module, used for inputting the target image and the target resolution into a pre-trained image feature comprehensive processing component to perform image feature comprehensive processing;
[0054] An image feature and optimization factor acquisition module, used for receiving the image features and optimization factors output by the image feature comprehensive processing component;
[0055] An image feature and optimization factor input module, used for inputting the image feature and the optimization factor into a pre-trained image super-resolution reconstruction component;
[0056] A reconstruction control parameter screening module, used to screen out the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor;
[0057] The image super-resolution processing module is used to perform image super-resolution processing based on the reconstruction control parameter, the image feature and the image super-resolution reconstruction component.
[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0059] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned image super-resolution method when executing the computer-readable instructions.
[0060] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0061] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the image super-resolution method as described above.
[0062] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0063] The image super-resolution method described in the embodiment of the present application obtains the target image and target resolution; inputs the target image and target resolution into the image feature comprehensive processing component to perform image feature comprehensive processing; receives the image features and optimization factors output by the image feature comprehensive processing component; inputs the image features and optimization factors into the pre-trained image super-resolution reconstruction component; selects the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performs image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. When super-resolution optimization is performed on a low-resolution image, the image details can be efficiently and accurately reconstructed by automatically adjusting the reconstruction control parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0065] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0066] Figure 2 is a flowchart of an embodiment of an image super-resolution method according to the present application;
[0067] Figure 3 It is a flowchart of a specific embodiment of training the image feature comprehensive processing component in the image super-resolution method described in the present application;
[0068] Figure 4 is a flowchart of a specific embodiment of training the image super-resolution reconstruction component in the image super-resolution method of the present application;
[0069] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown in FIG.
[0070] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 206 is shown in FIG.
[0071] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 207 is shown in FIG.
[0072] Figure 8 is a structural schematic diagram of an embodiment of an image super-resolution device according to the present application;
[0073] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0075] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0077] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0078] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0079] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0080] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0081] It should be noted that the image super-resolution method provided in the embodiments of the present application is generally executed by a server, and accordingly, the image super-resolution device is generally arranged in the server.
[0082] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0083] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image super-resolution method according to the present application. The image super-resolution method comprises the following steps:
[0084] Step 201: Acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing.
[0085] In this embodiment, "super-resolution" refers to optimizing a low-resolution image to a high-resolution one. Image resolution refers to the number of pixels contained in an image per inch. The higher the image resolution, the clearer the image is, and the corresponding image feature extraction is more accurate. Super-resolution processing of images can be considered as a processing method for image feature enhancement. Applying it to the pre-processing of medical images, document recognition or accident images can ensure that the medical images, document recognition or accident images after super-resolution are subsequently input into the corresponding medical image processing model, financial image processing model or vehicle image processing model, respectively, so that image features can be extracted more accurately to assist relevant business personnel in making task decisions.
[0086] Step 202: acquiring a target resolution, wherein the target resolution is an expected resolution after super-resolution is performed on the target image.
[0087] In this embodiment, the target resolution refers to the expected resolution after super-resolution of the target image. More specifically, assuming that the resolution of the image uploaded required by a certain image recognition software needs to exceed a certain resolution value, or the resolution required when uploading the image needs to exceed a certain resolution value, but the resolution value of the target image currently being uploaded does not reach the resolution value, super-resolution processing is required to make the resolution of the target image reach the required resolution value. At this time, the required resolution value can be the expected resolution.
[0088] In this embodiment, the target resolution may be determined by the user through interface interaction instruction input or selection according to actual application requirements.
[0089] Step 203: input the target image and the target resolution into the pre-trained image feature comprehensive processing component to perform image feature comprehensive processing.
[0090] In this embodiment, by inputting the target image and the target resolution into a pre-trained image feature comprehensive processing component, image feature comprehensive processing is performed, thereby realizing the use of a pre-trained intelligent processing component to process the target image, which is more intelligent and scientific.
[0091] In this embodiment, before executing the step of inputting the target image and the target resolution into the pre-trained image feature comprehensive processing component, the method also includes the step of training the image feature comprehensive processing component. By pre-training the image feature comprehensive processing component to obtain the trained image feature comprehensive processing component, it is achieved that when actually performing super-resolution processing, the trained image feature comprehensive processing component can be directly called.
[0092] For details, please refer to Figure 3 , Figure 3 is a flowchart of a specific embodiment of training the image feature comprehensive processing component in the image super-resolution method of the present application, including:
[0093] Step 301, obtaining N groups of training images, wherein each group of training images in the N groups of training images is composed of a low-resolution image and M high-resolution images corresponding to the low-resolution image, the resolutions of the M high-resolution images have gradient differences, and N and M are positive integers;
[0094] In this embodiment, the N groups of training images, assuming that N is 10, have a total of 10 groups of training images, each group of training images is composed of a low-resolution image and 9 high-resolution images corresponding to the low-resolution image, M is 9, and the N groups of training images contain a total of 100 images. Since the expected resolutions are different when performing super-resolution processing on images, the corresponding processing parameters will also be different. Assuming that the group numbers of the 10 groups of training images are A, B, C, D, E, F, G, H, I, and J, the images in each group are the same image, and the only difference is The image resolutions are different. For example, the images in group A are front-facing photos of an ID card. The image numbers are A-0, A-1, A-2, A-3, A-4, A-5, A-6, A-7, A-8, and A-9. The only difference is that the image resolutions of these 10 front-facing photos of ID cards are 50PPI, 100PPI, 120PPI, 160PPI, 200PPI, 210PPI, 270PPI, 350PPI, 500PPI, and 550PPI, respectively. PPI is the unit of image resolution, which refers to pixels per inch. At this time, the image resolution of A-0 is a front photo of the ID card with an image resolution of 50PPI, that is, A-0 corresponds to the low-resolution image, and the front photos of the ID card with image resolutions of 100PPI, 120PPI, 160PPI, 200PPI, 210PPI, 270PPI, 350PPI, 500PPI, and 550PPI are the 9 high-resolution images corresponding to the low-resolution image.
[0095] Step 302, inputting the N groups of training images into an image super-resolution model to be trained, wherein the image super-resolution model is constructed in an end-to-end mode by an image feature comprehensive processing component to be trained and an image super-resolution reconstruction component to be trained;
[0096] In this embodiment, the N groups of training images are input into the image super-resolution model to be trained. The purpose is to use the image feature comprehensive processing component in the image super-resolution model to extract image features, and to train the image feature comprehensive processing component to obtain the corresponding optimization factors when converting low-resolution images to different high-resolution images through training.
[0097] In this embodiment, after executing the step of inputting the N groups of training images into the image super-resolution model to be trained, the method also includes a step of training the image super-resolution reconstruction component to obtain the correspondence between the optimization factor and the reconstruction control parameter of the image super-resolution reconstruction component.
[0098] Continue to refer Figure 4 , Figure 4 is a flowchart of a specific embodiment of training the image super-resolution reconstruction component in the image super-resolution method of the present application, comprising:
[0099] Step 401, screening out low-resolution images in each group of training images from the N groups of training images, and inputting all the low-resolution images into the image super-resolution reconstruction component to be trained;
[0100] Step 402, completing super-resolution reconstruction of the low-resolution image based on the image super-resolution reconstruction component to obtain a reconstructed image;
[0101] Step 403, selecting M high-resolution images from each group of training images from the N groups of training images as verification images;
[0102] Step 404, inputting the reconstructed image into the image feature comprehensive processing component to obtain reconstructed image features;
[0103] Step 405, based on the reconstructed image features and the image features of the M high-resolution images in each group of training images, adversarial training is performed on the image super-resolution reconstruction component until the image feature loss value meets a preset loss threshold. The training of the image super-resolution reconstruction component is completed, and a pre-trained image super-resolution reconstruction component is obtained.
[0104] In this embodiment, the image super-resolution reconstruction component can be understood as a generator in an adversarial training network, which reconstructs a low-resolution image in each group of training images screened from the N groups of training images to generate a corresponding high-resolution image;
[0105] At this time, the image feature comprehensive processing component can be understood as a discriminator in an adversarial training network, which performs image feature extraction processing on the reconstructed image and compares it with the image features of the high-resolution image in the N groups of training images, thereby training and verifying the image super-resolution reconstruction component in an adversarial learning manner.
[0106] In this embodiment, the step of performing adversarial training on the image super-resolution reconstruction component until the image feature loss value meets the preset loss threshold, and the step of completing the training of the image super-resolution reconstruction component specifically includes: using a preset loss function: L total =L structure +λ1L texture +λ2L edge , calculate the image feature loss value for each adversarial training, where L total Represents the image feature loss value, L structure Represents the image structure feature loss value, L texture Represents the image texture feature loss value, L edge represents the image edge feature loss value, λ1 and λ2 are weight coefficients used to balance the importance of the image structure feature loss value, the image texture feature loss value and the image edge feature loss value; if the image feature loss value of the current adversarial training does not meet the preset loss threshold, the reconstruction control parameters of the image super-resolution reconstruction component are dynamically adjusted through the back propagation algorithm, and the adversarial training is performed again; if the image feature loss value of the current adversarial training meets the preset loss threshold, the current adversarial training is completed, the reconstruction control parameters of the image super-resolution reconstruction component when the current adversarial training is completed are recorded, the next verification image is replaced, and the adversarial training is performed again; until the image super-resolution reconstruction component has completed the adversarial training based on all verification images, the reconstruction control parameters recorded at the completion of each adversarial training are sorted out, and the reconstruction control parameter record log is output, the training of the image super-resolution reconstruction component is completed.
[0107] Since the images used in the adversarial training are all low-resolution images, and the corresponding high-resolution images are all generated, it is possible to obtain the reconstruction control parameters corresponding to the low-resolution images when performing super-resolution optimization to images with different high resolutions. Furthermore, when the image feature comprehensive processing component is pre-trained before, the image feature comprehensive processing component has learned the optimization factors corresponding to the super-resolution optimization of images from different low-resolution intervals to images from different high-resolution intervals. Therefore, after executing the step of obtaining the pre-trained image super-resolution reconstruction component, the reconstruction control parameters recorded each time the adversarial training is completed can be obtained based on the reconstruction control parameter record log, wherein the reconstruction control parameters include the parameters of each network structure in the image super-resolution reconstruction component; according to the correspondence between the reconstructed image output by the image super-resolution reconstruction component and the verification image, a correspondence between the reconstruction control parameters and the optimization factors is established.
[0108] In addition, during the training process of the image super-resolution reconstruction component, the image structure feature loss value, the image texture feature loss value and the image edge feature loss value are combined to construct a total loss value, thereby ensuring accurate reconstruction of image details during actual super-resolution optimization.
[0109] Step 303, extracting the image features of all the images in the N groups of training images respectively through the feature extraction layer of the image feature comprehensive processing component;
[0110] In this embodiment, the feature extraction layer may be composed of a plurality of convolutional layers, and in order to ensure that multi-dimensional image features can be extracted, the plurality of convolutional layers and the convolution kernel sizes and / or step sizes may be set differently.
[0111] Step 304, inputting the image features into the pooling processing layer of the image feature comprehensive processing component to obtain unique global feature vectors corresponding to all images output by the pooling processing layer;
[0112] In this embodiment, the pooling processing layer may be a Pooling layer, and the Pooling layer may be configured to perform global pooling processing on the image features extracted by the convolution layer to obtain a unique global feature vector corresponding to each image.
[0113] Step 305, grouping and sorting all unique global feature vectors according to preset image group numbers to obtain N groups of sorting results, wherein each group of sorting results in the N groups of sorting results is composed of a unique global feature vector of a low-resolution image and unique global feature vectors of M high-resolution images corresponding to the low-resolution image;
[0114] Continuing with the above example, according to the above image group numbers, the unique global feature vectors corresponding to the images in groups A, B, C, D, E, F, G, H, I, and J are sorted out, and a fully connected layer can be added after the Pooling layer to realize classification (grouping) processing of the image feature processing results.
[0115] Step 306, integrating information of the unique global feature vectors in the N groups of sorting results through the feature integration layer of the image feature comprehensive processing component, and obtaining optimization factors when the unique global feature vectors of the low-resolution images in each group of sorting results are converted into the unique global feature vectors of the corresponding M high-resolution images;
[0116] Continuing with the above example, group A contains 10 unique global feature vectors, which are the unique global feature vectors of A-0, A-1, A-2, A-3, A-4, A-5, A-6, A-7, A-8, and A-9. The feature integration layer can be composed of multiple fully connected layers. Since the image resolutions corresponding to A-0, A-1, A-2, A-3, A-4, A-5, A-6, A-7, A-8, and A-9 are different, A-0, A-1, A-2, A-3, A-4, A-5, A-6, A-7, A-8, and A-9 are different. , A-5, A-6, A-7, A-8, and A-9 are also different. Through the multiple fully connected layers and all the unique global feature vectors corresponding to the A group, the optimization factors when A-0 is super-resolution processed to A-1 and the optimization factors when A-0 is super-resolution processed to A-2 are obtained respectively. Similarly, the optimization factors when A-0 is super-resolution processed to A-3, A-4, and A-5 respectively, or the optimization factors when A-3 is super-resolution processed to A-6 and A-9 respectively, can be obtained.
[0117] Step 307, based on the optimization factors when converting the unique global feature vectors of the low-resolution images in each group of sorting results to the unique global feature vectors of the corresponding M high-resolution images, classification and integration are performed to obtain the optimization factors corresponding to the super-resolution optimization of images in different low-resolution intervals to images in different high-resolution intervals, and obtain the pre-trained image feature comprehensive processing component.
[0118] In this embodiment, since the image resolution is represented by a mathematical value, it involves optimization from a low-resolution interval to another high-resolution interval. After the results of each group are sorted, since the image resolutions of the N groups of training images are not the same, it is more appropriate to obtain the optimization factors corresponding to the super-resolution optimization of images from different low-resolution intervals to images from different high-resolution intervals. In specific implementation, assuming that there are 10 images in the N groups with a resolution between 30PPI and 100PPI, and when these 10 images are optimized to a resolution of 200PPI to 300PPI, the corresponding optimization factors are not all the same, then the optimization factor corresponding to the optimization of the image from the resolution interval of 30PPI to 100PPI to the interval of 200PPI to 300PPI is determined by taking the majority.
[0119] By pre-training the image feature comprehensive processing component, the image feature comprehensive processing component learns the corresponding optimization factors when super-resolution optimization is performed from images in different low-resolution intervals to images in different high-resolution intervals. In actual use, the corresponding optimization factors can be screened out only when the expected resolution and the resolution of the input image are known.
[0120] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:
[0121] Step 501, extracting the image features of the target image through the feature extraction layer of the image feature comprehensive processing component, wherein the image features include image resolution;
[0122] Step 502, identifying the resolution range corresponding to the target image before optimization through the image resolution;
[0123] Step 503, identifying the resolution interval corresponding to the target image after super-resolution processing through the target resolution;
[0124] Step 504 , determining an optimization factor corresponding to optimizing the target image to the target resolution according to the resolution range corresponding to the target image before optimization and the resolution range corresponding to the target image after super-resolution processing.
[0125] Step 204: receiving the image features and optimization factors output by the image feature comprehensive processing component.
[0126] Step 205: input the image features and the optimization factors into the pre-trained image super-resolution reconstruction component.
[0127] Step 206: Filter out reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor.
[0128] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 206 shown includes:
[0129] Step 601, according to the correspondence between the reconstruction control parameter and the optimization factor, determining the target reconstruction control parameter corresponding to the optimization factor;
[0130] Step 602: setting the target reconstruction control parameter as the reconstruction control parameter of the image super-resolution reconstruction component.
[0131] Step 207: performing image super-resolution processing based on the reconstruction control parameter, the image feature and the image super-resolution reconstruction component.
[0132] Continue to refer Figure 7 , Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 207 shown includes:
[0133] Step 701, dynamically adjusting the image super-resolution reconstruction component by using the reconstruction control parameter to obtain a dynamically adjusted image super-resolution reconstruction component;
[0134] Step 702: Input the image features into the dynamically adjusted image super-resolution reconstruction component, and after decoding and output, obtain a super-resolution processed image.
[0135] The present application pre-trains an image feature comprehensive processing component so that the image feature comprehensive processing component learns the optimization factors corresponding to the super-resolution optimization of images in different low-resolution intervals to images in different high-resolution intervals, and subsequently trains the image super-resolution reconstruction component using an adversarial training method so that the image super-resolution reconstruction component learns the reconstruction control parameters that need to be dynamically adjusted when the super-resolution optimization of images in different low-resolution intervals to images in different high-resolution intervals is performed, and constructs a corresponding relationship between the optimization factors and the reconstruction control parameters, so that when the image super-resolution processing is actually performed subsequently, only the optimization factors need to be known to select appropriate reconstruction control parameters to set the image super-resolution reconstruction component, thereby obtaining an image of the expected resolution, which is more intelligent and automated, and realizes automatic dynamic adjustment of the super-resolution optimization of the target image.
[0136] The present application obtains the target image and target resolution; inputs the target image and target resolution into the image feature comprehensive processing component to perform image feature comprehensive processing; receives the image features and optimization factors output by the image feature comprehensive processing component; inputs the image features and optimization factors into the pre-trained image super-resolution reconstruction component; selects the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performs image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. When super-resolution optimization is performed on low-resolution images, the image details can be efficiently and accurately reconstructed by automatically adjusting the reconstruction control parameters.
[0137] The embodiments of the present application can acquire and process relevant data based on image processing technology. Among them, image processing (Artificial Intelligence, AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0138] Image processing basic technologies generally include sensors, dedicated image processing chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Image processing software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0139] In the embodiment of the present application, by acquiring the target image and target resolution; inputting the target image and target resolution into the image feature comprehensive processing component to perform image feature comprehensive processing; receiving the image features and optimization factors output by the image feature comprehensive processing component; inputting the image features and optimization factors into the pre-trained image super-resolution reconstruction component; screening the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performing image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. When super-resolution optimization is performed on low-resolution images, the image details can be efficiently and accurately reconstructed by automatically adjusting the reconstruction control parameters.
[0140] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of an image super-resolution device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0141] like Figure 8As shown, the image super-resolution device 800 described in this embodiment includes: a target image acquisition module 801, a target resolution acquisition module 802, an image feature comprehensive processing module 803, an image feature and optimization factor acquisition module 804, an image feature and optimization factor input module 805, a reconstruction control parameter screening module 806 and an image super-resolution processing module 807. Among them:
[0142] A target image acquisition module 801 is used to acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing;
[0143] A target resolution acquisition module 802 is used to acquire a target resolution, wherein the target resolution is an expected resolution after super-resolution is performed on the target image;
[0144] An image feature comprehensive processing module 803 is used to input the target image and the target resolution into a pre-trained image feature comprehensive processing component to perform image feature comprehensive processing;
[0145] An image feature and optimization factor acquisition module 804 is used to receive the image features and optimization factors output by the image feature comprehensive processing component;
[0146] An image feature and optimization factor input module 805 is used to input the image feature and the optimization factor into a pre-trained image super-resolution reconstruction component;
[0147] A reconstruction control parameter screening module 806 is used to screen the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor;
[0148] The image super-resolution processing module 807 is used to perform image super-resolution processing based on the reconstruction control parameters, the image features and the image super-resolution reconstruction component.
[0149] The present application obtains the target image and target resolution; inputs the target image and target resolution into the image feature comprehensive processing component to perform image feature comprehensive processing; receives the image features and optimization factors output by the image feature comprehensive processing component; inputs the image features and optimization factors into the pre-trained image super-resolution reconstruction component; selects the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performs image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. When super-resolution optimization is performed on low-resolution images, the image details can be efficiently and accurately reconstructed by automatically adjusting the reconstruction control parameters.
[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0151] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0152] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.
[0153] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 with components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0154] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0155] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of an image super-resolution method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.
[0156] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions stored in the memory 9a or process data, such as computer-readable instructions for running the image super-resolution method.
[0157] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0158] The computer device proposed in this embodiment belongs to the field of image processing technology and is applied to image resolution optimization scenarios. This application obtains a target image and a target resolution; inputs the target image and the target resolution into an image feature comprehensive processing component to perform image feature comprehensive processing; receives image features and optimization factors output by the image feature comprehensive processing component; inputs the image features and optimization factors into a pre-trained image super-resolution reconstruction component; selects reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performs image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. This realizes efficient and accurate reconstruction of image details by automatically adjusting the reconstruction control parameters when super-resolution optimization is performed on low-resolution images.
[0159] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the image super-resolution method as described above.
[0160] The computer-readable storage medium proposed in this embodiment belongs to the field of image processing technology and is applied to image resolution optimization scenarios. This application obtains a target image and a target resolution; inputs the target image and the target resolution into an image feature comprehensive processing component to perform image feature comprehensive processing; receives image features and optimization factors output by the image feature comprehensive processing component; inputs the image features and optimization factors into a pre-trained image super-resolution reconstruction component; screens out reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factors; and performs image super-resolution processing based on the reconstruction control parameters, image features, and the image super-resolution reconstruction component. This realizes efficient and accurate reconstruction of image details by automatically adjusting the reconstruction control parameters when super-resolution optimization is performed on low-resolution images.
[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0162] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. An image super-resolution method, characterized in that: The steps include: Acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing; Acquire a target resolution, wherein the target resolution is an expected resolution after super-resolution of the target image; Inputting the target image and the target resolution into a pre-trained image feature comprehensive processing component to perform image feature comprehensive processing; Receiving the image features and optimization factors output by the image feature comprehensive processing component; Inputting the image features and the optimization factors into a pre-trained image super-resolution reconstruction component; Filtering the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor; Image super-resolution processing is performed based on the reconstruction control parameters, the image features and the image super-resolution reconstruction component.
2. The image super-resolution method according to claim 1, characterized in that: Before executing the step of inputting the target image to the pre-trained image feature comprehensive processing component, the method further includes: Obtain N groups of training images, wherein each group of training images in the N groups of training images consists of a low-resolution image and M high-resolution images corresponding to the low-resolution image, resolutions of the M high-resolution images have gradient differences, and N and M are positive integers; Inputting the N groups of training images into an image super-resolution model to be trained, wherein the image super-resolution model is constructed in an end-to-end mode by an image feature comprehensive processing component to be trained and an image super-resolution reconstruction component to be trained; Extracting the image features of all the images in the N groups of training images respectively through the feature extraction layer of the image feature comprehensive processing component; Inputting the image features into the pooling processing layer of the image feature comprehensive processing component to obtain unique global feature vectors corresponding to all images output by the pooling processing layer; According to a preset image group number, all unique global feature vectors are grouped and sorted to obtain N groups of sorting results, wherein each group of sorting results in the N groups of sorting results is composed of a unique global feature vector of a low-resolution image and unique global feature vectors of M high-resolution images corresponding to the low-resolution image; The unique global feature vectors in the N groups of sorting results are integrated by the feature integration layer of the image feature comprehensive processing component to obtain optimization factors when the unique global feature vectors of the low-resolution images in each group of sorting results are converted into the unique global feature vectors of the corresponding M high-resolution images; Based on the optimization factors when the unique global feature vector of the low-resolution image in each group of sorting results is converted to the unique global feature vector of the corresponding M high-resolution images, classification and integration are performed to obtain the corresponding optimization factors when super-resolution optimization is performed on images in different low-resolution intervals to images in different high-resolution intervals, and obtain the pre-trained image feature comprehensive processing component.
3. The image super-resolution method according to claim 2, characterized in that: After executing the step of inputting the N groups of training images into the image super-resolution model to be trained, the method further includes: Screening out low-resolution images in each group of training images from the N groups of training images, and inputting all the low-resolution images into the image super-resolution reconstruction component to be trained; Based on the image super-resolution reconstruction component, super-resolution reconstruction of the low-resolution image is completed to obtain a reconstructed image; Selecting M high-resolution images from each group of training images from the N groups of training images as verification images; Inputting the reconstructed image into the image feature comprehensive processing component to obtain reconstructed image features; According to the reconstructed image features and the image features of the M high-resolution images in each group of training images, adversarial training is performed on the image super-resolution reconstruction component until the image feature loss value meets a preset loss threshold. The training of the image super-resolution reconstruction component is completed, and a pre-trained image super-resolution reconstruction component is obtained.
4. The image super-resolution method according to claim 3, characterized in that: The step of performing adversarial training on the image super-resolution reconstruction component until the image feature loss value meets a preset loss threshold and the training of the image super-resolution reconstruction component is completed specifically includes: Use the preset loss function: L total =L structure +λ1L texture +λ2L edge , Calculate the image feature loss value for each adversarial training, where L total Represents the image feature loss value, L structure Represents the image structure feature loss value, L texture Represents the image texture feature loss value, L edge represents the image edge feature loss value, λ1 and λ2 are weight coefficients, which are used to balance the importance of image structure feature loss value, image texture feature loss value and image edge feature loss value; If the image feature loss value of the current adversarial training does not meet the preset loss threshold, the reconstruction control parameters of the image super-resolution reconstruction component are dynamically adjusted through the back propagation algorithm, and the adversarial training is performed again; If the image feature loss value of the current adversarial training meets the preset loss threshold, the current adversarial training is completed, the reconstruction control parameters of the image super-resolution reconstruction component when the current adversarial training is completed are recorded, the next verification image is replaced, and the adversarial training is performed again; The training of the image super-resolution reconstruction component is completed until the image super-resolution reconstruction component completes the adversarial training based on all verification images, sorts out the reconstruction control parameters recorded when each adversarial training is completed, and outputs the reconstruction control parameter record log.
5. The image super-resolution method according to claim 4, characterized in that: After executing the step of obtaining the pre-trained image super-resolution reconstruction component, the method further includes: Based on the reconstruction control parameter record log, the reconstruction control parameters recorded when each adversarial training is completed are obtained, wherein the reconstruction control parameters include parameters of each network structure in the image super-resolution reconstruction component; According to the corresponding relationship between the reconstructed image output by the image super-resolution reconstruction component and the verification image, a corresponding relationship between the reconstruction control parameters and the optimization factors is established.
6. The image super-resolution method according to claim 1, characterized in that: The step of inputting the target image and the target resolution into the pre-trained image feature comprehensive processing component to perform image feature comprehensive processing specifically includes: Extracting image features of the target image through the feature extraction layer of the image feature comprehensive processing component, wherein the image features include image resolution; Identifying the resolution range corresponding to the target image before optimization through the image resolution; Identifying, by means of the target resolution, a resolution interval corresponding to the target image after super-resolution processing; An optimization factor corresponding to optimizing the target image to the target resolution is determined according to a resolution interval corresponding to the target image before optimization and a resolution interval corresponding to the target image after super-resolution processing.
7. The image super-resolution method according to claim 1 or 6, characterized in that: The step of selecting the reconstruction control parameter of the image super-resolution reconstruction component according to the optimization factor specifically includes: Determining the target reconstruction control parameter corresponding to the optimization factor according to the corresponding relationship between the reconstruction control parameter and the optimization factor; Setting the target reconstruction control parameter as a reconstruction control parameter of the image super-resolution reconstruction component; The step of performing image super-resolution processing based on the reconstruction control parameter, the image feature and the image super-resolution reconstruction component specifically includes: Dynamically adjusting the image super-resolution reconstruction component by using the reconstruction control parameter to obtain a dynamically adjusted image super-resolution reconstruction component; The image features are input into the dynamically adjusted image super-resolution reconstruction component, and after decoding and output, a super-resolution processed image is obtained.
8. An image super-resolution device, characterized in that: include: A target image acquisition module, used to acquire a target image, wherein the target image includes an image to be subjected to super-resolution processing; A target resolution acquisition module, used to acquire a target resolution, wherein the target resolution is an expected resolution after super-resolution of the target image; An image feature comprehensive processing module, used for inputting the target image and the target resolution into a pre-trained image feature comprehensive processing component to perform image feature comprehensive processing; An image feature and optimization factor acquisition module, used for receiving the image features and optimization factors output by the image feature comprehensive processing component; An image feature and optimization factor input module, used for inputting the image feature and the optimization factor into a pre-trained image super-resolution reconstruction component; A reconstruction control parameter screening module, used to screen out the reconstruction control parameters of the image super-resolution reconstruction component according to the optimization factor; The image super-resolution processing module is used to perform image super-resolution processing based on the reconstruction control parameter, the image feature and the image super-resolution reconstruction component.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image super-resolution method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image super-resolution method according to any one of claims 1 to 7 are implemented.