Image super-resolution processing method, device and equipment based on zero-shot learning

Abstract

The present disclosure provides a zero-shot learning-based image super-resolution processing method, device and equipment, which can be applied to the technical field of image processing. The zero-shot learning-based image super-resolution processing method comprises: acquiring a wide-angle image and a long-focus image; cropping the wide-angle image to obtain a cropped image; aligning the cropped image to the long-focus image by using a trained alignment network to obtain an aligned image; cropping a first image pair to obtain a plurality of second image pairs, wherein the first image pair represents an image pair composed of the aligned image and the long-focus image; training a super-resolution model by using the plurality of second image pairs to obtain a target super-resolution model; and processing the wide-angle image by using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image super-resolution processing method, apparatus and device based on zero-shot learning. Background Technology

[0002] Current mobile devices typically feature asymmetric camera systems, such as the camera modules in smartphones. Asymmetric camera systems are used to capture images of the same scene with different field-of-view sizes. These systems usually consist of a wide-angle lens and a telephoto lens. The wide-angle lens has a wider field of view than the telephoto lens and is typically the main lens in the system. However, due to the constraint between field of view and resolution, within the overlapping field of view of the wide-angle and telephoto lenses, the resolution of the wide-angle lens is lower than that of the telephoto lens.

[0003] To obtain images with a large field of view and high resolution, a deep learning-based super-resolution processing method is proposed. This method predefines degradation, trains a super-resolution model in a supervised manner based on a large amount of sample data, and then processes the wide-angle image using the trained super-resolution model to obtain a super-resolution image corresponding to the wide-angle image. The sample data consists of wide-angle and telephoto images acquired multiple times by the same asymmetric camera system.

[0004] In realizing the present invention, the inventors discovered at least the following problems in the related technologies: the quality of the super-resolution images corresponding to wide-angle images obtained by the related deep learning-based super-resolution processing methods cannot meet the needs of practical applications. Summary of the Invention

[0005] In view of the above problems, this disclosure provides an image super-resolution processing method, apparatus and device based on zero-shot learning.

[0006] According to a first aspect of this disclosure, an image super-resolution processing method based on zero-shot learning is provided, comprising:

[0007] Acquire wide-angle and telephoto images, where wide-angle and telephoto images represent images of the same scene acquired by two acquisition devices with different focal lengths, and the resolution of the wide-angle image is lower than that of the telephoto image;

[0008] A cropped image is obtained by cropping a wide-angle image, wherein the field of view of the cropped image overlaps with that of the telephoto image;

[0009] The cropped image is aligned to the telephoto image using a trained alignment network to obtain the aligned image;

[0010] The first image pair is cropped to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of an aligned image and a telephoto image;

[0011] The super-resolution model is trained using multiple second images to obtain the target super-resolution model;

[0012] The target super-resolution model is used to process the wide-angle image to obtain the super-resolution image corresponding to the wide-angle image.

[0013] According to embodiments of this disclosure, the above-described method of aligning a cropped image to a telephoto image using a trained alignment network to obtain an aligned image includes:

[0014] The telephoto image is downsampled to obtain a downsampled image, where the resolution of the downsampled image is the same as that of the wide-angle image;

[0015] The cropped image and the downsampled image are input into the alignment network to obtain the aligned image.

[0016] According to embodiments of this disclosure, the training method for the alignment network includes:

[0017] The cropped image and the downsampled image are input into the alignment network to obtain the initial aligned image;

[0018] The initial aligned image is input into the spatial local discriminator to obtain the first discrimination result corresponding to the initial aligned image;

[0019] The spectrogram corresponding to the initial alignment image is input into the frequency domain global discriminator to obtain the second discrimination result corresponding to the initial alignment image;

[0020] The initial aligned image, cropped image, and downsampled image are input into a deep convolutional neural network to obtain intermediate feature maps corresponding to the initial aligned image, cropped image, and downsampled image, respectively.

[0021] Calculate the quadratic norm between the downsampled image and the initially aligned image;

[0022] Based on the first discrimination result, the second discrimination result, the intermediate feature map and the second norm, the alignment loss corresponding to the initial alignment network is obtained;

[0023] Update the network parameters of the initial alignment network based on the alignment loss.

[0024] According to embodiments of this disclosure, the training method for the spatial local discriminator includes:

[0025] The cropped image is input into the spatial local discriminator to obtain the third discrimination result corresponding to the cropped image;

[0026] Based on the first and third discrimination results, the spatial loss corresponding to the spatial local discriminator is obtained;

[0027] The parameters of the spatial local discriminator are updated based on the spatial loss.

[0028] According to embodiments of this disclosure, the training method for the frequency domain global discriminator includes:

[0029] Input the spectrogram corresponding to the cropped image into the frequency domain global discriminator to obtain the fourth discrimination result corresponding to the cropped image;

[0030] Based on the second and fourth discrimination results, the frequency domain loss corresponding to the frequency domain global discriminator is obtained;

[0031] Update the parameters of the frequency domain global discriminator based on the frequency domain loss.

[0032] According to embodiments of this disclosure, the alignment loss corresponding to the initial alignment network is obtained based on the first discrimination result, the second discrimination result, the intermediate feature map, and the quadratic norm, calculated according to the following formula (a):

[0033]

[0034] Among them, L align The alignment loss is represented by L2, and the quadratic norm is represented by L2. The loss term is characterized by its association with the first discrimination result. L represents the loss term associated with the second discrimination result. cl The loss term is characterized by its association with the intermediate feature map, and λ1, λ2, and λ3 represent the weight coefficients.

[0035] According to embodiments of this disclosure, the loss term associated with the intermediate feature map is calculated according to the following formula (ii):

[0036]

[0037] Among them, L cl The loss characterizes the loss associated with the intermediate feature maps. The image representing the output of the i-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ i (Y * The output of the i-th layer of a deep convolutional neural network and the cropped image Y * The corresponding intermediate feature map, The image representing the output of the j-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ j (X ↓ The output of the j-th layer of a deep convolutional neural network and the downsampled image X ↓The corresponding intermediate feature map, n represents the nth layer of the deep convolutional neural network, m represents the mth layer of the deep convolutional neural network, and m and n are both integers greater than or equal to 1.

[0038] According to embodiments of this disclosure, cropping the first image pair to obtain a plurality of second image pairs includes:

[0039] The aligned image is processed using a trained spatial local discriminator to obtain a similarity probability map, where the pixel value of each pixel in the similarity probability map represents the similarity between the aligned image and the telephoto image at that pixel.

[0040] The first image pair is cropped based on the similarity probability map to obtain multiple second image pairs.

[0041] A second aspect of this disclosure provides an image super-resolution processing apparatus based on zero-shot learning, comprising:

[0042] The acquisition module is used to acquire wide-angle and telephoto images. The wide-angle and telephoto images represent images acquired by two acquisition devices with different focal lengths from the same scene. The resolution of the wide-angle image is smaller than that of the telephoto image.

[0043] The first module is used to crop the wide-angle image to obtain a cropped image, wherein the field of view of the cropped image overlaps with that of the telephoto image;

[0044] The second module is used to align the cropped image to the telephoto image using the trained alignment network, thus obtaining the aligned image.

[0045] The third obtaining module is used to crop the first image pair to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of an aligned image and a telephoto image;

[0046] The fourth module is used to train the super-resolution model using multiple second images to obtain the target super-resolution model.

[0047] The fifth module is used to process the wide-angle image using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image.

[0048] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0049] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0050] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0051] According to embodiments of this disclosure, by acquiring a wide-angle image and a telephoto image, a low-resolution image with a large field of view (i.e., a wide-angle image) and a high-resolution image with a small field of view (i.e., a telephoto image) for the same scene are obtained. The wide-angle image is cropped to obtain a cropped image whose field of view overlaps with that of the telephoto image. Then, a trained alignment network is used to align the cropped image to the telephoto image to obtain an aligned image. The first image pair composed of the aligned image and the telephoto image is cropped to obtain multiple second image pairs. The super-resolution model is trained using the multiple second image pairs to obtain a target super-resolution model, such that the target super-resolution model... The model fully learns about the degradation of the current wide-angle image, and then uses the target super-resolution model to process the wide-angle image to obtain a super-resolution image corresponding to the wide-angle image. This allows the target super-resolution model to improve the super-resolution of the current wide-angle image based on its degradation, thus making the target super-resolution model more suitable for improving the super-resolution of wide-angle images obtained under unknown shooting conditions. This enhances the quality of the super-resolution image corresponding to the current wide-angle image, ensuring that the super-resolution image corresponding to the current wide-angle image meets the needs of practical applications. Attached Figure Description

[0052] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0053] Figure 1 This illustration schematically depicts an application scenario of the image super-resolution processing method based on zero-shot learning according to embodiments of the present disclosure.

[0054] Figure 2 A flowchart illustrating an image super-resolution processing method based on zero-shot learning according to an embodiment of the present disclosure is shown schematically.

[0055] Figure 3 The flowchart illustrating the training of an alignment network, a spatial local discriminator, and a frequency domain global discriminator according to embodiments of the present disclosure is shown in the schematic diagram.

[0056] Figure 4 A flowchart illustrating the cropping of a first image pair according to an embodiment of the present disclosure is shown schematically;

[0057] Figure 5 A flowchart illustrating another embodiment of the image super-resolution processing method based on zero-shot learning according to this disclosure is shown schematically.

[0058] Figure 6 A schematic diagram of a super-resolution image obtained by an image super-resolution processing method according to an embodiment of the present disclosure is shown.

[0059] Figure 7 A schematic diagram illustrating the structure of an image super-resolution processing apparatus based on zero-shot learning according to an embodiment of the present disclosure is shown.

[0060] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a zero-shot learning-based image super-resolution processing method according to embodiments of the present disclosure. Detailed Implementation

[0061] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0062] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0063] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0064] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0065] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0066] According to embodiments of this disclosure, an asymmetric camera system may include multiple lenses with fixed focal lengths, the multiple lenses having different focal lengths.

[0067] According to embodiments of this disclosure, due to minute camera motion during the acquisition of image pairs (wide-angle and telephoto images) by an asymmetric camera system, the degradation of each wide-angle image acquired by the same asymmetric camera system varies. This makes the super-resolution model trained based on predefined degradation unsuitable for super-resolution enhancement of the resolution of each wide-angle image. Therefore, the quality of the super-resolution image corresponding to the wide-angle image obtained by the relevant deep learning-based super-resolution processing method cannot meet the needs of practical applications.

[0068] In order to at least partially solve the technical problems existing in the related technologies, the embodiments of this disclosure provide an image super-resolution processing method, apparatus and device based on zero-shot learning, which can be applied to the field of image processing.

[0069] Embodiments of this disclosure provide an image super-resolution processing method based on zero-shot learning, comprising: acquiring a wide-angle image and a telephoto image, wherein the wide-angle image and the telephoto image represent images acquired by two acquisition devices with different focal lengths of the same scene respectively, and the resolution of the wide-angle image is less than that of the telephoto image; cropping the wide-angle image to obtain a cropped image, wherein the field of view of the cropped image overlaps with that of the telephoto image; aligning the cropped image to the telephoto image using a trained alignment network to obtain an aligned image; cropping a first image pair to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of the aligned image and the telephoto image; training a super-resolution model using the multiple second image pairs to obtain a target super-resolution model; and processing the wide-angle image using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image.

[0070] Figure 1 The illustration shows an application scenario of the image super-resolution processing method based on zero-shot learning according to an embodiment of the present disclosure.

[0071] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0072] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0073] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0074] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0075] It should be noted that the zero-shot learning-based image super-resolution processing method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the zero-shot learning-based image super-resolution processing apparatus provided in this disclosure embodiment can generally be located in server 105. The zero-shot learning-based image super-resolution processing method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the zero-shot learning-based image super-resolution processing apparatus provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0076] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0077] The following will be based on Figure 1 The described scene, through Figures 2-6 The image super-resolution processing method based on zero-shot learning according to the disclosed embodiments is described in detail.

[0078] Figure 2 A flowchart illustrating an image super-resolution processing method based on zero-shot learning according to an embodiment of the present disclosure is shown.

[0079] like Figure 2 As shown, the image super-resolution processing method based on zero-shot learning in this embodiment includes operations S210 to S230.

[0080] In operation S210, a wide-angle image and a telephoto image are acquired. The wide-angle image and the telephoto image represent images acquired by two acquisition devices with different focal lengths from the same scene. The resolution of the wide-angle image is smaller than that of the telephoto image.

[0081] According to embodiments of this disclosure, the two acquisition devices with different focal lengths can be an acquisition device corresponding to a wide-angle lens and an acquisition device corresponding to a telephoto lens in an asymmetric camera system. The field of view of the wide-angle lens is larger than that of the telephoto lens.

[0082] According to embodiments of this disclosure, for the same scene, a wide-angle image can be obtained by using an acquisition device corresponding to a wide-angle lens, and a telephoto image can be obtained by using an acquisition device corresponding to a telephoto lens.

[0083] According to embodiments of this disclosure, the number of wide-angle images and telephoto images in operation S210 is 1 each.

[0084] In operation S220, the wide-angle image is cropped to obtain a cropped image, wherein the field of view of the cropped image overlaps with that of the telephoto image.

[0085] According to embodiments of this disclosure, a wide-angle image can be cropped based on the multiple relationship between the focal length of a wide-angle lens and the focal length of a telephoto lens to obtain a cropped image.

[0086] According to embodiments of this disclosure, for example, when the focal length of a telephoto lens is twice the focal length of a wide-angle lens, the central region of a wide-angle image with a large field of view can be cropped to obtain a cropped image that overlaps with the field of view of the telephoto image, such that the size of the cropped image is half the size of the wide-angle image.

[0087] In operation S230, the trained alignment network is used to align the cropped image to the telephoto image to obtain the aligned image.

[0088] According to embodiments of this disclosure, the alignment network can be a network based on a convolutional neural network architecture. The embodiments of this disclosure do not limit the alignment network; it can be selected according to actual business needs.

[0089] According to embodiments of this disclosure, the alignment network can be, for example, a supervised optical flow estimation network (FlowNet, Learning Optical Flow with Convolutional Networks).

[0090] According to embodiments of this disclosure, for example, spatially misaligned image pairs (telephoto image, cropped image) can be input into a trained FlowNet, which processes the misaligned image pairs to obtain aligned images.

[0091] In operation S240, the first image pair is cropped to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of an aligned image and a telephoto image.

[0092] According to embodiments of this disclosure, a first image pair can be cropped according to a predetermined size and a predetermined cropping quantity to obtain a plurality of second image pairs.

[0093] According to embodiments of this disclosure, there are no limitations on the predetermined size and predetermined cutting quantity; the selection can be made according to actual business needs.

[0094] According to embodiments of this disclosure, for example, when the size of each image in the first image pair is 1024*1024px (pixels), the predetermined size can be 512*512px, and the predetermined cropping number can be 20. The predetermined size can also be 256*256px, and the predetermined cropping number can also be 30. The predetermined sizes can also be 512*512px and 256*256px respectively, with the predetermined cropping number corresponding to the size 512*512px being 10, and the predetermined cropping number corresponding to the size 256*256px being 50.

[0095] According to embodiments of this disclosure, for example, if the size of each image in the first image pair is 1024*1024px, the predetermined size is 512*512px, and the predetermined cropping quantity is 20, the first image pair can be randomly cropped into 20 second image pairs with a size of 512*512px, thus obtaining multiple second image pairs.

[0096] In operation S250, the super-resolution model is trained using multiple second images to obtain the target super-resolution model.

[0097] According to embodiments of this disclosure, the super-resolution model can be a model composed of deep convolutional networks based on an attention mechanism. The embodiments of this disclosure do not limit the super-resolution model; it can be selected according to actual business needs.

[0098] According to embodiments of this disclosure, the super-resolution model may be, for example, a deep residual channel convolutional network (RCAN, Image super-resolution using very deep residual channel attention networks).

[0099] According to embodiments of this disclosure, a super-resolution model is trained using multiple second image pairs to obtain a trained target super-resolution model. This allows the super-resolution model to fully learn the degradation of the current wide-angle image based on the multiple second image pairs, resulting in a target super-resolution model that has fully learned the degradation of the current wide-angle image.

[0100] In operation S260, the wide-angle image is processed using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image.

[0101] According to embodiments of this disclosure, since the target super-resolution model has fully learned the degradation of the current wide-angle image based on multiple second image pairs, the wide-angle image is processed using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image. This allows the target super-resolution model to improve the super-resolution of the current wide-angle image based on the degradation of the current wide-angle image, thereby obtaining a super-resolution image corresponding to the current wide-angle image. This improves the quality of the super-resolution image corresponding to the current wide-angle image, enabling the super-resolution image corresponding to the current wide-angle image to meet the needs of practical applications.

[0102] According to embodiments of this disclosure, by acquiring a wide-angle image and a telephoto image, a low-resolution image with a large field of view (i.e., a wide-angle image) and a high-resolution image with a small field of view (i.e., a telephoto image) for the same scene are obtained. The wide-angle image is cropped to obtain a cropped image whose field of view overlaps with that of the telephoto image. Then, a trained alignment network is used to align the cropped image to the telephoto image to obtain an aligned image. The first image pair composed of the aligned image and the telephoto image is cropped to obtain multiple second image pairs. The super-resolution model is trained using the multiple second image pairs to obtain a target super-resolution model, such that the target super-resolution model... The model fully learns about the degradation of the current wide-angle image, and then uses the target super-resolution model to process the wide-angle image to obtain a super-resolution image corresponding to the wide-angle image. This allows the target super-resolution model to improve the super-resolution of the current wide-angle image based on its degradation, thus making the target super-resolution model more suitable for improving the super-resolution of wide-angle images obtained under unknown shooting conditions. This enhances the quality of the super-resolution image corresponding to the current wide-angle image, ensuring that the super-resolution image corresponding to the current wide-angle image meets the needs of practical applications.

[0103] According to embodiments of this disclosure, for Figure 2 Operation S230, shown, uses a trained alignment network to align the cropped image to the telephoto image to obtain an aligned image, and may include the following operations:

[0104] The telephoto image is downsampled to obtain a downsampled image, where the resolution of the downsampled image is the same as that of the wide-angle image;

[0105] The cropped image and the downsampled image are input into the alignment network to obtain the aligned image.

[0106] According to embodiments of this disclosure, the method for downsampling telephoto images can be selected based on actual business needs, and the embodiments of this disclosure do not limit the downsampling method. For example, a bilinear interpolation algorithm can be used to downsample the telephoto image to obtain a downsampled image.

[0107] According to embodiments of this disclosure, a cropped image and a downsampled image are input into an alignment network to obtain an aligned image. The alignment network uses the downsampled image as a guide for alignment and performs spatial transformation on the cropped image so that the spatial position of the cropped image is aligned with the telephoto image, resulting in an aligned image that is aligned with the downsampled image.

[0108] Figure 3 The flowchart illustrating the training of an alignment network, a spatial local discriminator, and a frequency domain global discriminator according to embodiments of the present disclosure is shown.

[0109] like Figure 3 As shown, the training methods for the alignment network include:

[0110] Crop image Y * "301" and downsampled image X ↓ Inputting “302” into alignment network 310 yields the initial alignment image. "303".

[0111] Initial alignment image "303" Input Spatial Local Discriminator D s In “320”, the image aligned with the initial value is obtained. The first discrimination result corresponding to "303".

[0112] Align the image with the initial alignment The spectrum corresponding to "303" is input to the frequency domain global discriminator D. f In “330”, the image aligned with the initial value is obtained. The second discrimination result corresponding to “303”.

[0113] Initial alignment image "303", cropping image Y * "301" and downsampled image X ↓ Inputting “302” into the deep convolutional neural network 340, the resulting image is aligned with the initial image. "303", cropping image Y * "301" and downsampled image X ↓ The intermediate feature maps corresponding to “302”.

[0114] Calculate the downsampled image X ↓ "302" is aligned with the initial image. The quadratic norm between “303” is 304.

[0115] Based on the first discrimination result, the second discrimination result, the intermediate feature map, and the second norm 304, the alignment loss corresponding to the initial alignment network is obtained.

[0116] Update the network parameters of the initial alignment network based on the alignment loss.

[0117] According to embodiments of this disclosure, such as Figure 3 The alignment network 310 shown can be, for example, FlowNet, and the deep convolutional neural network 340 can be, for example, VGG (Very deep convolutional networks for large-scale image recognition).

[0118] According to embodiments of this disclosure, such as Figure 3 The spatial local discriminator D shown s "320" can include 8 convolutional layers and a global average pooling layer. For example... Figure 3 The frequency domain global discriminator D shown f "330" can include 8 convolutional layers.

[0119] like Figure 3 The loss term related to the first discrimination result can be calculated based on the first discrimination result. "305" indicates that the loss term related to the second discrimination result can be calculated based on the second discrimination result. "306" indicates that the loss term L related to the intermediate feature map can be calculated based on the intermediate feature map. cl "307". Then, based on the second norm of 304 and the loss term related to the first discrimination result... "305", loss item related to the second judgment result "306" and the loss term L associated with the intermediate feature map cl"307" obtains the alignment loss corresponding to the initial alignment network, and then updates the network parameters of the initial alignment network based on the alignment loss.

[0120] According to embodiments of this disclosure, the alignment loss corresponding to the initial alignment network, based on the first discrimination result, the second discrimination result, the intermediate feature map, and the quadratic norm, can be calculated according to the following formula (a):

[0121]

[0122] Among them, L align The alignment loss is represented by L2, and the quadratic norm is represented by L2. The loss term is characterized by its association with the first discrimination result. L represents the loss term associated with the second discrimination result. cl The loss term is characterized by its association with the intermediate feature map, and λ1, λ2, and λ3 represent the weight coefficients.

[0123] According to embodiments of this disclosure, the loss term associated with the intermediate feature map can be calculated using the following formula (ii):

[0124]

[0125] Among them, L cl The loss characterizes the loss associated with the intermediate feature maps. The image representing the output of the i-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ i (Y * The output of the i-th layer of a deep convolutional neural network and the cropped image Y * The corresponding intermediate feature map, The image representing the output of the j-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ j (X ↓ The output of the j-th layer of a deep convolutional neural network and the downsampled image X ↓ The corresponding intermediate feature map, n represents the nth layer of the deep convolutional neural network, m represents the mth layer of the deep convolutional neural network, and m and n are both integers greater than or equal to 1.

[0126] According to embodiments of this disclosure, for example, both m and n can be 10.

[0127] According to embodiments of this disclosure, the loss term related to the first discrimination result can be calculated according to the following formula (iii):

[0128]

[0129] in, Characterizes the loss associated with the first discrimination result. Characterizing the spatial local discriminator D s Output image aligned with the initial alignment The corresponding first discrimination result, Characterization with The expectation of the variable.

[0130] According to embodiments of this disclosure, the spatial local discriminator D s It will align with the initial image. The first discrimination result is obtained by processing multiple small-sized images to obtain the discrimination results corresponding to each small-sized image. Then, the discrimination results corresponding to each small-sized image can be input into the formula. Then calculate the average (i.e., ...) ), thus obtaining the loss term related to the first discrimination result.

[0131] According to embodiments of this disclosure, the loss term related to the second discrimination result can be calculated according to the following formula (iv):

[0132]

[0133] in, The loss characterizes the result related to the second discrimination. Characterizing the global discriminator D fs Output image aligned with the initial alignment The corresponding second discrimination result, Characterization and initial alignment image The corresponding spectrum diagram.

[0134] According to embodiments of this disclosure, the quadratic norm between the downsampled image and the initial aligned image can be calculated using the following formula (v).

[0135] L2=||X ↓ -Y * ||2 (Friday)

[0136] According to embodiments of this disclosure, during the training of the alignment network, the quadratic norm is used as supervision to ensure the initial alignment of the image. The spatial location is aligned with the telephoto image X. Using a loss term related to the first discrimination result as supervision ensures the initial alignment of the image. The local statistical consistency with the telephoto image X in the spatial domain is achieved. Using a loss term related to the second discrimination result as supervision ensures the initial aligned image... Global statistical consistency with the telephoto image X in the frequency domain. Using a loss term associated with intermediate feature maps as supervision, the initial alignment of the image can be constrained within the feature representation space. Get as close as possible to the cropped image Y * And far from the downsampled image X ↓ This ensures that the degradation during the alignment process remains unchanged, that is, it ensures the consistency of degradation between the aligned image and the cropped image.

[0137] According to embodiments of this disclosure, during the training of the alignment network, an initial alignment image is obtained by inputting a cropped image and a downsampled image into the alignment network. The initial alignment image is then input into a spatial local discriminator to obtain a first discrimination result corresponding to the initial alignment image. The spectrogram corresponding to the initial alignment image is input into a frequency domain global discriminator to obtain a second discrimination result corresponding to the initial alignment image. The initial alignment image, the cropped image, and the downsampled image are then input into a deep convolutional neural network to obtain intermediate feature maps corresponding to the initial alignment image, the cropped image, and the downsampled image, respectively. The quadratic norm between the downsampled image and the initial alignment image is calculated. Based on the first discrimination result, the second discrimination result, the intermediate feature maps, and the quadratic norm, an alignment loss corresponding to the initial alignment network is obtained. Based on the alignment loss, the network parameters of the initial alignment network are updated so that the alignment network aligns the spatial position of the cropped image to the telephoto image while maintaining sufficient consistency in quality degradation between the alignment image and the cropped image.

[0138] like Figure 3 As shown, the spatial local discriminator and the frequency global discriminator are trained simultaneously during the training of the alignment network.

[0139] like Figure 3 As shown, the training method for the spatial local discriminator includes:

[0140] Initial alignment image "303" Input Spatial Local Discriminator D s In “320”, the image aligned with the initial value is obtained. The first discrimination result corresponding to "303". The cropped image Y... * "301" Input Spatial Local Discriminator D s In step “320”, the third discrimination result corresponding to the cropped image is obtained. Based on the first and third discrimination results, the spatial domain loss corresponding to the spatial domain local discriminator is obtained. The parameters of the spatial domain local discriminator are updated based on the spatial domain loss.

[0141] According to embodiments of this disclosure, the spatial loss corresponding to the spatial local discriminator can be calculated according to the following formula (vi) based on the first discrimination result and the third discrimination result:

[0142]

[0143] in, Characterized by the spatial loss corresponding to the spatial local discriminator, D s (Y * Characterizes the spatial local discriminator D s The output is the cropped image Y * The corresponding third discrimination result. Characterized by Y * The expectation of the variable.

[0144] like Figure 3 As shown, the training methods for the frequency domain global discriminator include:

[0145] Align the image with the initial alignment The spectrum corresponding to "303" is input to the frequency domain global discriminator D. f In “330”, the image aligned with the initial value is obtained. The second discrimination result corresponding to "303" will be compared with the cropped image Y. * The spectrum diagram corresponding to "301" is input to the frequency domain global discriminator D. f "330" yields the fourth discrimination result corresponding to the cropped image. Based on the second and fourth discrimination results, the frequency domain loss corresponding to the frequency domain global discriminator is obtained. The parameters of the frequency domain global discriminator are updated based on the frequency domain loss.

[0146] According to embodiments of this disclosure, the frequency domain loss corresponding to the frequency domain global discriminator can be calculated according to the following formula (vii) based on the second and fourth discrimination results:

[0147]

[0148] in, Characterizing the frequency domain loss corresponding to the frequency domain global discriminator, [D f (fl[Y * Characterizes the global discriminator D fs The output is the cropped image Y * The corresponding fourth discrimination result.

[0149] According to embodiments of this disclosure, a trained alignment network can be obtained when the number of training iterations reaches a preset number, or when the alignment loss is less than a preset loss value. The termination condition for training the alignment network can be determined according to actual business needs, and embodiments of this disclosure do not limit the termination condition.

[0150] According to embodiments of this disclosure, for Figure 2Operation S240, which cropps the first image pair to obtain multiple second image pairs, may include the following operations:

[0151] The aligned image is processed using a trained spatial local discriminator to obtain a similarity probability map, where the pixel value of each pixel in the similarity probability map represents the similarity between the aligned image and the telephoto image at that pixel.

[0152] The first image pair is cropped based on the similarity probability map to obtain multiple second image pairs.

[0153] Figure 4 A flowchart illustrating the cropping of a first image pair according to an embodiment of the present disclosure is shown schematically.

[0154] like Figure 4 As shown, the image will be aligned. "403" is fed into the optimized spatial local discriminator D. s "420" yields the degradation identification map M. s "405". The degradation identification image M will be displayed. s Upsampling and normalization of “405” yields the aligned image. Similarity probability map P with consistent resolution "403" D Then, based on the similarity probability graph P D For the first image pair (telephoto image x "404"), align the image. Data augmentation is performed using "403" to obtain a set of low-to-high-resolution image pairs, i.e., the second set of image pairs, used for training the super-resolution model. "406". Here, K represents the number of the second image pairs. The set of second images corresponding to the aligned image in the set of second image pairs can be represented as follows: The set of second images corresponding to the telephoto image in the set of second image pairs can be represented as follows:

[0155] According to embodiments of this disclosure, an optimized spatial local discriminator D is achieved. s For example, it can be... Figure 3 The trained spatial local discriminator D s Upsampling methods can include, for example, bilinear interpolation. D Each pixel also represents the probability that the pixel will be used in the second image pair.

[0156] According to embodiments of this disclosure, although the consistency of degradation is constrained during the alignment process, in practical applications, the aligned image will still have some areas of degradation compared to the cropped image.

[0157] According to embodiments of this disclosure, a similarity probability map is obtained by processing the aligned image using a trained spatial local discriminator. In the similarity probability map, the pixel value of each pixel represents the similarity between the aligned image and the telephoto image at that pixel. Then, the first image pair is cropped according to the similarity probability map to obtain multiple second image pairs, such that the multiple second image pairs include more regions in the aligned image that remain unchanged despite degradation.

[0158] According to embodiments of this disclosure, the process of training a super-resolution model for an asymmetric camera system can be represented as follows: Where S Θ (·) characterizes the super-resolution model, min Θ The expression uses Θ as a variable to find the minimum value, where Θ represents the model parameters of the super-resolution model, and L(·) represents the loss function used in the super-resolution model. The degradation in resolution caused by the increase in the field of view can be called resolution-field-of-view degradation D. RV (·). Furthermore, Y * , The relationship with X can be represented by DRV(·) as follows: Therefore, during the training of the super-resolution model, the super-resolution model can implicitly learn the degradation of the current wide-angle image.

[0159] Figure 5 A flowchart illustrating a zero-shot learning-based image super-resolution processing method according to another embodiment of the present disclosure is shown.

[0160] like Figure 5 As shown, an asymmetric camera system can be used to capture the same scene, obtaining a wide-angle image Y "501" and a telephoto image X "502". The wide-angle image Y "501" is then cropped from the center to obtain the cropped image Y. * "503", image cropping Y * The field of view of “503” overlaps with the field of view of the telephoto image X.

[0161] The telephoto image X "502" is downsampled to obtain the downsampled image X. ↓ "504". The image Y will be cropped. * "503" and downsampled image X ↓ In the "504" input alignment network 510, using, for example Figure 3 The method shown trains the alignment network 510 to obtain a trained alignment network, and then uses the trained alignment network to crop the image Y. * "503" and downsampled image X ↓ The "504" error is processed to obtain an aligned image.

[0162] Simultaneously training the alignment network 510, a spatial local discriminator is also trained, resulting in a trained spatial local discriminator 520. The aligned image is then processed based on the trained spatial discriminator 520. And the telephoto image X are performed as follows Figure 4 The cropping process shown yields a second set of image pairs. "505", using the second image to analyze the set The data in “505” is used to train the super-resolution model 530 to obtain the target super-resolution model. The wide-angle image Y “501” is then input into the target super-resolution model to obtain the super-resolution image Y corresponding to the wide-angle image Y “501”. SR "506".

[0163] like Figure 5 As shown, by acquiring wide-angle and telephoto images, low-resolution wide-field-of-view images (i.e., wide-angle images) and high-resolution small-field-of-view images (i.e., telephoto images) for the same scene are obtained. The wide-angle image is cropped to obtain a cropped image whose field of view overlaps with that of the telephoto image. Then, a trained alignment network is used to align the cropped image to the telephoto image to obtain an aligned image. The first image pair composed of the aligned image and the telephoto image is cropped to obtain multiple second image pairs. The super-resolution model is trained using multiple second image pairs to obtain the target super-resolution model, making the target super-resolution model fully... The algorithm learns about the degradation of the current wide-angle image and then uses a target super-resolution model to process the wide-angle image, obtaining a super-resolution image corresponding to the wide-angle image. This allows the target super-resolution model to improve the super-resolution of the current wide-angle image based on its degradation, making it more suitable for improving the super-resolution of wide-angle images obtained under unknown shooting conditions. This enhances the quality of the super-resolution image corresponding to the current wide-angle image, ensuring that it meets the needs of practical applications.

[0164] According to embodiments of this disclosure, the effectiveness of the image super-resolution processing method based on zero-shot learning provided by the embodiments of this disclosure was verified using two simulated datasets and two real datasets.

[0165] According to embodiments of this disclosure, the HCI_new light field dataset and the Middlebury2021 binocular dataset are used in the simulation dataset experiments. The image data in the datasets is processed through simulation to obtain image data from an asymmetric camera system. Specifically, the image from the main viewpoint is downsampled to obtain a simulated wide-angle image, and then an image from another viewpoint is selected and cropped to obtain a simulated telephoto image. The original image from the main viewpoint is used as the real image. Peak signal-to-noise ratio and structural similarity are used as numerical indicators for quantitative comparison.

[0166] The experimental results are shown in Tables 1 and 2. Here, IG and AG represent symmetric Gaussian downsampling and asymmetric Gaussian downsampling (i.e., two methods of downsampling the main viewpoint image), respectively; IG_JEPG represents symmetric Gaussian downsampling combined with JPEG compression; the data before the slash " / " represents the peak signal-to-noise ratio; the data after the slash " / " represents structural similarity; 2×scale represents a 2x improvement in super-resolution for wide-angle images; and 4×scale represents a 4x improvement in super-resolution for wide-angle images. The comparison methods are categorized as follows: single-image super-resolution methods, reference image-based super-resolution methods, and blind super-resolution methods. Single-image super-resolution methods include RCAN (Image super-resolution using very deep residual channel attention networks) and KernelGAN (Blind super-resolution kernel estimation using an internal GAN). Reference image-based super-resolution methods include MASA (Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution), DCSR-SRA (Dual-camera super-resolution with aligned attention modules), and SelfDZSR (Self-supervised learning for real-world super-resolution from dual zoomed observations). Blind super-resolution methods include DANv2 (End-to-end alternating optimization for blind super-resolution) and DCLS (Deep constrained least squares for blind image super-resolution). ZeDuSR in Table 1 represents the image super-resolution processing method provided in the embodiments of this disclosure. Ours-ZS in Table 2 represents the image super-resolution processing method provided in the embodiments of this disclosure.

[0167]

[0168] Table 1

[0169]

[0170] Table 2

[0171] The experimental results in Tables 1 and 2 show that, even with different degradation rates in the two simulation data and different super-resolution ratios of the obtained super-resolution images relative to the wide-angle images, the image super-resolution processing method provided in this embodiment yields super-resolution images of higher quality than existing methods.

[0172] According to embodiments of this disclosure, in experiments using real datasets, the asymmetric camera systems of iPhone 11 and iPhone 12 were used to acquire wide-angle and telephoto image data. Specifically, the data from iPhone 11 underwent 2x super-resolution enhancement, while the data from iPhone 12 underwent 2x and 4x super-resolution enhancement. The visual comparison experimental results are as follows: Figure 6 As shown.

[0173] Figure 6 A schematic diagram of a super-resolution image obtained by an image super-resolution processing method according to an embodiment of the present disclosure is shown. Figure 6 ZeDuSR represents the image super-resolution processing method provided in the embodiments of this disclosure.

[0174] Depend on Figure 6 It is known that when the super-resolution image obtained from the real dataset has a different super-resolution factor than the wide-angle image, the super-resolution image obtained by the image super-resolution processing method provided in this disclosure can maintain more image details and the image details are clearer. The quality of the super-resolution image obtained by the image super-resolution processing method provided in this disclosure is better than that of existing methods.

[0175] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0176] Based on the above-described image super-resolution processing method based on zero-shot learning, this disclosure also provides an image super-resolution processing apparatus based on zero-shot learning. The following will be combined with... Figure 7 The device is described in detail.

[0177] Figure 7 A schematic block diagram of an image super-resolution processing apparatus based on zero-shot learning according to an embodiment of the present disclosure is shown.

[0178] like Figure 7As shown, the image super-resolution processing apparatus 700 based on zero-shot learning in this embodiment includes an acquisition module 710, a first acquisition module 720, a second acquisition module 730, a third acquisition module 740, a fourth acquisition module 750, and a fifth acquisition module 760.

[0179] The acquisition module 710 is used to acquire a wide-angle image and a telephoto image, wherein the wide-angle image and the telephoto image represent images obtained by two acquisition devices with different focal lengths from the same scene, and the resolution of the wide-angle image is lower than that of the telephoto image. In one embodiment, the acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0180] The first obtaining module 720 is used to crop the wide-angle image to obtain a cropped image, wherein the field of view of the cropped image overlaps with that of the telephoto image. In one embodiment, the first obtaining module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0181] The second obtaining module 730 is used to align the cropped image to the telephoto image using the trained alignment network, thereby obtaining an aligned image. In one embodiment, the second obtaining module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0182] The third obtaining module 740 is used to crop the first image pair to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of an aligned image and a telephoto image. In one embodiment, the third obtaining module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0183] The fourth obtaining module 750 is used to train the super-resolution model using multiple second images to obtain the target super-resolution model. In one embodiment, the fourth obtaining module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0184] The fifth obtaining module 760 is used to process the wide-angle image using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image. In one embodiment, the fifth obtaining module 760 can be used to perform the operation S260 described above, which will not be repeated here.

[0185] According to embodiments of this disclosure, the second obtaining module includes a downsampled image obtaining submodule and an aligned image obtaining submodule.

[0186] The downsampled image acquisition submodule is used to downsample the telephoto image to obtain a downsampled image, wherein the resolution of the downsampled image is the same as the resolution of the wide-angle image.

[0187] The alignment image acquisition submodule is used to input the cropped image and the downsampled image into the alignment network to obtain the aligned image.

[0188] According to embodiments of this disclosure, an image super-resolution processing apparatus based on zero-shot learning includes an alignment network training module. The alignment network training module is used for:

[0189] The cropped image and the downsampled image are input into the alignment network to obtain the initial aligned image;

[0190] The initial aligned image is input into the spatial local discriminator to obtain the first discrimination result corresponding to the initial aligned image;

[0191] The spectrogram corresponding to the initial alignment image is input into the frequency domain global discriminator to obtain the second discrimination result corresponding to the initial alignment image;

[0192] The initial aligned image, cropped image, and downsampled image are input into a deep convolutional neural network to obtain intermediate feature maps corresponding to the initial aligned image, cropped image, and downsampled image, respectively.

[0193] Calculate the quadratic norm between the downsampled image and the initially aligned image;

[0194] Based on the first discrimination result, the second discrimination result, the intermediate feature map and the second norm, the alignment loss corresponding to the initial alignment network is obtained;

[0195] Update the network parameters of the initial alignment network based on the alignment loss.

[0196] According to embodiments of this disclosure, an image super-resolution processing apparatus based on zero-shot learning includes a spatial local discriminator training module. The spatial local discriminator training module is used for:

[0197] The cropped image is input into the spatial local discriminator to obtain the third discrimination result corresponding to the cropped image;

[0198] Based on the first and third discrimination results, the spatial loss corresponding to the spatial local discriminator is obtained;

[0199] The parameters of the spatial local discriminator are updated based on the spatial loss.

[0200] According to embodiments of this disclosure, an image super-resolution processing apparatus based on zero-shot learning includes a frequency domain global discriminator training module. The frequency domain global discriminator training module is used for:

[0201] Input the spectrogram corresponding to the cropped image into the frequency domain global discriminator to obtain the fourth discrimination result corresponding to the cropped image;

[0202] Based on the second and fourth discrimination results, the frequency domain loss corresponding to the frequency domain global discriminator is obtained;

[0203] Update the parameters of the frequency domain global discriminator based on the frequency domain loss.

[0204] According to embodiments of this disclosure, the alignment network training module includes a computation submodule.

[0205] The calculation submodule is used to calculate according to the following formula (I):

[0206]

[0207] Among them, L align The alignment loss is represented by L2, and the quadratic norm is represented by L2. The loss term is characterized by its association with the first discrimination result. L represents the loss term associated with the second discrimination result. cl The loss term is characterized by its association with the intermediate feature map, and λ1, λ2, and λ3 represent the weight coefficients.

[0208] According to embodiments of this disclosure, the computing submodule includes a computing unit.

[0209] The calculation unit is used to calculate according to the following formula (II):

[0210]

[0211] Among them, L cl The loss characterizes the loss associated with the intermediate feature maps. The image representing the output of the i-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ i (Y * The output of the i-th layer of a deep convolutional neural network and the cropped image Y * The corresponding intermediate feature map, The image representing the output of the j-th layer of a deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, φ j (X ↓ The output of the j-th layer of a deep convolutional neural network and the downsampled image X ↓ The corresponding intermediate feature map, n represents the nth layer of the deep convolutional neural network, m represents the mth layer of the deep convolutional neural network, and m and n are both integers greater than or equal to 1.

[0212] According to embodiments of this disclosure, the third obtaining module includes a probability map obtaining submodule and a second image pair obtaining submodule.

[0213] The probability map submodule is used to process the aligned image using a trained spatial local discriminator to obtain a similarity probability map. In the similarity probability map, the pixel value of each pixel represents the similarity between the aligned image and the telephoto image at that pixel.

[0214] The second image pair acquisition submodule is used to crop the first image pair according to the similarity probability map to obtain multiple second image pairs.

[0215] According to embodiments of this disclosure, any plurality of modules among the acquisition module 710, the first obtaining module 720, the second obtaining module 730, the third obtaining module 740, the fourth obtaining module 750, and the fifth obtaining module 760 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 710, the first obtaining module 720, the second obtaining module 730, the third obtaining module 740, the fourth obtaining module 750, and the fifth obtaining module 760 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 710, the first acquisition module 720, the second acquisition module 730, the third acquisition module 740, the fourth acquisition module 750, and the fifth acquisition module 760 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0216] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a zero-shot learning-based image super-resolution processing method according to embodiments of the present disclosure.

[0217] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0218] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0219] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0220] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image super-resolution processing method based on zero-shot learning according to embodiments of this disclosure.

[0221] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0222] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the image super-resolution processing method based on zero-shot learning provided in embodiments of this disclosure.

[0223] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0224] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0225] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0226] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0227] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0228] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0229] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image super-resolution processing method based on zero-shot learning, comprising: Acquire wide-angle and telephoto images, wherein the wide-angle image and the telephoto image represent images acquired by two acquisition devices with different focal lengths from the same scene, and the resolution of the wide-angle image is smaller than the resolution of the telephoto image; The wide-angle image is cropped to obtain a cropped image, wherein the cropped image overlaps with the field of view of the telephoto image; The cropped image is aligned to the telephoto image using a trained alignment network to obtain an aligned image. The first image pair is cropped to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of the aligned image and the telephoto image; The super-resolution model is trained using multiple pairs of the second images to obtain the target super-resolution model; The wide-angle image is processed using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image; The training method for the alignment network includes: The cropped image and the downsampled image are input into the alignment network to obtain the initial alignment image; The initial aligned image is input into the spatial local discriminator to obtain the first discrimination result corresponding to the initial aligned image; The spectrogram corresponding to the initial alignment image is input into the frequency domain global discriminator to obtain the second discrimination result corresponding to the initial alignment image; The initial aligned image, the cropped image, and the downsampled image are input into a deep convolutional neural network to obtain intermediate feature maps corresponding to the initial aligned image, the cropped image, and the downsampled image, respectively. Calculate the quadratic norm between the downsampled image and the initially aligned image; Based on the first discrimination result, the second discrimination result, the intermediate feature map, and the second norm, the alignment loss corresponding to the initial alignment network is obtained; The network parameters of the initial alignment network are updated based on the alignment loss.

2. The method according to claim 1, wherein, The step of aligning the cropped image to the telephoto image using a trained alignment network to obtain the aligned image includes: The telephoto image is downsampled to obtain a downsampled image, wherein the resolution of the downsampled image is the same as the resolution of the wide-angle image; The cropped image and the downsampled image are input into the alignment network to obtain the aligned image.

3. The method according to claim 1, wherein, The training method for the spatial local discriminator includes: The cropped image is input into a spatial local discriminator to obtain a third discrimination result corresponding to the cropped image; Based on the first discrimination result and the third discrimination result, the spatial loss corresponding to the spatial local discriminator is obtained; The parameters of the spatial local discriminator are updated based on the spatial loss.

4. The method according to claim 1, wherein, The training method for the frequency domain global discriminator includes: The spectrogram corresponding to the cropped image is input into the frequency domain global discriminator to obtain the fourth discrimination result corresponding to the cropped image; Based on the second discrimination result and the fourth discrimination result, the frequency domain loss corresponding to the frequency domain global discriminator is obtained; The parameters of the frequency domain global discriminator are updated based on the frequency domain loss.

5. The method according to claim 1, wherein, The alignment loss corresponding to the initial alignment network, obtained based on the first discrimination result, the second discrimination result, the intermediate feature map, and the quadratic norm, is calculated according to the following formula (I): (one) in, Characterizing the alignment loss, Characterizing the quadratic norm, Characterizes the loss term related to the first discrimination result. The loss term that characterizes the result of the second discrimination is described. The loss term characterizes the intermediate feature map. , and Characterizing weighting coefficients.

6. The method according to claim 5, wherein, The loss term associated with the intermediate feature map is calculated according to the following formula (II): (two) in, The loss characterizing the intermediate feature map The image representing the output of the i-th layer of the deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, The cropped image represents the output of the i-th layer of the deep convolutional neural network. The corresponding intermediate feature map, The image representing the output of the j-th layer of the deep convolutional neural network aligned with the initial image. The corresponding intermediate feature map, The output of the j-th layer of the deep convolutional neural network is compared with the downsampled image. The corresponding intermediate feature map, n represents the nth layer of the deep convolutional neural network, m represents the mth layer of the deep convolutional neural network, and m and n are both integers greater than or equal to 1.

7. The method according to claim 1, wherein, The cropping of the first image pair to obtain multiple second image pairs includes: The aligned image is processed using the trained spatial local discriminator to obtain a similarity probability map, wherein the pixel value of each pixel in the similarity probability map represents the similarity between the aligned image and the telephoto image at that pixel; The first image pair is cropped according to the similarity probability map to obtain the plurality of second image pairs.

8. An image super-resolution processing apparatus based on zero-shot learning, comprising: The acquisition module is used to acquire wide-angle images and telephoto images, wherein the wide-angle images and the telephoto images represent images acquired by two acquisition devices with different focal lengths from the same scene, and the resolution of the wide-angle images is smaller than the resolution of the telephoto images; The first obtaining module is used to crop the wide-angle image to obtain a cropped image, wherein the cropped image overlaps with the field of view of the telephoto image; The second obtaining module is used to align the cropped image to the telephoto image using a trained alignment network to obtain an aligned image. The third obtaining module is used to crop the first image pair to obtain multiple second image pairs, wherein the first image pair represents an image pair composed of the aligned image and the telephoto image; The fourth module is used to train the super-resolution model using multiple pairs of the second images to obtain the target super-resolution model. The fifth module is used to process the wide-angle image using the target super-resolution model to obtain a super-resolution image corresponding to the wide-angle image; The zero-shot learning-based image super-resolution processing device also includes an alignment network training module; the alignment network training module is used for: The cropped image and the downsampled image are input into the alignment network to obtain the initial aligned image; The initial aligned image is input into the spatial local discriminator to obtain the first discrimination result corresponding to the initial aligned image; The spectrogram corresponding to the initial alignment image is input into the frequency domain global discriminator to obtain the second discrimination result corresponding to the initial alignment image; The initial aligned image, cropped image, and downsampled image are input into a deep convolutional neural network to obtain intermediate feature maps corresponding to the initial aligned image, cropped image, and downsampled image, respectively. Calculate the quadratic norm between the downsampled image and the initially aligned image; Based on the first discrimination result, the second discrimination result, the intermediate feature map and the second norm, the alignment loss corresponding to the initial alignment network is obtained; Update the network parameters of the initial alignment network based on the alignment loss.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 7.

Images