Zero-shot learning super-resolution reconstruction method and system based on polarization image
By using a plane array polarization camera and a bicubital interpolation algorithm under polarization visual conditions, combining degradation characterization learning and Stokes parameters, the problems of polarization image acquisition difficulties and slow neural network operation speed are solved, and high-precision super-resolution is achieved Rate reconstruction and polarization information retention.
Patent Information
- Application Number
- CN202411274186.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Under polarization vision conditions, it is difficult for the prior art to obtain large amounts of polarized images, and large neural networks operate at a slower speed, making it difficult to meet industrial needs.
By acquiring the original polarization image of the plane array polarization camera, downsampling is performed using a bicubic interpolation algorithm, combining degradation characterization learning and Stokes parameters, the neural network is trained based on the loss function that retains polarization information to generate high-precision super-resolution rate image.
It significantly improves the super-resolution reconstruction accuracy and network operation speed, and can effectively retain the polarization information in the super-resolution image and generate high-precision polarization super-resolution reconstruction results.
Smart Images
Figure CN118799191B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision, and in particular to a zero-sample learning super-resolution reconstruction method and system based on polarization images. Background Art
[0002] With the rapid development of computer vision and image processing technology, super-resolution reconstruction has become a research field that has attracted much attention. In many application fields, such as virtual reality, augmented reality, industrial measurement, biomedical imaging, etc., it has become an important task to accurately and quickly obtain and reconstruct real-world information. Super-resolution reconstruction based on deep learning is a common super-resolution reconstruction method. Through a unique neural network structure and corresponding loss function, super-resolution images of corresponding images can be obtained through training. It has the advantages of high reconstruction accuracy and good imaging quality, and has been widely used in various scene environments.
[0003] Traditional super-resolution reconstruction algorithms mainly rely on intelligent algorithms to supplement the values of pixels. This type of method mainly includes: expanding pixels based on mathematical principles, using interpolation algorithms to fill in the pixels, and super-resolution reconstruction algorithms based on learning. The super-resolution reconstruction method based on deep learning uses a special neural network algorithm structure to learn the features of image information, so as to achieve super-resolution reconstruction and retain image information. The basic principle of super-resolution reconstruction based on deep learning is: prepare a large number of low-resolution and high-resolution image pairs as training data sets. These image pairs are used to train neural networks to learn how to convert low-resolution images into high-resolution images; design suitable deep neural network structures, such as convolutional neural networks (CNNs). Network design can include a variety of structures, such as linear networks, residual networks, recursive networks, and attention networks, to better capture the local structure and global information of the image; extract features from low-resolution images through the forward propagation process of the neural network. These features are designed to capture the texture, edges, and other key information of the image; use high-resolution images in the training data set to guide the network to learn the mapping from low resolution to high resolution. This process is usually achieved by minimizing the difference between the reconstructed image and the high-resolution image, which is usually measured by a loss function; the weights of the network are continuously adjusted through optimization techniques such as backpropagation algorithms and gradient descent until the output of the network is close to the high-resolution image; after training, the network performance is evaluated using a test set. The output of the network is a reconstructed high-resolution image, and the evaluation indicators can include image quality, resolution, visual effects, etc. Under the conditions of polarized vision, it is currently difficult to obtain the large number of polarized images required by the network. In addition, large neural networks run very slowly and are difficult to meet industrial needs. Summary of the invention
[0004] Based on the problems mentioned in the above background technology, the present disclosure provides a zero-sample learning super-resolution reconstruction method and system based on polarization images, which can solve the problems that the existing technology is difficult to obtain a large number of polarization images required by the network under the condition of polarization vision, and the operation speed of large neural networks is very slow and difficult to meet industrial needs. In order to solve the above technical problems, the present disclosure provides the following technical solutions:
[0005] As one aspect of an embodiment of the present disclosure, a zero-sample learning super-resolution reconstruction method based on polarization images is provided, comprising the following steps:
[0006] Acquire an original polarization image from a polarization camera; the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90°, and 135°, and is concentrated in one image without the need for segmentation;
[0007] The collected original polarization image is downsampled using the bicubic interpolation algorithm, and degradation representation learning is used to obtain low-resolution images with different degradation forms;
[0008] The degraded low-resolution image is paired with the original polarization image, and the neural network is trained based on the loss function that preserves the polarization information using Stokes parameters.
[0009] The original polarization image is used as input to the trained neural network to output a reconstructed super-resolution image.
[0010] Optionally, the collected original polarization image is downsampled using a bicubic interpolation algorithm, and degradation representation learning is used to obtain low-resolution images with different degradation forms, including:
[0011] Set the downsampling factor of the bicubic interpolation algorithm , where bicubic interpolation uses a 4×4 pixel neighborhood to calculate the value of the target pixel;
[0012] Using degradation representation learning, noise, compression and Gaussian blur are added to the downsampled images to obtain low-resolution images with different degradation forms.
[0013] Optionally, noise is added, compression is performed, and Gaussian blur is applied to obtain low-resolution images with different degradation forms, including:
[0014] Add noise n to the downsampled image, as shown in the following formula:
[0015] ,
[0016] Among them, r is the scale factor, k is the fuzzy degradation kernel, is the original polarization image, is the processed low-resolution image, n is the noise;
[0017] The compression of the JPEG format image with noise added is expressed as follows:
[0018] ,
[0019] in, is the compressed image;
[0020] Then the compressed JPEG image is anisotropically blurred, expressed as:
[0021] ,
[0022] in, is the blurred image at position The pixel value at is a compressed JPEG image at location The pixel value at represents an anisotropic Gaussian kernel, whose shape and blur level depend on the gradient information of the input image. is the standard deviation of the Gaussian kernel representing the blur degree;
[0023] in, ,
[0024] is a weight function related to gradient information, ,in, The gradient size of the local area of the image is normalized to the range of [0, 1] and then applied to the weight function to obtain the corresponding weight value.
[0025] Optionally, the Stokes parameter is expressed as:
[0026] ,
[0027] ,
[0028] ,
[0029] in, The corresponding polarization angle is Polarization image of , , as well as Represent 4 polarization state images respectively. The polarization state images are represented by digital images as brightness values of the images, where: is an image containing all polarization angles; is the polarization difference image with polarization angles of 0° and 90°; is the polarization difference image with polarization angles of 45° and 135°, is the polarization degree image, indicating the degree of polarization; It is the polarization angle image, indicating the direction of polarization.
[0030] Optionally, a loss function that preserves polarization information L It is expressed as follows:
[0031] ,
[0032] in, represents the loss function of the super-resolution part, which is obtained by matching the degraded image with the original polarization image and making a difference; The loss of polarization information contained in the polarization degree information image Dolp is represented. The loss of polarization information contained in the polarization angle information image Aolp is represented. The two are used together to finally restore the complete polarization information. and is the weight indicator, and adaptive adjustment is achieved through continuous iteration of the network; , as well as The corresponding expressions are:
[0033] ,
[0034] ,
[0035] ,
[0036] in, Represents the pixels at each point in the original image. In the polarization image, it represents The pixel value of each point in the full intensity image; Represents the pixel value of each point in the image after super-resolution reconstruction; represents the value calculated from the original polarization image The pixel value of each point in the image, represents the value calculated from the polarization image after super-resolution reconstruction Pixel value of each point in the image; Represents the polarization image calculated after super-resolution reconstruction Structural similarity index of images; Represents the polarization image calculated after super-resolution reconstruction The pixel value of each point in the image, Represents the original polarization image calculated The pixel value of each point in the image.
[0037] Optionally, the zero-shot training process is expressed as follows:
[0038] The degraded image obtained by degradation representation learning is used as the network input, and the original polarization image is used as the output. The neural network is initially iterated to obtain a network parameter that can restore image information and polarization information.
[0039] Use the preliminarily trained neural network, take the original polarization image as input, and output the polarization image that retains the polarization information.
[0040] Alternatively, bicubic interpolation is expressed as follows:
[0041] ,
[0042] f ( m,n ) represents the value of the target pixel, m and n are the position coordinates of the target pixel in the horizontal and vertical directions, i and j Respectively represent the position offset of the pixel points in the neighborhood, p ( i, j ) is the value of the corresponding pixel in the neighborhood, and are bicubic interpolation functions, calculated in the horizontal and vertical directions respectively, C is the normalization coefficient, , , , are parameters related to interpolation.
[0043] As another aspect of the embodiments of the present disclosure, a zero-sample learning super-resolution reconstruction system based on polarization images is provided, comprising:
[0044] An original polarization image acquisition unit is used to acquire an original polarization image of a polarization camera; the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90°, and 135°, and is concentrated in one image without the need for segmentation;
[0045] The degradation representation unit downsamples the collected original polarization image using a bicubic interpolation algorithm and uses degradation representation learning to obtain low-resolution images with different degradation forms;
[0046] Zero-shot learning unit, which pairs the degraded low-resolution image with the original polarization image and trains the neural network based on the loss function that preserves the polarization information using Stokes parameters;
[0047] An image reconstruction unit takes the original polarization image as input to the trained neural network to output a reconstructed super-resolution image.
[0048] As another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the zero-sample learning super-resolution reconstruction method based on polarization images when executing the computer program.
[0049] As another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the zero-sample learning super-resolution reconstruction method based on polarization images are implemented.
[0050] Compared with the prior art, the present disclosure can significantly improve the super-resolution reconstruction accuracy and network operation speed; the present disclosure also proposes the calculation of the polarization loss function, which can generate high-precision polarization super-resolution reconstruction results and effectively retain the polarization information in the super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 This is a flow chart of a degradation-aware zero-shot learning super-resolution reconstruction method based on polarization images in an embodiment of the present disclosure;
[0052] Figure 2 A schematic diagram of super-resolution image reconstruction in Embodiment 1 of the present disclosure;
[0053] Figure 3 This is a block diagram of the degradation-aware zero-sample learning super-resolution reconstruction system based on polarization images in Example 2 of the present disclosure. DETAILED DESCRIPTION
[0054] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0055] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0056] In this embodiment, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in this embodiment, the term "at least one" represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0057] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some embodiments, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0058] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.
[0059] Traditional super-resolution reconstruction algorithms mainly rely on intelligent algorithms to supplement the values of pixels. This type of method mainly includes: expanding pixels based on mathematical principles, using interpolation algorithms to fill in the pixels, and super-resolution reconstruction algorithms based on learning. The super-resolution reconstruction method based on deep learning is a type of structured light technology. It uses a special neural network algorithm structure to learn the features of image information, so as to achieve super-resolution reconstruction and retain image information. The basic principle of super-resolution reconstruction based on deep learning is: prepare a large number of low-resolution and high-resolution image pairs as training data sets. These image pairs are used to train neural networks to learn how to convert low-resolution images into high-resolution images; design suitable deep neural network structures, such as convolutional neural networks (CNNs). Network design can include a variety of structures, such as linear networks, residual networks, recursive networks, and attention networks, to better capture the local structure and global information of the image; extract features from low-resolution images through the forward propagation process of the neural network. These features are designed to capture the texture, edges, and other key information of the image; use high-resolution images in the training data set to guide the network to learn the mapping from low resolution to high resolution. This process is usually achieved by minimizing the difference between the reconstructed image and the high-resolution image, which is usually measured by a loss function; the weights of the network are continuously adjusted through optimization techniques such as backpropagation algorithms and gradient descent until the output of the network is close to the high-resolution image; after training, the network performance is evaluated using a test set. The output of the network is a reconstructed high-resolution image, and evaluation indicators can include image quality, resolution, visual effects, etc. Under the conditions of polarized vision, it is currently difficult to obtain the large amount of data required by the network - polarized images. In addition, large neural networks run very slowly and are difficult to meet industrial needs.
[0060] In order to generate a corresponding super-resolution image in one shot, the inventors used a polarization camera and an RTX4070-12G graphics card to construct a shooting and processing model. The following embodiments illustrate the technical solution to be implemented in the embodiments of the present disclosure.
[0061] Example 1
[0062] The disclosed embodiments further provide a zero-sample learning super-resolution reconstruction method based on polarization images, such as Figure 1 As shown, the following steps are included:
[0063] S10, obtaining an original polarization image of a polarization camera; obtaining the original polarization image by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging; the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90° and 135°, and the original polarization image that does not need to be segmented is concentrated in one image;
[0064] S20, down-sampling the collected original polarization image using a bicubic interpolation algorithm, and using degradation representation learning to obtain low-resolution images with different degradation forms;
[0065] S30, pairing the degraded low-resolution image with the original polarization image, and using Stokes parameters to train the neural network based on a loss function that preserves polarization information;
[0066] S40, using the original polarization image as input to the trained neural network to output a reconstructed super-resolution image.
[0067] Based on the above configuration, the disclosed embodiment can significantly improve the super-resolution reconstruction accuracy and network operation speed; can generate high-precision polarization super-resolution reconstruction results, and can effectively retain the polarization information in the super-resolution image.
[0068] The following takes a polarization super-resolution imaging system including two main components: a polarization camera and an operating system with a processor as an example to explain in detail each step of the embodiment of the present disclosure.
[0069] S10, obtaining an original polarization image of a polarization camera; the original polarization image is obtained by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging; the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90° and 135°, and the original polarization image that does not need to be segmented is concentrated in one image;
[0070] Among them, the polarization state image is mainly obtained by using a camera with a specific model of focal plane segmentation area array sensor (such as IMX250MYR model). The principle of the polarization sensor in the polarization camera is to add a layer of polarizer above the photodiode based on the traditional CMOS image sensor. Four polarizers with different angles (90°, 45°, 135° and 0°) are placed on a single pixel, and each group of four pixels is used as a calculation unit. The degree and direction of polarization can be calculated by the association between polarizers in different directions. By performing relevant operations on the image directly obtained by the polarization camera, the steps of segmentation, reorganization and recalculation are omitted, which reduces the complexity of system use and improves the running speed of the overall process.
[0071] S20, down-sampling the collected original polarization image using a bicubic interpolation algorithm, and using degradation representation learning to obtain low-resolution images with different degradation forms;
[0072] There are currently three main benchmark models for super-resolution reconstruction networks: convolutional neural network (CNN), generative adversarial network (GAN), and Transformer model.
[0073] The Transformer model improves the super-resolution reconstruction accuracy as much as possible by sacrificing the network's running speed; the generative adversarial network is mainly based on the convolutional neural network, and the functions of some modules are divided, which makes the model more complicated to a certain extent. Therefore, this embodiment uses the convolutional neural network as the benchmark model.
[0074] This embodiment is based on zero-shot learning, so that the model can recover a high-resolution image from a single low-resolution image without any high-resolution training samples of the category. The key idea of this method is to use a pre-trained deep neural network to extract common features of low-resolution images and use these features to generate corresponding high-resolution images.
[0075] In one embodiment, downsampling the collected original polarization image using a bicubic interpolation algorithm includes:
[0076] The original polarization image is downsampled using the bicubic interpolation algorithm, where the downsampling factor is set , where bicubic interpolation uses a 4×4 pixel neighborhood to calculate the value of the target pixel. Where bicubic interpolation is expressed as follows:
[0077] ,
[0078] f ( m,n ) represents the value of the target pixel, m and n are the position coordinates of the target pixel in the horizontal and vertical directions, i and j Respectively represent the position offset of the pixel points in the neighborhood, p ( i, j ) is the value of the corresponding pixel in the neighborhood, and are bicubic interpolation functions, calculated in the horizontal and vertical directions respectively, C is the normalization coefficient, , , , is a parameter related to interpolation and a constant that can be adjusted according to specific circumstances.
[0079] In an optional embodiment, degradation representation learning is used to obtain low-resolution images with different degradation forms, including:
[0080] Noise, compression and Gaussian blur are added to the downsampled images to obtain low-resolution images with different degradation forms.
[0081] In an optional embodiment, adding noise, compression and Gaussian blur to obtain low-resolution images with different degradation forms includes:
[0082] Add noise n to the downsampled image, as shown in the following formula:
[0083] ,
[0084] Among them, r is the scale factor, k is the fuzzy degradation kernel, is the original polarization image, is the processed low-resolution image, n is the noise;
[0085] The compression of the JPEG format image with noise added is expressed as follows:
[0086] ,
[0087] in, is the compressed image;
[0088] Then the compressed JPEG image is anisotropically blurred, expressed as:
[0089] ,
[0090] in, is the blurred image at position The pixel value at is a compressed JPEG image at location The pixel value at represents an anisotropic Gaussian kernel, whose shape and blur level depend on the gradient information of the input image. The standard deviation of the Gaussian kernel representing the degree of blur is used to control the amplitude and diffusion of noise. The value of can control the characteristics of Gaussian noise.
[0091] in, ,
[0092] is a weight function related to gradient information, ,in, The gradient size of the local area of the image is normalized to the range of [0, 1] and then applied to the weight function to obtain the corresponding weight value. By adjusting the value of the weight value, the degree of edge preservation of the blur effect can be controlled. Using various types of degradation forms, a large number of random degradation combinations are formed, and the corresponding low-resolution images are generated to match the original polarization images, creating a large number of training image pairs.
[0093] S30, pairing the degraded low-resolution image with the original polarization image, using Stokes parameters, and training the neural network based on a loss function that retains polarization information.
[0094] As an optional embodiment, the Stokes parameter is expressed as:
[0095] ,
[0096] ,
[0097] ,
[0098] in, The corresponding polarization angle is Polarization image of , , as well as Represent 4 polarization state images respectively. The polarization state images are represented by digital images as brightness values of the images, where: is an image containing all polarization angles; is the polarization difference image with polarization angles of 0° and 90°; is the polarization difference image with polarization angles of 45° and 135°, is the polarization degree image, indicating the degree of polarization; It is the polarization angle image, indicating the direction of polarization.
[0099] Optionally, a loss function that preserves polarization information L It is expressed as follows:
[0100] ,
[0101] in, represents the loss function of the super-resolution part, which is obtained by matching the degraded image with the original polarization image and making a difference; The loss of polarization information contained in the polarization degree information image Dolp is represented. The loss of polarization information contained in the polarization angle information image Aolp is represented. The two are used together to finally restore the complete polarization information. and is the weight indicator, and adaptive adjustment is achieved through continuous iteration of the network; , as well as The corresponding expressions are:
[0102] ,
[0103] ,
[0104] ,
[0105] in, Represents the pixels of each point in the original image. In the polarization image, it mainly represents The pixel value of each point in the full intensity image; Represents the pixel value of each point in the image after super-resolution reconstruction; represents the value calculated from the original polarization image The pixel value of each point in the image, represents the value calculated from the polarization image after super-resolution reconstruction Pixel value of each point in the image; Represents the polarization image calculated after super-resolution reconstruction Structural similarity index of images; Represents the polarization image calculated after super-resolution reconstruction The pixel value of each point in the image, Represents the original polarization image calculated The pixel value of each point in the image. Through the constraints of these three losses, it is possible to restore and retain the polarization information while achieving high-precision super-resolution reconstruction.
[0106] S40, using the original polarization image as input to the trained neural network to output a reconstructed super-resolution image.
[0107] Among them, Figure 2 As shown in the figure, the original polarization image is downsampled and degraded to generate a low-resolution-high-resolution image pair to learn various information inside the image. At this time, the original polarization image is a high-resolution image, and the image generated by downsampling is a low-resolution image, which serve as the input (low-resolution image generated by downsampling) and output (high-resolution image as the original polarization image) of the neural network respectively. After multiple iterations, the neural network will obtain a network parameter for generating a high-resolution image from a low-resolution image for further super-resolution image reconstruction operations. In the final super-resolution image reconstruction step, the original polarization image is used as the network input to implement the super-resolution reconstruction operation to obtain a super-resolution image with a higher resolution than the original image.
[0108] This embodiment uses zero-shot learning as a basis to enable the model to recover a high-resolution image from a single low-resolution image without requiring any high-resolution training samples of the category. The key idea of this method is to use a pre-trained deep neural network to extract common features of low-resolution images and use these features to generate the corresponding high-resolution images. This process usually involves the following steps:
[0109] Step 1: Feature extraction, using a pre-trained network to extract deep features of low-resolution images.
[0110] Step 2: Feature mapping, mapping these features to high-resolution space through a mapping function, which is obtained by training and learning a large number of low-resolution and high-resolution image pairs.
[0111] Step 3: Image reconstruction, using the mapped features to reconstruct a high-resolution image, usually achieved through an upsampling layer (such as interpolation or deconvolution).
[0112] Step 4: Iterative optimization. During the reconstruction process, multiple iterations may be required to optimize the reconstructed image to make it closer to the real high-resolution image.
[0113] In the overall step, in order to make the training process as diverse as possible, degenerate representation learning is used to solve the problem of model overfitting caused by the single data form and too scarce data in the learning process.
[0114] Example 2
[0115] The disclosed embodiment also provides a zero-sample learning super-resolution reconstruction system 100 based on polarization images, such as Figure 3 As shown, including:
[0116] The original polarization image acquisition unit 1 acquires the original polarization image of the polarization camera; the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90° and 135°, and is concentrated in one image without the need for segmentation;
[0117] The degradation representation unit 2 downsamples the collected original polarization image using a bicubic interpolation algorithm and uses degradation representation learning to obtain low-resolution images with different degradation forms;
[0118] Zero-shot learning unit 3, pairs the degraded low-resolution image with the original polarization image, uses Stokes parameters, and trains the neural network based on a loss function that preserves polarization information;
[0119] The image reconstruction unit 4 uses the original polarization image as an input of the trained neural network to output a reconstructed super-resolution image.
[0120] Based on the above configuration, the disclosed embodiment can significantly improve the super-resolution reconstruction accuracy and network operation speed; can generate high-precision polarization super-resolution reconstruction results, and can effectively retain the polarization information in the super-resolution image.
[0121] In addition, the above-mentioned zero-sample learning super-resolution reconstruction system based on polarization images can be used to implement any method in the above-mentioned embodiment 1.
[0122] Example 3
[0123] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the zero-sample learning super-resolution reconstruction method based on polarization images in Example 1 when executing the computer program.
[0124] The electronic device in the embodiment of the present disclosure is merely an example and can be used to implement the functions of a host computer or a light curtain controller, and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.
[0125] The electronic device may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including the memory and the processor).
[0126] The bus includes data bus, address bus and control bus.
[0127] The memory may include volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).
[0128] The memory may also include a program tool having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.
[0129] The processor executes various functional applications and data processing by running computer programs stored in the memory.
[0130] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, etc.). Such communication may be performed through an input / output (I / O) interface. In addition, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
[0131] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.
[0132] Example 4
[0133] An embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, the steps of the zero-sample learning super-resolution reconstruction method based on polarization images in the above embodiment are implemented.
[0134] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.
[0135] In a possible implementation, the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the zero-sample learning super-resolution reconstruction method based on polarization images described in Example 1.
[0136] Among them, the program code for executing the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.
[0137] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A zero-shot learning super-resolution reconstruction method based on polarization images, characterized in that: The steps include: Acquire an original polarization image from a polarization camera; the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90°, and 135°, and is concentrated in one image without the need for segmentation; The collected original polarization image is downsampled using the bicubic interpolation algorithm, and degradation representation learning is used to obtain low-resolution images with different degradation forms; The degraded low-resolution image is paired with the original polarization image, and the neural network is trained based on the loss function that preserves the polarization information using Stokes parameters. The original polarization image is used as input to the trained neural network to output a reconstructed super-resolution image; The collected original polarization image is downsampled using the bicubic interpolation algorithm, and degradation representation learning is used to obtain low-resolution images with different degradation forms, including: Set the downsampling factor of the bicubic interpolation algorithm , where bicubic interpolation uses a 4×4 pixel neighborhood to calculate the value of the target pixel; Using degradation representation learning, noise, compression and Gaussian blur are added to the downsampled images to obtain low-resolution images with different degradation forms. The Stokes parameter is expressed as: ， ， ， in, The corresponding polarization angle is Polarization image of , , as well as Represent 4 polarization state images respectively. The polarization state images are represented by digital images as brightness values of the images, where: is an image containing all polarization angles; is the polarization difference image with polarization angles of 0° and 90°; is the polarization difference image with polarization angles of 45° and 135°, is the polarization degree image, indicating the degree of polarization; is the polarization angle image, indicating the direction of polarization; Loss function that preserves polarization information L It is expressed as follows: ， in, represents the loss function of the super-resolution part, which is obtained by matching the degraded image with the original polarization image and making a difference; The loss of polarization information contained in the polarization degree information image Dolp is represented. The loss of polarization information contained in the polarization angle information image Aolp is represented. The two are used together to finally restore the complete polarization information. and is the weight indicator, and adaptive adjustment is achieved through continuous iteration of the network; , as well as The corresponding expressions are: ， ， ， in, Represents the pixels at each point in the original image. In the polarization image, it represents The pixel value of each point in the full intensity image; Represents the pixel value of each point in the image after super-resolution reconstruction; represents the value calculated from the original polarization image The pixel value of each point in the image, represents the value calculated from the polarization image after super-resolution reconstruction Pixel value of each point in the image; Represents the polarization image calculated after super-resolution reconstruction Structural similarity index of images; Represents the polarization image calculated after super-resolution reconstruction The pixel value of each point in the image, Represents the original polarization image calculated Pixel value of each point in the image; The zero-shot training process is expressed as follows: The degraded image obtained by degradation representation learning is used as the network input, and the original polarization image is used as the output. The neural network is initially iterated to obtain a network parameter that can restore image information and polarization information. Use the preliminarily trained neural network, take the original polarization image as input, and output the polarization image that retains the polarization information.
2. The zero-shot learning super-resolution reconstruction method based on polarization images according to claim 1, characterized in that: Add noise, compress, and apply Gaussian blur to obtain low-resolution images with different forms of degradation, including: Add noise n to the downsampled image, as shown in the following formula: ， Among them, r is the scale factor, k is the fuzzy degradation kernel, is the original polarization image, is the processed low-resolution image, n is the noise; The compression of the JPEG format image with noise added is expressed as follows: ， in, is the compressed image; Then the compressed JPEG image is anisotropically blurred, expressed as: ， in, is the blurred image at position The pixel value at is a compressed JPEG image at location The pixel value at represents an anisotropic Gaussian kernel, whose shape and blur level depend on the gradient information of the input image. is the standard deviation of the Gaussian kernel representing the degree of blur; in, , is a weight function related to gradient information, ,in, The gradient size of the local area of the image is normalized to the range of [0, 1] and then applied to the weight function to obtain the corresponding weight value.
3. The zero-shot learning super-resolution reconstruction method based on polarization image according to claim 2, characterized in that: Bicubic interpolation is expressed as follows: ， f ( m,n ) represents the value of the target pixel, m and n are the position coordinates of the target pixel in the horizontal and vertical directions, i and j Respectively represent the position offset of the pixel points in the neighborhood, p ( i, j ) is the value of the corresponding pixel in the neighborhood, and are bicubic interpolation functions, calculated in the horizontal and vertical directions respectively, C is the normalization coefficient, , , , are parameters related to interpolation.
4. Zero-shot learning super-resolution reconstruction system based on polarization image, characterized in that: include: An original polarization image acquisition unit is used to acquire an original polarization image of a polarization camera; the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90°, and 135°, and is concentrated in one image without the need for segmentation; The degradation representation unit downsamples the collected original polarization image using a bicubic interpolation algorithm and uses degradation representation learning to obtain low-resolution images with different degradation forms; Zero-shot learning unit, which pairs the degraded low-resolution image with the original polarization image and trains the neural network based on the loss function that preserves the polarization information using Stokes parameters; an image reconstruction unit, taking the original polarization image as input to the trained neural network to output a reconstructed super-resolution image; The collected original polarization image is downsampled using the bicubic interpolation algorithm, and degradation representation learning is used to obtain low-resolution images with different degradation forms, including: Set the downsampling factor of the bicubic interpolation algorithm , where bicubic interpolation uses a 4×4 pixel neighborhood to calculate the value of the target pixel; Using degradation representation learning, noise, compression and Gaussian blur are added to the downsampled images to obtain low-resolution images with different degradation forms. The Stokes parameter is expressed as: ， ， ， in, The corresponding polarization angle is Polarization image of , , as well as Represent 4 polarization state images respectively. The polarization state images are represented by digital images as brightness values of the images, where: is an image containing all polarization angles; is the polarization difference image with polarization angles of 0° and 90°; is the polarization difference image with polarization angles of 45° and 135°, is the polarization degree image, indicating the degree of polarization; is the polarization angle image, indicating the direction of polarization; Loss function that preserves polarization information L It is expressed as follows: ， in, represents the loss function of the super-resolution part, which is obtained by matching the degraded image with the original polarization image and making a difference; The loss of polarization information contained in the polarization degree information image Dolp is represented. The loss of polarization information contained in the polarization angle information image Aolp is represented. The two are used together to finally restore the complete polarization information. and is the weight indicator, and adaptive adjustment is achieved through continuous iteration of the network; , as well as The corresponding expressions are: ， ， ， in, Represents the pixels at each point in the original image. In the polarization image, it represents The pixel value of each point in the full intensity image; Represents the pixel value of each point in the image after super-resolution reconstruction; represents the value calculated from the original polarization image The pixel value of each point in the image, represents the value calculated from the polarization image after super-resolution reconstruction Pixel value of each point in the image; Represents the polarization image calculated after super-resolution reconstruction Structural similarity index of images; Represents the polarization image calculated after super-resolution reconstruction The pixel value of each point in the image, Represents the original polarization image calculated Pixel value of each point in the image; The zero-shot training process is expressed as follows: The degraded image obtained by degradation representation learning is used as the network input, and the original polarization image is used as the output. The neural network is initially iterated to obtain a network parameter that can restore image information and polarization information. Use the preliminarily trained neural network, take the original polarization image as input, and output the polarization image that retains the polarization information.
5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the zero-sample learning super-resolution reconstruction method based on polarization images according to any one of claims 1 to 3 is implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the zero-sample learning super-resolution reconstruction method based on polarization images according to any one of claims 1 to 3 are implemented.