Image super-division method and device based on implicit expression flow model, equipment and storage medium

Abstract

The invention discloses an implicit expression flow model-based image super-division method, device and equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a feature vector, a prior noise and a Fourier feature vector, and extracting the feature vector, the prior noise and the Fourier feature vector through a local implicit module; determining a condition parameter set based on the feature vector and the Fourier feature vector; and determining a high-resolution image based on the condition parameter set and the prior noise to complete the super-division of each proportion of the image. The local implicit module is used for processing the image to obtain the feature vector, the prior noise and the Fourier feature vector, the feature extraction quality is more effectively improved, the calculation efficiency is higher, the high-resolution image is determined, multi-proportion super-division of the image is realized by learning and adapting to super-division coefficients under multiple proportions, and the image multi-proportion super-division efficiency is improved. The flexibility and the application range of the super-resolution image are improved, and the training and calculation cost is remarkably reduced while the image quality is maintained or improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image super-resolution method, device, equipment and storage medium based on an implicit expression flow model. Background Art

[0002] With the rapid development of digital image processing technology, the requirements for image quality are getting higher and higher. In the field of image processing, the purpose of image super-resolution is to improve the resolution of the image through algorithms, so that low-resolution images can be converted into high-resolution images while maintaining or enhancing the visual quality of the image. However, in many application scenarios, due to hardware limitations or data compression during transmission, the obtained image resolution is low and the necessary details cannot be obtained from it. Therefore, it is necessary to restore high-resolution images from low-resolution images.

[0003] At present, existing practices mainly rely on deep learning models, including autoencoder architecture, residual network structure and reversible flow model based on wavelet transform. By building a defect image database and using encoder and decoder structure to map and reconstruct images, defect areas can be detected, or high-resolution images can be reconstructed by extracting interference features and fusing image features, or multi-scale signal representation and variational dequantization technology can be used to improve training effects and generalization capabilities.

[0004] However, the existing practice of using the autoencoder architecture network in image reconstruction tasks has too high training and computational costs, the training process is unstable, and the quality of the generated samples is low. In addition, the convolutional network using the residual network structure cannot extract the high-frequency information of the image features well. The conditional input of the flow model obtained by sampling with low-pass filtering will not only lose the high-frequency features of the image, but also lose the relative position relationship between pixels. At the same time, a single model cannot complete multi-scale arbitrary scale super-resolution, and can only super-resolution images at a fixed scale, which lacks flexibility. Therefore, how to accurately and effectively perform personalized image super-resolution has become an urgent problem to be solved.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide an image super-resolution method, device, equipment and storage medium based on an implicit expression flow model, aiming to solve the technical problem of how to accurately and effectively perform personalized image super-resolution.

[0007] To achieve the above objectives, the present application proposes an image super-resolution method based on an implicit expression flow model, the method comprising:

[0008] Obtaining a feature vector, a priori noise, and a Fourier feature vector, wherein the feature vector, the priori noise, and the Fourier feature vector are extracted by a local implicit module;

[0009] determining a set of conditional parameters based on the eigenvector and the Fourier eigenvector;

[0010] The high-resolution image is determined based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image.

[0011] In one embodiment, the step of obtaining the feature vector, the prior noise and the Fourier feature vector comprises:

[0012] Obtain low-resolution image information, original image information and image segmentation method;

[0013] Extracting a feature vector from the low-resolution image information through a local implicit module;

[0014] Encoding and decoding the feature vector by a decoder to generate prior noise;

[0015] A Fourier eigenvector is determined based on the eigenvector, the original image information and the image segmentation method.

[0016] In one embodiment, the step of determining the Fourier eigenvector based on the eigenvector, the original image information and the image segmentation method comprises:

[0017] Get the image scaling ratio;

[0018] Determine each segmented image information set based on the image segmentation method and the original image information;

[0019] Extracting the center position of each segmented image information set to generate a center coordinate vector matrix;

[0020] Determine frequency domain vector information based on the eigenvector and the center coordinate vector matrix;

[0021] determining phase domain vector information based on the image scaling;

[0022] A Fourier eigenvector is generated based on the frequency domain vector information and the phase domain vector information.

[0023] In one embodiment, the step of determining a conditional parameter set based on the eigenvector and the Fourier eigenvector comprises:

[0024] Get Fourier characteristic function;

[0025] Based on the Fourier characteristic function, the characteristic vector and the Fourier characteristic vector are input into a target sensor to obtain a conditional parameter set.

[0026] In one embodiment, the step of determining the high-resolution image based on the conditional parameter set and the priori noise to complete super-resolution of each proportion of the image comprises:

[0027] Get image textures and high-resolution textures;

[0028] The image texture and the high-resolution texture are feature added to obtain a high-resolution image to complete super-resolution of each proportion of the image.

[0029] In one embodiment, the step of acquiring the image texture and the high-resolution texture comprises:

[0030] Obtain low-resolution image information and target flow model;

[0031] Upsampling the low-resolution image to obtain an image texture;

[0032] The conditional parameter set and the prior noise are input into the target flow model to determine a high-resolution texture.

[0033] In one embodiment, the step of inputting the conditional parameter set and the prior noise into the target flow model to determine the high-resolution texture comprises:

[0034] Get the flow model mapping relationship and loop times;

[0035] Calculate each high-resolution texture set based on the condition parameter set, the prior noise, the flow model mapping relationship and the number of loops;

[0036] The high-resolution texture sets are connected in series to obtain a high-resolution texture.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes an image super-resolution device based on an implicit expression flow model, and the image super-resolution device based on the implicit expression flow model comprises:

[0038] An acquisition module, used for acquiring a feature vector, a priori noise and a Fourier feature vector, wherein the feature vector, a priori noise and the Fourier feature vector are extracted through a local implicit module;

[0039] A processing module, configured to determine a conditional parameter set based on the eigenvector and the Fourier eigenvector;

[0040] An execution module is used to determine the high-resolution image based on the condition parameter set and the prior noise to complete the super-resolution of each proportion of the image.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes an image super-resolution device based on an implicit expression flow model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image super-resolution method based on the implicit expression flow model as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image super-resolution method based on the implicit expression flow model as described above are implemented.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] The present application proposes an image super-resolution method based on an implicit expression flow model, which obtains feature vectors, prior noise and Fourier feature vectors, which are extracted through local implicit modules; a conditional parameter set is determined based on the feature vectors and the Fourier feature vectors; a high-resolution image is determined based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image. The present application processes images through local implicit modules to obtain feature vectors, prior noise and Fourier feature vectors, which more effectively improves the quality of feature extraction and improves resolution, and has higher computational efficiency, thereby determining high-resolution images, learning and adapting to super-resolution coefficients at multiple scales to achieve multi-scale super-resolution of images, improving the flexibility and application scope of super-resolution images, and significantly reducing training and computational costs while maintaining or improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart diagram of the first embodiment of the image super-resolution method based on the implicit expression flow model of the present application is provided;

[0048] Figure 2 A flowchart diagram of the second embodiment of the image super-resolution method based on the implicit expression flow model of the present application is provided;

[0049] Figure 3 A brief flowchart of an image super-resolution method based on an implicit expression flow model provided in an embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of the module structure of an image super-resolution device based on an implicit expression flow model according to an embodiment of the present application;

[0051] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the image super-resolution method based on the implicit expression flow model in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtaining feature vectors, prior noise and Fourier feature vectors, wherein the feature vectors, prior noise and Fourier feature vectors are extracted through a local implicit module; determining a conditional parameter set based on the feature vectors and the Fourier feature vectors; and determining a high-resolution image based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image.

[0056] In this embodiment, for the convenience of description, the following description is made by taking the image super-resolution device based on the implicit expression flow model as the execution subject.

[0057] Because the existing technology uses the network of autoencoder architecture in image reconstruction tasks, the training and computational costs are too high, the training process is unstable, the quality of generated samples is low, and the convolutional network using the residual network structure cannot extract the high-frequency information of image features well. The conditional input of the flow model obtained by sampling with low-pass filtering will not only lose the high-frequency features of the image, but also lose the relative position relationship between pixels. At the same time, a single model cannot complete multi-scale arbitrary ratio super-resolution, and can only perform super-resolution on images at a fixed ratio, which lacks flexibility.

[0058] The present application provides a solution to obtain a feature vector, a priori noise and a Fourier feature vector, wherein the feature vector, the priori noise and the Fourier feature vector are extracted through a local implicit module; a conditional parameter set is determined based on the feature vector and the Fourier feature vector; and a high-resolution image is determined based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image.

[0059] It can be seen from the above embodiments that the present application processes images through a local implicit module to obtain feature vectors, prior noise and Fourier feature vectors, which more effectively improves the quality of feature extraction and improves the resolution, and has higher computational efficiency, thereby determining high-resolution images, learning and adapting to super-resolution coefficients at multiple scales to achieve multi-scale super-resolution of images, improving the flexibility and application scope of super-resolution images, and significantly reducing training and computing costs while maintaining or improving image quality.

[0060] Based on this, the embodiment of the present application provides an image super-resolution method based on an implicit expression flow model. Figure 1 , Figure 1 This is a flowchart of the first embodiment of the image super-resolution method based on the implicit expression flow model of the present application.

[0061] In this embodiment, the image super-resolution method based on the implicit expression flow model includes steps S10 to S30:

[0062] Step S10, obtaining a feature vector, a priori noise and a Fourier feature vector, wherein the feature vector, a priori noise and the Fourier feature vector are extracted through a local implicit module;

[0063] It should be noted that the feature vector reflects the characteristics of the extracted image visual information, the prior noise reflects the characteristics of the error information generated during the encoding and decoding process of the image, and the Fourier feature vector reflects the characteristics of the frequency domain and phase domain of the image.

[0064] It can be understood that the local implicit module can more conveniently and effectively process low-resolution images, thereby extracting the feature vector, including the image contour, texture and color distribution, for image super-resolution reconstruction, thereby restoring the high-resolution image. By encoding and decoding the feature vector through the decoder to generate the prior noise, the image information loss can be simulated, thereby compensating for the image quality degradation caused by information loss in the super-resolution process, making the reconstructed high-resolution image closer to the original image. At the same time, the Fourier feature vector represents the frequency distribution information of the image, thereby capturing the periodic and structural features in the image, enabling the flow model to better understand the global structure of the image, thereby more accurately reconstructing the details in the super-resolution process.

[0065] In addition, it should be noted that by processing the image through a local implicit module, the feature vector of the image can be extracted more effectively, and the prior noise is used to make the flow model learn information loss, while the Fourier feature vector enables the flow model to efficiently process the global and local features of the image. The combined use of prior noise and Fourier feature vectors can enable the flow model to significantly reduce the training and computing costs while maintaining image quality, while maintaining or improving image quality.

[0066] For ease of understanding, the acquisition of eigenvectors, prior noise and Fourier eigenvectors is taken as an example for explanation, wherein the information acquisition device is an information acquisition module, and the storage device is a memory.

[0067] The information acquisition module obtains low-resolution image information, original image information, image segmentation method and image scaling ratio, and extracts feature vectors from the low-resolution image information through the local implicit module, that is, the downsampled low-resolution image is passed through the encoder and convolution module to obtain the feature vector v * .

[0068] The feature vector is input into the local Fourier feature extraction layer to provide an amplitude coefficient for the Fourier feature, and then the feature vector is encoded and decoded by a decoder to generate a priori noise z.

[0069] The Fourier eigenvector is determined based on the eigenvector, the original image information and the image segmentation method, and each segmented image information set is determined based on the image segmentation method and the original image information. For example, if the image segmentation method is to segment the image in a 48*48 manner, a 48*48 segmented image can be obtained after segmenting the original image in 48*48, i.e., each segmented image information set, so as to extract the center position of each segmented image information set to generate a central coordinate vector matrix, i.e., 48*48 coordinates are generated at this time.

[0070] The frequency domain vector information is determined based on the eigenvector and the center coordinate vector matrix, that is, the center coordinate vector matrix is ​​combined with the eigenvector v * After feature multiplication, the frequency domain vector information v can be obtained f And the frequency domain vector information is input into the local Fourier feature extraction layer to provide frequency domain information for the Fourier feature.

[0071] The phase domain vector information c is determined based on the image scaling ratio. At this time, the phase domain vector information c is:

[0072]

[0073] Wherein, s is the scaling ratio from the original image to the low-resolution image, that is, the image scaling ratio.

[0074] The phase domain vector information passes through a linear layer and is input into the local Fourier feature extraction layer, which provides the Fourier feature with phase domain information. By learning any scaling ratio, the flow model can be fitted to reason about any scale super-resolution task.

[0075] A Fourier eigenvector is generated based on the frequency domain vector information and the phase domain vector information.

[0076] The feature vector, the prior noise and the Fourier feature vector are stored in a memory, and subsequent processing is performed based on the feature vector, the prior noise and the Fourier feature vector.

[0077] In a feasible implementation, step S10 may include steps A11 to A14:

[0078] Step A11, obtaining low-resolution image information, original image information and image segmentation method;

[0079] It should be noted that the low-resolution image information reflects the characteristics of the visual effects of the image after compression or downsampling, the original image information reflects the characteristics of the complete details of the image, and the image segmentation method reflects the characteristics of regional segmentation of the image.

[0080] It is understandable that the details in the low-resolution image information will become blurred or lost due to the reduction in resolution. Therefore, the image needs to be enhanced and the details restored. The original image information provides the original visual content and image structure in the image. At the same time, the image segmentation method can divide the image into multiple regions or objects, thereby identifying and locating specific features in the image.

[0081] In addition, it should be noted that by combining the low-resolution image information and the original image information, the image features can be accurately processed, thereby more accurately restoring the details and texture of the image and improving the quality of the super-resolution image. Through image segmentation and feature extraction, different areas of the image can be processed in a targeted manner, which not only improves the computational efficiency, but also allows the super-resolution strategies and parameters to be flexibly adjusted according to the characteristics of different areas, thereby improving computational efficiency and flexibility.

[0082] Step A12, extracting a feature vector from the low-resolution image information through a local implicit module;

[0083] It can be understood that the local implicit module can be used to analyze the local area of ​​the image, identify and extract key local features such as image edges, textures and shapes.

[0084] In addition, it should be noted that by processing images through local implicit modules, local features of the image can be extracted more accurately, the efficiency of feature extraction can be improved, and different image contents and conditions can be adapted to achieve more flexible and adaptable super-resolution processing, which reduces the amount of calculation and improves the quality of the reconstructed image.

[0085] Step A13, encoding and decoding the feature vector by a decoder to generate prior noise;

[0086] It can be understood that the local implicit module provides the input data required for the encoding and decoding process, namely the feature vector, and the encoding and decoding process processes these data to simulate the loss and recovery of image information. The decoder encodes and decodes the feature vector to generate prior noise, which characterizes the loss and disturbance that the image information may encounter during the processing. The prior noise is used to simulate and compensate for the quality degradation of the image due to various factors in practical applications, such as image compression and image transmission.

[0087] Step A14, determining a Fourier eigenvector based on the eigenvector, the original image information and the image segmentation method.

[0088] It can be understood that by combining the feature vector, the original image information, the image segmentation method and the Fourier feature vector, the flow model can learn to restore the details and structure of the image more comprehensively, thereby significantly improving the quality of the super-resolution image.

[0089] In a feasible implementation, step A14 may include steps B11 to B16:

[0090] Step B11, obtaining the image scaling ratio;

[0091] It should be noted that the image scaling ratio reflects the characteristics of a desired super-resolution image of any scale.

[0092] It can be understood that the image scaling ratio represents a personalized scaling of the image, including equal-proportional scaling and unequal-proportional scaling, which adjusts the image ratio more personalized and affects the details and quality of the image.

[0093] In addition, it should be noted that by precisely controlling the image scaling ratio, the details of the image can be restored more accurately, the quality of the super-resolution image can be improved, and it can adapt to different application scenarios. Whether the image is enlarged or reduced, the visual quality of the image can be maintained. At the same time, unnecessary calculations can be reduced and the efficiency of the algorithm can be improved.

[0094] Step B12, determining each segmented image information set based on the image segmentation method and the original image information;

[0095] It should be noted that each segmented image information set reflects the characteristics of different regions or objects after image segmentation.

[0096] It can be understood that each segmented image information set represents the application scope of the segmentation result, more accurately extracts the features of the segmented image, and adapts to different application scenarios. Reasonable image segmentation can reduce unnecessary calculations, especially when processing large-scale image data. By targetedly processing different areas of the image, the computing efficiency can be significantly improved.

[0097] Step B13, extracting the center position of each segmented image information set to generate a center coordinate vector matrix;

[0098] It should be noted that the center coordinate vector matrix reflects the characteristics of the center position information of each region after image segmentation.

[0099] It can be understood that the central coordinate vector matrix provides a clear geometric center point for each segmented area, that is, it provides spatial positioning information of the image segmented area, characterizes the relative position relationship between the segmented areas, and thus can infer the relationship between objects in the image, such as proximity, direction, arrangement and distribution.

[0100] In addition, it should be noted that the central coordinate vector matrix provides stable spatial positioning information, which can maintain the stability of the analysis results even when the image content changes or the image quality decreases. By using the central coordinate vector matrix, the center point of the image segmentation area can be more accurately located, thereby improving the accuracy of the overall image processing and reducing unnecessary calculations. Especially when processing large-scale image data, the computing efficiency can be significantly improved by targeted processing of different areas of the image.

[0101] Step B14, determining frequency domain vector information based on the eigenvector and the center coordinate vector matrix;

[0102] It should be noted that the frequency domain vector information reflects the characteristics of the image in the frequency domain.

[0103] It can be understood that the frequency domain vector information includes the frequency distribution, amplitude and phase information of the image, which is used to capture periodic features in the image, such as repetitive textures and patterns, and describes the structure of the image, including the distribution and intensity of different frequency components in the image.

[0104] In addition, it should be noted that the use of the frequency domain vector information can make it more stable in the face of noise and quality degradation, thereby more accurately analyzing the periodicity and structural characteristics of the image, identifying and compressing redundant information in the image, thereby optimizing the storage and transmission efficiency of the image, and by removing or reducing non-essential frequency components, the size of the image data can be reduced while maintaining the visual quality of the image.

[0105] Step B15, determining phase domain vector information based on the image scaling ratio;

[0106] It should be noted that the phase domain vector information reflects the characteristics of the image in the phase space.

[0107] It can be understood that the phase domain vector information includes the phase information and phase relationship of the image, and represents the phase difference between different frequency components in the image.

[0108] Additionally, it should be noted that by utilizing the phase domain vector information, the phase characteristics of the image can be analyzed more accurately and more stably in the face of noise and quality degradation. Redundant information in the image is identified and compressed, thereby optimizing the storage and transmission efficiency of the image. By removing or reducing unnecessary phase components, the size of the image data can be reduced while maintaining the visual quality of the image.

[0109] Step B16, generating a Fourier eigenvector based on the frequency domain vector information and the phase domain vector information.

[0110] It can be understood that by combining frequency domain and phase domain information to generate Fourier eigenvectors, the high-frequency details of the image can be reconstructed more accurately and more stable against noise and interference, thereby improving the quality of image super-resolution reconstruction, enhancing image analysis capabilities, reducing unnecessary calculations, and significantly improving computing efficiency.

[0111] Step S20, determining a conditional parameter set based on the eigenvector and the Fourier eigenvector;

[0112] It should be noted that the conditional parameter set reflects the characteristics of the data expected to be input into the flow model after being processed by the local implicit module.

[0113] It can be understood that the conditional parameter set can be dynamically adjusted according to the characteristics of the input image and the requirements of the super-resolution task, adapting to different image contents and super-resolution ratios, thereby enhancing the flexibility and adaptability of the algorithm.

[0114] For ease of understanding, the determination of a conditional parameter set is taken as an example for explanation, wherein the information collection device is an information collection module, the storage device is a memory, and the processing device is a processing module.

[0115] The information acquisition module obtains the feature vector and the Fourier feature vector and stores them in the memory, and obtains the Fourier feature function. The calculation method of the Fourier feature function is simplified as follows:

[0116]

[0117] Among them, E a is the corresponding amplitude, E f is the corresponding frequency, E p is the corresponding phase estimation function, which is implemented by convolution in this model.

[0118] Based on the Fourier characteristic function, the characteristic vector and the Fourier characteristic vector are input into a target sensor to obtain a condition parameter set, and subsequent processing is performed based on the condition parameter set.

[0119] In a feasible implementation, step S20 may include steps C11 to C12:

[0120] Step C11, obtaining Fourier characteristic function;

[0121] It should be noted that the Fourier characteristic function reflects the characteristics of the frequency components and phase information of the image, and provides the amplitude and phase information of the image at different frequencies.

[0122] It can be understood that by utilizing the Fourier characteristic function, the frequency domain characteristics of the image can be analyzed more accurately, reducing the size of the image data while maintaining the visual quality of the image.

[0123] Step C12: inputting the feature vector and the Fourier feature vector into a target sensor based on the Fourier feature function to obtain a conditional parameter set.

[0124] It can be understood that accurately obtaining the conditional parameter set can more effectively restore the details and texture of the image, improve the visual quality of the super-resolution image, adapt to different images and application scenarios, and enhance the practicality of the algorithm.

[0125] Step S30, determining the high-resolution image based on the condition parameter set and the priori noise to complete the super-resolution of each proportion of the image.

[0126] It should be noted that the high-resolution image reflects the characteristics of the image information that is expected to be finally obtained.

[0127] It is understandable that through image super-resolution technology, high-resolution images can be restored from low-resolution images, significantly improving the clarity and visual quality of the image.

[0128] For ease of understanding, the determination of a high-resolution image is taken as an example for explanation, wherein the information acquisition device is an information acquisition module, the storage device is a memory, and the execution device is an execution module.

[0129] The information acquisition module obtains low-resolution image information and a target flow model, upsamples the low-resolution image to obtain image texture, obtains the flow model mapping relationship and the number of cycles i, and the target flow model can obtain the target distribution by learning the flow model mapping relationship, where the flow model mapping relationship can be simplified as:

[0130] f(z→m)=g 1 *g 2 ... i-1 *g i

[0131] Among them, the high-resolution texture is calculated based on the number of loops i, and the noise distribution (z) uses the prior noise z generated after the low-resolution image is encoded and decoded. Each g is composed of a linear layer and an affine injection layer, wherein the input of the affine injection layer is the conditional parameter generated by the local implicit module. The high-resolution texture sets are calculated based on the conditional parameter set, the prior noise, the flow model mapping relationship and the number of loops, that is, the high-resolution texture m is finally obtained by the flow model connected in series with multiple g. The high-resolution texture m is obtained by adding features to the image texture obtained by upsampling the low-resolution image to obtain a high-resolution image, thereby completing the super-resolution of each proportion of the image.

[0132] The present application proposes an image super-resolution method based on an implicit expression flow model, which obtains feature vectors, prior noise and Fourier feature vectors, which are extracted through a local implicit module; a conditional parameter set is determined based on the feature vectors and the Fourier feature vectors; and a high-resolution image is determined based on the conditional parameter set and the prior noise to complete super-resolution of each image proportion. The technical problem of how to perform personalized image super-resolution more effectively is solved. Compared with the prior art, the present application uses a local implicit module with Fourier feature extraction to effectively extract feature vectors, prior noise and Fourier feature vectors, thereby determining the conditional parameter set, and combining the prior noise to reconstruct the high-resolution image in a personalized manner, thereby achieving super-resolution of each image proportion, significantly improving the clarity and visual effect of the image, while optimizing the computational efficiency and reducing the consumption of computing resources.

[0133] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0134] In this embodiment, refer to Figure 2 , Figure 2 The flowchart of the second embodiment of the image super-resolution method based on the implicit expression flow model of the present application is provided, and step S30 specifically includes steps S31 to S32:

[0135] Step S31, acquiring image texture and high-resolution texture;

[0136] It should be noted that the image texture reflects a feature of a certain change in the grayscale or color of pixels in the image, and the high-resolution texture reflects a feature of more detailed texture information.

[0137] It can be understood that the image texture includes recurring texture primitives and arrangement rules, and the high-resolution image texture includes texture detail features and clear texture structure, so as to more accurately analyze the texture and structure of the image and significantly enhance the visual quality of the image.

[0138] For ease of understanding, obtaining image texture and high-resolution texture is taken as an example for explanation, wherein the information acquisition device is the information acquisition module, the storage device is the memory, and the execution device is the execution module.

[0139] The information acquisition module acquires low-resolution image information, upsamples the low-resolution image to obtain image texture, acquires a condition parameter set, a priori noise, a flow model mapping relationship and a number of loops, calculates each high-resolution texture set based on the condition parameter set, the prior noise, the flow model mapping relationship and the number of loops, concatenates the each high-resolution texture set to obtain a high-resolution texture, and performs subsequent processing based on the image texture and the high-resolution texture.

[0140] In a feasible implementation, step S31 may include steps D11 to D13:

[0141] Step D11, obtaining low-resolution image information and target flow model;

[0142] It should be noted that the low-resolution image information reflects the characteristics of the visual effect of the image after compression or downsampling, and the target flow model reflects the characteristics of the flow model obtained by learning the mapping relationship from the low-resolution image to the high-resolution image.

[0143] It can be understood that the low-resolution image information includes the basic outline, color and texture of the image, but due to the reduction in resolution, the image details will become blurred or lost, and the target flow model is a deep learning model for image super-resolution, which is used to train input image features and optimize to generate high-resolution output features, effectively restore the details and texture of the image, improve the visual quality of the super-resolution image, and can accurately capture and learn the features of low-resolution images, more accurately reconstruct the details of the image in the super-resolution process, more efficiently process a large amount of low-resolution image information, and reduce the consumption of computing resources.

[0144] Step D12, upsampling the low-resolution image to obtain image texture;

[0145] It can be understood that upsampling a low-resolution image is to increase the number of pixels of the image so that it approaches or reaches the size of the high-resolution image, and to enlarge the pixel size of the low-resolution image according to a certain ratio to make the image larger in size and higher in resolution, providing better visual quality.

[0146] Step D13: input the conditional parameter set and the prior noise into the target flow model to determine the high-resolution texture.

[0147] It is understandable that the use of prior noise can make images closer to real-world images. By combining the conditional parameter set and prior noise, the target flow model can more accurately restore the details and texture of the image, thereby improving the quality of the super-resolution image, and can process image data more efficiently, reducing unnecessary calculations, and improving the computational efficiency of the super-resolution algorithm. At the same time, it can adapt to different image content and super-resolution requirements, enhancing the adaptability and flexibility of the model.

[0148] In a feasible implementation, step D13 may include steps E11 to E13:

[0149] Step E11, obtaining the flow model mapping relationship and the number of loops;

[0150] It should be noted that the flow model mapping relationship reflects the characteristics of the conversion rule from low-resolution image features to high-resolution image features, and the number of cycles reflects the characteristics of the number of times the flow model needs to be iterated.

[0151] It is understandable that the mapping relationship between prior noise and high-resolution texture can be learned, that is, the flow model mapping relationship can be learned to obtain the target distribution, so as to predict and reconstruct the structure and texture of the high-resolution image through the known low-resolution image information, and the number of cycles represents the degree to which the model refines the image details during the generation process. The more cycles, the more carefully the model processes the image, but too many cycles will consume a lot of training time. Therefore, a reasonable setting of the number of cycles can improve the accuracy of the conversion from low-resolution to high-resolution images, thereby generating higher quality super-resolution images.

[0152] Step E12, calculating each high-resolution texture set based on the condition parameter set, the prior noise, the flow model mapping relationship and the number of loops;

[0153] It should be noted that each high-resolution texture set reflects the characteristics of a series of high-resolution image textures obtained after cyclic calculation of the target flow model.

[0154] It can be understood that each high-resolution texture set includes image edge information, texture information and pattern information. By generating the high-resolution texture set, the high-frequency detail information of the image can be represented, and the quality of the super-resolution image can be significantly improved.

[0155] Step E13: Connect the high-resolution texture sets in series to obtain a high-resolution texture.

[0156] It can be understood that by connecting a set of high-resolution textures in series, the details and texture of the image can be displayed in detail, making the reconstructed image clearer and more realistic, and significantly improving the quality of the super-resolution image.

[0157] Step S32, adding features of the image texture and the high-resolution texture to obtain a high-resolution image to complete super-resolution of each proportion of the image.

[0158] It can be understood that feature addition is a commonly used technique in the field of image processing and computer vision. It combines image features at different levels or resolutions to enhance the details and quality of the image. It can be achieved through simple pixel-level addition or through more complex fusion strategies, such as weighted addition and feature fusion networks, to enhance the details of the image and restore the high-frequency information lost due to downsampling or compression, thereby generating clearer and more natural high-resolution images.

[0159] For ease of understanding, the example of obtaining a high-resolution image is used for explanation, wherein the information acquisition device is the information acquisition module, the storage device is the memory, and the execution device is the execution module.

[0160] The information acquisition module obtains low-resolution image information, upsamples the low-resolution image to obtain image texture, obtains a conditional parameter set, a priori noise, a flow model mapping relationship and a number of loops, calculates each high-resolution texture set based on the conditional parameter set, the priori noise, the flow model mapping relationship and the number of loops, connects the each high-resolution texture set in series to obtain a high-resolution texture, and performs feature addition of the obtained high-resolution texture m and the image texture obtained by upsampling the low-resolution image to obtain a high-resolution image, thereby completing super-resolution of each proportion of the image.

[0161] This application proposes an image super-resolution method based on an implicit expression flow model, which obtains image texture and high-resolution texture; adds the features of the image texture and the high-resolution texture to obtain a high-resolution image to complete the super-resolution of each proportion of the image. This solves the technical problem of how to perform personalized image super-resolution more accurately. Compared with the prior art, this application obtains the texture information and high-resolution texture of the image, adds their features, uses implicit expression of the image, and simultaneously learns the super-resolution coefficients at multiple scales during training, so as to achieve multi-scale arbitrary scale super-resolution with one model, thereby maintaining the naturalness and consistency of the image texture under different scaling, significantly improving the visual clarity and realism of the super-resolution image, while improving processing efficiency and image quality.

[0162] For example, in order to help understand the implementation process of the image super-resolution method based on the implicit expression flow model obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flowchart of an image super-resolution method based on an implicit expression flow model is provided. Specifically:

[0163] Referring to Example 1, a feature vector, a priori noise and a Fourier feature vector are obtained, and the feature vector, the priori noise and the Fourier feature vector are extracted by a local implicit module; a conditional parameter set is determined based on the feature vector and the Fourier feature vector; a high-resolution image is determined based on the conditional parameter set and the prior noise to complete the super-resolution of each proportion of the image. Referring to Example 2, an image texture and a high-resolution texture are obtained; the image texture and the high-resolution texture are feature-added to obtain a high-resolution image to complete the super-resolution of each proportion of the image. Conditional parameters and feature vectors are extracted from a low-resolution image by a local implicit module, prior noise is generated using the features, and the conditional parameters are input into a flow model, the flow model learns the mapping relationship between the high-resolution texture and the prior noise, and a high-resolution image is obtained by combining the upsampling of the low-resolution image and the generated high-resolution texture.

[0164] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image super-resolution method based on the implicit expression flow model of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0165] This application also provides an image super-resolution device based on an implicit expression flow model, please refer to Figure 4 , the image super-resolution device based on the implicit expression flow model comprises:

[0166] An acquisition module 10, used to acquire a feature vector, a priori noise and a Fourier feature vector, wherein the feature vector, a priori noise and the Fourier feature vector are extracted through a local implicit module;

[0167] A processing module 20, configured to determine a condition parameter set based on the eigenvector and the Fourier eigenvector;

[0168] The execution module 30 is used to determine the high-resolution image based on the condition parameter set and the priori noise to complete the super-resolution of each proportion of the image.

[0169] The acquisition module 10 is also used to acquire low-resolution image information, original image information and image segmentation method;

[0170] Extracting a feature vector from the low-resolution image information through a local implicit module;

[0171] Encoding and decoding the feature vector by a decoder to generate prior noise;

[0172] A Fourier eigenvector is determined based on the eigenvector, the original image information and the image segmentation method.

[0173] The acquisition module 10 is also used to acquire the image scaling ratio;

[0174] Determine each segmented image information set based on the image segmentation method and the original image information;

[0175] Extracting the center position of each segmented image information set to generate a center coordinate vector matrix;

[0176] Determine frequency domain vector information based on the eigenvector and the center coordinate vector matrix;

[0177] determining phase domain vector information based on the image scaling;

[0178] A Fourier eigenvector is generated based on the frequency domain vector information and the phase domain vector information.

[0179] The processing module 20 is also used to obtain the Fourier characteristic function;

[0180] Based on the Fourier characteristic function, the characteristic vector and the Fourier characteristic vector are input into a target sensor to obtain a conditional parameter set.

[0181] The execution module 30 is also used to obtain image texture and high-resolution texture;

[0182] The image texture and the high-resolution texture are feature added to obtain a high-resolution image to complete super-resolution of each proportion of the image.

[0183] The execution module 30 is also used to obtain low-resolution image information and a target flow model;

[0184] Upsampling the low-resolution image to obtain an image texture;

[0185] The conditional parameter set and the prior noise are input into the target flow model to determine a high-resolution texture.

[0186] The execution module 30 is also used to obtain the flow model mapping relationship and the number of loops;

[0187] Calculate each high-resolution texture set based on the condition parameter set, the prior noise, the flow model mapping relationship and the number of loops;

[0188] The high-resolution texture sets are connected in series to obtain a high-resolution texture.

[0189] The image super-resolution device based on the implicit expression flow model provided by the present application adopts the image super-resolution method based on the implicit expression flow model in the above-mentioned embodiment, which can solve the technical problem of how to accurately and effectively perform personalized image super-resolution. Compared with the prior art, the beneficial effects of the image super-resolution device based on the implicit expression flow model provided by the present application are the same as the beneficial effects of the image super-resolution method based on the implicit expression flow model provided by the above-mentioned embodiment, and the other technical features of the image super-resolution device based on the implicit expression flow model are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0190] The present application provides an image super-resolution device based on an implicit expression flow model, and the image super-resolution device based on the implicit expression flow model includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image super-resolution method based on the implicit expression flow model in the above-mentioned embodiment one.

[0191] Reference below Figure 5 , which shows a schematic diagram of the structure of an image super-resolution device based on an implicit expression flow model suitable for implementing an embodiment of the present application. The image super-resolution device based on an implicit expression flow model in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The image super-resolution device based on the implicit expression flow model shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] like Figure 5As shown, the image super-resolution device based on the implicit expression flow model may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image super-resolution device based on the implicit expression flow model are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the image super-resolution device based on the implicit expression flow model to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an image super-resolution device based on the implicit expression flow model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0193] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0194] The image super-resolution device based on the implicit expression flow model provided by the present application adopts the image super-resolution method based on the implicit expression flow model in the above-mentioned embodiment, which can solve the technical problem of how to accurately and effectively perform personalized image super-resolution. Compared with the prior art, the beneficial effects of the image super-resolution device based on the implicit expression flow model provided by the present application are the same as the beneficial effects of the image super-resolution method based on the implicit expression flow model provided by the above-mentioned embodiment, and the other technical features in the image super-resolution device based on the implicit expression flow model are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0195] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the image super-resolution method based on the implicit expression flow model in the above-mentioned embodiment.

[0198] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0199] The computer-readable storage medium may be included in an image super-resolution device based on an implicit expression flow model; or may exist independently without being assembled into an image super-resolution device based on an implicit expression flow model.

[0200] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an image super-resolution device based on an implicit expression flow model, the image super-resolution device based on the implicit expression flow model: obtains a feature vector, prior noise and a Fourier feature vector, wherein the feature vector, prior noise and Fourier feature vector are extracted through a local implicit module; determines a conditional parameter set based on the feature vector and the Fourier feature vector; and determines a high-resolution image based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image.

[0201] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0204] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image super-resolution method based on the implicit expression flow model, and can solve the technical problem of how to accurately and effectively perform personalized image super-resolution. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the image super-resolution method based on the implicit expression flow model provided by the above-mentioned embodiment, and will not be repeated here.

[0205] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image super-resolution method based on an implicit expression flow model, characterized in that: The method includes: Obtaining a feature vector, a priori noise, and a Fourier feature vector, wherein the feature vector, the priori noise, and the Fourier feature vector are extracted by a local implicit module; determining a set of conditional parameters based on the eigenvector and the Fourier eigenvector; The high-resolution image is determined based on the conditional parameter set and the prior noise to complete super-resolution of each proportion of the image.

2. The method according to claim 1, characterized in that The steps of obtaining the eigenvector, the priori noise and the Fourier eigenvector include: Obtain low-resolution image information, original image information and image segmentation method; Extracting a feature vector from the low-resolution image information through a local implicit module; Encoding and decoding the feature vector by a decoder to generate prior noise; A Fourier eigenvector is determined based on the eigenvector, the original image information and the image segmentation method.

3. The method according to claim 2, characterized in that The step of determining the Fourier eigenvector based on the eigenvector, the original image information and the image segmentation method comprises: Get the image scaling ratio; Determine each segmented image information set based on the image segmentation method and the original image information; Extracting the center position of each segmented image information set to generate a center coordinate vector matrix; Determine frequency domain vector information based on the eigenvector and the center coordinate vector matrix; determining phase domain vector information based on the image scaling; A Fourier eigenvector is generated based on the frequency domain vector information and the phase domain vector information.

4. The method according to claim 1, characterized in that The step of determining a conditional parameter set based on the eigenvector and the Fourier eigenvector comprises: Get Fourier characteristic function; Based on the Fourier characteristic function, the characteristic vector and the Fourier characteristic vector are input into a target sensor to obtain a conditional parameter set.

5. The method according to claim 1, characterized in that The step of determining the high-resolution image based on the condition parameter set and the priori noise to complete the super-resolution of each proportion of the image comprises: Get image textures and high-resolution textures; The image texture and the high-resolution texture are feature added to obtain a high-resolution image to complete super-resolution of each proportion of the image.

6. The method according to claim 5, characterized in that The steps of obtaining image texture and high-resolution texture include: Obtain low-resolution image information and target flow model; Upsampling the low-resolution image to obtain an image texture; The conditional parameter set and the prior noise are input into the target flow model to determine a high-resolution texture.

7. The method according to claim 6, characterized in that The step of inputting the conditional parameter set and the prior noise into the target flow model to determine the high-resolution texture comprises: Get the flow model mapping relationship and loop times; Calculate each high-resolution texture set based on the condition parameter set, the prior noise, the flow model mapping relationship and the number of loops; The high-resolution texture sets are connected in series to obtain a high-resolution texture.

8. An image super-resolution device based on an implicit expression flow model, characterized in that: The device comprises: An acquisition module, used for acquiring a feature vector, a priori noise and a Fourier feature vector, wherein the feature vector, a priori noise and the Fourier feature vector are extracted through a local implicit module; A processing module, configured to determine a conditional parameter set based on the eigenvector and the Fourier eigenvector; An execution module is used to determine the high-resolution image based on the condition parameter set and the prior noise to complete the super-resolution of each proportion of the image.

9. An image super-resolution device based on an implicit expression flow model, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image super-resolution method based on the implicit expression flow model as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image super-resolution method based on the implicit expression flow model as described in any one of claims 1 to 7 are implemented.

Images