Image Processing Method and System Based on Generative Adversarial Networks

Abstract

This invention relates to an image processing method and system based on generative adversarial networks (GANs). Before compressing and transmitting the image, this application uses high-frequency information to grid the image, quantifies the complexity of the high-frequency information in each grid, and then uses a pre-constructed information loss probability model to determine the probability of high-frequency information loss in each grid after compression and super-resolution reconstruction. If the probability is high, the high-frequency information of the corresponding grid is extracted and sent along with the compressed image to the target terminal. Super-resolution reconstruction based on GANs is then performed at the target terminal, and the high-frequency information from the original image is fused with the reconstructed image to obtain a fused image. This application can simultaneously ensure the transmission speed of large batches of image files while accurately restoring the details of the reconstructed image, overcoming the shortcomings of existing GAN super-resolution reconstruction methods that easily lose high-frequency details.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image processing method and system based on generative adversarial networks. Background Technology

[0002] Generative Adversarial Networks (GANs) are models that generate high-quality data through adversarial training of two neural networks. Their core idea originates from zero-sum games in game theory. They consist of a generator (G) and a discriminator (D), which, through mutual competition and optimization, ultimately achieve the goal of generating realistic data.

[0003] Generative adversarial networks (GANs) are widely used in image super-resolution processing. Low-resolution images have a significant advantage during transmission, and after transmission, using GANs for super-resolution reconstruction can achieve rapid transmission of large batches of high-quality images. However, existing GANs are prone to losing high-frequency information in super-resolution reconstruction, resulting in information loss in the image. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an image processing method and system based on generative adversarial networks to solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The image processing method based on generative adversarial networks of the present invention includes the following steps:

[0007] Obtain the image to be processed and its size information; and preprocess the image to be processed to obtain a preprocessed image;

[0008] Calculate the high-frequency energy ratio of the preprocessed image, and construct a grid that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the grid size;

[0009] The preprocessed image is compressed to obtain a compressed image; and high-frequency information is extracted from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise.

[0010] The high-frequency information image is divided based on the grid to obtain multiple high-frequency information grids; and the complexity of the high-frequency information in each high-frequency information grid is quantitatively evaluated to obtain a quantitative evaluation vector.

[0011] The information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid is determined based on a pre-constructed information loss probability model, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information.

[0012] The high-frequency information grid with an information loss probability higher than a preset probability threshold is selected as the target grid, and the high-frequency information of the target grid is extracted; the high-frequency information of the target grid and the compressed image are sent to the target terminal;

[0013] The compressed image in the target terminal is super-resolution reconstructed based on a pre-built generative adversarial network to obtain a reconstructed image. The high-frequency information of the target grid is then fused with the reconstructed image to obtain a fused image.

[0014] In one embodiment of this application, the image to be processed is preprocessed to obtain a preprocessed image, including:

[0015] The image to be processed is converted to grayscale to obtain a grayscale image;

[0016] The grayscale image is subjected to a high-pass filter to obtain a filtered image;

[0017] The filtered image is then contrast-enhanced to obtain a preprocessed image.

[0018] In one embodiment of this application, the high-frequency energy ratio of the preprocessed image is calculated, and a grid covering the preprocessed image is constructed based on the high-frequency energy ratio and the size information of the image to be processed, including:

[0019] The preprocessed image is normalized to obtain a normalized image;

[0020] Perform a Fast Fourier Transform on the normalized image to obtain a normalized frequency domain image;

[0021] The normalized frequency domain image is centered to obtain a centered image, wherein the low-frequency component of the centered image is located at the center and the high-frequency component is located at the edge.

[0022] The center point of the centered image is determined, and a high-low frequency dividing circle is constructed based on a set radius threshold and the center point, wherein the inside of the high-low frequency dividing circle is the low-frequency component and the outside of the high-low frequency dividing circle is the high-frequency component.

[0023] Calculate the total energy of the centered image. and the high-frequency energy outside the high- and low-frequency dividing circle And based on the total energy and the high-frequency energy Calculate the proportion of high-frequency energy , ;

[0024] The proportion of high-frequency energy The ratio is compared with a preset threshold, and the high-frequency energy percentage is... When the proportion exceeds a preset threshold, based on the high-frequency energy ratio Determine the number of grid height divisions in the image. and width division quantity Number of height divisions and width division quantity The formula for calculation is:

[0025] ;

[0026] in, The preset baseline value for the number of divisions in the height direction. The preset baseline value for the number of divisions in the width direction. This is the preset percentage benchmark value;

[0027] Based on the number of height divisions The number of width divisions The size information of the image to be processed is used to construct a grid that covers the preprocessed image.

[0028] In one embodiment of this application, high-frequency information extraction is performed on the preprocessed image to obtain a high-frequency information image, including:

[0029] Perform a Fast Fourier Transform on the preprocessed image to obtain a frequency domain image;

[0030] The frequency domain image is high-pass filtered based on a pre-constructed high-pass filter to obtain a filtered image.

[0031] Perform an inverse Fourier transform on the filtered image to obtain a high-frequency information image.

[0032] In one embodiment of this application, the method for constructing the information loss probability model includes:

[0033] Multiple image samples are acquired and compressed to obtain multiple compressed image samples;

[0034] The compressed image samples are input into a pre-constructed generative adversarial network to obtain reconstructed samples;

[0035] Extract high-frequency information image samples from the image sample and high-frequency information image samples from the reconstructed sample; perform gridding processing on the high-frequency information image samples of the image sample to obtain gridded samples; perform gridding processing on the high-frequency information image samples of the reconstructed sample to obtain gridded reconstructed samples; and construct sample pairs based on the gridded samples and the gridded reconstructed samples.

[0036] Quantization evaluation is performed on multiple grids in the gridded sample to obtain quantization evaluation vectors for multiple grids. ,in, Represents grid coordinates;

[0037] For any pair of samples, normalize and subtract the two grids with the same grid coordinates to obtain a difference grid image, and calculate the sum of the pixel values ​​of the difference grid image to obtain the difference value representing the difference between the two. ;

[0038] With the aforementioned quantitative evaluation vector and the difference value Constructing the fundamental vectors for analysis ;

[0039] Using the quantitative evaluation vector as the clustering benchmark, multiple analytical basis vectors are clustered to obtain multiple vector clusters;

[0040] Calculate the reference range for multiple quantization evaluation vectors in each vector cluster, and calculate the probability value of the anomaly analysis basis vectors with difference values ​​greater than the difference threshold in each vector cluster relative to the total number of analysis basis vectors in the vector cluster. ;

[0041] Based on the reference range of multiple quantization evaluation vectors and the probability values ​​corresponding to the reference range of multiple quantization evaluation vectors. Construct an information loss probability model.

[0042] In one embodiment of this application, the complexity of high-frequency information in each high-frequency information grid is quantitatively evaluated to obtain a quantitative evaluation vector, including:

[0043] The high-frequency information in each high-frequency information grid is segmented to obtain the high-frequency information and its types. The types of high-frequency information include contours, noise, textures, and abrupt changes.

[0044] The high-frequency information is converted to the mid-frequency domain, and the energy of each type of high-frequency information is calculated from the frequency. Information entropy Non-zero coefficient ratio and variance Among them, energy Information entropy The calculation formulas are as follows:

[0045] ;

[0046] In the formula, Represents pixels in high-frequency information , frequency components, This indicates that the pixel frequency components in the high-frequency information are The probability of;

[0047] For the energy respectively The information entropy The non-zero coefficient ratio and the variance Normalization is performed to obtain normalized energy. Normalized information entropy Normalized non-zero coefficient proportion and normalized variance ;

[0048] For the normalized energy The normalized information entropy The normalized non-zero coefficient ratio and the normalized variance We perform weighted summation to obtain the quantized values ​​of high-frequency information;

[0049] For any high-frequency information grid, a quantization evaluation vector is constructed based on the quantization values ​​of the contour, noise, texture, and abrupt change points.

[0050] In one embodiment of this application, the information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid is determined based on a pre-built information loss probability model, including:

[0051] Substitute the quantized evaluation vector of the high-frequency information in each high-frequency information grid into the information loss probability model;

[0052] When the quantized evaluation vector falls into any target reference range in the information loss probability model, the probability corresponding to the target reference range is taken as the information loss probability corresponding to the quantized evaluation vector.

[0053] In one embodiment of this application, a generative adversarial network is pre-constructed, including:

[0054] Obtain a training dataset containing pairs of low-resolution and high-resolution images;

[0055] Construct a generator and a discriminator, and define a loss function; train the generator and discriminator based on the training dataset and the loss function to obtain a generative adversarial network, wherein the loss function includes adversarial loss, content loss and pixel-level loss.

[0056] In one embodiment of this application, the high-frequency information of the target mesh is fused with the reconstructed image to obtain a fused image, including:

[0057] The high-frequency information of the target mesh is superimposed onto the corresponding position of the reconstructed image, and the superimposed edges are Gaussian blurred to obtain the fused image.

[0058] This application provides an image processing system based on generative adversarial networks, including:

[0059] The acquisition module is used to acquire the image to be processed and its size information; and to preprocess the image to be processed to obtain a preprocessed image;

[0060] A meshing module is used to calculate the high-frequency energy ratio of the preprocessed image and construct a mesh that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the mesh size;

[0061] The compression and feature extraction module is used to compress the preprocessed image to obtain a compressed image; and to extract high-frequency information from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise.

[0062] The quantization module is used to divide the high-frequency information image based on the grid to obtain multiple high-frequency information grids; and to quantize and evaluate the complexity of the high-frequency information in each high-frequency information grid to obtain a quantization evaluation vector.

[0063] The probability evaluation module is used to determine the information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid based on a pre-built information loss probability model, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information.

[0064] The transmission module is used to select high-frequency information grids with a probability of information loss higher than a preset probability threshold as target grids, extract high-frequency information from the target grids, and send the high-frequency information of the target grids and the compressed image to the target terminal.

[0065] The restoration module is used to perform super-resolution reconstruction of the compressed image in the target terminal based on a pre-built generative adversarial network to obtain a reconstructed image, and to fuse the high-frequency information of the target grid with the reconstructed image to obtain a fused image.

[0066] The beneficial effects of this invention are as follows: The image processing method and system based on generative adversarial networks (GANs) of this invention first evaluates the proportion of high-frequency information in the image before compression and transmission. If the proportion of high-frequency information is large, it indicates the presence of numerous contours or details. Therefore, the image is gridded using high-frequency information, and the complexity of high-frequency information in each grid is quantitatively evaluated. Then, a pre-constructed information loss probability model is used to determine the probability of high-frequency information loss in each grid after compression and super-resolution reconstruction. If the probability is high, the high-frequency information of the corresponding grid is extracted and sent along with the compressed image to the target terminal. Super-resolution reconstruction based on GANs is then performed at the target terminal, and the high-frequency information from the original image is fused with the reconstructed image to obtain a fused image. This invention can simultaneously ensure the transmission speed of large batches of file images while accurately restoring the details of the reconstructed image, overcoming the shortcomings of existing GAN super-resolution reconstruction methods that easily lose high-frequency details. Attached Figure Description

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0068] Figure 1 This is a structural diagram of a file transfer system according to an embodiment of this application;

[0069] Figure 2 This is a flowchart illustrating an image processing method based on generative adversarial networks in one embodiment of this application;

[0070] Figure 3 This is a comparative schematic diagram of the original image and the high-frequency image in one embodiment of this application;

[0071] Figure 4 This is a schematic diagram illustrating the extraction of information from a target grid in one embodiment of this application;

[0072] Figure 5 This is a structural diagram of an image processing system based on a generative adversarial network shown in one embodiment of this application;

[0073] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0074] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0075] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the actual number, shape and size ratio of the layers in the actual implementation. In the actual implementation, the form and number of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0076] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of the invention; however, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details.

[0077] Figure 1 This is a structural diagram of a file transfer system according to an embodiment of this application, such as... Figure 1 As shown, the file transfer system includes a compression module, an image analysis and extraction module, and a target terminal. In this application, the input image is analyzed by the image analysis and extraction module to determine whether the input image contains high-frequency information that is easily lost in subsequent processing. If so, the high-frequency information is extracted accordingly. The input image is compressed, and the extracted high-frequency information and the compressed image are sent together to the target terminal. A generative adversarial network deployed in the target terminal performs super-resolution reconstruction of the compressed image and fuses the resulting reconstructed image with the high-frequency information to obtain a fused image that can accurately restore image details.

[0078] Figure 2 This is a flowchart illustrating an image processing method based on generative adversarial networks in one embodiment of this application, as shown below. Figure 2 As shown: The image processing method based on generative adversarial networks in this embodiment may include steps S210 to S270:

[0079] S210, acquire the image to be processed and its size information; and preprocess the image to be processed to obtain a preprocessed image;

[0080] The image to be processed is the original image, with dimensions of H*W and color channels of RGB.

[0081] Before analysis and processing, the original image needs to be preprocessed to reduce the amount of information to be processed and to optimize image details, facilitating subsequent analysis and processing. The preprocessing process includes:

[0082] S211, perform grayscale conversion on the image to be processed to obtain a grayscale image. ;

[0083] Converting a color image to grayscale involves removing color information and retaining only brightness information. A common method is to weight the RGB channels. After removing color interference, the brightness distribution of the image is clearer, facilitating subsequent edge detection, texture analysis, and other operations.

[0084] S212, for the grayscale image Perform high-pass filtering to obtain the filtered image;

[0085] High-pass filtering can preserve high-frequency information (such as edges and textures) while suppressing low-frequency information (such as smooth areas).

[0086] S213, perform contrast enhancement on the filtered image to obtain a preprocessed image. .

[0087] The image is made clearer by adjusting the dynamic range of pixel values. This is achieved using histogram equalization.

[0088] S220, calculate the high-frequency energy ratio of the preprocessed image, and construct a grid that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the grid size;

[0089] High-frequency information, such as contours, textures, and noise, is easily lost in subsequent super-resolution reconstruction. Therefore, the higher the proportion of high-frequency energy in an image, the greater the probability of loss. Thus, this application first calculates the preprocessed image... The proportion of mid-to-high frequency energy is used to further refine the image, increasing granularity. The mesh generation process includes:

[0090] S221, Normalize the preprocessed image to obtain a normalized image;

[0091] The pixel values ​​of the preprocessed image are scaled to a uniform range (e.g., [0,1] or [-1,1]) to eliminate brightness differences between different images. This application uses a max-min normalization method for normalization processing.

[0092] S222, Perform a fast Fourier transform on the normalized image to obtain a normalized frequency domain image;

[0093] The normalized image is transformed from the spatial domain to the frequency domain, separating low-frequency (smooth regions) and high-frequency (edges, texture) components. Decomposing the image into different frequency components facilitates subsequent separation of high and low frequency information.

[0094] S223, The normalized frequency domain image is centered to obtain a centered image, wherein the low-frequency component of the centered image is located at the center and the high-frequency component is located at the edge.

[0095] Centering moves the low-frequency components of the frequency domain image to the center, while the high-frequency components are distributed at the edges.

[0096] S224, determine the center point of the centered image, and construct a high-low frequency dividing circle based on a set radius threshold and the center point, wherein the inside of the high-low frequency dividing circle is the low-frequency component and the outside of the high-low frequency dividing circle is the high-frequency component.

[0097] In a centered frequency domain image, a dividing circle is constructed with the center point as the center and a set radius. The radius can be adjusted according to task requirements to flexibly adapt to different image characteristics.

[0098] S225, Calculate the total energy of the centered image. and the high-frequency energy outside the high- and low-frequency dividing circle And based on the total energy and the high-frequency energy Calculate the proportion of high-frequency energy , ;

[0099] The formula for calculating the total energy is as follows:

[0100] ;

[0101] In the formula, Represents the centralized processing of pixels in an image. Frequency components;

[0102] High-frequency energy The calculation formula is:

[0103] ;

[0104] Represents pixels in high-frequency information Frequency components;

[0105] S226, the high-frequency energy ratio The ratio is compared with a preset threshold, and the high-frequency energy percentage is... When the proportion exceeds a preset threshold, based on the high-frequency energy ratio Determine the number of grid height divisions in the image. and width division quantity Number of height divisions and width division quantity The formula for calculation is:

[0106] ;

[0107] in, The preset baseline value for the number of divisions in the height direction. The preset baseline value for the number of divisions in the width direction. This is the preset percentage benchmark value;

[0108] High-frequency energy ratio Reflecting the richness of image details (such as texture density and edge complexity), when the proportion of high-frequency energy is high, the number of divisions is increased to capture more details; conversely, the number of divisions is reduced to lower computational costs.

[0109] S227, based on the number of height divisions The number of width divisions The size information of the image to be processed is used to construct a grid that covers the preprocessed image.

[0110] Finally, a grid is generated to evenly cover the entire image, ensuring no areas are missed. The grid density is dynamically adjusted according to the image content, balancing efficiency and accuracy.

[0111] In the above process, high-frequency regions of the image are quickly located through frequency domain analysis and energy calculation, avoiding global overprocessing. The number of grid divisions is dynamically adjusted according to the image content to balance computational resources and processing efficiency. Normalization and centering processes reduce noise interference and ensure the accuracy of energy calculation.

[0112] S230, the preprocessed image is compressed to obtain a compressed image; and high-frequency information is extracted from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise.

[0113] The image compression method is JPEG compression based on Discrete Cosine Transform (DCT). The compressed image has reduced resolution and data size, which facilitates fast transmission.

[0114] The process of extracting high-frequency information includes:

[0115] S231, Perform a fast Fourier transform on the preprocessed image to obtain a frequency domain image;

[0116] The preprocessed image is converted from the spatial domain to the frequency domain to separate low-frequency and high-frequency components.

[0117] S232, perform high-pass filtering on the frequency domain image based on a pre-constructed high-pass filter to obtain a filtered image;

[0118] This application uses a Butterworth high-pass filter (HPF) for filtering to ensure a smooth transition and reduce ringing effects.

[0119] S233, Perform an inverse Fourier transform on the filtered image to obtain a high-frequency information image.

[0120] The filtered frequency domain image is converted back to the spatial domain to obtain an image containing only high-frequency information. Figure 3 This is a comparative schematic diagram of the original image and high-frequency image in one embodiment of this application, as shown below. Figure 3 As shown, the upper image is a high-frequency image containing only high-frequency information, including contours, noise, etc., while the lower image is a grayscale image.

[0121] S240, the high-frequency information image is divided based on the grid to obtain multiple high-frequency information grids; and the complexity of the high-frequency information in each high-frequency information grid is quantitatively evaluated to obtain a quantitative evaluation vector;

[0122] High-frequency information images inherently possess a certain amount of data; if all of it were used for transmission, it would slow down the transmission rate. Therefore, to extract the parts of high-frequency information images prone to information loss, this application performs gridding processing on the high-frequency information and then quantifies the complexity of the high-frequency information in each grid. Generally, the more complex the high-frequency information, the more likely it is to be lost in subsequent super-resolution reconstruction. Therefore, the process of quantifying the complexity of high-frequency information includes:

[0123] S241, the high-frequency information in each high-frequency information grid is segmented to obtain the high-frequency information and the types of high-frequency information. The types of high-frequency information include contour, noise, texture and abrupt change points.

[0124] In this embodiment, an edge extraction operator (such as the Canny operator) is used to segment the high-frequency information grid to obtain edge features. Then, a texture segmenter (such as a Gabor filter) is used to segment the high-frequency information grid to obtain texture features. Next, noise regions are separated through statistical analysis or morphological operations, such as smoothing the image using Gaussian filtering to separate high-frequency noise. The remaining unsegmentable high-frequency information may be some abrupt changes. Based on the above process, high-frequency information can be classified.

[0125] Since different types of high-frequency information have different loss probabilities during reconstruction, this application quantizes each type of high-frequency information in the high-frequency information grid to form a quantization evaluation vector. The specific process is as follows.

[0126] S242, convert the high-frequency information to the intermediate frequency domain, and calculate the energy of each type of high-frequency information from the frequency. Information entropy Non-zero coefficient ratio and variance Among them, energy Information entropy The calculation formulas are as follows:

[0127] ;

[0128] In the formula, This represents the total number of pixels in the high-frequency information grid. The average frequency component of the pixels in the high-frequency information grid. Represents pixels in high-frequency information , frequency components, This indicates that the pixel frequency components in the high-frequency information are The probability is determined by statistically analyzing the histogram of high-frequency coefficients.

[0129] S243, respectively, the energy The information entropy The non-zero coefficient ratio and the variance Normalization is performed to obtain normalized energy. Normalized information entropy Normalized non-zero coefficient proportion and normalized variance ;

[0130] Normalization is calculated using max-min normalization, resulting in quantized values ​​in the range of 0-1.

[0131] S244, regarding the normalized energy The normalized information entropy The normalized non-zero coefficient ratio and the normalized variance We perform weighted summation to obtain the quantized value of high-frequency information. , Indicates the types of high-frequency information.

[0132] Final quantization value for:

[0133] ;

[0134] in, As the first weight, As the second weight, As the third weight, It is the fourth weight.

[0135] S245, for any high-frequency information grid, construct a quantization evaluation vector based on the quantization values ​​of the contour, noise, texture, and abrupt change points.

[0136] Finally, the grid The corresponding quantitative evaluation vector is , The quantization value representing the contour. The quantization value representing the noise. This represents the quantization value of the texture. The quantization value representing the mutation point.

[0137] S250, Based on the pre-constructed information loss probability model, determine the information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information.

[0138] In this application, the information loss probability is pre-constructed based on the actual performance of the generative adversarial network to characterize the information loss probability corresponding to different evaluation vectors, specifically including:

[0139] (1) Acquire multiple image samples and compress the multiple image samples to obtain multiple compressed image samples;

[0140] The compression method used here is the same as that described earlier, and will not be repeated here.

[0141] (2) Input the compressed image samples into a pre-constructed generative adversarial network to obtain reconstructed samples;

[0142] Generative Adversarial Networks (GANs) are pre-trained, and the training process may include:

[0143] Obtain a training dataset containing pairs of low-resolution images [LR] and high-resolution images [HR];

[0144] Construct a generator and a discriminator, and define a loss function; train the generator and discriminator based on the training dataset and the loss function to obtain a generative adversarial network, wherein the loss function includes adversarial loss, content loss and pixel-level loss.

[0145] The goal of a generator (G) is to generate samples (such as images or text) that resemble real data from random noise. The input is a random noise vector (e.g., normally or uniformly distributed). The output is generated fake data, with the aim of approximating the real data distribution as closely as possible.

[0146] The goal of a discriminator (D) is to distinguish whether input data comes from a real dataset or fake data generated by a generator. It takes either real or generated data as input and outputs a probability value representing the probability that the input data is "real".

[0147] After training, the generator can output high-resolution images from low-resolution images, achieving super-resolution image reconstruction.

[0148] (3) Extract the high-frequency information image samples of the image sample and the high-frequency information image samples of the reconstructed sample; and perform gridding processing on the high-frequency information image samples of the image sample to obtain gridded samples; and perform gridding processing on the high-frequency information image samples of the reconstructed sample to obtain gridded reconstructed samples; and construct sample pairs based on the gridded samples and the gridded reconstructed samples;

[0149] (4) Quantize and evaluate multiple grids in the gridded sample to obtain the quantization evaluation vectors of multiple grids. ,in, Represents grid coordinates;

[0150] Please refer to the previous text for details on gridding and quantization processing; they will not be repeated here.

[0151] (5) For any pair of samples, normalize and subtract the two grids with the same grid coordinates to obtain a difference grid image, and calculate the sum of the pixel values ​​of the difference grid image to obtain the difference value representing the difference between the two. ;

[0152] For two grids with the same coordinates, normalization and subtraction (the absolute value of the difference) are performed respectively to obtain a difference image. The pixel values ​​of the difference image are then summed, and the total difference is used to measure the difference between the two grids; the larger the value, the greater the difference. Difference value This is a quantitative value that reflects the differences between the two.

[0153] (6) Using the aforementioned quantitative evaluation vector and the difference value Constructing the fundamental vectors for analysis ;

[0154] Then, based on the grid, the quantization evaluation vector and the difference value Build the basic data to analyze the discrepancies. The correlation between the quantization evaluation vector and the grid.

[0155] (7) Using the quantitative evaluation vector as the clustering benchmark, multiple analytical basis vectors are clustered to obtain multiple vector clusters;

[0156] In this embodiment, density clustering is used to cluster analytical basis vectors that are similar to the quantization evaluation vectors into one cluster. Therefore, vector clusters corresponding to various typical quantization evaluation vectors can be obtained.

[0157] (8) Calculate the reference range of multiple quantification evaluation vectors in each vector cluster, and calculate the probability value of the abnormal analysis basis vector with a difference value greater than the difference threshold in each vector cluster relative to the total number of analysis basis vectors in the vector cluster. ;

[0158] The reference range is a reference range that conforms to three standard deviations, constructed from the mean and standard deviation of each parameter in the quantitative evaluation vector. The quantitative evaluation vector includes four parameters, so the reference range includes four corresponding ranges of three standard deviations.

[0159] Then, based on the difference threshold, all data in the cluster are filtered to obtain the anomaly analysis base vector. The ratio of this part of the vector to the total number of vectors is the corresponding probability. That is to say, when the value of the quantized evaluation vector falls within one of the reference ranges, the probability that the high-frequency information of the super-resolution reconstruction differs significantly from that of the original image is the corresponding probability value. .

[0160] (9) The reference range of multiple quantization evaluation vectors and the probability values ​​corresponding to the reference range of multiple quantization evaluation vectors. Construct an information loss probability model.

[0161] After obtaining the information loss probability model, the quantized evaluation vector of high-frequency information in each high-frequency information grid is substituted into the information loss probability model; when the quantized evaluation vector falls into any target reference range in the information loss probability model, the probability corresponding to the target reference range is taken as the information loss probability corresponding to the quantized evaluation vector.

[0162] S260, select the high-frequency information grid with an information loss probability higher than a preset probability threshold as the target grid, and extract the high-frequency information of the target grid; send the high-frequency information of the target grid and the compressed image to the target terminal;

[0163] For grids with a high probability of loss, high-frequency information is extracted. Then, the high-frequency information is packaged with the compressed image and sent to the target terminal. Figure 4 This is a schematic diagram of information extraction from a target mesh in one embodiment of this application, as shown below. Figure 4As shown, the grid to be filled is the grid with a high probability of being lost. The high-frequency information of the grid with a high probability of being lost is extracted to obtain the position and high-frequency information of multiple target high-frequency grids.

[0164] S270, the compressed image in the target terminal is super-resolution reconstructed based on a pre-built generative adversarial network to obtain a reconstructed image, and the high-frequency information of the target grid is fused with the reconstructed image to obtain a fused image.

[0165] When the target terminal receives the data packet, it parses the compressed image and high-frequency information, and uses a generative adversarial network deployed locally or in the cloud to perform super-resolution reconstruction of the compressed image. Since the high-frequency information preserves positional information, it can be aligned with the reconstructed image based on this positional information, and Gaussian blurring is applied to the overlapping edges to obtain a fused image. This avoids obvious stitching artifacts and results in a more natural stitching.

[0166] The image processing method based on generative adversarial networks (GANs) of this invention first evaluates the proportion of high-frequency information in the image before compression and transmission. A high proportion of high-frequency information indicates the presence of numerous contours or details. Therefore, the image is gridded using high-frequency information, and the complexity of high-frequency information in each grid is quantitatively evaluated. Then, a pre-constructed information loss probability model is used to determine the probability of high-frequency information loss in each grid after compression and super-resolution reconstruction. If the probability is high, the high-frequency information of the corresponding grid is extracted and sent along with the compressed image to the target terminal. Super-resolution reconstruction based on GANs is then performed at the target terminal, and the high-frequency information from the original image is fused with the reconstructed image to obtain a fused image. This invention can simultaneously ensure the transmission speed of large batches of image files while accurately restoring the details of the reconstructed image, overcoming the drawback of existing GAN super-resolution reconstruction methods that easily lose high-frequency details.

[0167] like Figure 5 As shown, this application provides an image processing system based on generative adversarial networks, comprising:

[0168] The acquisition module is used to acquire the image to be processed and its size information; and to preprocess the image to be processed to obtain a preprocessed image;

[0169] A meshing module is used to calculate the high-frequency energy ratio of the preprocessed image and construct a mesh that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the mesh size;

[0170] The compression and feature extraction module is used to compress the preprocessed image to obtain a compressed image; and to extract high-frequency information from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise.

[0171] The quantization module is used to divide the high-frequency information image based on the grid to obtain multiple high-frequency information grids; and to quantize and evaluate the complexity of the high-frequency information in each high-frequency information grid to obtain a quantization evaluation vector.

[0172] The probability evaluation module is used to determine the information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid based on a pre-built information loss probability model, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information.

[0173] The transmission module is used to select high-frequency information grids with a probability of information loss higher than a preset probability threshold as target grids, extract high-frequency information from the target grids, and send the high-frequency information of the target grids and the compressed image to the target terminal.

[0174] The restoration module is used to perform super-resolution reconstruction of the compressed image in the target terminal based on a pre-built generative adversarial network to obtain a reconstructed image, and to fuse the high-frequency information of the target grid with the reconstructed image to obtain a fused image.

[0175] The present invention relates to an image processing method and system based on generative adversarial networks (GANs). Before compressing and sending an image, this application first evaluates the proportion of high-frequency information in the image. If the proportion of high-frequency information is large, it indicates the presence of more contours or details. Therefore, the image is gridded using high-frequency information, and the complexity of high-frequency information in each grid is quantitatively evaluated. Then, a pre-constructed information loss probability model is used to determine the probability of high-frequency information loss in each grid after compression and super-resolution reconstruction. If the probability is high, the high-frequency information of the corresponding grid is extracted and sent along with the compressed image to the target terminal. Super-resolution reconstruction based on GANs is then performed on the target terminal, and the high-frequency information from the original image is fused with the reconstructed image to obtain a fused image. This application can simultaneously ensure the transmission speed of large batches of image files while accurately restoring the details of the reconstructed image, overcoming the shortcomings of existing GAN super-resolution reconstruction methods that easily lose high-frequency details.

[0176] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0177] like Figure 6 As shown, the computer system includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from storage portion 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0178] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0179] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0180] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0183] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0184] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0185] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. An image processing method based on generative adversarial networks, characterized in that, Including the following steps: Obtain the image to be processed and its size information; and preprocess the image to be processed to obtain a preprocessed image; Calculate the high-frequency energy ratio of the preprocessed image, and construct a grid that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the grid size; The preprocessed image is compressed to obtain a compressed image; and high-frequency information is extracted from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise. The high-frequency information image is divided based on the grid to obtain multiple high-frequency information grids; and the complexity of the high-frequency information in each high-frequency information grid is quantitatively evaluated to obtain a quantitative evaluation vector. The information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid is determined based on a pre-constructed information loss probability model, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information. The high-frequency information grid with an information loss probability higher than a preset probability threshold is selected as the target grid, and the high-frequency information of the target grid is extracted; the high-frequency information of the target grid and the compressed image are sent to the target terminal; The compressed image in the target terminal is super-resolution reconstructed based on a pre-built generative adversarial network to obtain a reconstructed image. The high-frequency information of the target grid is then fused with the reconstructed image to obtain a fused image.

2. The image processing method based on generative adversarial networks according to claim 1, characterized in that, The image to be processed is preprocessed to obtain a preprocessed image, including: The image to be processed is converted to grayscale to obtain a grayscale image; The grayscale image is subjected to a high-pass filter to obtain a filtered image; The filtered image is then contrast-enhanced to obtain a preprocessed image.

3. The image processing method based on generative adversarial networks according to claim 1, characterized in that, Calculate the high-frequency energy percentage of the preprocessed image, and construct a grid covering the preprocessed image based on the high-frequency energy percentage and the size information of the image to be processed, including: The preprocessed image is normalized to obtain a normalized image; Perform a Fast Fourier Transform on the normalized image to obtain a normalized frequency domain image; The normalized frequency domain image is centered to obtain a centered image, wherein the low-frequency component of the centered image is located at the center and the high-frequency component is located at the edge. The center point of the centered image is determined, and a high-low frequency dividing circle is constructed based on a set radius threshold and the center point, wherein the inside of the high-low frequency dividing circle is the low-frequency component and the outside of the high-low frequency dividing circle is the high-frequency component. Calculate the total energy of the centered image. and the high-frequency energy outside the high- and low-frequency dividing circle And based on the total energy and the high-frequency energy Calculate the proportion of high-frequency energy , ; The proportion of high-frequency energy The ratio is compared with a preset threshold, and the high-frequency energy percentage is... When the proportion exceeds a preset threshold, based on the high-frequency energy ratio Determine the number of grid height divisions in the image. and width division quantity Number of height divisions and width division quantity The formula for calculation is: ； in, The preset baseline value for the number of divisions in the height direction. The preset baseline value for the number of divisions in the width direction. This is the preset percentage benchmark value; Based on the number of height divisions The number of width divisions The size information of the image to be processed is used to construct a grid that covers the preprocessed image.

4. The image processing method based on generative adversarial networks according to claim 1, characterized in that, High-frequency information extraction is performed on the preprocessed image to obtain a high-frequency information image, including: Perform a Fast Fourier Transform on the preprocessed image to obtain a frequency domain image; The frequency domain image is high-pass filtered based on a pre-constructed high-pass filter to obtain a filtered image. Perform an inverse Fourier transform on the filtered image to obtain a high-frequency information image.

5. The image processing method based on generative adversarial networks according to claim 1, characterized in that, The method for constructing the information loss probability model includes: Multiple image samples are acquired and compressed to obtain multiple compressed image samples; The compressed image samples are input into a pre-constructed generative adversarial network to obtain reconstructed samples; Extract high-frequency information image samples from the image sample and high-frequency information image samples from the reconstructed sample; perform gridding processing on the high-frequency information image samples of the image sample to obtain gridded samples; perform gridding processing on the high-frequency information image samples of the reconstructed sample to obtain gridded reconstructed samples; and construct sample pairs based on the gridded samples and the gridded reconstructed samples. Quantization evaluation is performed on multiple grids in the gridded sample to obtain quantization evaluation vectors for multiple grids. ,in, Represents grid coordinates; For any pair of samples, normalize and subtract the two grids with the same grid coordinates to obtain a difference grid image, and calculate the sum of the pixel values ​​of the difference grid image to obtain the difference value representing the difference between the two. ; With the aforementioned quantitative evaluation vector and the difference value Constructing the fundamental vectors for analysis ; Using the quantitative evaluation vector as the clustering benchmark, multiple analytical basis vectors are clustered to obtain multiple vector clusters; Calculate the reference range for multiple quantization evaluation vectors in each vector cluster, and calculate the probability value of the anomaly analysis basis vectors with difference values ​​greater than the difference threshold in each vector cluster relative to the total number of analysis basis vectors in the vector cluster. ; Based on the reference range of multiple quantization evaluation vectors and the probability values ​​corresponding to the reference range of multiple quantization evaluation vectors. Construct an information loss probability model.

6. The image processing method based on generative adversarial networks according to claim 1 or 5, characterized in that, The complexity of high-frequency information in each high-frequency information grid is quantitatively evaluated to obtain a quantitative evaluation vector, including: The high-frequency information in each high-frequency information grid is segmented to obtain the high-frequency information and its types. The types of high-frequency information include contours, noise, textures, and abrupt changes. The high-frequency information is converted to the mid-frequency domain, and the energy of each type of high-frequency information is calculated from the frequency. Information entropy Non-zero coefficient ratio and variance Among them, energy Information entropy The calculation formulas are as follows: ； In the formula, Represents pixels in high-frequency information , frequency components, This indicates that the pixel frequency components in the high-frequency information are The probability of; For the energy respectively The information entropy The non-zero coefficient ratio and the variance Normalization is performed to obtain normalized energy. Normalized information entropy The normalized non-zero coefficient ratio and the normalized variance We perform weighted summation to obtain the quantized values ​​of high-frequency information; For any high-frequency information grid, a quantization evaluation vector is constructed based on the quantization values ​​of the contour, noise, texture, and abrupt change points.

7. The image processing method based on generative adversarial networks according to claim 1, characterized in that, The information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid is determined based on a pre-built information loss probability model, including: Substitute the quantized evaluation vector of the high-frequency information in each high-frequency information grid into the information loss probability model; When the quantized evaluation vector falls into any target reference range in the information loss probability model, the probability corresponding to the target reference range is taken as the information loss probability corresponding to the quantized evaluation vector.

8. The image processing method based on generative adversarial networks according to claim 2, characterized in that, Pre-constructing a generative adversarial network includes: Obtain a training dataset containing pairs of low-resolution and high-resolution images; Construct a generator and a discriminator, and define a loss function; train the generator and discriminator based on the training dataset and the loss function to obtain a generative adversarial network, wherein the loss function includes adversarial loss, content loss and pixel-level loss.

9. The image processing method based on generative adversarial networks according to claim 1, characterized in that, The high-frequency information of the target grid is fused with the reconstructed image to obtain a fused image, including: The high-frequency information of the target mesh is superimposed onto the corresponding position of the reconstructed image, and the superimposed edges are Gaussian blurred to obtain the fused image.

10. An image processing system based on generative adversarial networks, characterized in that, include: The acquisition module is used to acquire the image to be processed and its size information. The image to be processed is then preprocessed to obtain a preprocessed image; A meshing module is used to calculate the high-frequency energy ratio of the preprocessed image and construct a mesh that covers the preprocessed image based on the high-frequency energy ratio and the size information of the image to be processed, wherein the higher the high-frequency energy ratio, the smaller the mesh size; The compression and feature extraction module is used to compress the preprocessed image to obtain a compressed image; and to extract high-frequency information from the preprocessed image to obtain a high-frequency information image, wherein the high-frequency information includes contour features and noise. The quantization module is used to divide the high-frequency information image based on the grid to obtain multiple high-frequency information grids; and to quantize and evaluate the complexity of the high-frequency information in each high-frequency information grid to obtain a quantization evaluation vector. The probability evaluation module is used to determine the information loss probability corresponding to the quantization evaluation vector of each high-frequency information grid based on a pre-built information loss probability model, wherein the information loss probability model characterizes the high-frequency information loss probability corresponding to the quantization evaluation vector of high-frequency information. The transmission module is used to select high-frequency information grids with a probability of information loss higher than a preset probability threshold as target grids, extract high-frequency information from the target grids, and send the high-frequency information of the target grids and the compressed image to the target terminal. The restoration module is used to perform super-resolution reconstruction of the compressed image in the target terminal based on a pre-built generative adversarial network to obtain a reconstructed image, and to fuse the high-frequency information of the target grid with the reconstructed image to obtain a fused image.

Images