Method and apparatus for reconstructing image resolution, electronic device, and storage medium

Abstract

The present invention discloses a method and apparatus for reconstructing image resolution, an electronic device, and a storage medium, which relate to the field of fintech or other related fields. The reconstruction method includes: using a preset generator to extract image feature information of an image to be processed, and based on the image feature information, reconstructing the image to be processed into an image with a target resolution; using a first discriminator to downsample the image with the target resolution to obtain an original image; using a second discriminator to extract feature information of the original image, and based on the feature information, constructing a feature map; calculating a loss value of the feature map based on a third discriminator and a preset loss strategy, and when the loss value is less than a first preset threshold, determining that the resolution of the image with the target resolution meets the resolution required for image reconstruction. The present invention solves the technical problem in the related art that the resolution of the reconstructed image cannot be guaranteed to meet the business requirements.

Description

Technical Field

[0001] The present invention relates to the field of fintech, and in particular, to a method and apparatus for reconstructing image resolution, an electronic device, and a storage medium. Background Art

[0002] Image resolution refers to the number of pixels contained per inch, and as an index for evaluating image quality, the higher its value, the better the image quality. Currently, there are a large number of business scenarios in financial institutions that require images (for example, businesses that require face recognition, etc.), and there are different requirements for the clarity of the captured images. However, the clarity of the images captured by different devices is inconsistent, which easily leads to problems such as business failures.

[0003] In related technologies, image reconstruction methods include: an image super-resolution method based on a convolutional neural network. In this method, there are three convolutional layers, which respectively implement extracting feature information of a low-resolution image and mapping it to a high dimension, mapping of high-dimensional feature vectors, and image reconstruction; the EDSR (Enhanced Deep Super-Resolution Network) method, which can optimize the residual block structure, remove the normalization module, and deepen the network depth by stacking 65 convolutional layers, effectively improving the image reconstruction effect.

[0004] However, the image super-resolution method based on a convolutional neural network needs to first upsample a low-resolution image to the size of a high-resolution image by means of bicubic interpolation, and image feature information will be lost during preprocessing, which is not conducive to extracting the connection between pixels. Moreover, since only three convolutional layers are used, it will result in the inability to extract rich high-frequency information and texture information. Even when resources are sufficient or abundant, the quality of image reconstruction cannot be guaranteed. The EDSR method only simply stacks convolutional layers, which will cause a huge increase in model parameters, and this method cannot be used for image reconstruction on devices with limited resources. In addition, the image reconstructed by this method still has the problem of image blurring. Furthermore, existing image reconstruction methods only satisfy the resource limitations of a certain type of device and cannot meet the requirements of a reconstruction model for devices with different resources.

[0005] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0006] Embodiments of the present invention provide a method and apparatus for reconstructing image resolution, an electronic device, and a storage medium, so as to at least solve the technical problem in related technologies that the resolution of the reconstructed image cannot meet business requirements.

[0007] According to one aspect of the embodiments of the present invention, there is provided a method for reconstructing image resolution, including: using a preset generator to extract image feature information of an image to be processed, and based on the image feature information, reconstructing the image to be processed into a target resolution image; using a first discriminator to downsample the target resolution image to obtain an original image; using a second discriminator to extract feature information of the original image, and based on the feature information, constructing a feature map; based on a third discriminator and a preset loss strategy, calculating a loss value of the feature map, and when the loss value is less than a first preset threshold, determining that the resolution of the target resolution image reaches the resolution required for image reconstruction.

[0008] Optionally, the preset generator includes: a first module of the preset generator, a second module of the preset generator, and a third module of the preset generator. Among them, the first type of device reconstructs the image to be processed using the first module of the preset generator, the second type of device reconstructs the image to be processed using the first module and the second module of the preset generator, and the third type of device reconstructs the image to be processed using the first module, the second module, and the third module of the preset generator.

[0009] Optionally, before reconstructing the image to be processed into a target resolution image, it further includes: obtaining multiple sample images with different resolutions; training the first discriminator based on the sample images, and when a first loss function value of the first discriminator is less than a second preset threshold, determining that the training of the first discriminator is completed; when it is determined that the training of the first discriminator is completed, training the first module of the preset generator.

[0010] Optionally, the step of training the first module of the preset generator includes: training an initial generator based on a first loss function and the first discriminator to obtain the trained initial generator; using the trained initial generator to generate a resolution image, and based on the second discriminator, obtaining a corresponding downsampled feature map of the resolution image; using the first loss function to calculate a difference between a highest loss value and a lowest loss value of the downsampled feature map within a preset time period; when the difference is less than a third preset threshold, evaluating the downsampled feature map using a third loss function based on the third discriminator, and when an obtained evaluation difference is less than a fourth preset threshold, determining that the training of the first module of the preset generator is completed.

[0011] Optionally, the step of training the initial generator based on the first loss function and the first discriminator includes: using the first loss function to calculate a second loss function value of the initial generator; and when the second loss function value is less than a fifth preset threshold, using the first discriminator to train the initial generator to obtain the trained initial generator.

[0012] Optionally, the step of using the trained initial generator to generate a resolution image and obtaining a downsampled feature map corresponding to the resolution image based on the second discriminator includes: using the trained initial generator to generate a resolution image; downsampling the resolution image to obtain a downsampled image; using a second loss function to evaluate the downsampled image to obtain an evaluation value; and when the evaluation value is less than a sixth preset threshold, extracting the downsampled feature map of the downsampled image based on the second discriminator.

[0013] Optionally, after determining that the training of the first module of the preset generator is completed, it further includes: locking first training parameters of the first module of the preset generator; and combining the first training parameters and repeatedly using a training strategy of the first module of the preset generator to train a second module of the preset generator.

[0014] Optionally, after training the second module of the preset generator, it further includes: locking the first training parameters and second training parameters of the second module of the preset generator; and combining the first training parameters and the second training parameters and repeatedly using the training strategy of the first module of the preset generator to train a third module of the preset generator.

[0015] According to another aspect of the embodiments of the present invention, there is also provided an image resolution reconstruction device, including: a reconstruction unit configured to use a preset generator to extract image feature information of a to-be-processed image and reconstruct the to-be-processed image into a target resolution image based on the image feature information; a downsampling unit configured to use a first discriminator to downsample the target resolution image to obtain an original image; a construction unit configured to use a second discriminator to extract feature information of the original image and construct a feature map based on the feature information; and a calculation unit configured to calculate a loss value of the feature map based on a third discriminator and a preset loss strategy, and determine that the resolution of the target resolution image reaches a resolution required for image reconstruction when the loss value is less than a first preset threshold.

[0016] Optionally, the preset generator includes: a first module of the preset generator, a second module of the preset generator, and a third module of the preset generator. Among them, the first type of device uses the first module of the preset generator to reconstruct the image to be processed, the second type of device uses the first module and the second module of the preset generator to reconstruct the image to be processed, and the third type of device uses the first module, the second module, and the third module of the preset generator to reconstruct the image to be processed.

[0017] Optionally, the reconstruction device further includes: a first acquisition module, configured to acquire multiple sample images with different resolutions before reconstructing the image to be processed into a target resolution image; a first training module, configured to train the first discriminator based on the sample images, and determine that the first discriminator is trained when a first loss function value of the first discriminator is less than a second preset threshold; a second training module, configured to train the first module of the preset generator when it is determined that the first discriminator is trained.

[0018] Optionally, the second training module includes: a first training sub-module, configured to train an initial generator based on a first loss function and the first discriminator to obtain the trained initial generator; a first generation sub-module, configured to use the trained initial generator to generate a resolution image, and obtain a downsampled feature map corresponding to the resolution image based on the second discriminator; a first calculation sub-module, configured to use the first loss function to calculate a difference between a highest loss value and a lowest loss value of the downsampled feature map within a preset time period; a first determination sub-module, configured to, when the difference is less than a third preset threshold, evaluate the downsampled feature map based on a third discriminator using a third loss function, and determine that the first module of the preset generator is trained when an evaluation difference obtained is less than a fourth preset threshold.

[0019] Optionally, the first training sub-module includes: a second calculation sub-module, configured to calculate a second loss function value of the initial generator using the first loss function; a second training sub-module, configured to train the initial generator using the first discriminator when the second loss function value is less than a fifth preset threshold to obtain the trained initial generator.

[0020] Optionally, the first generation sub-module includes: a second generation sub-module for generating a resolution image by using the trained initial generator; a first downsampling sub-module for downsampling the resolution image to obtain a downsampled image; a first evaluation sub-module for evaluating the downsampled image by using a second loss function to obtain an evaluation value; and a first extraction sub-module for extracting a downsampled feature map of the downsampled image based on the second discriminator when the evaluation value is less than a sixth preset threshold.

[0021] Optionally, the reconstruction device further includes: a first locking module for locking first training parameters of the first module of the preset generator after determining that the training of the first module of the preset generator is completed; and a third training module for repeatedly using the training strategy of the first module of the preset generator to train the second module of the preset generator in combination with the first training parameters.

[0022] Optionally, the reconstruction device further includes: a second locking module for locking the first training parameters and second training parameters of the second module of the preset generator after training the second module of the preset generator; and a fourth training module for repeatedly using the training strategy of the first module of the preset generator to train the third module of the preset generator in combination with the first training parameters and the second training parameters.

[0023] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned method for reconstructing the image resolution.

[0024] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory. The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned method for reconstructing the image resolution.

[0025] In the present disclosure, a preset generator is adopted to extract the image feature information of the image to be processed, and based on the image feature information, the image to be processed is reconstructed into an image with a target resolution. A first discriminator is used to downsample the image with the target resolution to obtain the original image. A second discriminator is used to extract the feature information of the original image and construct a feature map based on the feature information. Based on a third discriminator and a preset loss strategy, the loss value of the feature map is calculated, and when the loss value is less than a first preset threshold, it is determined that the resolution of the image with the target resolution meets the resolution requirements for image reconstruction. In the present application, after reconstructing the image to be processed into an image with a target resolution using the preset generator, the image with the target resolution can be downsampled by the first discriminator, and the feature map of the downsampled original image can be extracted by the second discriminator. The loss value of the feature map is calculated using the third discriminator and the preset loss strategy, and when the loss value is less than the preset threshold, it is determined that the resolution of the image with the target resolution meets the resolution requirements for image reconstruction, which can ensure that the resolution of the reconstructed image meets the service requirements, and can also ensure that the resolutions of images collected using different types of devices all meet the service requirements, thereby solving the technical problem in the related art that the resolution of the reconstructed image cannot be guaranteed to meet the service requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 is a flowchart of an optional method for reconstructing the image resolution according to an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of an optional generator structure according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of an optional first-stage structure of the generator according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of an optional second-stage structure of the generator according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of an optional third-stage structure of the generator according to an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of an optional discriminator structure according to an embodiment of the present invention;

[0033] Figure 7Schematic diagram of an optional image resolution reconstruction device according to an embodiment of the present invention;

[0034] Figure 8 Hardware structure block diagram of an electronic device (or mobile device) for an image resolution reconstruction method according to an embodiment of the present invention. Detailed implementation manners

[0035] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] To facilitate the understanding of the present invention by those skilled in the art, the following explains some terms or nouns involved in each embodiment of the present invention:

[0038] Image resolution: The number of pixels contained per inch.

[0039] Low-resolution image: An image with low image resolution, blurred or even distorted.

[0040] High-resolution image: An image with high image resolution, rich in colors and without blur.

[0041] Super-resolution image: An image reconstructed from a low-resolution image and having the same size as the original high-resolution image.

[0042] It should be noted that the method and apparatus for reconstructing image resolution in the present disclosure can be used in the field of fintech when reconstructing image resolution, and can also be used in any field other than the field of fintech when reconstructing image resolution. The application field of the method and apparatus for reconstructing image resolution in the present disclosure is not limited.

[0043] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set between the present system and relevant users or institutions. Before obtaining relevant information, a request for acquisition needs to be sent to the aforementioned users or institutions through the interface, and after receiving the consent information feedback from the aforementioned users or institutions, the relevant information is obtained.

[0044] The following embodiments of the present invention can be applied to various systems / applications / devices for reconstructing image resolution. The present invention provides a multi-stage single-image super-resolution algorithm, and an algorithm for obtaining reconstructed images that meet the requirements of different types of devices can be obtained through training. Different stages of the generator reconstructed in the present invention have different numbers of parameters. The number of parameters in the first stage is relatively low, which can meet the requirements of resource-constrained devices (i.e., devices that can process a small amount of data); the second stage can use a residual structure to deepen the network depth, which can meet the requirements of devices with relatively rich resources (i.e., devices that can process more data); the third stage can use a dense structure to enhance the feature extraction ability, which can meet the requirements of devices with rich resources (i.e., devices that can process a large amount of data). At the same time, the discriminator structure of the present invention is segmented, and different stages of the generator are evaluated respectively, which can achieve comprehensive, multi-faceted comparison and effectively improve the quality of image reconstruction.

[0045] The present invention will be described in detail below in conjunction with each embodiment.

[0046] Embodiment 1

[0047] According to an embodiment of the present invention, an embodiment of a method for reconstructing image resolution is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0048] Figure 1 is a flowchart of an optional method for reconstructing image resolution according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0049] Step S101: Use a preset generator to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into an image with a target resolution.

[0050] Step S102: Use a first discriminator to downsample the image with the target resolution to obtain the original image.

[0051] Step S103: Use a second discriminator to extract the feature information of the original image, and based on the feature information, construct a feature map.

[0052] Step S104: Based on a third discriminator and a preset loss strategy, calculate the loss value of the feature map, and when the loss value is less than a first preset threshold, determine that the resolution of the image with the target resolution reaches the resolution required for image reconstruction.

[0053] Through the above steps, a preset generator can be used to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into an image with a target resolution. Then use a first discriminator to downsample the image with the target resolution to obtain the original image, use a second discriminator to extract the feature information of the original image, and based on the feature information, construct a feature map. Based on a third discriminator and a preset loss strategy, calculate the loss value of the feature map, and when the loss value is less than a first preset threshold, determine that the resolution of the image with the target resolution reaches the resolution required for image reconstruction. In the embodiments of the present invention, after reconstructing the image to be processed into an image with a target resolution using a preset generator, the image with the target resolution can be downsampled by a first discriminator, and the feature map of the downsampled original image can be extracted by a second discriminator. Use a third discriminator and a preset loss strategy to calculate the loss value of the feature map, and when the loss value is less than a preset threshold, determine that the resolution of the image with the target resolution reaches the resolution required for image reconstruction, which can ensure that the resolution of the reconstructed image meets the business requirements, and can also ensure that the resolutions of images collected by different types of devices also meet the business requirements, thereby solving the technical problem in the related art that the resolution of the reconstructed image cannot be guaranteed to meet the business requirements.

[0054] The embodiments of the present invention will be described in detail below in combination with the above steps.

[0055] In an embodiment of the present invention, optionally, the preset generator includes: a first module of the preset generator, a second module of the preset generator, and a third module of the preset generator. Among them, the first type of device uses the first module of the preset generator to reconstruct the image to be processed, the second type of device uses the first module and the second module of the preset generator to reconstruct the image to be processed, and the third type of device uses the first module, the second module, and the third module of the preset generator to reconstruct the image to be processed.

[0056] In an embodiment of the present invention, a generative adversarial network structure (e.g., the RaGAN (Relative Average) structure) can be used to reconstruct a generator (i.e., a preset generator). The preset generator in this embodiment is used to reconstruct a low-resolution image into a high-resolution image. To meet the requirements of different types of devices, the generator can be divided into three stages (i.e., the preset generator can be divided into three modules, including: the first module of the preset generator, the second module of the preset generator, and the third module of the preset generator). By controlling the number of parameters and the amount of computation of each stage (i.e., each module of the preset generator), different device requirements can be met. For lightweight devices (i.e., the first type of devices that can process a small amount of data), only the first stage of the generator can be used to reconstruct the image (i.e., the first type of devices use the first module of the preset generator to reconstruct the image to be processed). For devices with relatively rich resources (i.e., the second type of devices that can process more data), the second stage and the first stage of the generator can be used to jointly reconstruct the image (i.e., the second type of devices use the first module and the second module of the preset generator to reconstruct the image to be processed). For devices with rich resources (i.e., the third type of devices that can process a large amount of data), the third stage can be selected, and the second stage and the first stage can be used to jointly reconstruct a high-quality image (i.e., the third type of devices use the first module, the second module, and the third module of the preset generator to reconstruct the image to be processed).

[0057] Figure 2 is a schematic diagram of an optional generator structure according to an embodiment of the present invention, such as Figure 2 shown, the generator can be divided into three stages. In the first stage, the low-resolution image can be reconstructed into a high-resolution image generated in the first stage through a reconstruction unit. In the second stage, the low-resolution image can be reconstructed into a high-resolution image generated in the second stage through a reconstruction unit. In the third stage, the low-resolution image can be reconstructed into a high-resolution image generated in the third stage through a reconstruction unit.

[0058] In this embodiment, the first stage is for devices with limited resources (i.e., the first type of devices). Figure 3 is a schematic diagram of an optional first-stage structure of the generator according to an embodiment of the present invention, such as Figure 3 shown, in order to reduce the number of parameters and the amount of computation, a residual block structure of first dilating and then compressing can be used. Among them, the residual block structure can be composed of two 3*3 convolutional layers (i.e., two CONV convolutional layers) and a RELU activation function. The residual block structures can be connected through residuals (denoted by +), and all residual block structures can be connected through a 1X1 CONV convolutional layer. This 1X1 CONV convolutional layer can be used to increase the dimension of convolutional layers of other dimensions.

[0059] In this embodiment, the second stage is for devices with relatively rich resources (i.e., the second type of device). Figure 4 It is a schematic diagram of an optional second-stage structure of a generator according to an embodiment of the present invention. As Figure 4 shown, this second-stage structure uses a residual structure (the same as connecting all residual block structures through a 1X1CONV convolutional layer in the first-stage structure). By increasing the network depth, a large amount of image feature information can be extracted, thereby reconstructing a high-quality image.

[0060] In this embodiment, the third stage is for devices with rich resources (i.e., the third type of device). Figure 5 It is a schematic diagram of an optional third-stage structure of a generator according to an embodiment of the present invention. As Figure 5 shown, this third-stage structure uses a dense network structure (connecting multiple residual structures in the second stage). Through the dense network structure, a large amount of image feature information can be obtained. Each dense block will merge the feature information of each convolutional layer inside. It is screened by the last 1*1 convolutional layer (for example, 1X1CONV convolutional layer). Finally, the feature information extracted by each dense block is screened again by a 3*3 convolutional layer. By screening the effective feature information twice, a high-quality image can be reconstructed.

[0061] Optionally, before reconstructing the image to be processed into an image with a target resolution, it further includes: obtaining multiple sample images with different resolutions; training a first discriminator based on the sample images, and determining that the training of the first discriminator is completed when the first loss function value of the first discriminator is less than a second preset threshold; and training the first module of a preset generator when it is determined that the training of the first discriminator is completed.

[0062] In an embodiment of the present invention, the discriminator can be used to reconstruct a high-resolution image into a low-resolution image and score the reconstruction quality. The discriminator can be set into three stages (in this embodiment, the first stage of the discriminator corresponds to the first discriminator, the second stage of the discriminator corresponds to the second discriminator, and the third stage of the discriminator corresponds to the third discriminator). The first loss function (such as, L1 loss function), the second loss function (such as, L2 loss function), and the third loss function (such as, RaGAN loss function) are respectively used to discriminate the reconstructed image and the real image. Figure 6 It is a schematic diagram of an optional discriminator structure according to an embodiment of the present invention. As Figure 6 shown, a high / ultra-high resolution image can be judged by the first stage for the first-stage loss, judged by the second stage for the second-stage loss, and judged by the third stage for the third-stage loss.

[0063] In an embodiment of the present invention, the discriminator in the first stage can be trained using a high-resolution image and a low-resolution image (i.e., sample images with different resolutions) (i.e., training the first discriminator through multiple acquired sample images with different resolutions). When the first loss function value of the first discriminator is less than a second preset threshold (for example, the second preset threshold can be set to 6), it is determined that the training of the first discriminator is completed, so as to ensure that the discriminator in the first stage can be downsampled to a low-resolution image. And when it is determined that the training of the first discriminator is completed, the training of the first module of the preset generator is started.

[0064] Optionally, the step of training the first module of the preset generator includes: training the initial generator based on the first loss function and the first discriminator to obtain the trained initial generator; using the trained initial generator to generate a resolution image, and based on the second discriminator, obtaining the corresponding downsampled feature map of the resolution image; using the first loss function to calculate the difference between the highest loss value and the lowest loss value of the downsampled feature map within a preset time period; when the difference is less than a third preset threshold, evaluating the downsampled feature map based on the third discriminator using a third loss function, and when the obtained evaluation difference is less than a fourth preset threshold, determining that the training of the first module of the preset generator is completed.

[0065] In an embodiment of the present invention, the steps of training the first module of the preset generator are as follows: the initial generator can be trained using the first loss function (such as, L1 loss function) and the first discriminator. Using the trained initial generator to generate a resolution image, and based on the second discriminator, obtaining the corresponding downsampled feature map of the resolution image (that is, using the discriminator in the second stage to extract the feature map of the image downsampled by the discriminator in the first stage (i.e., the image obtained by downsampling the resolution image), to obtain the downsampled feature map). The difference between the highest loss value and the lowest loss value of the downsampled feature map within a preset time period (for example, the time period for training the first module of the preset generator) can be calculated through the first loss function. When the difference is less than a third preset threshold (for example, 0.3), the downsampled feature map can be evaluated based on the third discriminator using a third loss function (such as, RaGAN loss function). When the obtained evaluation difference is less than a fourth preset threshold (such as, 0.2), it is determined that the training of the first module of the preset generator is completed (i.e., the first stage of the generator is trained and can be used for resource-constrained devices).

[0066] Optionally, the step of training the initial generator based on the first loss function and the first discriminator includes: using the first loss function to calculate the second loss function value of the initial generator; when the second loss function value is less than a fifth preset threshold, training the initial generator using the first discriminator to obtain the trained initial generator.

[0067] In an embodiment of the present invention, a first loss function may be adopted to initially train an initial generator. When the second loss function value of the initial generator is calculated to be less than a fifth preset threshold (e.g., 5), the initial training is completed. Then, a first discriminator may be adopted to continue training the initial generator to obtain a trained initial generator.

[0068] Optionally, the steps of generating a resolution image using the trained initial generator and obtaining a downsampled feature map corresponding to the resolution image based on a second discriminator include: generating a resolution image using the trained initial generator; downsampling the resolution image to obtain a downsampled image; evaluating the downsampled image using a second loss function to obtain an evaluation value; and extracting the downsampled feature map of the downsampled image based on the second discriminator when the evaluation value is less than a sixth preset threshold.

[0069] In an embodiment of the present invention, a resolution image may be generated using the trained initial generator. Then, the resolution image may be downsampled to obtain a downsampled image. A second loss function (e.g., L2 loss function) may be used to evaluate the downsampled image to obtain an evaluation value. When the evaluation value is less than a sixth preset threshold (e.g., it may be set to 10), a second discriminator may be adopted to extract the downsampled feature map of the downsampled image (wherein, the second discriminator may use the loss function of the 13th convolutional layer in the VGG16 network and extract the downsampled feature map before the activation function).

[0070] Optionally, after it is determined that the training of the first module of the preset generator is completed, it further includes: locking the first training parameters of the first module of the preset generator; and repeating the training strategy of the first module of the preset generator in combination with the first training parameters to train the second module of the preset generator.

[0071] In an embodiment of the present invention, before the second stage of training the generator, the first training parameters obtained in the first stage of the generator may be locked first. Then, in combination with the first training parameters, the training strategy of the first module of the preset generator may be repeated to train the second module of the preset generator. The discriminator used in the second stage may be the discriminator used in the first stage of training the generator. The limit value of the loss value during the training process of the second stage remains unchanged. When the training is completed, the first stage and the second stage of the generator can be used in devices with relatively rich resources.

[0072] Optionally, after training the second module of the preset generator, it further includes: locking the first training parameters and the second training parameters of the second module of the preset generator; and repeating the training strategy of the first module of the preset generator in combination with the first training parameters and the second training parameters to train the third module of the preset generator.

[0073] In an embodiment of the present invention, before the third stage of training the generator, the first training parameters obtained in the first stage of the generator and the second training parameters obtained in the second stage can be locked first. Combining the first training parameters and the second training parameters, the training strategy of the first module of the preset generator is repeatedly used to train the third module of the preset generator. The discriminator used in the third stage can be the discriminator in the second stage of training the generator. The limit value of the loss value in the training process of this third stage remains unchanged. When the training is completed, the first stage, the second stage, and the third stage of the generator can be used in devices with rich resources. Thus, the training of the generator is completed.

[0074] Step S101: Use a preset generator to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into an image with a target resolution.

[0075] In an embodiment of the present invention, a trained preset generator can be used to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into an image with a target resolution (the target resolution image refers to a high-resolution image that meets the service requirements).

[0076] Step S102: Use a first discriminator to downsample the target resolution image to obtain the original image.

[0077] In an embodiment of the present invention, a first discriminator can be used to gradually downsample the target resolution image to the size of the original low-resolution image (i.e., the original image).

[0078] Step S103: Use a second discriminator to extract the feature information of the original image, and based on the feature information, construct a feature map.

[0079] In an embodiment of the present invention, a second discriminator can be used to extract the feature information of the original image (for example, the VGG16 network can be used, and its first 13 layers can be selected to extract the feature information of the original image obtained in the first stage of the discriminator), and based on the feature information, construct a feature map.

[0080] Step S104: Based on a third discriminator and a preset loss strategy, calculate the loss value of the feature map, and when the loss value is less than a first preset threshold, determine that the resolution of the target resolution image reaches the resolution required for image reconstruction.

[0081] In an embodiment of the present invention, the loss value of the feature map can be calculated through a third discriminator and a preset loss strategy (for example, calculate the loss value through the VGG19 network), and when the loss value is less than a first preset threshold (which can be specifically set according to the actual situation), determine that the resolution of the target resolution image reaches the resolution required for image reconstruction.

[0082] The following is a detailed description in combination with another optional specific implementation manner.

[0083] This embodiment provides a multi-stage single-image super-resolution reconstruction algorithm, which can reconstruct a low-resolution image into a high-resolution image and can be used to reconstruct images of types such as blurred and noisy images. For example, in this embodiment, an adversarial neural network structure (such as, RaGAN structure) can be used to reconstruct the generator and discriminator. Among them, the generator is used to reconstruct a low-resolution image into a high-resolution image, and the discriminator is used to reconstruct a high-resolution image into a low-resolution image, and the overall structure can be set to a spindle shape.

[0084] In this embodiment, the discriminator can be set in three stages, and different types of loss functions (such as, L2, L1, and RaGAN loss functions) are used to discriminate between the reconstructed image and the real image, and limit amounts are set for the loss functions respectively. When the loss value is higher than the limit amount, the generator will be used to reconstruct the image again until the limit amount is satisfied.

[0085] The specific structure of the generator in this embodiment is as follows:

[0086] The generator is used to reconstruct a low-resolution image into a high-resolution image. In order to meet the requirements of different types of devices, the generator can be divided into three stages. By controlling the number of parameters and the amount of calculation in each stage, it is ensured to meet the requirements of different types of devices. For lightweight devices, only the first stage of the generator can be used to reconstruct the image. For devices with more resources, the second stage and the first stage can be used to jointly reconstruct the image. For devices with rich resources, the third stage can be selected, and the second stage and the first stage can be used to jointly reconstruct a high-quality image.

[0087] In this embodiment, the first stage is for devices with limited resources. In order to reduce the number of parameters and the amount of calculation, a residual block structure of first expanding and then compressing can be used. This residual block can be composed of two 3*3 convolutional layers and a Relu activation function through residual connection.

[0088] The second stage is for devices with relatively rich resources. This structure can use a residual structure to extract a large amount of image feature information by increasing the network depth, so as to reconstruct a high-quality image.

[0089] The third stage is for devices with rich resources. This structure can use a dense network structure. Through the dense network structure, a large amount of image feature information can be obtained. Each dense block will merge the feature information of each convolutional layer inside, and the last 1*1 convolutional layer will perform screening. Finally, the feature information extracted by each dense block will be screened again by a 3*3 convolutional layer. By screening the effective feature information twice, a higher-quality image can be reconstructed.

[0090] The discriminator structure in this embodiment is as follows:

[0091] The discriminator is used to reconstruct high-resolution images into low-resolution images and to score the reconstructed images. The discriminator can be set into three stages, using different loss functions to distinguish the reconstructed images from the real images.

[0092] In this embodiment, the discriminator can use 10 layers of 5*5 convolutional layers in the first stage, and gradually downsample to the original low-resolution image size. In the second stage, the VGG16 network can be used to select its first 13 layers to extract the feature map of the image obtained in the first stage. In the third stage, the feature map extracted in the second stage can be discriminated through the VGG19 network to determine the probability that the image is a true high-resolution image.

[0093] The training method of the generator and the discriminator in this embodiment is as follows:

[0094] (1) Use high-resolution images and low-resolution images to train the first stage of the discriminator to ensure that the first stage of the discriminator can be downsampled to low-resolution images. When the calculated loss function value is within the preset threshold, the discriminator training is completed.

[0095] (2) Use the first loss function (e.g., L1 loss function) to train the first stage of the generator until the loss function value drops below a preset threshold, otherwise continue training.

[0096] (3) The first stage of the discriminator is then used to train the first stage of the generator, and the second loss function (e.g., L2 loss function) is used to evaluate the images sampled from the high-resolution image and the super-resolution image. The loss value should not be higher than a preset threshold, otherwise continue training.

[0097] (4) Use the second stage of the discriminator to extract feature maps from the images downsampled by the first stage of the discriminator. The loss function that can be used is the 13th convolutional layer in the VGG16 network, and extract feature maps before the activation function. After that, it is evaluated by the first loss function. During the training process, when the loss value fluctuation range is within the preset threshold (the highest loss value minus the lowest loss value), stop training and proceed to the next step, otherwise continue training.

[0098] (5) The third stage of the discriminator is used to perform probability evaluation on the feature map of the second stage of the discriminator, and the third loss function (e.g., RaGAN loss function) is used for evaluation. If the loss function fluctuates within the preset threshold, the training is completed. At this point, the first stage of the generator training is completed and can be used for resource-constrained devices.

[0099] (6) In the second stage of training the generator, the parameters obtained in the first stage of the generator can be locked first, and then steps (2) to (5) are repeated. The discriminator used can be the discriminator in the first stage of training the generator. At this time, the loss value limit remains unchanged. After training is completed, the first and second stages of the generator can be used in devices with relatively rich resources.

[0100] (7) In the third stage of training the generator, the parameters obtained in the first and second stages of the generator can be locked first, and then steps (2) to (5) are repeated. The discriminator used can be the discriminator in the second stage of training the generator. At this time, the loss value limit remains unchanged. After training is completed, the first, second, and third stages of the generator can be used in devices with rich resources. Thus, the training of the generator is completed.

[0101] In the embodiments of the present invention, in order to meet the requirements of different types of devices, a multi-stage single-image super-resolution reconstruction algorithm is proposed. Through this algorithm, an image reconstruction method that meets different types of devices can be trained, which can ensure that the images collected by different types of devices meet the requirements of different business scenarios.

[0102] Embodiment 2

[0103] An image resolution reconstruction device provided in this embodiment includes a plurality of implementation units, and each implementation unit corresponds to each implementation step in Embodiment 1 above.

[0104] Figure 7 It is a schematic diagram of an optional image resolution reconstruction device according to an embodiment of the present invention. As Figure 7 shown, the reconstruction device may include: a reconstruction unit 70, a downsampling unit 71, a construction unit 72, and a calculation unit 73, where

[0105] The reconstruction unit 70 is configured to use a preset generator to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into an image with a target resolution;

[0106] The downsampling unit 71 is configured to use a first discriminator to downsample the image with the target resolution to obtain the original image;

[0107] The construction unit 72 is configured to use a second discriminator to extract the feature information of the original image, and based on the feature information, construct a feature map;

[0108] The calculation unit 73 is configured to calculate the loss value of the feature map based on a third discriminator and a preset loss strategy, and determine that the resolution of the image with the target resolution reaches the resolution required for image reconstruction when the loss value is less than a first preset threshold.

[0109] The above reconstruction device can use a preset generator through the reconstruction unit 70 to extract the image feature information of the image to be processed, and based on the image feature information, reconstruct the image to be processed into a target resolution image. The downsampling unit 71 uses a first discriminator to downsample the target resolution image to obtain the original image. The construction unit 72 uses a second discriminator to extract the feature information of the original image and construct a feature map based on the feature information. The calculation unit 73 calculates the loss value of the feature map based on the third discriminator and a preset loss strategy, and determines that the resolution of the target resolution image meets the resolution requirements for image reconstruction when the loss value is less than the first preset threshold. In the embodiment of the present invention, after reconstructing the image to be processed into a target resolution image using the preset generator, the first discriminator can be used to downsample the target resolution image, and the second discriminator can be used to extract the feature map of the downsampled original image. The third discriminator and the preset loss strategy are used to calculate the loss value of the feature map, and when the loss value is less than the preset threshold, it is determined that the resolution of the target resolution image meets the resolution requirements for image reconstruction, which can ensure that the resolution of the reconstructed image meets the business requirements, and can also ensure that the resolutions of images collected by different types of devices also meet the business requirements, thus solving the technical problem in the related art that the resolution of the reconstructed image cannot be guaranteed to meet the business requirements.

[0110] Optionally, the preset generator includes: the first module of the preset generator, the second module of the preset generator, and the third module of the preset generator. Among them, the first type of device uses the first module of the preset generator to reconstruct the image to be processed, the second type of device uses the first module and the second module of the preset generator to reconstruct the image to be processed, and the third type of device uses the first module, the second module, and the third module of the preset generator to reconstruct the image to be processed.

[0111] Optionally, the reconstruction device further includes: a first acquisition module, configured to acquire multiple sample images with different resolutions before reconstructing the image to be processed into a target resolution image; a first training module, configured to train the first discriminator based on the sample images, and determine that the first discriminator is trained when the first loss function value of the first discriminator is less than the second preset threshold; and a second training module, configured to train the first module of the preset generator when it is determined that the first discriminator is trained.

[0112] Optionally, the second training module includes: a first training sub-module, configured to train an initial generator based on a first loss function and a first discriminator to obtain a trained initial generator; a first generation sub-module, configured to use the trained initial generator to generate a resolution image, and based on a second discriminator, obtain a downsampled feature map corresponding to the resolution image; a first calculation sub-module, configured to use the first loss function to calculate a difference between a highest loss value and a lowest loss value of the downsampled feature map within a preset time period; a first determination sub-module, configured to, when the difference is less than a third preset threshold, evaluate the downsampled feature map based on a third discriminator using a third loss function, and when an obtained evaluation difference is less than a fourth preset threshold, determine that the training of the first module of the preset generator is completed.

[0113] Optionally, the first training sub-module includes: a second calculation sub-module, configured to use the first loss function to calculate a second loss function value of the initial generator; a second training sub-module, configured to, when the second loss function value is less than a fifth preset threshold, use the first discriminator to train the initial generator to obtain a trained initial generator.

[0114] Optionally, the first generation sub-module includes: a second generation sub-module, configured to use the trained initial generator to generate a resolution image; a first downsampling sub-module, configured to downsample the resolution image to obtain a downsampled image; a first evaluation sub-module, configured to use the second loss function to evaluate the downsampled image to obtain an evaluation value; a first extraction sub-module, configured to, when the evaluation value is less than a sixth preset threshold, extract a downsampled feature map of the downsampled image based on the second discriminator.

[0115] Optionally, the reconstruction device further includes: a first locking module, configured to lock first training parameters of the first module of the preset generator after determining that the training of the first module of the preset generator is completed; a third training module, configured to repeat the training strategy of the first module of the preset generator in combination with the first training parameters to train a second module of the preset generator.

[0116] Optionally, the reconstruction device further includes: a second locking module, configured to lock the first training parameters and second training parameters of the second module of the preset generator after training the second module of the preset generator; a fourth training module, configured to repeat the training strategy of the first module of the preset generator in combination with the first training parameters and the second training parameters to train a third module of the preset generator.

[0117] The above-mentioned reconstruction device may further include a processor and a memory. The above-mentioned reconstruction unit 70, downsampling unit 71, construction unit 72, calculation unit 73, etc. are all stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions.

[0118] The above-mentioned processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and by adjusting the kernel parameters, the resolution of the target resolution image is determined to reach the resolution required for image reconstruction.

[0119] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). The memory includes at least one memory chip.

[0120] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program initialized with the following method steps: using a preset generator, extracting the image feature information of the image to be processed, and based on the image feature information, reconstructing the image to be processed into a target resolution image; using a first discriminator to downsample the target resolution image to obtain the original image; using a second discriminator to extract the feature information of the original image, and based on the feature information, constructing a feature map; based on a third discriminator and a preset loss strategy, calculating the loss value of the feature map, and when the loss value is less than a first preset threshold, determining that the resolution of the target resolution image reaches the resolution required for image reconstruction.

[0121] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned image resolution reconstruction method.

[0122] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, which includes one or more processors and a memory. The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned image resolution reconstruction method.

[0123] Figure 8 It is a hardware structure block diagram of an electronic device (or mobile device) for an image resolution reconstruction method according to an embodiment of the present invention. As Figure 8As shown, the electronic device may include one or more processors 102 (illustrated as 102a, 102b, ……, 102n in the figure), where the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, and a memory 104 for storing data. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 8 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components than those Figure 8 shown in, or have a different configuration from that Figure 8 shown.

[0124] The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0125] In the above-mentioned embodiments of the present invention, the descriptions of the respective embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0126] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units may be a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other may be through some interfaces, and the indirect couplings or communication connections of the units or modules may be in electrical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, in each embodiment of the present invention, the functional units may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0129] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0130] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for reconstructing image resolution, characterized in that, it includes: Using a preset generator to extract the image feature information of the image to be processed, and based on the image feature information, reconstructing the image to be processed into an image with a target resolution. The preset generator includes: the first module of the preset generator, the second module of the preset generator, and the third module of the preset generator. Among them, the first type of device reconstructs the image to be processed using the first module of the preset generator, the second type of device reconstructs the image to be processed using the first module and the second module of the preset generator, and the third type of device reconstructs the image to be processed using the first module, the second module, and the third module of the preset generator; Using a first discriminator to downsample the target resolution image to obtain an original image; Using a second discriminator to extract the feature information of the original image and construct a feature map based on the feature information; Based on a third discriminator and a preset loss strategy, calculating the loss value of the feature map, and when the loss value is less than a first preset threshold, determining that the resolution of the target resolution image reaches the resolution required for image reconstruction.

2. The reconstruction method according to claim 1, characterized in that, Before reconstructing the image to be processed into an image with a target resolution, it further includes: Obtaining multiple sample images with different resolutions; Training the first discriminator based on the sample images, and when the first loss function value of the first discriminator is less than a second preset threshold, determining that the training of the first discriminator is completed; When it is determined that the training of the first discriminator is completed, training the first module of the preset generator.

3. The reconstruction method according to claim 2, characterized in that, The step of training the first module of the preset generator includes: Training an initial generator based on a first loss function and the first discriminator to obtain the trained initial generator; Using the trained initial generator to generate a resolution image, and based on the second discriminator, obtaining a corresponding downsampled feature map of the resolution image; Using the first loss function to calculate the difference between the highest loss value and the lowest loss value of the downsampled feature map within a preset time period; When the difference is less than a third preset threshold, evaluating the downsampled feature map using a third loss function based on the third discriminator, and when the obtained evaluation difference is less than a fourth preset threshold, determining that the training of the first module of the preset generator is completed.

4. The reconstruction method according to claim 3, characterized in that, The step of training the initial generator based on a first loss function and the first discriminator includes: Using the first loss function to calculate the second loss function value of the initial generator; When the second loss function value is less than a fifth preset threshold, using the first discriminator to train the initial generator to obtain the trained initial generator.

5. The reconstruction method according to claim 3, characterized in that, The steps of using the trained initial generator to generate a resolution image and obtaining the downsampled feature map corresponding to the resolution image based on the second discriminator include: Using the trained initial generator to generate a resolution image; Downsampling the resolution image to obtain a downsampled image; Evaluating the downsampled image using a second loss function to obtain an evaluation value; When the evaluation value is less than a sixth preset threshold, extracting the downsampled feature map of the downsampled image based on the second discriminator.

6. The reconstruction method according to claim 3, wherein, after determining that the training of the first module of the preset generator is completed, it further includes: Locking the first training parameters of the first module of the preset generator; Combining the first training parameters and repeatedly adopting the training strategy of the first module of the preset generator to train the second module of the preset generator.

7. The reconstruction method according to claim 6, wherein, after training the second module of the preset generator, it further includes: Locking the first training parameters and the second training parameters of the second module of the preset generator; Combining the first training parameters and the second training parameters and repeatedly adopting the training strategy of the first module of the preset generator to train the third module of the preset generator.

8. An image resolution reconstruction device, wherein, it includes: A reconstruction unit for using a preset generator to extract image feature information of a to-be-processed image and reconstructing the to-be-processed image into a target resolution image based on the image feature information. The preset generator includes: the first module of the preset generator, the second module of the preset generator, and the third module of the preset generator. Among them, the first type of device reconstructs the to-be-processed image using the first module of the preset generator, the second type of device reconstructs the to-be-processed image using the first module and the second module of the preset generator, and the third type of device reconstructs the to-be-processed image using the first module, the second module, and the third module of the preset generator; A downsampling unit for downsampling the target resolution image using a first discriminator to obtain an original image; A construction unit for using a second discriminator to extract the feature information of the original image and constructing a feature map based on the feature information; A calculation unit for calculating the loss value of the feature map based on a third discriminator and a preset loss strategy, and determining that the resolution of the target resolution image reaches the resolution required for image reconstruction when the loss value is less than a first preset threshold.

9. A computer-readable storage medium, wherein, the computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the image resolution reconstruction method according to any one of claims 1 to 7.

10. An electronic device, wherein, Comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for reconstructing the image resolution according to any one of claims 1 to 7.

Images