Image and video super-resolution and super-definition reconstruction system and method

Abstract

The present invention discloses a system and method for super-resolution and super-definition reconstruction of images and videos. The reconstruction system includes a feature extraction module, a super-resolution and super-definition reconstruction module, and at least one feature processing module; the feature extraction module is used to extract the set feature layer of the image and video to be processed; the super-resolution and super-definition reconstruction module is used to perform feature sampling and feature extraction to obtain the reconstructed super-resolution and super-definition image and video; each specific feature processing module is used to obtain the feature information output corresponding to the image and video to be processed according to different computer vision task settings. The present invention can execute multiple specific feature processing modules in parallel according to different computer vision tasks to obtain a variety of specific feature information outputs, and finally combine the super-resolution and super-definition reconstruction modules to form accurate high-resolution and high-definition images, thereby effectively improving the performance of subsequent computer vision tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image and video processing, and relates to an image and video processing system, and in particular to an image and video super-resolution and super-definition reconstruction system and method applied to specific computer vision tasks. Background Art

[0002] A deep neural network is a complex mathematical model. The input data is used to obtain the corresponding output data through the deep neural network. The loss function is constructed by the difference between the output data and the labeled data. The loss function calculates the gradient of the parameters in the deep neural network. The parameters in the deep neural network are updated through gradient back propagation. By continuously updating the parameters, the difference between the output data and the labeled data is continuously reduced. The input data and the labeled data constitute the training data required for deep neural network training. The performance of the deep neural network is related to the structure of the neural network and the training data. Deep neural networks have achieved performance superior to traditional methods in the fields of image, speech, natural language processing, etc., and have been widely used.

[0003] Image and video super-resolution reconstruction refers to reconstructing the corresponding high-resolution image and video from the observed low-resolution image. With the rapid development of deep learning technology, the image and video super-resolution reconstruction method based on deep neural network is currently the best image and video super-resolution reconstruction method.

[0004] The existing image and video super-resolution reconstruction system based on deep neural network usually includes two major modules: a feature extraction module 21 and a super-resolution image reconstruction module 31, which obtains a reconstructed super-resolution image video 41. During training, the image and video loss function 51 between the reconstructed super-resolution image video 41 and the high-resolution image video corresponding to the image video 11 to be processed is calculated, and the image and video training gradient back propagation is performed based on the image and video loss function 51, and the parameters of the feature extraction module 21 and the super-resolution image reconstruction module 31 are updated, so that the feature extraction module 21 can extract the image and video information of the image video 11 to be processed. The overall Figure 1 The existing image and video super-resolution reconstruction method based on deep neural network achieves good visual effects after image and video reconstruction, but there is a problem of low credibility of the reconstructed image and video.

[0005] The above-mentioned existing image and video super-resolution reconstruction is essentially an ill-posed problem, that is, for the original low-resolution and low-definition image and video, there are multiple high-resolution and high-definition images corresponding to it using the existing method. This ill-posed problem will cause the reconstructed super-resolution image and video information to change, resulting in the reconstructed super-resolution image and video having low credibility and cannot be used in computer vision tasks (text recognition, face recognition, ADAS, gesture recognition, etc.).

[0006] In view of this, there is an urgent need to design a new image and video reconstruction method to overcome at least some of the above-mentioned defects of the existing image and video reconstruction method. Summary of the invention

[0007] The present invention provides an image and video super-resolution and super-definition reconstruction system and method, which can be centered around different computer vision tasks, and can configure and fuse multiple features, ultimately improving the image and video resolution and clarity while ensuring the credibility of the reconstructed images and videos, and can then be used by computer vision tasks, thereby improving the actual performance of existing computer vision tasks.

[0008] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0009] An image and video super-resolution and super-definition reconstruction system, the reconstruction system comprising:

[0010] A feature extraction module is used to extract a set feature layer corresponding to the image or video to be processed;

[0011] A super-resolution and super-definition reconstruction module, connected to the feature extraction module, for performing feature sampling and feature extraction to obtain a reconstructed super-resolution image; and

[0012] At least one specific feature processing module, each specific feature processing module is respectively connected to the feature extraction module to perform feature processing on the feature layer, and obtains feature information output corresponding to the image video to be processed according to different computer vision task settings.

[0013] As an embodiment of the present invention, the system further comprises:

[0014] An image and video loss function acquisition module, used to calculate the image and video loss function according to the super-resolution and super-definition image and video acquired by the super-resolution and super-definition reconstruction module;

[0015] At least one feature information loss function acquisition module, each feature information loss function acquisition module is connected to a corresponding specific feature processing module, and is used to calculate a corresponding feature information loss function according to the feature information output acquired by the specific feature processing module;

[0016] The loss function fusion module is used to fuse the image and video loss function obtained by the image and video loss function acquisition module and the feature information loss function obtained by each feature information loss function acquisition module to obtain a fusion loss function; and use the fusion loss function for back propagation to train the image and video super-resolution and super-definition reconstruction network.

[0017] As an embodiment of the present invention, at least one corresponding specific feature processing module is designed according to different computer vision task requirements to perform specific feature processing on the feature layer to obtain specific feature information of the image video to be processed related to the computer vision task requirements.

[0018] As an implementation manner of the present invention, each feature information loss function acquisition module is used to acquire a feature information loss function by calculating the difference between the feature information output and the expected feature information.

[0019] As an implementation mode of the present invention, the feature extraction module is used to obtain a high-level feature layer of the image or video to be processed, and the high-level feature layer contains deep feature information of the image or video to be processed;

[0020] The super-resolution and super-definition reconstruction module is used to extract the features of the deep neural network from the high-level feature layer to obtain the features of the output of each layer of the deep neural network; the features of the output of the last layer of the deep neural network are determined as the reconstructed super-resolution and super-definition image video;

[0021] The specific feature processing module is used to perform feature processing of the high-level feature layer using a deep neural network to obtain features output by each layer of the deep neural network; and to determine the features output by the last layer of the deep neural network as feature information output;

[0022] The image video loss function acquisition module is used to calculate the image loss function, specifically including: calculating L by comparing the reconstructed super-resolution and super-definition image video with the high-resolution and high-definition image video corresponding to the image video to be processed. 1 Loss, so that the reconstructed super-resolution and super-definition image video has the pixel value of the corresponding high-resolution and high-definition image video;

[0023] The image and video loss function acquisition module is also used to transmit the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module have better visual effects;

[0024] The feature information loss function acquisition module is used to calculate the loss function of the feature information, and obtain the feature information loss function by calculating the difference between the feature information output and the expected feature information;

[0025] The feature information loss function acquisition module is also used to back-propagate the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility;

[0026] The loss function fusion module performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module.

[0027] According to one aspect of the present invention, the following technical solution is adopted: a method for super-resolution and super-definition reconstruction of an image or video, the reconstruction method comprising:

[0028] The feature extraction module extracts a set feature layer corresponding to the image or video to be processed;

[0029] The super-resolution and super-definition reconstruction module performs feature sampling and feature extraction to obtain a reconstructed super-resolution image; and

[0030] At least one specific feature processing module performs feature processing on the feature layer, and obtains feature information output corresponding to the image or video to be processed according to different computer vision task settings.

[0031] As an embodiment of the present invention, the reconstruction method further comprises:

[0032] The image and video loss function acquisition module calculates the image and video loss function according to the super-resolution and super-definition image and video acquired by the super-resolution and super-definition reconstruction module;

[0033] At least one feature information loss function acquisition module calculates a corresponding feature information loss function according to the feature information output acquired by the specific feature processing module;

[0034] The loss function fusion module fuses the image and video loss function obtained by the image and video loss function acquisition module and the feature information loss function obtained by each feature information loss function acquisition module to obtain a fusion loss function; and uses the fusion loss function for back propagation to train the image and video super-resolution and super-definition reconstruction network.

[0035] As an embodiment of the present invention, at least one corresponding specific feature processing module is designed according to different computer vision task requirements to perform specific feature processing on the feature layer to obtain specific feature information of the image video to be processed related to the computer vision task requirements.

[0036] As an implementation manner of the present invention, each feature information loss function acquisition module acquires the feature information loss function by calculating the difference between the feature information output and the expected feature information.

[0037] As an implementation mode of the present invention, the feature extraction module obtains a high-level feature layer of the image or video to be processed, and the high-level feature layer contains deep feature information of the image or video to be processed;

[0038] The super-resolution and super-definition reconstruction module extracts features of a deep neural network from the high-level feature layer to obtain features output by each layer of the deep neural network; and determines the features output by the last layer of the deep neural network as the reconstructed super-resolution and super-definition image video;

[0039] The specific feature processing module performs feature processing of a deep neural network on the high-level feature layer to obtain features output by each layer of the deep neural network; and determines the features output by the last layer of the deep neural network as feature information output;

[0040] The image video loss function acquisition module calculates the image loss function, specifically including: calculating L by comparing the reconstructed super-resolution and super-definition image video with the high-resolution and high-definition image video corresponding to the image video to be processed. 1 Loss, so that the reconstructed super-resolution and super-definition image video has the pixel value of the corresponding high-resolution and high-definition image video;

[0041] The image and video loss function acquisition module also transmits the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module have better visual effects;

[0042] The feature information loss function acquisition module calculates the loss function of the feature information, and obtains the feature information loss function by calculating the difference between the feature information output and the expected feature information;

[0043] The feature information loss function acquisition module also back-propagates the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility;

[0044] The loss function fusion module performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module.

[0045] The beneficial effects of the present invention are as follows: the image and video super-resolution and super-definition reconstruction system and method proposed in the present invention can be centered around different computer vision tasks, and can configure and fuse multiple features, ultimately improving the image and video resolution and clarity while ensuring the credibility of the reconstructed image and video, and can then be used by computer vision tasks, thereby improving the actual performance of existing computer vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the composition of existing image and video super-resolution reconstruction systems.

[0047] Figure 2 The figure is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention.

[0048] Figure 3 The figure is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention.

[0049] Figure 4 The figure is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention.

[0050] Figure 5 The figure is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention.

[0051] Figure 6 The figure is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0054] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0055] The description of the steps in each embodiment in the specification is only for the convenience of explanation, and the implementation of the present application is not limited by the order of the steps. The "connection" in the specification includes both direct connection and indirect connection.

[0056] In the specification, "image video" refers to images, or videos, or images and videos.

[0057] The present invention discloses an image and video super-resolution and super-definition reconstruction system. Figure 2 , Figure 3 FIG. 1 is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention; please refer to FIG. Figure 2 , Figure 3 The reconstruction system may be a configurable multi-feature fusion reconstruction system. The reconstruction system includes: a feature extraction module 100, a super-resolution and super-definition reconstruction module 200 and at least one specific feature processing module 300.

[0058] The feature extraction module 100 is used to extract the set feature layer corresponding to the image video to be processed; the super-resolution and super-definition reconstruction module 200 is connected to the feature extraction module 100 to perform feature sampling and feature extraction to obtain the reconstructed super-resolution and super-definition image video; each specific feature processing module 300 is respectively connected to the feature extraction module 100 to perform feature processing on the feature layer, and obtain the feature information output corresponding to the image video to be processed according to different computer vision task settings.

[0059] Figure 4 FIG. 1 is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention; please refer to FIG. Figure 4 In one embodiment of the present invention, the system further includes: an image and video loss function acquisition module 400, at least one feature information loss function acquisition module 500 and a loss function fusion module 600.

[0060] The image and video loss function acquisition module 400 is used to calculate the image and video loss function according to the super-resolution and super-definition image and video acquired by the super-resolution and super-definition image reconstruction module 200; each feature information loss function acquisition module 500 is connected to the corresponding specific feature processing module 300 to calculate the corresponding feature information loss function according to the feature information output acquired by the specific feature processing module. The loss function fusion module 600 is used to fuse the image and video loss function acquired by the image and video loss function acquisition module 400 and the feature information loss function acquired by each feature information loss function acquisition module 500 to obtain a fusion loss function; the fusion loss function is used for back propagation to train the image and video super-resolution and super-definition reconstruction network.

[0061] In one embodiment of the present invention, the reconstruction system of the present invention can design at least one corresponding specific feature processing module 300 according to different computer vision task requirements to perform specific feature processing on the feature layer to obtain specific feature information of the image video to be processed related to the computer vision task requirements. Each feature information loss function acquisition module 500 is used to obtain a feature information loss function by calculating the difference between the feature information output and the expected feature information.

[0062] According to different computer tasks, the feature processing module, feature information output, and feature information loss function acquisition module of the reconstruction system of the present invention are set differently; the number of feature processing is also different. The reconstruction system of the present invention uses multiple features to ensure the accuracy of super-resolution and super-definition reconstruction; for the requirements of different computer vision tasks, different features are selected and the number of features is also different. However, no matter how many features there are, the processing method for a single feature is similar. At the same time, no matter how many features there are, each feature will generate a feature information loss function, and all feature information loss functions can be fused with the image video loss function to obtain a "total" loss function. The "total" loss function is back-propagated during the algorithm training process, ultimately affecting and optimizing the feature extraction module, and then greatly improving the credibility of the reconstruction effect of the super-resolution and super-definition reconstruction module.

[0063] In one embodiment of the present invention, the feature extraction module 100 is used to obtain a high-level feature layer of the image video to be processed, and the high-level feature layer contains deep feature information of the image video to be processed. The super-resolution and super-definition reconstruction module 200 is used to perform feature extraction of the high-level feature layer through a deep neural network to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as the reconstructed super-resolution and super-definition image video. The specific feature processing module 300 is used to perform feature processing of the high-level feature layer through a deep neural network to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as feature information output.

[0064] The image video loss function acquisition module 400 is used to calculate the image loss function, specifically including: calculating L between the reconstructed super-resolution and super-definition image video and the high-resolution and high-definition image video corresponding to the image video to be processed. 1The image and video loss function acquisition module 400 is also used to transmit the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module has better visual effects.

[0065] The feature information loss function acquisition module 500 is used to calculate the loss function of the feature information, and obtain the feature information loss function by calculating the difference between the feature information output and the expected feature information. The feature information loss function acquisition module 500 is also used to back-propagate the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility.

[0066] The loss function fusion module 600 performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module.

[0067] Figure 5 FIG. 1 is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention; please refer to FIG. Figure 5 In one embodiment of the present invention, at least one feature processing module 300 includes a semantic segmentation module 310, and the semantic segmentation module 310 is used to extract features from the feature layer to obtain a semantic segmentation map corresponding to the image to be processed. At least one specific information loss function acquisition module 500 includes a semantic segmentation loss function acquisition module 510, and the semantic segmentation loss function acquisition module 510 is used to calculate the semantic segmentation loss function according to the semantic segmentation map acquired by the semantic segmentation module 310. The loss function fusion module 600 is used to fuse the image loss function acquired by the image and video loss function acquisition module 400 and the semantic segmentation loss function acquired by the semantic segmentation loss function acquisition module 510 to obtain a fusion loss function; and use the fusion loss function to train a multi-task image super-resolution reconstruction network.

[0068] In one embodiment, the feature extraction module 100 is used to obtain a high-level feature layer of the image video to be processed, and the high-level feature layer contains deep feature information of the image video to be processed. The super-resolution and super-definition reconstruction module 200 is used to perform feature extraction of the high-level feature layer through a deep neural network to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as the reconstructed super-resolution and super-definition image video. The semantic segmentation module 310 is used to perform feature extraction of the high-level feature layer through a deep neural network to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as the semantic segmentation map of the image to be processed.

[0069] The image video loss function acquisition module 400 is used to calculate the image loss function, specifically including: calculating L between the reconstructed super-resolution and super-definition image video and the high-resolution and high-definition image video corresponding to the image video to be processed. 1 The image and video loss function acquisition module 400 is also used to transmit the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module has better visual effects.

[0070] The semantic segmentation loss function acquisition module 510 is used to calculate the semantic segmentation loss function, specifically including: calculating the multi-category cross entropy loss function by using the semantic segmentation map of the image to be processed obtained by the semantic segmentation module and the corresponding semantic segmentation annotation map, so that the semantic segmentation map obtained by the semantic segmentation module is more accurate. The semantic segmentation loss function acquisition module 510 is used to back-propagate the calculated semantic segmentation loss function to the feature extraction module through the semantic segmentation training gradient, so that the feature layer extracted by the feature extraction module contains rich semantic information, thereby helping the super-resolution and super-definition reconstruction module to reconstruct the super-resolution image pixel value more accurately, and improve the clarity of the reconstructed super-resolution image.

[0071] The loss function fusion module 600 performs weighted summation on the image loss function and the semantic segmentation loss function to obtain a fusion loss function; the loss function fusion module 600 is also used to back-propagate the fusion loss function to the image and video loss function acquisition module 400 and the semantic segmentation loss function acquisition module 510.

[0072] Figure 6 FIG. 1 is a schematic diagram of the composition of an image and video super-resolution and super-definition reconstruction system in one embodiment of the present invention; please refer to FIG. Figure 6In one embodiment of the present invention, at least one feature processing module 300 includes a text recognition module 320, and the text recognition module 320 downsamples the feature layer, extracts temporal features from the downsampled feature layer, and performs text recognition on the extracted temporal features to obtain the text content in the text image to be processed. At least one specific information loss function acquisition module 500 includes a text loss function acquisition module 520, and the text loss function acquisition module 520 calculates the text loss function according to the text content acquired by the text recognition module 320. The loss function fusion module 600 fuses the image loss function acquired by the image and video loss function acquisition module 400 and the text loss function acquired by the text loss function acquisition module 520 to obtain a fusion loss function; and uses the fusion loss function to train a multi-task text image super-resolution reconstruction network. In one embodiment, the reconstruction system may also include a super-resolution gradient map reconstruction module 700.

[0073] Specifically, in one embodiment of the present invention, the feature extraction module 100 is used to extract a set feature layer corresponding to the image to be processed; the super-resolution and super-definition reconstruction module 200 is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution text image; the text recognition module 320 is used to downsample the feature layer, perform temporal feature extraction on the downsampled feature layer, perform text recognition on the extracted temporal features, and obtain the text content in the text image to be processed; the super-resolution gradient map reconstruction module 700 is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution gradient map.

[0074] In one embodiment of the present invention, the feature extraction module 100 is used to obtain a high-level feature layer of the text image to be processed, and the high-level feature layer contains deep feature information of the text image to be processed. In one embodiment, the text image to be processed can be input into the feature extraction module in the ESRGAN generation network to obtain the high-level feature layer.

[0075] The super-resolution and super-definition reconstruction module 200 is used to upsample the feature layer through a deep neural network, extract features from the upsampled feature layer, and obtain features output by each layer of the deep neural network; and determine the features output by the last layer of the deep neural network as the reconstructed super-resolution text image.

[0076] The text recognition module 320 is used to downsample the high-level feature layer using a deep neural network including a pooling layer, so that the height of the feature layer after downsampling is a set value of 1; the downsampled feature layer is sent to a bidirectional LSTM network to extract temporal features to obtain the output of the temporal features of the text image to be processed; the temporal features are further extracted through a fully connected layer and a softmax function, and the features of the last layer are determined as the text content of the text image to be processed.

[0077] The super-resolution gradient map reconstruction module 700 is used to upsample the high-level feature layer through a deep neural network, perform feature extraction on the upsampled feature layer, and obtain the features of the output of each layer of the deep neural network; and determine the features of the output of the last layer of the deep neural network as the reconstructed super-resolution gradient map.

[0078] like Figure 6 As shown, in one embodiment of the present invention, the reconstruction system includes an image and video loss function acquisition module 400, a text loss function acquisition module 520, a gradient loss function acquisition module 800 and a loss function fusion module 600.

[0079] The image and video loss function acquisition module 400 is used to calculate the image loss function according to the super-resolution text image acquired by the super-resolution and super-definition reconstruction module; the text loss function acquisition module 520 is used to calculate the text loss function according to the text content acquired by the text recognition module; the gradient loss function acquisition module 800 is used to calculate the gradient loss function according to the super-resolution gradient map acquired by the super-resolution gradient map reconstruction module. The loss function fusion module 600 is used to fuse the image loss function acquired by the image and video loss function acquisition module 400, the text loss function acquired by the text loss function acquisition module 520, and the gradient loss function acquired by the gradient loss function acquisition module 800 to obtain a fusion loss function; and the fusion loss function is used to train the multi-task text image super-resolution reconstruction network.

[0080] In one embodiment of the present invention, the image video loss function acquisition module 400 is used to calculate the image loss function, specifically comprising: calculating L between the reconstructed super-resolution text image and the high-resolution text image corresponding to the text image to be processed 1 Loss, so that the reconstructed super-resolution text image has the pixel value of the corresponding high-resolution text image.

[0081] The text loss function acquisition module 520 is used to calculate the text loss function, specifically including: calculating the CTC loss between the text content of the to-be-processed text image obtained by the text recognition module and the corresponding annotated text content, so that the text content recognized by the text recognition module is more correct.

[0082] The gradient loss function acquisition module 800 is used to calculate the gradient loss function, specifically including: calculating the gradient map of the high-resolution text image corresponding to the to-be-processed text image through the Sobel operator to obtain the target gradient map; calculating L by comparing the target gradient map with the reconstructed super-resolution gradient map 1 Loss, so that the reconstructed super-resolution gradient map has the pixel value of the target gradient map.

[0083] The loss function fusion module 600 performs weighted summation on the three loss functions of the image loss function, the text loss function, and the gradient loss function to obtain a fusion loss function.

[0084] The image video loss function acquisition module 400 is used to back-propagate the calculated image loss function to the feature extraction module through the image training gradient; so that the high-level feature layer extracted by the feature extraction module contains rich image information, thereby making the super-resolution text image reconstructed by the super-resolution reconstruction module more realistic.

[0085] The text loss function acquisition module 520 is used to back-propagate the calculated text loss function to the feature extraction module through the text training gradient, so that the feature layer extracted by the feature extraction module contains rich text information, thereby helping the super-resolution and super-clarity reconstruction module to reconstruct the super-resolution text image with more complete text content and improve the credibility of the reconstructed super-resolution text image.

[0086] The gradient loss function acquisition module 800 is used to back-propagate the calculated gradient loss function to the feature extraction module through gradient training, so that the high-level feature layer extracted by the feature extraction module contains rich gradient information, thereby helping the super-resolution and super-clarity reconstruction module to reconstruct the super-resolution text image and text edges more clearly, thereby improving the clarity of the reconstructed super-resolution text image.

[0087] The loss function fusion module 600 is also used to back-propagate the fusion loss function to the image and video loss function acquisition module 400 , the text loss function acquisition module 520 and the gradient loss function acquisition module 800 .

[0088] In one usage scenario of the present invention, the present invention can reconstruct a text image with reduced resolution into a super-resolution text image, thereby solving the problem that when the existing image super-resolution reconstruction method based on deep neural network is applied to text image reconstruction, the text edges of the reconstructed super-resolution text image are blurred and the credibility of the text content is low, thereby providing clear and reliable images for high-level tasks such as semantic analysis of text images.

[0089] When the present invention is used in a text image recognition scenario, compared with the existing image super-resolution reconstruction method based on a deep neural network, the present invention has the following two advantages:

[0090] (1) The text edges of the reconstructed super-resolution text image are clear:

[0091] The multi-task text image super-resolution reconstruction method provided by the present invention adds a super-resolution gradient map reconstruction module in parallel on the basis of super-resolution and super-definition reconstruction modules, calculates the gradient loss function, and when the network parameters are updated, gradient training and gradient back propagation are performed so that the high-level feature layer extracted by the feature extraction module contains rich gradient information, thereby making the text edges of the super-resolution text image reconstructed by the super-resolution and super-definition reconstruction modules clearer.

[0092] (2) The text content of the reconstructed super-resolution text image is highly credible:

[0093] The multi-task text image super-resolution reconstruction method proposed in the present invention adds a text recognition module in parallel on the basis of the super-resolution and super-definition reconstruction modules, calculates the text loss function, and when the network parameters are updated, uses the text training gradient back propagation to make the high-level feature layer extracted by the feature extraction module contain rich text information, so that the text content of the super-resolution text image reconstructed by the super-resolution and super-definition reconstruction modules is correct and highly credible.

[0094] The present invention also discloses a method for reconstructing image and video super-resolution and super-definition, the reconstruction method comprising:

[0095] The feature extraction module extracts a set feature layer corresponding to the image or video to be processed;

[0096] The super-resolution and super-definition reconstruction modules perform feature sampling and feature extraction to obtain reconstructed super-resolution and super-definition image videos;

[0097] At least one specific feature processing module performs feature processing on the feature layer, and obtains feature information output corresponding to the image or video to be processed according to different computer vision task settings.

[0098] In one embodiment of the present invention, the reconstruction method further comprises:

[0099] The image and video loss function acquisition module calculates the image and video loss function according to the super-resolution and super-definition image and video acquired by the super-resolution and super-definition reconstruction module;

[0100] At least one feature information loss function acquisition module calculates a corresponding feature information loss function according to the feature information output acquired by the specific feature processing module;

[0101] The loss function fusion module fuses the image and video loss function obtained by the image and video loss function acquisition module and the feature information loss function obtained by each feature information loss function acquisition module to obtain a fusion loss function; and uses the fusion loss function for back propagation to train the image and video super-resolution and super-definition reconstruction network.

[0102] In one embodiment of the present invention, at least one corresponding specific feature processing module is designed according to different computer vision task requirements to perform specific feature processing on the feature layer to obtain specific feature information of the image video to be processed related to the computer vision task requirements. Each feature information loss function acquisition module obtains a feature information loss function by calculating the difference between the feature information output and the expected feature information.

[0103] In one embodiment of the present invention, the feature extraction module obtains a high-level feature layer of the image video to be processed, and the high-level feature layer contains deep feature information of the image video to be processed. The super-resolution and super-definition reconstruction module performs feature extraction of a deep neural network on the high-level feature layer to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as the reconstructed super-resolution and super-definition image video. The specific feature processing module performs feature processing of a deep neural network on the high-level feature layer to obtain features output by each layer of the deep neural network; the features output by the last layer of the deep neural network are determined as feature information output.

[0104] The image video loss function acquisition module calculates the image loss function, specifically including: calculating L by comparing the reconstructed super-resolution and super-definition image video with the high-resolution and high-definition image video corresponding to the image video to be processed. 1 The image and video loss function acquisition module also transmits the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module have better visual effects.

[0105] The feature information loss function acquisition module calculates the loss function of the feature information, and obtains the feature information loss function by calculating the difference between the feature information output and the expected feature information. The feature information loss function acquisition module also back-propagates the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility.

[0106] The loss function fusion module performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module.

[0107] In one embodiment of the present invention, at least one feature processing module 300 includes a semantic segmentation module 310, which can be used to complete semantic segmentation; the semantic segmentation module 310 can work in conjunction with the semantic segmentation loss function acquisition module 510. In another embodiment of the present invention, at least one feature processing module 300 includes a text recognition module 320, which can be used to complete text recognition; the text recognition module 320 can work in conjunction with the text loss function acquisition module 520. In addition, at least one feature processing module 300 can also include a super-resolution gradient map reconstruction module 700, which can work in conjunction with the gradient loss function acquisition module 800.

[0108] Of course, in one embodiment of the present invention, at least one feature processing module 300 may simultaneously include a semantic segmentation module 310 and a text recognition module 320 (it may also be other modules, such as modules for completing face recognition, advanced driver assistance system ADAS, posture recognition and other processing processes), and each process may be processed in parallel.

[0109] In summary, the image and video super-resolution and super-definition reconstruction system and method proposed in the present invention can be centered around different computer vision tasks, and can configure and fuse multiple features, ultimately improving the image and video resolution and clarity while ensuring the credibility of the reconstructed images and videos, and can then be used by computer vision tasks, thereby improving the actual performance of existing computer vision tasks.

[0110] It should be noted that the present application can be implemented in software and / or a combination of software and hardware; for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium; for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented in hardware; for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0111] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the embodiments may not be embodied in the embodiments due to interference from various factors, and the description of the effects or advantages is not used to limit the embodiments. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. An image and video super-resolution and super-definition reconstruction system, characterized in that: The reconstruction system comprises: A feature extraction module is used to extract a set feature layer corresponding to the image or video to be processed; A super-resolution and super-definition reconstruction module, connected to the feature extraction module, for performing feature sampling and feature extraction to obtain reconstructed super-resolution and super-definition image videos; and At least one specific feature processing module, each of which is connected to the feature extraction module to perform feature processing on the feature layer and obtain feature information output corresponding to the image or video to be processed according to different computer vision task settings; The system further comprises: An image and video loss function acquisition module, used to calculate the image and video loss function according to the super-resolution image acquired by the super-resolution and super-definition reconstruction module; At least one feature information loss function acquisition module, each feature information loss function acquisition module is connected to a corresponding specific feature processing module, and is used to calculate a corresponding feature information loss function according to the feature information output acquired by the specific feature processing module; and A loss function fusion module, used to fuse the image video loss function obtained by the image video loss function acquisition module and the feature information loss function obtained by each feature information loss function acquisition module to obtain a fusion loss function; Use the fusion loss function for back propagation to train the image and video super-resolution and super-definition reconstruction network; The feature extraction module is used to obtain a high-level feature layer of the image or video to be processed, wherein the high-level feature layer contains deep feature information of the image or video to be processed; The super-resolution and super-definition reconstruction module is used to extract the features of the deep neural network from the high-level feature layer to obtain the features of the output of each layer of the deep neural network; the features of the output of the last layer of the deep neural network are determined as the reconstructed super-resolution and super-definition image video; The specific feature processing module is used to perform feature processing of the high-level feature layer using a deep neural network to obtain features output by each layer of the deep neural network; and to determine the features output by the last layer of the deep neural network as feature information output; The image and video loss function acquisition module is used to calculate the image loss function, specifically including: calculating the L1 loss between the reconstructed super-resolution and super-definition image and video and the high-resolution and high-definition image and video corresponding to the image and video to be processed, so that the reconstructed super-resolution and super-definition image and video have the pixel values ​​of the corresponding high-resolution and high-definition image and video; The image and video loss function acquisition module is also used to transmit the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module have better visual effects; The feature information loss function acquisition module is used to calculate the loss function of the feature information, and obtain the feature information loss function by calculating the difference between the feature information output and the expected feature information; The feature information loss function acquisition module is also used to back-propagate the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility; The loss function fusion module performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module; The feature extraction module includes a text recognition module, which downsamples the feature layer, extracts temporal features from the downsampled feature layer, and performs text recognition on the extracted temporal features to obtain text content in the text image to be processed; A super-resolution gradient map reconstruction module is added in parallel on the basis of the super-resolution and super-definition reconstruction modules to calculate the gradient loss function. When the network parameters are updated, the gradient back propagation is performed through gradient training so that the high-level feature layer extracted by the feature extraction module contains rich gradient information. The feature extraction module is used to extract the set feature layer corresponding to the image to be processed; the super-resolution and super-definition reconstruction module is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution text image; the text recognition module is used to downsample the feature layer, perform temporal feature extraction on the downsampled feature layer, perform text recognition on the extracted temporal features, and obtain the text content in the text image to be processed; the super-resolution gradient map reconstruction module is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution gradient map; The feature extraction module is used to input the text image to be processed into the feature extraction module in the ESRGAN generation network, so as to obtain a high-level feature layer; The super-resolution and super-definition reconstruction module is used to perform up-sampling of the feature layer by a deep neural network, perform feature extraction on the up-sampled feature layer, and obtain features output by each layer of the deep neural network; and determine the features output by the last layer of the deep neural network as the reconstructed super-resolution text image; The text recognition module is used to downsample the high-level feature layer using a deep neural network including a pooling layer, so that the height of the feature layer after downsampling is a set value of 1; the downsampled feature layer is sent to a bidirectional LSTM network to extract time series features, and the output of the time series features of the text image to be processed is obtained; the time series features are further extracted through a fully connected layer and a softmax function, and the features of the last layer are determined as the text content of the text image to be processed; The super-resolution gradient map reconstruction module is used to upsample the high-level feature layer through a deep neural network, extract features from the upsampled feature layer, and obtain features output by each layer of the deep neural network; and determine the features output by the last layer of the deep neural network as the reconstructed super-resolution gradient map.

2. The image and video super-resolution and super-definition reconstruction system according to claim 1, characterized in that: At least one specific feature processing module is designed corresponding to different computer vision task requirements to perform specific feature processing on the feature layer to obtain feature information output of the image or video to be processed related to the computer vision task requirements.

3. The image and video super-resolution and super-definition reconstruction system according to claim 1, characterized in that: Each feature information loss function acquisition module is used to obtain a feature information loss function by calculating the difference between the feature information output and the expected feature information.

4. A method for super-resolution and super-definition reconstruction of images and videos, characterized in that: The reconstruction method comprises: The feature extraction module extracts a set feature layer corresponding to the image or video to be processed; The super-resolution and super-definition reconstruction module performs feature sampling and feature extraction to obtain reconstructed super-resolution and super-definition image videos; and At least one specific feature processing module performs feature processing on the feature layer, and obtains feature information output corresponding to the image or video to be processed according to different computer vision task settings; The reconstruction method further comprises: The image and video loss function acquisition module calculates the image and video loss function according to the super-resolution and super-definition image and video acquired by the super-resolution and super-definition reconstruction module; At least one feature information loss function acquisition module calculates a corresponding feature information loss function according to the feature information output acquired by the specific feature processing module; and The loss function fusion module fuses the image and video loss function obtained by the image and video loss function acquisition module and the feature information loss function obtained by each feature information loss function acquisition module to obtain a fusion loss function; the fusion loss function is used for back propagation to train the image and video super-resolution and super-definition reconstruction network; The feature extraction module obtains a high-level feature layer of the image or video to be processed, wherein the high-level feature layer contains deep feature information of the image or video to be processed; The super-resolution and super-definition reconstruction module extracts features of a deep neural network from the high-level feature layer to obtain features output by each layer of the deep neural network; and determines the features output by the last layer of the deep neural network as the reconstructed super-resolution and super-definition image video; The specific feature processing module performs feature processing of a deep neural network on the high-level feature layer to obtain features output by each layer of the deep neural network; and determines the features output by the last layer of the deep neural network as feature information output; The image and video loss function acquisition module calculates the image loss function, specifically including: calculating the L1 loss between the reconstructed super-resolution and super-definition image and video and the high-resolution and high-definition image and video corresponding to the image and video to be processed, so that the reconstructed super-resolution and super-definition image and video have the pixel values ​​of the corresponding high-resolution and high-definition image and video; The image and video loss function acquisition module also transmits the calculated image and video loss function to the feature extraction module through back propagation, so that the high-level feature layer extracted by the feature extraction module contains rich image and video information, so that the super-resolution and super-definition image and video reconstructed by the super-resolution and super-definition reconstruction module have better visual effects; The feature information loss function acquisition module calculates the loss function of the feature information, and obtains the feature information loss function by calculating the difference between the feature information output and the expected feature information; The feature information loss function acquisition module also back-propagates the calculated feature information loss function to the feature extraction module, so that the feature layer extracted by the feature extraction module contains rich specific feature information, thereby helping the super-resolution and super-definition image video reconstructed by the super-resolution and super-definition reconstruction module to have higher credibility; The loss function fusion module performs weighted summation on the image and video loss function and the feature information loss function to obtain a fusion loss function; the loss function fusion module is also used to back-propagate the fusion loss function to the image and video loss function acquisition module and the feature information loss function acquisition module; The feature extraction module includes a text recognition module, which downsamples the feature layer, extracts temporal features from the downsampled feature layer, and performs text recognition on the extracted temporal features to obtain text content in the text image to be processed; A super-resolution gradient map reconstruction module is added in parallel on the basis of the super-resolution and super-definition reconstruction modules to calculate the gradient loss function. When the network parameters are updated, the gradient back propagation is performed through gradient training so that the high-level feature layer extracted by the feature extraction module contains rich gradient information. The feature extraction module is used to extract the set feature layer corresponding to the image to be processed; the super-resolution and super-definition reconstruction module is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution text image; the text recognition module is used to downsample the feature layer, perform temporal feature extraction on the downsampled feature layer, perform text recognition on the extracted temporal features, and obtain the text content in the text image to be processed; the super-resolution gradient map reconstruction module is used to upsample the feature layer, perform feature extraction on the upsampled feature layer, and obtain a reconstructed super-resolution gradient map; The feature extraction module is used to input the text image to be processed into the feature extraction module in the ESRGAN generation network, so as to obtain a high-level feature layer; The super-resolution and super-definition reconstruction module is used to perform up-sampling of the feature layer by a deep neural network, perform feature extraction on the up-sampled feature layer, and obtain features output by each layer of the deep neural network; and determine the features output by the last layer of the deep neural network as the reconstructed super-resolution text image; The text recognition module is used to downsample the high-level feature layer using a deep neural network including a pooling layer, so that the height of the feature layer after downsampling is a set value of 1; the downsampled feature layer is sent to a bidirectional LSTM network to extract time series features, and the output of the time series features of the text image to be processed is obtained; the time series features are further extracted through a fully connected layer and a softmax function, and the features of the last layer are determined as the text content of the text image to be processed; The super-resolution gradient map reconstruction module is used to upsample the high-level feature layer through a deep neural network, extract features from the upsampled feature layer, and obtain features output by each layer of the deep neural network; and determine the features output by the last layer of the deep neural network as the reconstructed super-resolution gradient map.

5. The image and video super-resolution and super-definition reconstruction method according to claim 4, characterized in that: At least one corresponding specific feature processing module is designed according to different computer vision task requirements to perform specific feature processing on the feature layer to obtain specific feature information of the image video to be processed related to the computer vision task requirements.

6. The image and video super-resolution and super-definition reconstruction method according to claim 5, characterized in that: Each feature information loss function acquisition module obtains a feature information loss function by calculating the difference between the feature information output and the expected feature information.

Images