Lightweight image super-resolution reconstruction method and device based on convolutional neural network

Abstract

The present application proposes a lightweight image super-resolution reconstruction method and device based on a convolutional neural network, including: obtaining an image to be reconstructed; inputting the image to be reconstructed into a pre-trained super-resolution generation network, and outputting a corresponding super-resolution image; the super-resolution generation network includes multiple convolutional layers, DTS layers, Resize layers and output layers, and the multiple convolutional layers are used to extract features from the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first enlarged image; the Resize layer is obtained using nearest neighbor interpolation and is used to enlarge the image to be reconstructed to generate a second enlarged image; the output layer is used to fuse and add the first enlarged image and the second enlarged image to map and obtain a super-resolution image. The super-resolution generation network of the present application is built based on a lightweight convolutional neural network, which not only ensures the super-resolution quality of the image, but also reduces the number of parameters and complexity of the network.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lightweight image super-resolution reconstruction method and device based on a convolutional neural network. Background Art

[0002] Thermal imaging is not sensitive to lighting conditions, so it has a wide range of applications in smart cities, forest fire prevention, gas leaks, and agricultural production. However, due to the size limitation of the sensor, it is very difficult to obtain high-resolution thermal infrared images on drone gimbal cameras, which widely use thermal imaging.

[0003] At present, the image super-resolution algorithm based on deep learning can effectively reconstruct high-resolution images from low-resolution images and improve image quality. The Chinese patent with application number 202311713718.0 discloses an image super-resolution enhancement method and device, training method, equipment and medium. The super-resolution image model used is an adversarial generation network model, including a first generator, a second generator and a discriminator. The first generator is a super-resolution generation network, which is responsible for generating a high-quality high-resolution image from a low-resolution image through a super-resolution generation branch network; the second generator is a degradation parameter generation network, which is responsible for generating the control parameters required in the degradation process, so that the high-resolution image can be degraded to a low-resolution image with a quality similar to that of the low-resolution image through the degradation process; and the discriminator is responsible for judging whether the image is a low-resolution image / high-resolution image collected truly, or image data generated by the super-resolution generation / degradation process, so that the high-quality high-resolution image can be used for forward circulation, and the low-quality low-resolution image to be super-resolved can be used for reverse circulation to synchronously train the super-resolution generation network and the degradation parameter generation network.

[0004] The super-resolution image model in the above-mentioned prior art solution can adapt to the adaptive degradation of images of different scene types to improve the quality of super-resolution enhancement, and has good generalization ability and high model transplantation efficiency. However, in this solution, the super-resolution image model needs to use high-quality high-resolution images for forward circulation and low-quality low-resolution images to be super-resolved for reverse circulation to synchronously train the super-resolution generation network and the degradation parameter generation network. The model is highly complex, with a large number of parameters and calculations, and has great requirements on the computing power of the chip / processor. It is usually only applicable to fixed equipment. For the application field of drones, drone gimbal cameras can usually only be equipped with embedded devices, and the computing power of the chips / processors they carry is relatively low, which cannot meet the requirements at all.

[0005] In view of this, it is particularly important to provide a lightweight image super-resolution reconstruction method that reduces the number and complexity of model parameters while ensuring image super-resolution performance and improving operating speed, and meets the requirements of deployment on mobile or embedded devices. Summary of the invention

[0006] In order to solve the technical problem that the image super-resolution algorithm based on deep learning in the prior art has high requirements on computing power and cannot meet the low computing power requirements of drones, the present application proposes a lightweight image super-resolution reconstruction method and device based on a convolutional neural network.

[0007] According to the first aspect of the present application, a lightweight image super-resolution reconstruction method based on a convolutional neural network is proposed, comprising:

[0008] Acquire an image to be reconstructed;

[0009] Input the image to be reconstructed into a pre-trained super-resolution generation network, and output a corresponding super-resolution image;

[0010] Among them, the super-resolution generation network includes multiple convolution layers, DTS layers, Resize layers and output layers. The multiple convolution layers are used to extract features of the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first enlarged image with a preset super-resolution multiple; the Resize layer is obtained using nearest neighbor interpolation and is used to enlarge the image to be reconstructed to generate a second enlarged image with a preset super-resolution multiple; the output layer is used to fuse and add the first enlarged image and the second enlarged image to map and obtain the super-resolution image.

[0011] Preferably, the process of converting the depth information of the feature image into plane information is as follows:

[0012] [N, K*K*C, H, W] → [N, C, K*H, K*W]

[0013] Among them, N is the number of feature images, C is the number of channels of the feature image, H is the pixel height value of the feature image, W is the pixel width value of the feature image, and K is the preset super-resolution multiple.

[0014] Preferably, the super-resolution generation network is deployed through a dual-core heterogeneous processor, and the dual-core heterogeneous processor includes a first processor and a second processor, wherein a plurality of the convolutional layers and the Resize layers are deployed in parallel on the first processor, and the DTS layer and the output layer are deployed on the second processor, and the convolution operation efficiency of the first processor is higher than the convolution operation efficiency of the second processor.

[0015] Preferably, in the process of reconstructing a single frame of the image to be reconstructed, the running time of the first processor is a first time, the running time of the second processor is a second time, and the super-resolution generation network is configured as: the first time is greater than the second time.

[0016] Preferably, the convolution kernel size of the convolution layer is 3x3, the output channel is 64, the span is 1, the number of convolution layers is configured to be 13 layers, and the first time consumption is 50ms.

[0017] Preferably, the training process of the super-resolution generation network includes:

[0018] Constructing a training set, wherein the training set includes a first training sample including a plurality of low-resolution images and a second training sample including a plurality of high-resolution images;

[0019] Constructing the super-resolution generation network, taking the first training sample as the input of the super-resolution generation network, and outputting the super-resolution image;

[0020] Constructing an adversarial network, wherein the adversarial network includes a first judgment network and a second judgment network;

[0021] Among them, the first judgment network takes the super-resolution image as input, and is used to judge the true or false value of each pixel in the super-resolution image relative to the original high-resolution image; the second judgment network takes the second training sample as input, and is used to judge the true or false value of each pixel in the high-resolution image relative to the original high-resolution image; the adversarial network adjusts its own weight parameters based on the judgment results of the first judgment network and the second judgment network to iteratively train until the loss functions of the adversarial network and the super-resolution generation network converge, thereby obtaining the pre-trained super-resolution generation network.

[0022] Preferably, the training process of the super-resolution generation network further includes:

[0023] A degradation generation network is constructed, wherein the degradation generation network takes the second training sample as input and is used to perform degradation processing on the high-resolution image to obtain a degraded low-resolution image as the first training sample.

[0024] According to the second aspect of the present application, a lightweight image super-resolution reconstruction device based on a convolutional neural network is proposed, comprising:

[0025] An acquisition module, configured to acquire an image to be reconstructed;

[0026] A super-resolution generation module is configured to input the image to be reconstructed into a pre-trained super-resolution generation network and output a corresponding super-resolution image;

[0027] Among them, the super-resolution generation network includes multiple convolution layers, DTS layers, Resize layers and output layers. The multiple convolution layers are used to extract features of the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first enlarged image with a preset super-resolution multiple; the Resize layer is obtained using nearest neighbor interpolation and is used to enlarge the image to be reconstructed to generate a second enlarged image with a preset super-resolution multiple; the output layer is used to fuse and add the first enlarged image and the second enlarged image to map and obtain the super-resolution image.

[0028] According to a third aspect of the present application, a pan-tilt camera is provided, comprising:

[0029] one or more processors;

[0030] A memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the gimbal camera implements the lightweight image super-resolution reconstruction method based on a convolutional neural network as provided in any embodiment of the first aspect above.

[0031] According to the fourth aspect of the present application, a computer-readable storage medium is proposed, on which a computer program is stored. When the program is executed by a processor, a lightweight image super-resolution reconstruction method based on a convolutional neural network is implemented as provided in any embodiment of the first aspect above.

[0032] The present application proposes a lightweight image super-resolution reconstruction method and device based on a convolutional neural network. The super-resolution generation network is built based on a lightweight convolutional neural network, including 13 3x3 convolutional layers, a DTS layer, a Resize layer and an output layer, wherein the 13 convolutional layers extract the depth information of the feature image of the image to be reconstructed, and the DTS layer converts the depth information of the feature image into plane information through a channel-to-plane operation, thereby generating a first enlarged image of a preset super-resolution multiple, and at the same time using the nearest neighbor interpolation to obtain the Resize layer and enlarge the image to be reconstructed according to the preset super-resolution multiple to obtain a second enlarged image, so that the output layer fuses the first enlarged image and the second enlarged image and performs addition calculation, and the output super-resolution image is closer to the original high-resolution image, thereby ensuring the quality of the super-resolution image.

[0033] At the same time, in view of the characteristics of the NPU processors currently on the market, that is, the NPU processors cannot combine the two characteristics of efficient convolution operation efficiency and support for channel-to-plane operations, the super-resolution generation network of the present application is deployed on a dual-core heterogeneous processor, including a first processor and a second processor, wherein the first processor has efficient convolution operation efficiency but does not support channel-to-plane operations, and the second processor supports channel-to-plane operations but has low convolution operation efficiency. Therefore, 13 3x3 convolution layers and the Resize layer are deployed on the first processor, and the DTS layer and the output layer are deployed on the second processor. The first processor and the second processor run in parallel, thereby improving the operation speed of the super-resolution generation network.

[0034] Furthermore, the super-resolution generation network is configured so that a first time consumption of the first processor is greater than a second time consumption of the second processor, so that the final running time of the super-resolution generation network to reconstruct multiple consecutive frames of images depends on the first time consumption of the first processor, and by increasing or decreasing the number of convolutional layers, the first time consumption of the first processor can be linearly adjusted, so that the display frame rate of the super-resolution video can be adaptively adjusted.

[0035] In summary, the present application provides a lightweight image super-resolution reconstruction method and device based on a convolutional neural network. A super-resolution generation network is built based on a lightweight convolutional neural network. At the same time, the super-resolution generation network is deployed on a dual-core heterogeneous processor. While ensuring the image super-resolution performance and improving the operating speed, the network parameters and complexity are reduced, which meets the requirements for deployment on mobile or embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0037] Figure 1 is a flowchart of a lightweight image super-resolution reconstruction method based on a convolutional neural network according to a specific embodiment of the present application;

[0038] Figure 2 is a network structure diagram of a super-resolution generation network according to a specific embodiment of the present application;

[0039] Figure 3 is a working principle diagram of the DTS layer according to a specific embodiment of the present application;

[0040] Figure 4is a structural diagram of a training model of a super-resolution generation network according to a specific embodiment of the present application;

[0041] Figure 5 is a schematic diagram of an image to be enhanced according to a specific embodiment of the present application;

[0042] Figure 6 It is a schematic diagram of a super-resolution image obtained by using the prior art based on a bicubic interpolation algorithm;

[0043] Figure 7 is a schematic diagram of a super-resolution image obtained by using a super-resolution generation network of a specific embodiment of the present application;

[0044] Figure 8 It is a structural schematic diagram of a lightweight image super-resolution reconstruction device based on a convolutional neural network according to a specific embodiment of the present application. DETAILED DESCRIPTION

[0045] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0046] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] Figure 1 A flowchart of a lightweight image super-resolution reconstruction method based on a convolutional neural network according to a specific embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0048] S1. Obtain an image to be reconstructed.

[0049] Specifically, the image to be reconstructed is captured by an image acquisition device with a thermal imaging function, and the image to be reconstructed is a low-resolution thermal imaging image. In this embodiment, the image to be reconstructed is acquired by a thermal imaging image sensor carried by a drone gimbal camera.

[0050] S2: Input the image to be reconstructed into the pre-trained super-resolution generation network, and output the corresponding super-resolution image.

[0051] Figure 2 FIG. 4 shows a network structure diagram of a super-resolution generation network according to a specific embodiment of the present application. Figure 2 As shown in the figure, the super-resolution generation network is built based on a lightweight convolutional neural network. The super-resolution generation network includes an input layer, multiple convolutional layers, a DTS layer, a Resize layer, and an output layer.

[0052] Among them, the input layer is used to input the image to be reconstructed; multiple convolution layers and DTS layers constitute a branch, multiple convolution layers are used to extract features of the image to be reconstructed to obtain the depth information of the feature image, and the DTS layer (depth to space) is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first enlarged image with a preset super-resolution multiple; the Resize layer constitutes another branch, and the Resize layer is obtained using nearest neighbor interpolation, and is used to enlarge the image to be reconstructed to generate a second enlarged image with a preset super-resolution multiple; the output layer is used to fuse and add the first enlarged image and the second enlarged image to map and obtain a super-resolution image.

[0053] It should be noted that the number of convolutional layers configured will affect the image feature extraction capability, thereby affecting the accuracy of the super-resolution generation network. Within a certain range, the more convolutional layers are configured, the stronger the image feature extraction capability will be, but the convolution calculation time will also increase accordingly. As a result, when processing multiple consecutive frames of images to be reconstructed, the super-resolution image video frame rate generated by the final super-resolution will be low and the picture will be more jerky.

[0054] In this embodiment, the number of convolutional layers is configured to be 13 layers, and the parameters of the convolutional layers are: the convolution kernel size is 3x3, the output channel is 64, and the span is 1. In other implementations, the number of convolutional layers can be configured according to actual needs.

[0055] The function of the DTS layer is to convert the depth information of the feature image into plane information, thereby generating a first magnified image with a preset super-resolution multiple. In this embodiment, the process of converting the depth information of the feature image into plane information by the DTS layer through the channel-to-plane operation is as follows:

[0056] [N, K*K*C, H, W] → [N, C, K*H, K*W]

[0057] Among them, N is the number of feature images, C is the number of channels of the feature image, H is the pixel height value of the feature image, W is the pixel width value of the feature image, and K is the preset super-resolution multiple.

[0058] Figure 3 FIG. 4 shows a working principle diagram of the DTS layer according to a specific embodiment of the present application. Figure 3 As shown in the figure, taking the feature image with 4 channels (three RGB channels + image transparency channel), 1x1 pixel value, and 2x preset super-resolution as an example, the process of converting the depth information of the feature image into plane information through the channel-to-plane operation of the DTS layer is as follows:

[0059] [1, 16, 1, 1] → [1, 4, 2, 2]

[0060] The image to be enhanced is subjected to feature extraction by the convolution layer, and the depth information of the feature image is output through 16 output channels. The feature image is converted from the original 16 channel data with a pixel value of 1x1 to 4 channel data with a pixel value of 2x2. The image to be enhanced is transformed from a low-resolution image to a high-resolution image that is super-enlarged by 2 times.

[0061] In this embodiment, a resize operation is also performed on the enhanced image. The resize layer is obtained by using nearest neighbor interpolation. The resize layer enlarges the reconstructed image according to a preset super-resolution multiple to obtain a second enlarged image. In this way, the output layer fuses the first enlarged image and the second enlarged image for addition calculation, so that the output super-resolution image is closer to the original high-resolution image, thereby ensuring the quality of the super-resolution image.

[0062] It should be noted that the super-resolution generation network of this embodiment is deployed on the NPU processor. However, the characteristics of the NPU (neural network processor) currently on the market, that is, the NPU processor cannot combine the two characteristics of efficient convolution operation efficiency and support for channel-to-plane operation. The super-resolution generation network of this embodiment contains multiple convolution layers and DTS layers, and it is necessary to perform convolution operation calculations and channel-to-plane operations at the same time. Therefore, this embodiment also provides the following method to improve operation efficiency.

[0063] Continue to refer to Figure 2 In one embodiment provided in the present application, the super-resolution generation network is deployed through a dual-core heterogeneous processor, and the dual-core heterogeneous processor includes a first processor and a second processor for parallel processing. Among them, multiple convolution layers and Resize layers are deployed in parallel on the first processor, and the DTS layer and the output layer are deployed on the second processor. The convolution operation efficiency of the first processor is higher than the convolution operation efficiency of the second processor. The first processor does not support channel-to-plane operations, and the second processor supports channel-to-plane operations.

[0064] In this embodiment, the first processor is a dedicated NPU for image processing, and the second processor is a conventional NPU that can support channel-to-plane operations. It should be understood by those skilled in the art that, although the specific models of the first processor and the second processor are not disclosed in this embodiment, the first processor and the second processor can be selected according to the characteristics required by this embodiment, and therefore this reason cannot be used as a limitation to this embodiment.

[0065] In one embodiment provided in the present application, in the process of reconstructing a single frame of an image to be reconstructed, the operation time of the first processor is defined as the first time, and the operation time of the second processor is defined as the second time. The super-resolution generation network is configured such that the first time is greater than the second time.

[0066] Specifically, since the present embodiment adopts a dual-core heterogeneous processor, when reconstructing a single-frame image, the final running time of the super-resolution generation network is the sum of the first time and the second time, and when reconstructing continuous multiple-frame images, the first processor and the second processor can be in parallel, so the final running time of the super-resolution generation network depends on the processor with the larger running time. By configuring the first time of the first processor to be greater than the second time of the second processor, the final running time of the super-resolution generation network to reconstruct continuous multiple-frame images depends on the first time of the first processor. The first time of the first processor depends on the number of convolutional layers. Therefore, by increasing or decreasing the number of convolutional layers, the time consumed by the super-resolution generation network to reconstruct continuous multiple-frame images can be linearly adjusted, thereby adaptively adjusting the display frame rate of the super-resolution video.

[0067] In this embodiment, 13 convolutional layers are configured so that the first time consumption is greater than the second time consumption, and the first time consumption is specifically 50 ms.

[0068] It should be noted that there is a trade-off between the super-resolution quality and the display frame rate of super-resolution videos. Within a certain range, the more convolutional layers are configured, the stronger the image feature extraction capability, the higher the super-resolution quality, and the lower the video display frame rate; on the contrary, the fewer convolutional layers are configured, the weaker the image feature extraction capability, the worse the super-resolution quality, and the higher the video display frame rate.

[0069] In this embodiment, after many experiments, it was found that when the 3x3 convolutional layer is configured as 13 layers, the super-resolution generation network at this time can still maintain a good feature extraction capability. When the super-resolution generation network continuously reconstructs multiple frames of images to be reconstructed, the display frame rate of the obtained super-resolution video is exactly 20 frames. While ensuring the image super-resolution quality, there will be no obvious freeze in the video display. In other embodiments, for application scenarios with higher requirements for super-resolution image quality, you can choose to configure more layers of convolutional layers to reduce the display frame rate of the obtained super-resolution video. For application scenarios with lower requirements for super-resolution image quality, you can choose to configure a small number of convolutional layers to increase the display frame rate of the obtained super-resolution video. There is no limitation here.

[0070] Figure 4 FIG. 4 shows a training model structure diagram of a super-resolution generation network according to a specific embodiment of the present application, as shown in FIG. Figure 4 As shown, the training model structure of the super-resolution generation network includes a degradation generation network, a super-resolution generation network and an adversarial network, wherein the adversarial network includes a first judgment network and a second judgment network. The pre-training process of the super-resolution generation network includes:

[0071] A training set is constructed, the training set including a first training sample including a plurality of low-resolution images and a second training sample including a plurality of high-resolution images. In this embodiment, a high-resolution image of multi-angle thermal imaging is taken by a pan-tilt camera with a thermal imaging function as the second training sample.

[0072] A degradation generation network is constructed, and the degradation generation network takes the second training sample as input to perform degradation processing on the high-resolution image to obtain a degraded low-resolution image. In this embodiment, the low-resolution image generated by the degradation generation network of the second training sample is used as the first training sample.

[0073] A super-resolution generation network is constructed, which includes an input layer, multiple convolutional layers, a DTS layer, a Resize layer, and an output layer. The first training sample is used as the input of the super-resolution generation network, and a super-resolution image is output.

[0074] Construct an adversarial network, which includes a first judgment network and a second judgment network. The first judgment network takes the super-resolution image as input and is used to judge the true or false value of each pixel in the super-resolution image relative to the original high-resolution image; the second judgment network takes the second training sample as input and is used to judge the true or false value of each pixel in the high-resolution image relative to the original high-resolution image; the adversarial network adjusts its own weight parameters based on the judgment results of the first judgment network and the second judgment network to iteratively train until the loss functions of the adversarial network and the super-resolution generation network converge, thereby obtaining a pre-trained super-resolution generation network.

[0075] In this embodiment, by deploying the first judgment network and the second judgment network to compete with the super-resolution generation network, the convergence speed of the entire training model is fast, and the super-resolution generation network obtained by training has high accuracy.

[0076] In this embodiment, the first judgment network and the second judgment network adopt the Unet network (encoding-decoding judgment network), which has a simple and lightweight structure and a fast running speed, and meets the application scenario requirements of drone gimbal cameras. In other embodiments, the first judgment network and the second judgment network may also adopt other encoding-decoding judgment networks, which are not limited here.

[0077] In a specific embodiment, after the super-resolution generation network is obtained by training the training model, the super-resolution generation network can be transplanted to the device end. In this embodiment, the super-resolution generation network is transplanted to the embedded device after training, and finally mounted on the drone gimbal camera.

[0078] Figure 5 FIG. 4 shows a schematic diagram of an image to be enhanced according to a specific embodiment of the present application. Figure 6 FIG. 1 shows a schematic diagram of a super-resolution image obtained by using a bicubic interpolation algorithm based on the prior art. Figure 7 FIG. 4 shows a schematic diagram of a super-resolution image obtained by using a super-resolution generation network according to a specific embodiment of the present application. Figure 5 , Figure 6 , Figure 7 As shown, taking the 2x super-resolution effect as an example, after the original image to be enhanced is super-resolutioned 2x, compared with the super-resolution algorithm based on bicubic interpolation in the prior art, the super-resolution generation network of the embodiment of the present application has better signal recovery for low-resolution images, better image sharpening, and overall clarity.

[0079] In summary, the present application provides a lightweight image super-resolution reconstruction method based on a convolutional neural network. The super-resolution generation network is obtained through adversarial network training. The training difficulty of the super-resolution generation network is low and the training effect is good. Among them, the super-resolution generation network is built based on a lightweight convolutional neural network, which reduces the number of network parameters and complexity. At the same time, it cooperates with the super-resolution amplification operations of the DTS layer and the Resize layer, and the final fusion super-resolution image quality is good. The super-resolution generation network is deployed on a dual-core heterogeneous processor, which ensures the image super-resolution performance while improving the operating speed. Therefore, the super-resolution generation network of the present application meets the requirements of being deployed on mobile terminals or embedded devices, and is suitable for the application field of drone gimbal cameras.

[0080] Based on the above-mentioned lightweight image super-resolution reconstruction method based on convolutional neural network, the present application also proposes a lightweight image super-resolution reconstruction device based on convolutional neural network. Figure 8FIG. 1 shows a structural schematic diagram of a lightweight image super-resolution reconstruction device based on a convolutional neural network according to a specific embodiment of the present application. Figure 8 As shown, the device comprises:

[0081] An acquisition module 10, configured to acquire an image to be reconstructed;

[0082] The super-resolution generation module 20 is configured to input the image to be reconstructed into a pre-trained super-resolution generation network and output a corresponding super-resolution image;

[0083] Among them, the super-resolution generation network includes multiple convolutional layers, DTS layers, Resize layers and output layers. The multiple convolutional layers are used to extract features of the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first enlarged image with a preset super-resolution multiple; the Resize layer is used to enlarge the image to be reconstructed to generate a second enlarged image with a preset super-resolution multiple; the output layer is used to fuse and add the first enlarged image and the second enlarged image to map and obtain a super-resolution image.

[0084] Based on the above-mentioned lightweight image super-resolution reconstruction method based on convolutional neural network, the present application also proposes a gimbal camera, including: one or more processors, and a memory. The memory is used to store one or more programs, and when the one or more programs are executed by one or more processors, the gimbal camera implements the lightweight image super-resolution reconstruction method based on convolutional neural network provided in any of the above-mentioned embodiments.

[0085] Based on the above-mentioned lightweight image super-resolution reconstruction method based on convolutional neural network, the present application also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the lightweight image super-resolution reconstruction method based on convolutional neural network provided in any of the above-mentioned embodiments is implemented.

[0086] In the embodiments of the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device / system / method embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0090] Obviously, those skilled in the art can make various modifications and changes to the embodiments of the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and changes are within the scope of the claims of the present invention and their equivalents, the present invention is also intended to cover these modifications and changes. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to profit. Any reference numerals in the claims should not be considered to limit the scope.

Claims

1. A lightweight image super-resolution reconstruction method based on convolutional neural network, characterized in that: include: Acquire an image to be reconstructed; Input the image to be reconstructed into a pre-trained super-resolution generation network, and output a corresponding super-resolution image; The super-resolution generation network includes a plurality of convolutional layers, a DTS layer, a Resize layer and an output layer, wherein the plurality of convolutional layers are used to extract features of the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first magnified image of a preset super-resolution multiple; the Resize layer is obtained by using nearest neighbor interpolation and is used to magnify the image to be reconstructed to generate a second magnified image of the preset super-resolution multiple; the output layer is used to fuse and add the first magnified image and the second magnified image to map and obtain the super-resolution image; The super-resolution generation network is deployed through a dual-core heterogeneous processor, and the dual-core heterogeneous processor includes a first processor and a second processor, wherein a plurality of the convolutional layers and the Resize layer are deployed in parallel on the first processor, and the DTS layer and the output layer are deployed on the second processor, and the convolution operation efficiency of the first processor is higher than the convolution operation efficiency of the second processor; In the process of reconstructing a single frame of the image to be reconstructed, the operation time of the first processor is a first time, the operation time of the second processor is a second time, and the super-resolution generation network is configured such that: the first time is greater than the second time; The adjustment of the first time consumption includes: linearly adjusting the size of the first time consumption by increasing or decreasing the number of the convolutional layers.

2. The method according to claim 1, characterized in that The process of converting the depth information of the feature image into plane information is as follows: [N, K*K*C, H, W] → [N, C, K*H, K*W] Among them, N is the number of feature images, C is the number of channels of the feature image, H is the pixel height value of the feature image, W is the pixel width value of the feature image, and K is the preset super-resolution multiple.

3. The method according to claim 1, characterized in that: The convolution kernel size of the convolution layer is 3x3, the output channel is 64, the span is 1, the number of convolution layers is configured to be 13 layers, and the first time consumption is 50ms.

4. The method according to claim 1, characterized in that: The training process of the super-resolution generation network includes: Constructing a training set, wherein the training set includes a first training sample including a plurality of low-resolution images and a second training sample including a plurality of high-resolution images; Constructing the super-resolution generation network, taking the first training sample as the input of the super-resolution generation network, and outputting the super-resolution image; Constructing an adversarial network, wherein the adversarial network includes a first judgment network and a second judgment network; Among them, the first judgment network takes the super-resolution image as input, and is used to judge the true or false value of each pixel in the super-resolution image relative to the original high-resolution image; the second judgment network takes the second training sample as input, and is used to judge the true or false value of each pixel in the high-resolution image relative to the original high-resolution image; the adversarial network adjusts its own weight parameters based on the judgment results of the first judgment network and the second judgment network to iteratively train until the loss functions of the adversarial network and the super-resolution generation network converge, thereby obtaining the pre-trained super-resolution generation network.

5. The method according to claim 4, characterized in that The training process of the super-resolution generation network also includes: A degradation generation network is constructed, wherein the degradation generation network takes the second training sample as input and is used to perform degradation processing on the high-resolution image to obtain a degraded low-resolution image as the first training sample.

6. A lightweight image super-resolution reconstruction device based on convolutional neural network, characterized in that: include: An acquisition module, configured to acquire an image to be reconstructed; A super-resolution generation module is configured to input the image to be reconstructed into a pre-trained super-resolution generation network and output a corresponding super-resolution image; The super-resolution generation network includes a plurality of convolutional layers, a DTS layer, a Resize layer and an output layer, wherein the plurality of convolutional layers are used to extract features of the image to be reconstructed to obtain depth information of the feature image; the DTS layer is used to convert the depth information of the feature image into plane information through a channel-to-plane operation to generate a first magnified image of a preset super-resolution multiple; the Resize layer is obtained by using nearest neighbor interpolation and is used to magnify the image to be reconstructed to generate a second magnified image of the preset super-resolution multiple; the output layer is used to fuse and add the first magnified image and the second magnified image to map and obtain the super-resolution image; The super-resolution generation network is deployed through a dual-core heterogeneous processor, and the dual-core heterogeneous processor includes a first processor and a second processor, wherein a plurality of the convolutional layers and the Resize layer are deployed in parallel on the first processor, and the DTS layer and the output layer are deployed on the second processor, and the convolution operation efficiency of the first processor is higher than the convolution operation efficiency of the second processor; In the process of reconstructing a single frame of the image to be reconstructed, the operation time of the first processor is a first time, the operation time of the second processor is a second time, and the super-resolution generation network is configured such that: the first time is greater than the second time; The adjustment of the first time consumption includes: linearly adjusting the size of the first time consumption by increasing or decreasing the number of the convolutional layers.

7. A pan-tilt camera, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the gimbal camera to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images