Training method and system of infrared super-division network model and storage medium thereof

Abstract

The invention relates to the field of deep learning, and discloses a training method and system for an infrared super-division network model and a storage medium, and the method is characterized in that the infrared super-division network model is a lightweight super-division network model constructed through a heavy parameter network block and transposition convolution, and is suitable for end side equipment; the training method comprises the following steps: inputting a single-channel infrared image to a lightweight super-division network model; expanding the number of channels to a set number of channels through a convolutional layer; parameters of the lightweight super-division network model are adjusted through the multiple heavy parameter network blocks; a transposed convolutional layer is connected to perform up-sampling on the feature map; the channel number is recovered to be a single channel through a convolutional layer; and outputting a result. Compared with the prior art, more accurate and effective image reconstruction can be realized, the computing power of side end equipment is not increased, and the performance and speed of the equipment are not reduced.

Description

Technical Field

[0001] The present invention relates to deep learning, and in particular to a training method and system for an infrared super-resolution network model and a storage medium thereof. Background Art

[0002] Super-resolution technology is an important technology used to improve the resolution of images or videos. It aims to reconstruct high-resolution images from low-resolution input images. This technology plays a key role in the field of computer vision, enhancing the details and clarity of images and improving the visual experience. In recent years, super-resolution technology based on deep learning has received widespread attention and development. Through the learning and optimization of neural networks, more accurate and effective image reconstruction can be achieved. Super-resolution technology can not only be applied to image enhancement, video processing and other fields, but super-resolution networks have high requirements for computing power, mainly due to their complex algorithms and large amount of calculations. These networks use complex neural network structures to achieve image detail enhancement, which requires a lot of computing resources. Although super-resolution technology can significantly improve image quality, when deployed on end-side devices, due to limited computing power of the device, it often leads to performance degradation and extremely slow processing speed. Summary of the invention

[0003] Therefore, in order to solve the above-mentioned shortcomings, the present invention provides a training method, system and storage medium for an infrared super-resolution network model. Compared with the prior art, the present invention can achieve more accurate and effective image reconstruction without increasing the computing power of the side device and reducing the performance and speed of the device.

[0004] In a first aspect, the present invention provides a training method for an infrared super-resolution network model, wherein the infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolutions, and is applicable to a terminal device; the training method comprises the following steps:

[0005] Input a single-channel infrared image to the lightweight super-resolution network model;

[0006] After a convolution layer, the number of channels is expanded to the set number of channels;

[0007] Adjusting the parameters of the lightweight super-resolution network model through a plurality of the re-parameterized network blocks;

[0008] Followed by a transposed convolutional layer to upsample the feature map;

[0009] The number of channels is restored to a single channel through a convolution layer;

[0010] Output the result.

[0011] In one embodiment, the re-reference network block includes:

[0012] The first branch consists of a convolution layer with a convolution kernel of 3*3 and a BN layer;

[0013] A parallel branch consisting of a convolutional layer with a 1*1 kernel and a BN layer; and

[0014] An identity mapping branch consisting of a bn layer.

[0015] In one embodiment, the training method of the infrared super-resolution network model further includes calculating the loss of the output result and the target result by the following formula, back-propagating to calculate the gradient, and then back-propagating;

[0016]

[0017] m, n are the width and height of the model output (A) and target (B) images, μ a , μ b Represents the average value of A and B pixel values, δ a , δ b Represents the standard deviation of A and B pixel values, γ is the non-zero content, i and j represent the row index and column index of the pixel, respectively, δ ab is the covariance between A and B.

[0018] In one embodiment, when training the magnitude super-resolution network model, a server-based heavy super-resolution network model is first trained, and then the heavy super-resolution model is used as a teacher network to distill the lightweight model.

[0019] In one embodiment, the lightweight super-resolution network model is trained using multi-scale images.

[0020] In a second aspect, the present invention provides a training system for an infrared super-resolution network model, wherein the infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolutions, and is applicable to a terminal device. The training system includes the following modules:

[0021] The first module is used to input a single-channel infrared image into the lightweight super-resolution network model;

[0022] The second module is used to expand the number of channels to the set number of channels through a convolution layer;

[0023] A third module is used to adjust the parameters of the lightweight super-resolution network model through a plurality of the re-parameterized network blocks;

[0024] The fourth module is used to upsample the feature map through a transposed convolution layer;

[0025] The fifth module is used to restore the number of channels to a single channel through a convolution layer;

[0026] The sixth module is used to output the results.

[0027] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the training method of the infrared super-resolution network model

[0028] The present invention has the following advantages:

[0029] Compared with the prior art, the present invention can achieve more accurate and effective image reconstruction without increasing the computing power of the side device or reducing the performance and speed of the device.

[0030] The present invention uses a heavy parameter structure and introduces more branches to learn single-branch reasoning, which not only enriches the representation ability of the model, but also realizes the effective allocation of computing resources in the reasoning process. At the same time, the loss is truncated so that the model can still achieve comprehensive and balanced learning when the number of parameters is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a training method for an infrared super-resolution network model;

[0032] Figure 2 It is a systematic module diagram of a training system for an infrared super-resolution network model;

[0033] In the figure: 100, first module; 200, second module; 300, third module; 400, fourth module; 500, fifth module; 600, sixth module; 700, seventh module. DETAILED DESCRIPTION

[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0035] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0036] As described in the background technology, super-resolution technology is an important technology for improving the resolution of images or videos, aiming to reconstruct high-resolution images from low-resolution input images. This technology plays a key role in the field of computer vision, which can enhance the details and clarity of images and improve the visual experience. In recent years, super-resolution technology based on deep learning has received widespread attention and development. Through the learning and optimization of neural networks, more accurate and effective image reconstruction can be achieved. Super-resolution technology can not only be applied to image enhancement, video processing and other fields, but super-resolution networks have high requirements for computing power, mainly due to their complex algorithms and large amounts of calculations. These networks use complex neural network structures to achieve image detail enhancement, which requires a lot of computing resources. Although super-resolution technology can significantly improve image quality, when deployed on end-side devices, due to limited computing power of the device, it often leads to performance degradation and extremely slow processing speed.

[0037] Based on the above problems, this embodiment provides a training method for an infrared super-resolution network model, wherein the infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolutions, and is suitable for end-side devices; Figure 1 As shown, the training method comprises the following steps:

[0038] Step S100, inputting a single-channel infrared image into the lightweight super-resolution network model;

[0039] Step S200, expand the number of channels to the set number of channels through a convolution layer. Optionally, the convolution layer is a convolution layer with a convolution kernel of 1, a step size of 1, an input channel filled with 0, and an output channel of 16; the number of channels can be set according to the computing power of the terminal side, generally 128, 256, 512, 1024; the set channel can also pass through several convolution layers, each convolution layer increases the number of channels by two times, such as 1-2-4-8-16-32-64-128-256, etc. until the target number of channels is reached.

[0040] Step S300: adjusting the parameters of the lightweight super-resolution network model through several of the re-parameterized network blocks. Optionally, the number of the re-parameterized network blocks is arranged according to the computing power of the terminal side and the inference delay requirements, and the number of the re-parameterized network blocks is 1, 4, 8 or 16. This step realizes changing the structure or parameters of the model to improve the performance of the model.

[0041] Step S400, upsampling the feature map by using a transposed convolutional layer;

[0042] Exemplarily, a transposed convolution layer is connected, the feature map is upsampled twice, and then passed through several re-parameter network blocks. Optionally, the number of re-parameter network blocks in this step is 1, and the transposed convolution is a transposed convolution layer with parameters of a convolution kernel of 4, a step size of 2, an input channel of 16, and an output channel of 16;

[0043] Step S500, restoring the number of channels to a single channel through a convolutional layer;

[0044] Step S600: output the result.

[0045] In one embodiment, the training method of the infrared super-resolution network model further includes:

[0046] Step S700, calculate the loss of the output result and the target result by the following formula, calculate the gradient by back propagation, and then back propagate;

[0047]

[0048] m, n are the width and height of the model output (A) and target (B) images, μ a , μ b Represents the average value of A and B pixel values, δ a , δ b Represent the standard deviation of the pixel values ​​of A and B respectively, γ is the non-zero content to ensure that the denominator is not 0, i and j represent the row index and column index of the pixel respectively, δ ab is the covariance between A and B.

[0049] The above technical features use a lightweight super-resolution model with a heavy parameter structure, which can make it have lighter parameters without affecting the learning ability. The loss function uses multiple truncation, and many aspects will no longer generate gradient learning after learning to a certain extent, making the model's performance expressed in a limited number of parameters more comprehensive and more suitable for the end side.

[0050] In one embodiment, the re-reference network block includes:

[0051] The first branch consists of a convolution layer with a convolution kernel of 3*3 and a BN layer;

[0052] A parallel branch consisting of a convolutional layer with a 1*1 kernel and a BN layer; and

[0053] An identity mapping branch consisting of a bn layer.

[0054] This technical feature has three branches during training, which can learn more information from different branches. During reasoning, the linear characteristics of the convolution are continued, and the three branches can be merged into a 3*3 convolution layer, so that more parameters can be used for learning, and good results can be achieved with very few parameters for reasoning.

[0055] Based on this embodiment, the principle of the re-parameter structure is:

[0056] Add parallel 1×1 convolution branches and identity mapping branches to each 3×3 convolution layer to form a repblock. After training, perform equivalent transformation on the model to obtain the deployment model. According to the linearity of convolution (specifically, additivity), let the three 3×3 convolution kernels be W1, W2, and W3, and we have conv(x, W1) + conv(x, W2) + conv(x, W3) = conv(x, W1 + W2 + W3).

[0057] In one embodiment, when training the magnitude super-resolution network model, a server-based heavy super-resolution network model is first trained, and then the heavy super-resolution model is used as a teacher network to distill the lightweight model.

[0058] In one embodiment, the lightweight super-resolution network model is trained using multi-scale images, for example, input (1*1*512*512) output (1*1*1024*1024), input (1*1*320*320) output (1*1*640*640).

[0059] In another embodiment, a training system for an infrared super-resolution network model is provided. The infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolutions, and is suitable for end-side devices, such as Figure 2 As shown, the training model includes the following modules:

[0060] The first module 100 is used to input a single-channel infrared image into a lightweight super-resolution network model;

[0061] The second module 200 is used to expand the number of channels to a set number of channels through a convolution layer;

[0062] The third module 300 is used to adjust the parameters of the lightweight super-resolution network model through a plurality of the re-parameterized network blocks;

[0063] A fourth module 400 is used to upsample the feature map by connecting a transposed convolutional layer;

[0064] A fifth module 500 is used to restore the number of channels to a single channel through a convolution layer;

[0065] The sixth module 600 is used to output the result.

[0066] The seventh module 700 is used to calculate the loss of the output result and the target result by the following formula, calculate the gradient by back propagation, and then back propagate;

[0067]

[0068] m, n are the width and height of the model output (A) and target (B) images, μ a , μ b Represents the average value of A and B pixel values, δ a , δ b Represent the standard deviation of A and B pixel values, Υ is the non-zero content, i and j represent the row index and column index of the pixel respectively; δ ab is the covariance between A and B.

[0069] It should be noted that, in the above embodiments, the description of each embodiment has its own focus, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program codes.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the above-mentioned system method for the infrared super-resolution network model.

[0075] If the module / unit integrated in the system / terminal device of the infrared super-resolution network model is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0076] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0079] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The above embodiments are only used to illustrate the technical solution of the present application, not to limit it; although the present application is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that: it is still possible to modify the technical solutions recorded in the above embodiments, or to replace some of the technical features therein by equivalent; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

[0081] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for an infrared super-resolution network model, characterized in that: The infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolution, and is suitable for end-side devices; The training method comprises the following steps: Input a single-channel infrared image to the lightweight super-resolution network model; After a convolution layer, the number of channels is expanded to the set number of channels; Adjusting the parameters of the lightweight super-resolution network model through a plurality of the re-parameterized network blocks; Followed by a transposed convolutional layer to upsample the feature map; The number of channels is restored to a single channel through a convolution layer; Output the result.

2. According to the training method of an infrared super-resolution network model described in claim 1, it is characterized in that: The re-participation network block includes: The first branch consists of a convolution layer with a convolution kernel of 3*3 and a BN layer; A parallel branch consisting of a convolutional layer with a 1*1 kernel and a BN layer; and An identity mapping branch consisting of a bn layer.

3. The training method of an infrared super-resolution network model according to claim 1, characterized in that: It also includes calculating the loss of the output result and the target result through the following formula, back-propagating to calculate the gradient, and then back-propagating; m, n are the width and height of the model output (A) and target (B) images, μ a , μ b Represents the average value of A and B pixel values, δ a , δ b Represent the standard deviation of A and B pixel values, γ is the non-zero content, i and j represent the row index and column index of the pixel respectively; δ ab is the covariance between A and B.

4. The training method of an infrared super-resolution network model according to claim 1, characterized in that: When training the magnitude super-resolution network model, a server-based heavy super-resolution network model is first trained, and then the heavy super-resolution model is used as a teacher network to distill the lightweight model.

5. The training method of an infrared super-resolution network model according to claim 1, characterized in that: The lightweight super-resolution network model is trained using multi-scale images.

6. A training system for an infrared super-resolution network model, characterized in that: The infrared super-resolution network model is a lightweight super-resolution network model constructed by re-parameterizing network blocks and transposed convolution, which is suitable for end-side devices. The training model includes the following modules: The first module is used to input a single-channel infrared image into the lightweight super-resolution network model; The second module is used to expand the number of channels to the set number of channels through a convolution layer; The third module is used to adjust the parameters of the lightweight super-resolution network model through a plurality of the re-parameterized network blocks; The fourth module is used to upsample the feature map through a transposed convolution layer; The fifth module is used to restore the number of channels to a single channel through a convolution layer; The sixth module is used to output the results.

7. A storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the training method of the infrared super-resolution network model as described in any one of claims 1-5.

Images