Training method and system of image super-resolution reconstruction model and storage medium

Abstract

The embodiment of the invention discloses a training method and system of an image super-resolution reconstruction model and a storage medium, belongs to the technical field of image processing, and can realize calculation resource consumption breakthrough of the image super-resolution reconstruction model on equipment with the same calculation capability and improve the implementation cost of algorithm equipment. The method comprises the following steps: taking image blocks of an original image as training samples, and respectively inputting the training samples into a teacher model and a student model to obtain reference features and output features; inputting the output features into a jump-out model to obtain a jump-out score corresponding to each layer of network module; determining a loss value of a jumping-out model corresponding to the jumping-out score and a mark corresponding to the image block based on a first difference value between the output feature and a reference feature and a second difference value between an output feature of a next-layer network module of the target layer and the reference feature; and adjusting model parameters of the student model and model parameters of the jump-out model based on the joint loss value.

Description

Technical Field

[0001] This document relates to the field of image processing technology, and in particular to a training method, system and storage medium for an image super-resolution reconstruction model. Background Art

[0002] Image super-resolution (SR) technology is a branch of artificial intelligence. It can reconstruct low-resolution images into high-resolution images through algorithms. High-resolution images have higher pixel density and can carry more detailed information. Image super-resolution technology can be applied to various fields such as video compression, image rendering, medical imaging, aerospace satellites, etc. and has a wide range of application scenarios.

[0003] At present, image super-resolution technology usually implements the algorithm on embedded devices and / or servers that provide computing resources. The model of the image super-resolution algorithm often adopts large-scale convolutional networks, scale transformation modules, and stitching modules, which require a long operating time support and consume a large amount of storage space such as memory / hard disk, which will lead to increased equipment costs. At the same time, the algorithm processing efficiency on existing devices is very low, and it is difficult to use less power on existing devices, and it is also difficult to implement a larger-scale algorithm model. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a training method, system and storage medium for an image super-resolution reconstruction model, so as to avoid excessive equipment costs due to the large consumption of computing resources of the image super-resolution reconstruction model, and then when the model is used, achieve a breakthrough in the consumption of computing resources of the image super-resolution reconstruction model on devices with the same computing power, improve processing efficiency and improve the cost of implementing the algorithm equipment.

[0005] In order to achieve the above purpose, the embodiments of this specification adopt the following scheme:

[0006] In a first aspect, a training method for an image super-resolution reconstruction model is provided, the training method comprising:

[0007] The image blocks of the original image are used as training samples and are input into the teacher model and the student model respectively to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model;

[0008] Input the output features of each layer of network modules in the student model into the jump-out model to obtain the jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as the jump-out layer;

[0009] Determine a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature;

[0010] Based on the joint loss value, the model parameters of the student model and the model parameters of the jump-out model are adjusted, and the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0011] In a second aspect, an image super-resolution reconstruction method is provided, the image super-resolution reconstruction method comprising: acquiring an original image; inputting the original image into an image super-resolution reconstruction model to obtain a super-resolution image; wherein the image super-resolution reconstruction model is configured with a jump-out model as a sub-network; the image super-resolution reconstruction model is a student model in the aforementioned training method of the image super-resolution reconstruction model, and is a student model trained by the training method.

[0012] In a third aspect, a system for image super-resolution reconstruction is provided, the system comprising:

[0013] An input module, used to input the image blocks of the original image as training samples to the teacher model and the student model respectively, so as to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model;

[0014] A training module, used for inputting the output features of each layer of network modules in the student model into the jump-out model to obtain a jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as a jump-out layer;

[0015] The training module is further used to determine the loss value of the jump-out model corresponding to the jump-out score and the label corresponding to the image block based on a first difference value between the output feature of the network module of the target layer in the student model and the benchmark feature, and a second difference value between the output feature of the network module of the next layer of the target layer and the benchmark feature;

[0016] The training module is also used to adjust the model parameters of the student model and the model parameters of the jump-out model based on a joint loss value, wherein the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0017] In a fourth aspect, an electronic device is provided, the electronic device comprising:

[0018] at least one processor;

[0019] a memory connected to the at least one processor;

[0020] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned method by executing the instructions stored in the memory.

[0021] According to a fifth aspect, a machine-readable storage medium is provided, which stores machine instructions. When the machine instructions are executed on a machine, the machine executes the aforementioned method.

[0022] In the scheme of the embodiment of this specification, a distillation learning framework is adopted, and the original image and the teacher model are used to train the student model to obtain the benchmark features and the output features in the student model. The benchmark features can provide the student model with a reference knowledge basis for identifying image features, rather than reconstructing the convolutional layer structure of the student model and / or the teacher model. The jump-out score of the output features of each layer of network modules in the student model is obtained by jumping out of the model, so as to evaluate the perception level of the features of the current image block by each layer of network modules in the student model through the jump-out model. The higher the jump-out score corresponding to a layer of network modules, the more sufficient the output features of the layer of network modules are for the features required for reconstruction. When the output features are sufficient for the features required for reconstruction, the layer of network modules can be used as a network module of the jump-out layer, and can terminate the forward calculation process of the features of the image block perceived by the student model in advance when the model is used, so as to achieve the purpose of reducing the computing resource overhead. In the student model, the output features between the target layer and the next layer of network modules are used, and the difference between the benchmark features in the teacher model is used to construct the loss value and label of the jump-out model, which can evaluate the feature scoring ability of the jump-out model and automatically form labels corresponding to the image blocks. At the same time, there is no need to provide samples and sample labels for the jump-out model separately, nor is there a need to provide a reference model for the jump-out model. Finally, the joint loss value is used to adjust the student model and the jump-out model, and the model parameters of the two are optimized simultaneously, so that the change in the feature scoring ability of the jump-out model can be based on the change in the student model's perception level of the image block. Keeping the jump-out model and the student model optimized synchronously can ensure the accuracy of the image information of the high-resolution image reconstruction corresponding to the original image.

[0023] Other features and advantages of the embodiments of this specification will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0025] Figure 1 A schematic diagram of steps of a training method for an exemplary image super-resolution reconstruction model according to an embodiment of this specification;

[0026] Figure 2 This is a schematic diagram of a data processing flow in the training phase of an exemplary image super-resolution reconstruction model according to an embodiment of this specification;

[0027] Figure 3 A schematic diagram of a data processing flow in the test phase of an exemplary image super-resolution reconstruction model according to an embodiment of this specification;

[0028] Figure 4 This is a schematic diagram of an exemplary system module applied to image super-resolution reconstruction according to an embodiment of this specification;

[0029] Figure 5 This is a schematic diagram of an exemplary system module applied to image super-resolution reconstruction according to an embodiment of this specification;

[0030] Figure 6 This is a schematic diagram of an exemplary system module applied to image super-resolution reconstruction according to an embodiment of this specification;

[0031] Figure 7 The following is a schematic diagram of a module of an exemplary electronic device according to an embodiment of this specification. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0033] As mentioned above, the current image super-resolution (super-resolution can be referred to as "super-resolution") algorithm model is relatively complex, for example, it uses a large number of serial convolutional network modules, multi-level splicing cascade network modules, and large-scale high-resolution feature layer network modules for calculation. These network modules have greatly improved the high-resolution performance of the algorithm, but also brought more computing resource consumption, requiring the device to provide more storage space and processor resources. It is obviously unreasonable for a huge model to use the same amount of calculation for different parts of the image during the entire calculation process. For example, it is reasonable to use more calculations for areas with more high-frequency information in the image, but for smooth areas in the image, complex networks do not need to be involved. At this time, using the same network for calculations in different parts of the image will have obvious additional computing overhead.

[0034] In order to reduce computational overhead, we can try to use model parallel dynamic reasoning algorithm and model serial dynamic reasoning algorithm.

[0035] In the model parallel dynamic reasoning algorithm, first, the image can be divided into image blocks of different super-resolution difficulties through the difficulty classification strategy, and then the image blocks of each difficulty are sent to the super-resolution modules (SR-Module) with different parameter amounts in parallel. Finally, the super-resolution results of each image block are merged into a complete super-resolution image. Although this parallel algorithm can use super-resolution models with different computing power for different image blocks, the parameters of the entire parallel model become more, the overall width of the model is larger, and there is still a problem of excessive memory resource consumption.

[0036] In the model series dynamic reasoning algorithm, different image blocks can be sent to the same series model. During the forward reasoning process, different image blocks only use part of the model. For example, when a simple image block is inferred in the model, only the first few layers of the model are used, while a complex image block uses more layers of the network during reasoning. Different networks are subsets of the network of the same branch. In this way, different models with different computational amounts can be used for different image blocks, which can reduce the computational overhead in scenes without resolution requirements. However, this type of algorithm model is usually a direct series model, which does not have many multi-scale splicing, and the learning is directly consistent with the high-resolution image. In practical applications, there is a need to process a variety of common production / life images. In order to meet the specific resolution requirements of the hardware in the scene, the network model structure is usually more complex. Complex models often contain multi-level splicing, downsampling, upsampling, etc., which makes it difficult to apply the series algorithm to complex models that process common images.

[0037] In view of this, the solution of this specification provides a solution for image super-resolution reconstruction, which uses a complex model including adaptive reasoning and a jump-out (prediction) model to cooperate, and can perform super-resolution reconstruction on general images. At the same time, it achieves a breakthrough in the computing resource consumption of the image super-resolution reconstruction model on devices with equivalent computing power, thereby improving the reconstruction efficiency. It should be understood that the technical solutions provided in each embodiment of this specification can be implemented on a device with communication capabilities and computing and instruction processing capabilities, such as a server or electronic device.

[0038] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0039] One embodiment of the present specification provides a training method for an image super-resolution reconstruction model, which can be a machine (learning) model, and the image super-resolution reconstruction model can be deployed on a device with computing capabilities, such as the aforementioned server and electronic device. The image super-resolution reconstruction model may include a (can) adaptive reasoning subnetwork and a subnetwork for improving resolution after the adaptive reasoning subnetwork. It may also include subnetworks adapted to product features and application scenarios, such as a feature extraction subnetwork before the adaptive reasoning subnetwork, a subnetwork for establishing connections between subnetworks, and a subnetwork for splicing image blocks after the subnetwork for improving resolution. In addition, the image super-resolution reconstruction model may also include a scale-scaling subnetwork. The adaptive reasoning subnetwork can be used as one of the subnetworks in the forward calculation of the image super-resolution reconstruction model.

[0040] In some possible examples, the image super-resolution reconstruction model may include multiple resolution parallel branches, for example, according to the resolution size, it may include a high-resolution branch and a low-resolution branch, or it may include a high-resolution branch, a medium-resolution branch and a low-resolution branch. Multiple resolution parallel branches can be used to process information of the same image block, for example, a high-resolution branch can be used to process high-frequency information, and a low-resolution branch can be used to process low-frequency information (relative to high-frequency information). Each resolution branch may include an adaptive reasoning sub-network, and may also include a network module adapted to product features and application scenarios (for example, a network module for adjusting specified model parameter settings). Each adaptive reasoning sub-network includes multiple cascaded network modules, for example, it may include cascaded convolutional network modules. The entire image super-resolution reconstruction model includes a series-parallel model structure, and the image super-resolution reconstruction model can be called a complex model, wherein the image can be a separate picture or a frame of data in a video / data stream.

[0041] In an embodiment of the present specification, the aforementioned training method can be performed on the basis of a distillation learning framework to reduce the equipment implementation cost of the image reconstruction model and improve the training efficiency while maintaining the accuracy of the super-resolution image. In the distillation learning framework, the image super-resolution reconstruction model can be used as a student model, and the adaptive reasoning subnetwork and the subnetwork for improving resolution of the teacher model are respectively the same model structures as the adaptive reasoning subnetwork and the subnetwork for improving resolution of the student model, wherein the number of parallel branches containing the adaptive reasoning subnetwork in the teacher model and the student model and the structure of the corresponding branches can be the same.

[0042] Based on the distillation learning framework, a jump-out model can be used to evaluate the output features of the student model to determine the network module that can be used as the jump-out layer while maintaining the accuracy of the super-resolution image. The jump-out model can score and evaluate the output features of each layer of network modules to determine whether to use the network module as the jump-out layer. The output features of the network module as the jump-out layer carry sufficient feature information of the image block for the subsequent sub-network, that is, if the forward calculation continues, the output features of the next layer of the network module of the jump-out layer and the output features of the network module as the jump-out layer, for the subsequent sub-network, the difference between the two output features can be ignored, for example, the difference value is less than a specified threshold.

[0043] Among them, in the adaptive reasoning sub-network, each network module can be configured with a layer identifier to indicate the relative position number / sequence number and forward calculation progress of the network module in the adaptive reasoning sub-network. The layer identifier of the network module as the jump-out layer can be the current network module position number / sequence number in the current resolution branch. The layer identifier of the network module as the jump-out layer can indicate to stop the forward calculation of the current resolution branch and jump out of the resolution branch to the subsequent sub-network. For the jump-out method, the jump-out model can evaluate the network modules of each layer of multiple resolution branches (the branches can be automatically converted). Based on the size of the jump-out score, it can jump out of the adaptive reasoning sub-network in any resolution branch to the sub-network used to improve the resolution in the subsequent sub-network to complete super-resolution image reconstruction. In some possible application scenarios of adaptive inference sub-network types, the jump-out model can evaluate the network modules of each layer of the current resolution branch. Based on the jump-out score, it can jump from the position number / sequence number in the low-resolution branch to the corresponding position number / sequence number in the (medium / ) high-resolution branch. At this time, the subsequent sub-network can be a high-resolution branch. The jump-out model can further evaluate the output features of the network modules of each layer of the high-resolution branch to determine whether to jump to the sub-network used to improve the resolution to complete super-resolution image reconstruction.

[0044] In the embodiments of this specification, the original image can be used as a training sample, and the original image can be a low-resolution image relative to the output image predicted by the image super-resolution reconstruction model. Figure 1 , the aforementioned training method may include:

[0045] S1, using the image block of the original image as a training sample, and inputting it into the teacher model and the student model respectively, so as to obtain the reference features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model. The reference features and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model may correspond to the same input image block.

[0046] In some possible implementations, the original image can be preprocessed to obtain an image block of the original image. The preprocessing operation may include an image block operation, a mean reduction operation, an image flipping operation, and the like. The original image can be divided into a group of multiple image blocks by an image block operation, and the group of multiple image blocks can be subjected to a mean reduction operation and / or an image flipping operation. Among them, for the same image block, the low-resolution branch in the adaptive reasoning subnetwork can first perform reasoning calculations to perceive the feature information of the image block. After jumping out of the low-resolution branch, the medium / high-resolution branch can calculate the image block corresponding to the image block, and at the same time, the output features of the jump-out layer network module when the low-resolution branch jumps out can be fused. In this way, the jump-out mechanism (the medium-resolution branch can also jump out to the high-resolution branch) applied to the adaptive reasoning subnetwork in the student model by the jump-out model, the network modules in the resolution branch do not need to all participate in the forward calculation in sequence, which can reduce the computing overhead of the device. Multiple image blocks can be processed in parallel, for example, using multiple threads to reduce the running time.

[0047] For some possible preferred examples, see Figure 2 , the low-resolution image (original image) can be recorded as X l , the corresponding image block is The corresponding high-resolution image block generated by the algorithm is Real high-resolution image X gt , the corresponding image block is The teacher model used for benchmark features in output feature consistency learning is M t (·), the partial sub-network that can be used for adaptive reasoning is The single network module is The image super-resolution reconstruction model actually used online can be a student model with M s (·), the partial sub-network that can be used for adaptive reasoning is The single network module is The jump-out (prediction) model used to determine the model jump-out situation in adaptive reasoning is R(·).

[0048] The aforementioned step S1 may include:

[0049] S101, the low-resolution image X l As training data, the preprocessed image blocks As a training sample for the image super-resolution reconstruction model, l can be the image identifier, l p Can be a low resolution image X l The image block identifier;

[0050] S102, a teacher model can be obtained, and the model structure of the adaptive reasoning sub-network of the teacher model is the same as that of the adaptive reasoning sub-network of the student model, and the model structure of the sub-network for improving the resolution of the teacher model is also the same as that of the sub-network for improving the resolution of the student model.

[0051] The image blocks may be sent to the teacher model and the student model for forward (inference) calculation respectively. The aforementioned step S1 may also include:

[0052] S103, the image block The output feature F of the (most) terminal layer network module of the adaptive reasoning sub-network in the teacher model can be retained when the forward calculation is executed to the adaptive reasoning sub-network. t , as the benchmark feature for distillation learning;

[0053] S104, the image block The output features F of each layer of network modules in the adaptive reasoning sub-network in the student model can be retained when the forward calculation is executed to the adaptive reasoning sub-network. s· , s· can represent the network modules of each layer in the student model.

[0054] Among them, the benchmark features can carry sufficient image feature information and can be used for super-resolution image reconstruction.

[0055] In the embodiments of this specification, a jump-out (prediction) model can be used to evaluate the output features of the student model relative to the baseline features. The jump-out model can use any regression module, and the jump-out model can be used as a subnetwork of the student model or a scoring network parallel to the student model. The aforementioned training method can also include:

[0056] S2, inputting the output features of each layer of network modules in the student model into the jump-out model to obtain the jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as the jump-out layer.

[0057] The jump-out model scores the output features of each layer of network modules in the student model. The jump-out score can evaluate the difference between the output features of the target layer network module and the next layer network module (for example, in the same resolution branch) when they are used to reconstruct the image. In some possible examples of jump-out score calculation methods, the larger the jump-out score, the smaller the difference, thereby identifying the network module position that can terminate the current resolution branch in advance, thereby avoiding the consumption of computing resources and memory resources due to unreasonable forward calculations. In the embodiment of this specification, the aforementioned training method may also include:

[0058] S3, determining a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a next-layer network module of the target layer and the benchmark feature.

[0059] In some possible implementations, the network module of the target layer can be regarded as the network module of the i-th layer in the adaptive reasoning sub-network in the student model, and the network module of the next layer can be regarded as the network module of the i+1-th layer in the adaptive reasoning sub-network in the student model. The target layer and the next layer network modules can both be network modules in the same forward resolution branch.

[0060] In some possible preferred examples, the feature F output by each layer of the network module in the student model can be s· The corresponding jump-out score s is obtained by sending it to the jump-out model prediction. Then, the jump-out encoding value s is generated according to the difference between the output features of the network submodules of the teacher model and the student model. gt As a notation, step S3 may include:

[0061] S301: Determine a jump-out coding value based on a first difference value and a second difference value, and use the jump-out coding value as a mark corresponding to the image block.

[0062] Among them, the output feature F of the target layer si And the output feature F of the next layer network module of the target layer s(i+1) The corresponding jump code value can be calculated by the following formula:

[0063] s gti =1-tanh(-10*(log 10 mean((F s(i+1) -F t ) 2 )-log 10 mean((F si -F t ) 2 )))

[0064] In this formula, F si -F t can be the first difference value, F s(i+1) -F t It can be the second difference value, tanh is the hyperbolic tangent function, and mean is the mean function. The output features of the network submodules of the teacher model and the student model can be used as input samples of the jump-out model, and the jump-out coding value can be used as the sample (true value) label of the jump-out model, without the need to prepare additional samples and sample labels for the jump-out model. The mean square error (MSE) loss value can be selected as the loss value of the jump-out model. The jump-out model is optimized by the MSE loss. The aforementioned step S3 may also include:

[0065] S302: Determine a bounce loss value corresponding to the network module of the target layer based on the bounce score corresponding to the network module of the target layer and the bounce code value.

[0066] Among them, the bounce loss value corresponding to the network module of the target layer can be calculated by the following formula:

[0067] MSE(·)=(s i -s gti ) 2

[0068] In this formula, s i is the bounce score (of the output feature) corresponding to the network module of the target layer. The bounce loss values ​​corresponding to the network modules of each layer can be determined according to the calculation rule of the bounce loss values ​​corresponding to the network modules of the target layer. The aforementioned step S3 can also include:

[0069] S303: Determine the loss value of the jump-out model based on the jump-out loss value corresponding to each layer of network module.

[0070] In some possible preferred examples, the loss value L of the jump-out model is R It can be calculated by the following formula:

[0071]

[0072] In this formula, n is the number of network module layers in the adaptive reasoning sub-network, and the loss value L of the model is R It is the sum of the jump-out loss values ​​corresponding to each layer of network modules.

[0073] In the embodiments of this specification, a joint loss value can be constructed to bind the distillation learning with the learning process of jumping out of the model. The aforementioned training method can also include:

[0074] S4, adjusting the model parameters of the student model and the model parameters of the jump-out model based on the joint loss value, wherein the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0075] In some possible implementations, the joint loss value may be taken as the sum of the loss value of the jump-out model and the learning loss value; wherein the learning loss value may include the information enhancement loss value L of the student model e And the super-resolution loss value L S , and can be taken as the information enhancement loss value L of the student model e And the super-resolution loss value L S The sum of the information enhancement loss value can be used to simultaneously optimize the student model and the jump-out model to ensure the accuracy of the output features and the accuracy of the super-resolution image. The aforementioned training method can also include: determining the information enhancement loss value, and the information enhancement loss value can include:

[0076] E1, based on the first difference value, determine the information enhancement loss value corresponding to the network module of the target layer.

[0077] In some possible preferred examples, the information enhancement loss value e corresponding to the network module of the target layer is i It can be calculated by the following formula:

[0078] e i =(1-2·sigmoid(-avgpooling(|F si -F t |)) λ )·|F si -F t |

[0079] In this formula, sigmoid(·)=1 / (1+e u ), u=(-avgpooling(|F si -F t |)) λ , avgpooling is an average pooling function, λ is an adjustment coefficient and its value can be in the range of 0 to 1, which can provide edge information enhancement capability of the image block. The information enhancement loss value can also include:

[0080] E2, based on the information enhancement loss value corresponding to each layer of the network module, determine the information enhancement loss value of the student model, where the information enhancement loss value L of the student model E It can be calculated by the following formula:

[0081]

[0082] In this formula, the information enhancement loss value L of the student model isE is the sum of the information enhancement loss values ​​corresponding to the network modules of each layer. The aforementioned training method may also include: determining the super-resolution loss value, and the determining the super-resolution loss value may include:

[0083] V1, determining a super-resolution loss value based on the super-resolution image block predicted by the student model and the actual super-resolution image block.

[0084] In some possible preferred examples, the smoothL1loss(·) function can be used to determine the super-resolution loss value o corresponding to the j-th image block: j :

[0085]

[0086] In this formula, can be the jth image patch predicted by the student model, It can be the corresponding real high-resolution image block; VGG(·) is a network model proposed by the Visual Geometry Group, which can provide accuracy support; the smoothL1loss(·) function is:

[0087]

[0088] Determining the super-resolution loss value may also include:

[0089] V2, based on the super-resolution loss value of each image block, determines the super-resolution loss value of the student model.

[0090] The high-resolution image block X output by the entire student model prediction can be sp· and the corresponding real high-resolution image block X gtp , and sent to the VGG network model at the same time, and the corresponding features of the two are finally optimized using smoothL1loss(·). The super-resolution loss value of the student model is L S It can be calculated by the following formula:

[0091]

[0092] In this formula, N is the total number of image blocks of the original image. Based on the aforementioned loss value, the joint loss value is the sum of the loss value of the jump-out model, the information enhancement loss value of the student model, and the super-resolution loss value of the student model. The joint loss value L can be calculated by the following formula:

[0093] L=L E +L R +L S

[0094] During the training iteration process, the entire student model can be optimized through the back-propagation algorithm, while the teacher model can update the model parameters through the exponential moving average EMA strategy, and the changes in the model parameters of the teacher model will be more stable.

[0095] It should be noted that the above-mentioned loss value and jump code value calculation formulas are preferred examples, not the only limited implementation methods. The value size and design calculation formula can be adjusted according to specific demand scenarios and product characteristics, as well as based on testing and usage effects, such as linearly adjusting the loss value and jump code value, adding fixed fine-tuning values, etc.

[0096] In the embodiment of this specification, the capacity increment subnetwork in the feature self-distillation learning model, that is, the adaptive reasoning subnetwork in the model, is specifically to use the terminal layer output feature of the entire adaptive reasoning subnetwork in the teacher model as the benchmark feature, and then the output feature of each network module in the adaptive reasoning subnetwork in the student model is distilled and learned with the benchmark feature, and at the same time, in the image reconstruction process, the features of each module in the subnetwork are sent to the jump-out model to predict the model jump-out, which can adaptively jump out of the feature, and there is no need to prepare samples and sample labels for the jump-out model separately during training, so as to realize the image super-resolution reconstruction model of adaptive reasoning of any (sufficient information) network layer, which can reduce the computational overhead of the image reconstruction process, and on the equipment deployed by the same ordinary adaptive network model, the resource consumption is significantly reduced, and the network modules in each resolution branch do not need to be used for forward calculation. At the same time, it provides a basis for realizing large model applications in embedded devices. Further, the information enhancement loss value is used for distillation learning, which can enhance the edge features that are difficult to learn in the image features and reduce the block effect between image blocks, thereby improving the image super-resolution effect.

[0097] One embodiment of this specification provides an image super-resolution reconstruction method under the same inventive concept as the above embodiment, please refer to Figure 3 , which can be used in the test phase and practical application of the image super-resolution reconstruction model in the aforementioned embodiment, the image super-resolution reconstruction method may include:

[0098] N1, obtain the original image;

[0099] N2, inputting the original image into an image super-resolution reconstruction model to obtain a super-resolution image; wherein,

[0100] The image super-resolution reconstruction model is configured with a jump-out model as a sub-network; the image super-resolution reconstruction model is a student model in the aforementioned training method for the image super-resolution reconstruction model, and is a student model trained by the training method.

[0101] In some possible implementations, the foregoing step N2 may include:

[0102] N201, inputting multiple image patches of the original image into an image super-resolution reconstruction model to perform forward computation through an adaptive inference sub-network in the image super-resolution reconstruction model.

[0103] The original image can be preprocessed as described above, and multiple preprocessed image patches can be input into the trained image super-resolution reconstruction model. The foregoing step N2 may further include:

[0104] N202, inputting the output features of the network model of the target layer in the adaptive inference sub-network into a jump-out model to obtain a jump-out score corresponding to the network module of the target layer.

[0105] When the input image patch is forward-inferred in the network to the adaptive inference sub-network si part, according to the front-to-back order of each module in the sub-network, the output features F of the network module are obtained layer by layer

[0106] and the output features are fed into the jump-out model R(·) to predict the jump-out score. The foregoing step N2 may further include:

[0107] In the forward inference process, a jump-out model and a jump-out threshold can be used to judge the jump-out timing of the output features, that is, whether they are the output features of the network module of the jump-out layer. When the data in the forward computation passes through the adaptive inference sub-network in sequence, the forward inference is performed layer by layer according to the order of each layer network module in it. When the output features F si of the i-th (target layer) network module si fed into the jump-out network obtain a jump-out score higher than the jump-out threshold th, then jump out, and use the current output features F fed into the subsequent sub-network. At the same time the subsequent sub-modules do not need to perform forward computation. Therefore, if the target layer i < n, the computational consumption in the actual forward inference of the entire model can be reduced, where n is the number of network module layers in the adaptive inference sub-network (or the total number of network modules in the front-to-back order of the resolution branch). At this time, the jump-out score R(F si ) of the output features F si of the network module of the target layer by the jump-out model is:

[0108] R(Fsi )>th

[0109] The aforementioned step N2 may further include:

[0110] N204, if the jump-out score is less than or equal to the jump-out threshold, the next layer network module of the target layer in the adaptive reasoning sub-network continues to perform forward calculation to obtain the output features of the next layer network module;

[0111] At this time, the model jumps out of the target layer network module The output feature F si The bounce score R(F si ),for:

[0112] R(F si )≤th

[0113] The next layer network module can be The output features of are input to the jump-out model to obtain the jump-out score of the next layer of the target layer, and the current jump-out score can be returned to step N203. The above judgment using the jump-out threshold can be called an adaptive feature jump-out strategy.

[0114] In some possible implementations, the aforementioned step N2 may further include:

[0115] N205, the output features of the jump-out layer network module are input into the subsequent sub-network, and forward (relative to back propagation) reasoning is continued to obtain the super-resolution image block X sp. Then, multiple super-resolution image blocks are processed by splicing the image blocks to obtain the final complete super-resolution image X s .

[0116] In the embodiments of this specification, the image super-resolution reconstruction model trained by the feature self-distillation learning strategy has the capacity increment super-resolution capability, and the adaptive feature jump-out strategy is applied to the model adaptive reasoning (which can be performed by the adaptive sub-network), which reduces the computational overhead in the image reconstruction process, achieves lower computational consumption, and improves efficiency, so that larger models can be used in devices with equivalent hardware performance. The model adaptive reasoning of the embodiments of this specification can be applied to a variety of model structures (such as series-parallel composites), and the model structure of the sub-network used to improve the resolution is not limited, and any reconstruction sub-network can be used.

[0117] One embodiment of this specification provides a system for image super-resolution reconstruction under the same inventive concept as the above embodiment. Figure 4 , the system 400 may include:

[0118] An input module 401 is used to input the image blocks of the original image as training samples to the teacher model and the student model respectively, so as to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model;

[0119] A training module 402 is used to input the output features of each layer of network modules in the student model into the jump-out model to obtain a jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as a jump-out layer;

[0120] The training module 402 is further configured to determine a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature;

[0121] The training module 402 is also used to adjust the model parameters of the student model and the model parameters of the jump-out model based on a joint loss value, wherein the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0122] Optionally, the student model is an image super-resolution reconstruction model to be trained, the image super-resolution reconstruction model includes multiple resolution branches, each resolution branch includes an adaptive reasoning sub-network;

[0123] The adaptive reasoning sub-network of the teacher model and the adaptive reasoning sub-network of the student model have the same model structure;

[0124] The jump-out model is a sub-network of the student model.

[0125] Optionally, based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature, determining a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block comprises:

[0126] Determine a jump-out code value based on the first difference value and the second difference value, and use the jump-out code value as a mark corresponding to the image block;

[0127] Determine a bounce loss value corresponding to the network module of the target layer based on the bounce score corresponding to the network module of the target layer and the bounce code value;

[0128] Based on the jump-out loss values ​​corresponding to the network modules at each layer, the loss value of the jump-out model is determined.

[0129] Optionally, combine Figure 5 , the system 400 further includes a determination module 403, wherein the learning loss value includes an information enhancement loss value and a super-resolution loss value; the determination module 403 is used to:

[0130] Based on the first difference value, determining an information enhancement loss value corresponding to the network module of the target layer;

[0131] Based on the information enhancement loss values ​​corresponding to the network modules of each layer, the information enhancement loss value of the student model is determined.

[0132] Optionally, the determining module 403 is further configured to:

[0133] Based on the super-resolution image block predicted by the student model and the actual super-resolution image block, a super-resolution loss value of the student model is determined.

[0134] Optionally, the joint loss value is the sum of the loss value of the jump-out model, the information enhancement loss value of the student model, and the super-resolution loss value of the student model.

[0135] Optionally, the training module 402 is further used for:

[0136] Based on the model parameters of the student model and the exponential moving average strategy, the model parameters of the teacher model are updated.

[0137] Optionally, combine Figure 6 The system 400 further includes an image super-resolution reconstruction module 404, wherein the image super-resolution reconstruction module 404 is used to:

[0138] Get the original image;

[0139] The original image is input into an image super-resolution reconstruction model to obtain a super-resolution image; wherein,

[0140] The image super-resolution reconstruction model is configured with a jump-out model as a sub-network;

[0141] The image super-resolution reconstruction model is a student model in the aforementioned system, and is a trained student model.

[0142] Optionally, inputting the original image into an image super-resolution reconstruction model to obtain a super-resolution image comprises:

[0143] Inputting the plurality of image blocks of the original image into an image super-resolution reconstruction model to perform forward calculations through an adaptive inference subnetwork in the image super-resolution reconstruction model;

[0144] Inputting the output features of the network model of the target layer in the adaptive reasoning sub-network into the bounce model to obtain the bounce score corresponding to the network module of the target layer;

[0145] If the jump-out score is greater than the jump-out threshold, the output feature is input into a subsequent sub-network in the image super-resolution reconstruction model to obtain a plurality of super-resolution image blocks, and a super-resolution image is obtained by splicing.

[0146] Optionally, the image super-resolution reconstruction module is further used for:

[0147] If the jump-out score is less than or equal to the jump-out threshold, the next layer network module of the target layer in the adaptive reasoning sub-network continues to perform forward calculation to obtain the output features of the next layer network module;

[0148] The output features of the next-layer network module of the target layer are input into the bounce model to obtain a bounce score corresponding to the next-layer network module of the target layer.

[0149] An embodiment of this specification provides an electronic device under the same inventive concept as the above embodiment, and the electronic device may include: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method in the above embodiment by executing the instructions stored in the memory. Please refer to Figure 7 , provides an exemplary electronic device, whose internal structure diagram can be shown as Figure 7 As shown, it can be a server, an industrial computer, a user device, a terminal device, a microcontroller, etc. The electronic device includes a processor A01, a network interface A02, and a memory connected by a bus. Among them, the processor A01 of the electronic device is used to provide computing, instruction processing and control capabilities. The memory of the electronic device includes a memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01 and a computer program B02. The memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the electronic device is used to communicate with the network. When the computer program B02 is executed by the processor A01, the method in the aforementioned embodiment is implemented.

[0150] In one embodiment of the present specification, when the computer program B02 is executed by the processor A01, the implemented method may include:

[0151] The image blocks of the original image are used as training samples and are input into the teacher model and the student model respectively to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model;

[0152] Input the output features of each layer of network modules in the student model into the jump-out model to obtain the jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as the jump-out layer;

[0153] Determine a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature;

[0154] Based on the joint loss value, the model parameters of the student model and the model parameters of the jump-out model are adjusted, and the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0155] In another embodiment of the present specification, the electronic device may further include an image capturing unit (e.g., an image sensor) or may be connected to an image capturing device (e.g., a camera). When the computer program B02 is executed by the processor A01, the implemented method may include:

[0156] Get the original image;

[0157] The original image is input into an image super-resolution reconstruction model to obtain a super-resolution image; wherein,

[0158] The image super-resolution reconstruction model is configured with a jump-out model as a sub-network;

[0159] The image super-resolution reconstruction model is a student model in the aforementioned image super-resolution reconstruction model training method, and is a student model trained by the training method.

[0160] One embodiment of the present specification provides a machine-readable storage medium under the same inventive concept as the aforementioned embodiment, storing machine instructions. When the machine instructions are executed on a machine, the machine executes the method in the aforementioned embodiment. The machine may include a computer and a device with communication, computing and instruction processing capabilities.

[0161] In one embodiment of the present specification, when the machine instruction is executed on a machine, the method executed by the machine may include:

[0162] The image blocks of the original image are used as training samples and are input into the teacher model and the student model respectively to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model;

[0163] Input the output features of each layer of network modules in the student model into the jump-out model to obtain the jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as the jump-out layer;

[0164] Determine a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature;

[0165] Based on the joint loss value, the model parameters of the student model and the model parameters of the jump-out model are adjusted, and the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

[0166] In another embodiment of the present specification, when the machine instruction is executed on the machine, the method executed by the machine may include:

[0167] Get the original image;

[0168] The original image is input into an image super-resolution reconstruction model to obtain a super-resolution image; wherein,

[0169] The image super-resolution reconstruction model is configured with a jump-out model as a sub-network;

[0170] The image super-resolution reconstruction model is a student model in the aforementioned image super-resolution reconstruction model training method, and is a student model trained by the training method.

[0171] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this specification can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0172] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0173] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the protection scope of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the protection scope of this specification.

[0174] The systems or modules described in the above embodiments may be implemented by computers or other entities, or by products with certain functions. A typical implementation device is a computer.

[0175] Machine-readable storage media can be computer storage media, which can include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0177] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A training method for an image super-resolution reconstruction model, characterized in that: The training method comprises: The image blocks of the original image are used as training samples and are input into the teacher model and the student model respectively to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model; Input the output features of each layer of network modules in the student model into the jump-out model to obtain the jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as the jump-out layer; Determine a loss value of the jump-out model corresponding to the jump-out score and a label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature; Based on the joint loss value, the model parameters of the student model and the model parameters of the jump-out model are adjusted, and the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

2. The method for training an image super-resolution reconstruction model according to claim 1, characterized in that: The student model is an image super-resolution reconstruction model to be trained, and the image super-resolution reconstruction model includes multiple resolution branches, each resolution branch includes an adaptive reasoning sub-network; The adaptive reasoning sub-network of the teacher model and the adaptive reasoning sub-network of the student model have the same model structure; The jump-out model is a sub-network of the student model.

3. The training method of the image super-resolution reconstruction model according to claim 1, characterized in that: The step of determining the loss value of the jump-out model corresponding to the jump-out score and the label corresponding to the image block based on a first difference value between an output feature of a network module of a target layer in the student model and the benchmark feature, and a second difference value between an output feature of a network module of a next layer of the target layer and the benchmark feature, comprises: Determine a jump-out code value based on the first difference value and the second difference value, and use the jump-out code value as a mark corresponding to the image block; Determine a bounce loss value corresponding to the network module of the target layer based on the bounce score of the network module of the target layer and the bounce code value; Based on the jump-out loss values ​​corresponding to the network modules at each layer, the loss value of the jump-out model is determined.

4. The method for training an image super-resolution reconstruction model according to claim 3, characterized in that: The learning loss value includes an information enhancement loss value and a super-resolution loss value; the training method further includes determining the information enhancement loss value, and the determining the information enhancement loss value includes: Based on the first difference value, determining an information enhancement loss value corresponding to the network module of the target layer; Based on the information enhancement loss values ​​corresponding to the network modules of each layer, the information enhancement loss value of the student model is determined.

5. The method for training an image super-resolution reconstruction model according to claim 4, characterized in that: The training method further includes determining a super-resolution loss value, wherein determining the super-resolution loss value includes: Based on the super-resolution image block predicted by the student model and the actual super-resolution image block, a super-resolution loss value of the student model is determined.

6. The method for training an image super-resolution reconstruction model according to claim 5, characterized in that: The joint loss value is the sum of the loss value of the jump-out model, the information enhancement loss value of the student model, and the super-resolution loss value of the student model.

7. The method for training an image super-resolution reconstruction model according to claim 1, characterized in that: The training method further comprises: Based on the model parameters of the student model and the exponential moving average strategy, the model parameters of the teacher model are updated.

8. A method for super-resolution image reconstruction, characterized in that: The image super-resolution reconstruction method comprises: Get the original image; The original image is input into an image super-resolution reconstruction model to obtain a super-resolution image; wherein, The image super-resolution reconstruction model is configured with a jump-out model as a sub-network; The image super-resolution reconstruction model is a student model in the image super-resolution reconstruction model training method according to any one of claims 1 to 7, and is a student model trained by the training method.

9. The image super-resolution reconstruction method according to claim 8, characterized in that: The step of inputting the original image into an image super-resolution reconstruction model to obtain a super-resolution image comprises: Inputting the plurality of image blocks of the original image into an image super-resolution reconstruction model to perform forward calculations through an adaptive inference subnetwork in the image super-resolution reconstruction model; Inputting the output features of the network model of the target layer in the adaptive reasoning sub-network into the bounce model to obtain the bounce score corresponding to the network module of the target layer; If the jump-out score is greater than the jump-out threshold, the output feature is input into a subsequent sub-network in the image super-resolution reconstruction model to obtain a plurality of super-resolution image blocks, and a super-resolution image is obtained by splicing.

10. The image super-resolution reconstruction method according to claim 9, characterized in that: The image super-resolution reconstruction method also includes: If the jump-out score is less than or equal to the jump-out threshold, the next layer network module of the target layer in the adaptive reasoning sub-network continues to perform forward calculation to obtain the output features of the next layer network module; The output features of the next-layer network module of the target layer are input into the bounce model to obtain a bounce score corresponding to the next-layer network module of the target layer.

11. A system for image super-resolution reconstruction, characterized in that: The system comprises: An input module, used to input the image blocks of the original image as training samples to the teacher model and the student model respectively, so as to obtain the benchmark features output by the adaptive reasoning sub-network in the teacher model and the output features of each layer of the network module in the adaptive reasoning sub-network in the student model; A training module, used for inputting the output features of each layer of network modules in the student model into the jump-out model to obtain a jump-out score corresponding to each layer of network modules, wherein the jump-out score is used to indicate the network module in the student model as a jump-out layer; The training module is further used to determine the loss value of the jump-out model corresponding to the jump-out score and the label corresponding to the image block based on a first difference value between the output feature of the network module of the target layer in the student model and the benchmark feature, and a second difference value between the output feature of the network module of the next layer of the target layer and the benchmark feature; The training module is also used to adjust the model parameters of the student model and the model parameters of the jump-out model based on a joint loss value, wherein the joint loss value is determined based on the loss value of the jump-out model and the learning loss value.

12. A machine-readable storage medium storing machine instructions, which, when executed on a machine, enables the machine to execute the method of any one of claims 1 to 10.

Images