Multi-scale super-resolution reconstruction method and terminal equipment for marine ship images

Abstract

The present invention discloses a multi-scale super-resolution reconstruction method and terminal device for marine ship images, which obtains high-resolution marine ship images, synthesizes low-resolution images using the high-resolution marine ship images; uses the low-resolution images to train a degradation characterization estimation network to obtain a degradation characterization vector of the low-resolution image; uses the low-resolution image and the degradation characterization vector as inputs of a reconstruction model, trains the reconstruction model, and obtains a multi-scale super-resolution reconstruction model. The present invention can obtain more accurate degradation characterization estimation in realistic complex scenes, improve the long-tail distribution problem of degraded images, solve the problem that the existing super-resolution methods have poor performance in non-specific scenes, and can cope with various complex scenes of marine ship image super-resolution applications.

Description

Technical Field

[0001] The invention relates to image reconstruction technology, in particular to a multi-scale super-resolution reconstruction method and terminal equipment for marine ship images. Background Art

[0002] Image super-resolution is a basic task in image processing and computer vision, and its purpose is to reconstruct the observed low-resolution image into a high-resolution image. As an ill-posed problem, high-resolution images in reality often exist in a variety of different degradation states. The difficulty of the image super-resolution task lies in recovering the detailed features of the image such as texture and contour from a one-to-many mapping relationship. The existing image super-resolution methods can be divided into two categories according to the implementation method: traditional super-resolution reconstruction and super-resolution reconstruction based on deep learning. Traditional image super-resolution reconstruction methods such as bicubic interpolation can generate smoother pixel edges than basic algorithms, but this method oversimplifies the image super-resolution problem and usually leads to overly smooth textures. Traditional super-resolution methods based on learning require pre-modeling, have poor reusability and complex implementation, and are difficult to meet the requirements of real-time processing of ship images in reality. In addition, typical traditional super-resolution methods based on reconstruction include convex set projection method, Bayesian analysis method and maximum a posteriori probability method. Compared with interpolation methods, this type of method can alleviate the blurring effect in interpolation methods by introducing some prior assumptions of the image. However, such methods are limited by prior knowledge. In real-world applications of ship image super-resolution, when the prior information provided is insufficient or missing, the image reconstruction quality will be greatly reduced. Based on the above facts, the image processing application field urgently needs to achieve high-resolution images through new algorithms.

[0003] In recent years, with the rapid development of artificial intelligence technology, deep learning has achieved great success in the field of computer vision. Thanks to the powerful representation extraction and model fitting capabilities of deep convolutional neural networks, super-resolution methods based on deep learning have made significant progress and have greatly surpassed traditional methods in performance. Super-resolution methods based on deep learning complete the construction of the model by learning the mapping between low-resolution images and corresponding high-resolution images. Existing image super-resolution methods based on deep learning usually use known biquadratic interpolation to generate low-resolution images for training deep super-resolution models. However, in the problem of real-time ship image processing, the operation of image scaling, rotation, and spatial transformation is affected by many interference factors. The mapping relationship between the input low-resolution image and high-resolution often does not obey the ideal degradation assumption. In the real ship image application scenario, it is difficult for users to obtain the degradation mode of low-resolution images or accurately estimate the degraded images. When the predefined blur kernel is different from the real blur kernel, the learning-based method will have serious performance degradation problems. This kernel mismatch phenomenon will cause undesirable artifacts in the output super-resolution image. Specifically, if the input kernel is smoother than the real kernel, the output image will be blurred. On the contrary, if the input kernel is sharper than the correct kernel, the result will be over-distorted and produce obvious ringing effects. Some methods try to train super-resolution models with a wide range of degraded images to enhance their generalization ability. When the degradation space deepens, the invariant properties between images become difficult to capture, and the model will have inevitable estimation distortion problems. On the one hand, low-resolution ship images generated under different degradation conditions contain different degradation representations. When incompletely aligned or mislabeled degraded images are input into the estimator, the biased estimation produced by the network leads to serious artifacts in the results. On the other hand, low-resolution ship images with different degradations may produce degraded images with the same representation, and the degree of image degradation is difficult to control using linear parameters. Therefore, most of the low-resolution images are concentrated at a certain degradation level, especially severe degradation, which leads to model collapse and can only produce satisfactory super-resolution results in specific scenarios.

[0004] Existing methods require paired data with degraded feature labels. Collecting such image data is time-consuming and laborious, and does not cover real-world situations well. When the predefined blur kernel is different from the actual kernel label, this kernel mismatch phenomenon will cause undesirable artifacts in the output super-resolution image. Existing methods usually train super-resolution models under ideal conditions or special assumed degradation. When a large range of degraded images are used to train super-resolution models to enhance their generalization ability, the dataset is likely to have a long-tail distribution problem, affecting the effect of image super-resolution. Existing methods are often designed for general image super-resolution tasks. When faced with the complex imaging environment of marine ship images, when degraded images that are not fully aligned or incorrectly labeled are input, the model cannot accurately restore the image's texture, contour and other information.

[0005] CN114626989A provides a blind image super-resolution method and system based on domain adversarial conversion, which uses a known downsampling method to generate a low-resolution image (such as biquadratic downsampling, bilinear downsampling) or uses a simple degradation model in a blind background. However, real-world applications often do not obey the pre-assumptions, which will cause these methods to distort the representation estimation of the degraded image, thereby affecting the final super-resolution effect. This method ignores the pathological nature of the super-resolution problem. High-resolution images often have multiple different low-resolution degraded images, and the degree of degradation shows a long-tail distribution problem, which makes the training domain distribution and the test domain distribution mismatch, resulting in the super-resolution model performing well only in some scenarios and unable to cope with the wider range of degradation in real application scenarios. The blind image super-resolution method requires supervision with degraded image label information, but this label information is missing in real application scenarios. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-scale super-resolution reconstruction method and terminal device for marine ship images in view of the shortcomings of the existing technology, so as to obtain a more accurate degradation characterization estimation in real complex scenes, improve the long-tail distribution problem of degraded images, and solve the problem that the existing super-resolution methods only perform well in specific scenarios.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-scale super-resolution reconstruction method for marine ship images, comprising the following steps:

[0008] S1. Acquire a high-resolution marine ship image, and use the high-resolution marine ship image to synthesize a low-resolution image; use the low-resolution image to train a degradation representation estimation network to obtain a degradation representation vector of the low-resolution image;

[0009] S2, taking the low-resolution image and the degradation representation vector as inputs of a reconstruction model, training the reconstruction model, and obtaining a multi-scale super-resolution reconstruction model;

[0010] The reconstruction model includes a shallow feature extraction module and multiple cascaded degradation representation fusion groups, and the outputs of the shallow feature extraction module and the last degradation representation fusion group are spliced ​​and input into an upsampling layer;

[0011] The degradation representation fusion group includes sequentially connected degradation representation fusion convolution modules and multiple cascaded basic blocks; the degradation representation fusion convolution module includes multiple degradation units, the degradation unit includes a DRF convolution layer, the DRF convolution layer includes a convolution module, a fully connected module and a first convolution layer, the inputs of the convolution module and the fully connected module are both the degradation representation vector, the input of the first convolution layer is the low-resolution image, the first convolution layer is connected to the first reshaping layer, the output of the first reshaping layer and the output of the fully connected module are convolved and used as the input of the first activation layer, the first activation layer is connected to the second reshaping layer, and the second reshaping layer is connected to the second convolution layer; the output of the convolution module is convolved with the output of the first convolution layer, and then spliced ​​with the output of the second convolution layer, the spliced ​​result is used as the input of the second activation layer, the second activation layer is connected to the DRF convolution layer of the next degradation unit, and the DRF convolution layer is sequentially connected to the third activation layer, the third convolution layer and the fifth activation layer; the output of the fifth activation layer is spliced ​​with the degradation representation vector and used as the input of the next degradation representation fusion module.

[0012] The present invention can obtain more accurate degradation characterization estimation in real complex scenes, improve the long-tail distribution problem of degraded images, solve the problem that existing super-resolution methods have poor performance in non-specific scenes, and can cope with various complex scenes of marine ship image super-resolution applications.

[0013] In step S1, the low-resolution image The expression is: ;in, represents filtering, represents the convolution operation, Represents a scale factor The downsampling operation, represents the adjustable noise level, JPEG represents image compression, Indicates the compression level of JPEG. It is a high-resolution image of ships at sea.

[0014] The present invention uses a more complex degradation model, which can generate a wider range of degradation images and improve the generalization ability of the model.

[0015] In step S1, the specific implementation process of obtaining the degradation characterization vector includes:

[0016] The high-resolution marine ship images are aligned, and positive sample pairs are obtained using the aligned image blocks. The positive sample pairs are resampled to obtain degraded image blocks. The expression of the degraded image blocks is: ;in, is the degraded image block, Based on parameter weights The degradation model after resampling, is a high-resolution marine image. Express Perform random cropping to obtain paired image blocks ;

[0017] The degraded image block is used as the input of the degradation representation estimation network to obtain the degradation representation estimation vector : ; represents the aligned degraded image patches, is the output of the encoder.

[0018] The present invention uses standard deviation to constrain the positive sample definition, so that the aligned image blocks are input into the degradation representation encoder, and a better estimation result can be obtained. The present invention uses contrastive learning for the training of the degradation representation encoder, which does not require the degradation label of the image, which is also unavailable in the real world. This makes it feasible to train low-resolution ship images in the real world. In addition, the present invention can obtain more accurate degradation representation estimation in real complex scenes and improve the long-tail distribution problem of degraded images.

[0019] In step S1, the degradation characterization estimation network includes an encoder connected in sequence and two fully connected layers connected in series; the input of the encoder is a degraded image block.

[0020] The degradation characterization estimation network loss function It is expressed as: ;in, represents the batch size of degraded image patches, represents negative samples, is the number of negative samples in the momentum encoder, is a hyperparameter.

[0021] The last degradation representation fusion group is connected to the upsampling layer through an output convolution layer to facilitate the subsequent residual feature fusion and reconstruction.

[0022] In the present invention, the basic block includes one or more of a residual block, a residual dense block, a multi-scale residual block, a residual channel attention block, a dense projection unit, and a residual Transformer block.

[0023] In the present invention, the convolution module includes two convolution layers connected in series; the fully connected module includes two fully connected layers connected in series.

[0024] The method of the present invention further comprises:

[0025] S3. Acquire a ship image at sea, input the ship image at sea into a trained degradation representation network to obtain a degradation representation vector, input the degradation representation vector and a low-resolution image of the ship image at sea into a multi-scale super-resolution reconstruction model to obtain a reconstructed image.

[0026] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.

[0027] Compared with the prior art, the present invention has the following beneficial effects: the present invention can obtain more accurate degradation characterization estimation in real complex scenes, improve the long-tail distribution problem of degraded images, solve the problem that the existing super-resolution method only performs well in specific scenes, and the model supports and Super-resolution reconstruction at multiple scales to cope with diverse and complex scenarios of marine ship image super-resolution applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the principle of adaptive alignment comparison learning in an embodiment of the present invention.

[0029] Figure 2 4 is a structural diagram of a degradation representation fusion module for a degradation representation and a low-resolution image in an embodiment of the present invention.

[0030] Figure 3 This is a framework of a ship image super-resolution method based on adaptive alignment contrast learning in an embodiment of the present invention.

[0031] Figure 4 for Figure 2 The structure diagram of the convolution module.

[0032] Figure 5 for Figure 2 The structure diagram of the fully connected module. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] The embodiment of the present invention proposes a multi-scale super-resolution method for visible light images of marine ships based on adaptive alignment contrast learning. The super-resolution scheme based on deep learning includes two stages: training and reasoning.

[0036] Training phase: The technology consists of three steps: 1. Artificial low-resolution image synthesis channel; 2. Estimation of degradation representation of low-resolution images; 3. Multi-scale high-resolution ship image reconstruction.

[0037] Step 1 is the artificial low-resolution image synthesis channel, and its process can be expressed as ,in is a degradation model, is a high-resolution marine ship image. is a low-resolution marine ship image. Denotes the downsampling scale factor. Degradation model The closer the degradation process is to reality, the better the super-resolution model The better the effect. Previous methods usually use a classical degradation model with Gaussian filtering and additive Gaussian noise. However, the classical degradation model is not enough to cover real-world situations. A multi-order degradation model can be instantiated by using more operations, such as more generalized filters, different downsampling methods, more noise, and JPEG compression. For real-world marine ship image super-resolution application scenarios, the more complex degradation model used in the embodiment of the present invention can be expressed as:

[0038]

[0039] in represents filtering, represents the convolution operation, Represents a scale factor The downsampling operation, represents the adjustable noise level, JPEG represents image compression, is the degree of compression, Refers to the degree of compression JPEG compression. The degradation model blurs, downsamples, de-noises, and compresses the image in sequence. Each operation has multiple types of modules and parameters. Based on the above method, low-resolution images can be artificially synthesized from high-resolution images, thereby training super-resolution models for paired labeled data.

[0040] The second step is to perform self-supervised low-resolution marine ship degraded image representation estimation based on adaptive alignment contrast learning. Adaptive alignment contrast learning includes two parts: alignment and adaptive degradation sampling. By combining the two for degradation feature estimation, our method can obtain more accurate degradation representation estimation in realistic and complex marine ship scenes, improve the long-tail distribution problem of degraded images, and solve the problem that existing super-resolution methods only perform well in specific scenarios, so as to cope with the diverse and complex scenarios of marine ship image super-resolution applications.

[0041] Based on the low-resolution image obtained in step 1 , traditional contrastive learning methods usually Random cropping is performed to obtain two sub-image blocks to construct positive samples. This may lead to inaccurate alignment and thus cause degradation estimation errors. generate Depends on The parameter setting of random parameters may bring about the long-tail distribution problem of degraded images. In the embodiment of the present invention, these two processes will be subject to conditional constraints. Specifically, more compact alignment and greater concentration are the key to ensure that the degraded representation between low-resolution images maintains mapping invariance, which is very important for contrastive learning. The compactness and concentration of the alignment are guaranteed by the following two conditions:

[0042]

[0043] Our constraint goal is to find a A subset of , depends on and . express The set of all cropped sub-image patches, represents any positive sample pair, express degree of compactness. Respectively represent the number of samples In the collection The proportion of express The degradation and downsampling of high-resolution images will keep the image distance in the degraded space. The first criterion is used to control the set The second criterion is used to control the sample range of The number of samples in the medium. The smaller, The larger the value, the greater the distance degradation. In particular, Represents the degenerate distance:

[0044]

[0045] in, represents the degenerate representation encoder, represents the second-order norm. The essence is to rely on the degradation model Specifically, the first criterion is calculated as follows:

[0046]

[0047] in, It means calculating the sample standard deviation. Represents the upper bound of the constraint difference. By calculating the distance of all samples, the corresponding , substituting into the degenerate space, we can obtain γ. This constraint process is recorded as , then align the image blocks , the input is any pair of image patches , the output is the aligned image patch The second criterion calculates the first stage artificially synthesized low-resolution image by solving the optimization equation The parameter weights of the process:

[0048]

[0049] in, represents an image quality evaluation index, and SSIM is used in the embodiment of the present invention. Represents the calculation of the frequency histogram of the degraded image block, Indicates the sampling step size. represent uniform distribution and normal distribution respectively. represents the parameters in the degradation model, represents the sampling interval weight in the degradation model, Represents the interval length. This formula recalculates the parameter weights , control set Medium sample concentration .

[0050] In the above process, we obtained a specific subset The compactness and concentration , the next step is The process is recalculated, that is, the adaptive alignment process proposed in the embodiment of the present invention, which can be expressed as:

[0051]

[0052] in, represents a batch of high-resolution image patches, , is a batch of high-resolution images, Express Perform random cropping to obtain paired image blocks , indicating Paired image patches Align and get the aligned image blocks , forming batches of high-resolution image patches . Based on parameter weights The degradation model after resampling is finally output Batches of low-resolution image patches, namely:

[0053]

[0054] In the degradation representation extraction stage based on contrastive learning, we use the adaptively aligned image patches Perform batch operations, recorded as ,in represents the aligned degraded image patches, represents the batch size. We use As a loss function:

[0055]

[0056] in, represents negative samples, is the number of negative samples in the momentum encoder, is a hyperparameter. , we characterize the encoder for the degradation , and train for 100 epochs. Accepts two inputs , after a two-layer full connection, the output estimate is a degraded representation ,Right now:

[0057]

[0058] During training, the batch size is set to 32. For each high-resolution image, we use adaptive alignment to crop two image patches as positive samples, satisfying the two constraints in the second stage. The image patch size is set to 192×192. When the super-resolution downsampling factor is set to 4, the size of the degraded image patch is 48×48. The size of the blur kernel is set to 21×21. is 0.1. Set is 0.07, is 1024. During the optimization process, the Adam optimizer is used and the is 0.9, is 0.999, and the learning rate is set to .

[0059] Step 3: Based on adaptive alignment contrast learning, the embodiment of the present invention proposes a multi-scale super-resolution method for surface marine ship images. This part includes three stages: 1. shallow feature extraction and fusion; 2. deep feature extraction; 3. high-resolution image reconstruction.

[0060] Shallow feature extraction and fusion: given a low-resolution image patch (in The convolutional layer is better at early visual processing and has more stable optimization and results. We first use a The convolutional layer For shallow feature extraction:

[0061]

[0062] in Represents the shallow features of the image. Further, we use the degraded representation obtained in the representation estimation stage of the low-resolution degraded image Input into the super-resolution reconstruction network for feature fusion. Specifically, the degradation representation First, through a two-layer fully connected layer and a reshape layer (reshape function). Then we get an estimated kernel (in represents the size of the filter), Acts on To get The final fusion shallow features will be represented on a tensor scale by and Merge to get. That is:

[0063]

[0064] Deep feature extraction: We use shallow feature extraction and fusion to obtain As the input of the deep feature extraction module, it is as follows:

[0065]

[0066] in, Represents a deep feature extraction module, which is a typical block-group structure. Specifically: a deep feature extraction module consists of Degraded representation fusion group Composition, each group Include blocks. That is: The proposed method supports the free combination of various group-block structures. We use a 5×5 architecture with 5 groups and 5 blocks in each group. The first block in each group is the degradation representation fusion module (corresponding to Figure 2 , ), used for the above feature fusion operation, the remaining four blocks support the use of the following basic modules: residual block (RB), residual dense block (RDB), multi-scale residual block (MSRB), residual channel attention block (RCAB), dense projection unit (DBU) and residual Transformer block (RSTB), a feature fusion block and four basic modules form a degraded representation fusion group. In addition to the basic blocks, there is a 3×3 convolution layer at the end of each network for bias induction, so as to facilitate the subsequent residual feature fusion and reconstruction. In general: each group produces an intermediate feature:

[0067]

[0068]

[0069] High-resolution image reconstruction: There is an upsampling layer after the tail convolution layer for high-resolution image reconstruction. We use shallow features and deep features As input, reconstruct a high-resolution image:

[0070]

[0071] in To rebuild the module function, represents the reconstructed high-resolution image. Shallow features mainly contain low-frequency information in the image, while deep features mainly contain high-frequency information that is easy to lose. We use the sub-pixel convolution layer as the reconstruction function, connect the shallow features and deep features through residual jumps, and input them into the reconstruction function for high-resolution image reconstruction. We record the entire reconstruction process as , which accepts low-resolution images and degradation characterization As input, the output is In the high-resolution image reconstruction stage, we use Norm as loss function:

[0072]

[0073] in and They are the original high-resolution image block and the reconstructed high-resolution image block respectively.

[0074] In use Degraded Characterization Encoder After training for 100 epochs, we use The entire network is trained for 500 epochs, and the learning rate decays to , and the learning rate is adjusted using an adaptive method. During training, the batch size is set to 32. For each high-resolution image, we use adaptive alignment to crop two image patches as positive samples to satisfy the two constraints in the second stage. The image patch size is set to 192×192. When the super-resolution downsampling factor is set to 4, the size of the degraded image patch is 48×48. The size of the blur kernel is set to 21×21. Settings is 0.1. Set is 0.07, is 1024. During the optimization process, the Adam optimizer is used and the is 0.9, The experimental platform is Windows 11, and the hardware uses Intel CPU 13900K and NVIDIA GPU RTX 4090 24G. The super-resolution technology of ship images proposed in the embodiment of the present invention supports multi-scale applications to provide high-degree-of-freedom control for marine operators. Model support and Super-resolution reconstruction at multiple scales by modifying the scale parameters in the downsampling stage and the super-resolution reconstruction stage to achieve the goal.

[0075] Inference phase: In the training phase, we obtain the degraded representation encoder and super-resolution reconstruction models In the training phase, given a real-time image of a ship at sea , which is input into the degenerate representation encoder , in the inference phase, Takes an image as input and outputs an estimated degradation representation Then, the low-resolution ship image and degradation characterization As input super-resolution reconstruction model , the output is .

[0076] Example 2

[0077] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0078] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.

[0079] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0080] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0081] Example 3

[0082] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.

[0083] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0087] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0088] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multi-scale super-resolution reconstruction method for marine ship images, characterized in that: The following steps are involved: S1, obtaining a high-resolution marine ship image, and synthesizing a low-resolution image using the high-resolution marine ship image; Using the low-resolution image to train a degradation representation estimation network to obtain a degradation representation vector of the low-resolution image; S2, taking the low-resolution image and the degradation representation vector as inputs of a reconstruction model, training the reconstruction model, and obtaining a multi-scale super-resolution reconstruction model; The reconstruction model includes a shallow feature extraction module and multiple cascaded degradation representation fusion groups, and the outputs of the shallow feature extraction module and the last degradation representation fusion group are spliced ​​and input into an upsampling layer; The degradation representation fusion group includes sequentially connected degradation representation fusion convolution modules and multiple cascaded basic blocks; the degradation representation fusion convolution module includes multiple degradation units, the degradation unit includes a DRF convolution layer, the DRF convolution layer includes a convolution module, a fully connected module and a first convolution layer, the inputs of the convolution module and the fully connected module are both the degradation representation vector, the input of the first convolution layer is the low-resolution image, the first convolution layer is connected to the first reshaping layer, the output of the first reshaping layer and the output of the fully connected module are convolved and used as the input of the first activation layer, the first activation layer is connected to the second reshaping layer, and the second reshaping layer is connected to the second convolution layer; the output of the convolution module is convolved with the output of the first convolution layer, and then spliced ​​with the output of the second convolution layer, the spliced ​​result is used as the input of the second activation layer, the second activation layer is connected to the DRF convolution layer of the next degradation unit, and the DRF convolution layer is sequentially connected to the third activation layer, the third convolution layer and the fifth activation layer; the output of the fifth activation layer is spliced ​​with the degradation representation vector and used as the input of the next degradation representation fusion module.

2. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1 is characterized in that: In step S1, the low-resolution image The expression is: ;in, represents filtering, represents the convolution operation, Represents a scale factor The downsampling operation, represents the adjustable noise level, JPEG represents image compression, Indicates the compression level of JPEG. It is a high-resolution image of ships at sea.

3. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1 is characterized in that: In step S1, the specific implementation process of obtaining the degradation characterization vector includes: The high-resolution marine ship images are aligned, and positive sample pairs are obtained using the aligned image blocks. The positive sample pairs are resampled to obtain degraded image blocks. The expression of the degraded image blocks is: ;in, is the degraded image block, Based on parameter weights The degradation model after resampling, is a high-resolution marine image. Express Perform random cropping to obtain paired image patches , is the downsampling factor, is the alignment operation, represents the continuous multiplication of operators; The degraded image block is used as the input of the degradation representation estimation network to obtain the degradation representation estimation vector : ; represents the aligned degraded image patches, For the encoder, It means that the input of the encoder is used as the input of the 2-MLP, and the 2-MLP includes a first fully connected layer, an activation layer, a second fully connected layer, and a reshaping layer connected in sequence.

4. The multi-scale super-resolution reconstruction method for marine ship images according to claim 3 is characterized in that: In step S1, the degradation characterization estimation network includes an encoder connected in sequence and two fully connected layers connected in series; the input of the encoder is a degraded image block.

5. The multi-scale super-resolution reconstruction method for marine ship images according to claim 3 is characterized in that: The degradation characterization estimation network loss function It is expressed as: ;in, B represents the batch size of degraded image patches, represents negative samples, is the number of negative samples in the encoder, is a hyperparameter.

6. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1, characterized in that: The last degradation representation fusion group is connected to the upsampling layer through an output convolution layer.

7. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1, characterized in that: The basic block includes one or more of a residual block, a residual dense block, a multi-scale residual block, a residual channel attention block, a dense projection unit, and a residual Transformer block.

8. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1, characterized in that: The convolution module includes two convolution layers connected in series; the fully connected module includes two fully connected layers connected in series.

9. The multi-scale super-resolution reconstruction method for marine ship images according to claim 1, characterized in that: Also includes: S3. Acquire a ship image at sea, input the ship image at sea into a trained degradation representation network to obtain a degradation representation vector, input the degradation representation vector and a low-resolution image of the ship image at sea into a multi-scale super-resolution reconstruction model to obtain a reconstructed image.

10. A terminal device comprising a memory, a processor and a computer program stored in the memory; characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images