Super-resolution reconstruction method and device of image, equipment, medium and program product

Abstract

The invention provides a super-resolution reconstruction method of an image, which can be applied to the field of financial science and technology or the field of image reconstruction. The method comprises the following steps: acquiring an input image of an initial subnet of a super-resolution branch by using a zero sample method based on an image to be reconstructed; performing feature extraction on the input image of the initial subnet by using a convolutional layer to obtain shallow features of a super-resolution branch; processing the to-be-reconstructed image through a local implicit standardized stream to obtain reconstruction features, and transferring the reconstruction features corresponding to the subnets of the reconstruction branches to the super-resolution features corresponding to the subnets of the super-resolution branches through a task converter module to obtain artifact-free features; and performing up-sampling on the artifact-free features and the shallow features to obtain a super-resolution reconstructed image. Therefore, the super-resolution reconstruction method disclosed by the invention has high fidelity, high definition and corresponding degree of adaptation. The invention further provides an image super-resolution reconstruction device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of financial technology or the field of image reconstruction, and more specifically to a method, device, equipment, medium and program product for super-resolution reconstruction of an image. Background Art

[0002] In the daily counter business process, it is common for users to submit image materials. According to user submission data, on average, one out of every eight material submissions is rejected because the submitted materials are not clear. Depending on the business process, the notification time to the user may have different degrees of lag, which may bring a bad user experience and potential economic loss risks to the user. In order to further improve the efficiency of business processing, reduce unnecessary repeated submission processes for users, and optimize the user experience, the image super-resolution method is used to reconstruct the low-definition images uploaded by users into high-definition images that are easy for the system to recognize. In actual use scenarios, the pictures uploaded by users are often single images. The problem is regarded as a zero-sample problem, and a completely unsupervised or self-supervised method is used to learn the internal information unique to the image.

[0003] However, the existing zero-shot super-resolution methods cannot reconstruct images to meet the requirements for counter material submission. This is because in the actual transaction process, users often provide forms, certificates, and other images with fine regular structures or sharp transitions between adjacent pixels. In this case, the reconstruction process is inherently pathological, because the same low-definition image can produce many high-definition images, and the authenticity of the reconstructed image cannot be guaranteed, that is, the fidelity of the generated image is questionable. Secondly, previous deep learning-based super-resolution methods usually use upsampling of predefined scales in their network architectures, such as squeeze layers, transposed convolutions, and sub-pixel convolutions. Once the upsampling scale is determined, they cannot further adjust the output resolution without modifying the model architecture, which leads to a lack of versatility in practical applications. In the process of processing these user sensitive information, issues such as the fidelity, clarity, and security of the reconstructed image are very important. However, in the current scenario where users submit image materials, the images reconstructed using the existing zero-shot super-resolution methods are not fidelity, clarity, and versatility, and cannot ensure the convenience of user operations while ensuring the security and accuracy of user information, resulting in low counter efficiency. Summary of the invention

[0004] In view of at least one aspect of the above problems, embodiments of the present disclosure provide a method, apparatus, device, medium, and program product for super-resolution reconstruction of an image that improves the fidelity of the reconstructed image.

[0005] According to a first aspect of the present disclosure, a method for super-resolution reconstruction of an image is provided, comprising: based on the image to be reconstructed, using a zero-sample method to obtain an input image of an initial subnet of a super-resolution branch; using a convolutional layer to perform feature extraction on the input image of the initial subnet to obtain shallow features of the super-resolution branch; based on the shallow features, transferring the reconstruction features corresponding to the subnet of the reconstruction branch to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed with a local implicit normalization flow; and upsampling the artifact-free features and the shallow features to obtain a super-resolution reconstructed image.

[0006] According to an embodiment of the present disclosure, based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features, including: based on the shallow features, using the residual block corresponding to the initial subnet of the super-resolution branch to obtain the super-resolution features of the current subnet corresponding to the super-resolution branch; based on the super-resolution features of the current subnet, transferring the reconstruction features of the current subnet of the reconstruction branch to the current subnet of the super-resolution branch through the task converter module to obtain to the artifact-free feature of the current subnet; based on the artifact-free feature of the current subnet, updating the current super-resolution feature to the super-resolution feature of the next subnet corresponding to the super-resolution branch through the residual block corresponding to the next subnet of the super-resolution branch; and based on the super-resolution feature of the next subnet, transferring the reconstruction feature of the next subnet of the reconstruction branch to the next subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the next subnet, until the reconstruction feature transfer of all subnets in the reconstruction branch is completed to obtain the artifact-free feature.

[0007] According to an embodiment of the present disclosure, based on the super-resolution feature of the current subnet, the reconstruction feature of the current subnet of the reconstruction branch is transferred to the current subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the current subnet, including: sampling the reconstruction feature of the current subnet to obtain the sampled feature; correlation embedding the reconstruction feature of the current subnet and the super-resolution feature of the current subnet to obtain the similarity between the reconstruction branch and the super-resolution branch; based on the sampled feature, performing attention transfer on the similarity to obtain an attention map; and based on the attention map, performing soft attention processing on the sampled feature and the super-resolution feature of the current subnet to obtain the artifact-free feature of the current subnet.

[0008] According to an embodiment of the present disclosure, the step of transferring attention to the similarity based on the sampling feature to obtain an attention map includes: obtaining an attention map from the similarity; and indexing the sampling feature based on the attention map to obtain the structural feature.

[0009] According to an embodiment of the present disclosure, based on the attention map, the sampling features and the super-resolution features of the current subnet are subjected to soft attention processing to obtain the artifact-free features of the current subnet, including: convolutionally synthesizing the reconstructed features of the current subnet and the structural features to obtain synthetic features; calculating soft attention confidence based on the similarity and the structural features; and based on the soft attention confidence, feature fusion of the synthetic features and the super-resolution features of the current subnet to obtain the artifact-free features of the current subnet.

[0010] According to an embodiment of the present disclosure, the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed by a local implicit normalized flow, including: inputting the image to be reconstructed into the deep cascade network of the reconstruction branch to obtain the output features corresponding to the subnet of the reconstruction branch; and performing local implicit normalized flow processing on the output features corresponding to the subnet to extract the reconstruction features corresponding to the subnet of the reconstruction branch.

[0011] According to an embodiment of the present disclosure, the output features corresponding to the subnet are subjected to local implicit normalized flow processing to extract the reconstruction features corresponding to the subnet of the reconstruction branch, including: using a normalized flow model to map the output features corresponding to the subnet of the reconstruction branch to a latent space; extracting local Fourier features from the output features corresponding to the subnet and image parameters to generate conditional parameters; processing the local Fourier features using a local integration method to obtain a Fourier feature set; and based on the conditional parameters, mapping the Fourier feature set to the latent space through a forward transformation to obtain variables in the latent space, and recovering the reconstruction features corresponding to the subnet of the reconstruction branch from the variables in the latent space through an inverse transformation.

[0012] According to an embodiment of the present disclosure, the method of using a zero-sample method based on the image to be reconstructed to obtain an input image of an initial subnet of a super-resolution branch includes: performing data enhancement on the image to be reconstructed to obtain m pseudo-resolution sample pairs; wherein m is an integer greater than 1; and inputting the image to be reconstructed into a trained self-supervised convolutional neural network to obtain an input image of the initial subnet of the super-resolution branch; wherein the trained self-supervised convolutional neural network is obtained by training a convolutional neural network based on the m pseudo-resolution sample pairs.

[0013] According to an embodiment of the present disclosure, the method further includes: connecting the subnet of the reconstruction branch and the subnet of the super-resolution branch by dense connection between subnets.

[0014] A second aspect of the present disclosure provides a super-resolution reconstruction device for an image, comprising: a zero-sample module, for obtaining an input image of an initial subnet of a super-resolution branch using a zero-sample method based on an image to be reconstructed; a shallow feature extraction module, for performing feature extraction on the input image of the initial subnet using a convolutional layer to obtain shallow features of the super-resolution branch; an artifact-free processing module, for transferring, based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed using a local implicit normalization flow; and an upsampling module, for upsampling the artifact-free features and the shallow features to obtain a super-resolution reconstructed image.

[0015] According to an embodiment of the present disclosure, the artifact-free processing module includes: a first processing submodule, which is used to obtain the super-resolution feature of the current subnetwork corresponding to the super-resolution branch by using the residual block corresponding to the initial subnetwork of the super-resolution branch based on the shallow feature; a second processing submodule, which is used to transfer the reconstruction feature of the current subnetwork of the reconstruction branch to the current subnetwork of the super-resolution branch through the task converter module based on the super-resolution feature of the current subnetwork, so as to obtain the artifact-free feature of the current subnetwork; a third processing submodule, which is used to update the current super-resolution feature to the super-resolution feature of the next subnetwork corresponding to the super-resolution branch through the residual block corresponding to the next subnetwork of the super-resolution branch based on the artifact-free feature of the current subnetwork; and a fourth processing submodule, which is used to transfer the reconstruction feature of the next subnetwork of the reconstruction branch to the next subnetwork of the super-resolution branch through the task converter module based on the super-resolution feature of the next subnetwork, so as to obtain the artifact-free feature of the next subnetwork, until the reconstruction feature transfer of all subnetworks in the reconstruction branch is completed, and the artifact-free feature is obtained.

[0016] According to an embodiment of the present disclosure, the second processing submodule includes: a first processing unit, which is used to sample the reconstruction features of the current subnet to obtain sampling features; a second processing unit, which is used to perform correlation embedding on the reconstruction features of the current subnet and the super-resolution features of the current subnet to obtain the similarity between the reconstruction branch and the super-resolution branch; a third processing unit, which is used to transfer attention to the similarity based on the sampling features to obtain an attention map; and a fourth processing unit, which is used to perform soft attention processing on the sampling features and the super-resolution features of the current subnet based on the attention map to obtain artifact-free features of the current subnet.

[0017] According to an embodiment of the present disclosure, the third processing unit includes: an attention mapping subunit, used to obtain an attention map from the similarity; and an index selection subunit, used to perform index selection on the sampling features based on the attention map to obtain the structural features.

[0018] According to an embodiment of the present disclosure, the fourth processing unit includes: a convolution synthesis subunit, used to convolute the reconstructed features of the current subnet and the structural features to obtain synthetic features; a soft attention processing subunit, used to calculate the soft attention confidence based on the similarity and the structural features; and a feature fusion subunit, used to perform feature fusion of the synthetic features and the super-resolution features of the current subnet based on the soft attention confidence to obtain the artifact-free features of the current subnet.

[0019] According to an embodiment of the present disclosure, the artifact-free processing module also includes: a fifth processing sub-module, used to input the image to be reconstructed into the deep cascade network of the reconstruction branch to obtain the output features corresponding to the subnet of the reconstruction branch; and a sixth processing sub-module, used to perform local implicit normalized flow processing on the output features corresponding to the subnet to extract the reconstruction features corresponding to the subnet of the reconstruction branch.

[0020] According to an embodiment of the present disclosure, the sixth processing submodule includes: a spatial mapping unit, which is used to map the output features corresponding to the subnet of the reconstruction branch to the latent space using a standardized flow model; a parameter generation unit, which is used to extract local Fourier features from the output features corresponding to the subnet and the image parameters to generate conditional parameters; a local integration unit, which is used to process the local Fourier features using a local integration method to obtain a Fourier feature set; and a reconstruction unit, which is used to map the Fourier feature set to the latent space through a forward transformation based on the conditional parameters to obtain variables in the latent space, and to recover the reconstruction features corresponding to the subnet of the reconstruction branch from the variables in the latent space through an inverse transformation.

[0021] According to an embodiment of the present disclosure, the zero-sample module includes: a first zero-sample submodule, used to perform data enhancement on the image to be reconstructed to obtain m pseudo-resolution sample pairs; wherein m is an integer greater than 1; and a second zero-sample submodule, used to input the image to be reconstructed into a trained self-supervised convolutional neural network to obtain an input image of an initial subnet of the super-resolution branch; wherein the trained self-supervised convolutional neural network is obtained by training a convolutional neural network based on the m pseudo-resolution sample pairs.

[0022] According to an embodiment of the present disclosure, the device further includes: connecting the subnet of the reconstruction branch and the subnet of the super-resolution branch by dense connection between subnets.

[0023] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0024] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0025] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0026] According to an embodiment of the present disclosure, in a scenario where a user submits image materials, images reconstructed using existing zero-sample super-resolution methods are not faithful, have low definition, and are not very versatile. Therefore, in an embodiment of the present disclosure, based on the image to be reconstructed, a zero-sample method is used to obtain an input image of an initial subnet of a super-resolution branch; a convolutional layer is used to perform feature extraction on the input image of the initial subnet to obtain shallow features of the super-resolution branch; based on the shallow features, the reconstructed features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features; wherein the reconstructed features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed using a local implicit normalization flow; and the artifact-free features and the shallow features are upsampled to obtain a super-resolution reconstructed image. By implementing the embodiments of the present disclosure, a new task can be quickly learned with a small number of image samples or only a single image, and zero-sample super-resolution reconstruction based on local implicit normalization flow can be achieved, ensuring that the generated image is consistent with the semantic information contained in the original image, with high fidelity, high clarity and corresponding adaptability, thereby further improving counter service efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0028] Figure 1 A diagram schematically shows an application scenario of super-resolution reconstruction of an image according to an embodiment of the present disclosure;

[0029] Figure 2 A flowchart of a method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 3 A branch network model connection diagram of a method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 4 A subnetwork dense connection structure diagram of an image super-resolution reconstruction method according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 5 A flowchart of a zero-sample method of a super-resolution reconstruction method for an image according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 6 The following schematically shows a reconstruction feature extraction flow chart of a super-resolution reconstruction method for an image according to an embodiment of the present disclosure;

[0034] Figure 7 A local implicit normalized flow processing diagram of a super-resolution reconstruction method for an image according to an embodiment of the present disclosure is schematically shown;

[0035] Figure 8 A task conversion flow chart of a method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown;

[0036] Fig. 9 The following schematically shows an attention transfer flow chart of the image super-resolution reconstruction method according to an embodiment of the present disclosure;

[0037] Fig.10 A soft attention processing diagram of a super-resolution reconstruction method for an image according to an embodiment of the present disclosure is schematically shown;

[0038] Fig.11 A structural block diagram schematically shows a device for super-resolution reconstruction of an image according to an embodiment of the present disclosure; and

[0039] Fig.12 A block diagram of an electronic device suitable for implementing a super-resolution reconstruction method of an image according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0041] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0042] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0043] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0044] It should be noted that the super-resolution reconstruction method and device for images disclosed herein can be used in the financial technology field in scenarios where users submit image materials, and can also be used in any field outside the financial technology field, such as image processing, super-resolution reconstruction and other technical fields. The application field of the super-resolution reconstruction method and device for images disclosed herein is not limited.

[0045] With the development of super-resolution algorithms and image reconstruction, more and more users and enterprises are willing to use related methods for data reconstruction due to their low cost and cross-modal characteristics. However, due to the particularity of the banking and financial industry, we need to ensure the security and accuracy of customer information while ensuring the convenience of customer operations. In the scenario where users submit image materials, the images reconstructed by the existing zero-sample super-resolution methods are not faithful, have low clarity, and are not very versatile. In order to reconstruct the blurred image materials submitted by customers into clear images with higher resolution, while ensuring that the generated image is consistent with the semantic information contained in the original image, an embodiment of the present disclosure provides a super-resolution reconstruction method for an image, including: based on the image to be reconstructed, using a zero-sample method to obtain an input image of an initial subnet of a super-resolution branch; using a convolutional layer to extract features of the input image of the initial subnet to obtain shallow features of the super-resolution branch; based on the shallow features, transferring the reconstruction features corresponding to the subnet of the reconstruction branch to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed with a local implicit normalization flow; and upsampling the artifact-free features and the shallow features to obtain a super-resolution reconstructed image. It can quickly learn a new task with a small number of image samples or only a single image, and realize zero-sample super-resolution reconstruction based on local implicit normalization flow. The reconstruction can achieve good reconstruction results after only one gradient update. It can also learn artifact-free features without super-resolution motion artifacts from the reconstruction features of the reconstruction branch, and reconstruct the blurred image materials to be reconstructed submitted by the user into clear super-resolution reconstructed images with higher resolution, thereby improving the fidelity of the reconstructed images. Due to the design of the task transformation module, the artifact-free features after de-artifacting will be passed to the super-resolution branch to reduce the impact of artifacts. The reconstruction process is supervised to ensure the authenticity of the reconstructed image. The entire reconstruction process focuses on multi-task learning of the reconstruction and super-resolution subnets to effectively remove artifacts. Even when processing different image information uploaded by different users and images to be reconstructed with different texture features and semantic information, no pre-training is required. It only needs to optimize each input image separately to quickly adapt to any given image condition, ensuring that the generated image is consistent with the semantic information contained in the original image, with high fidelity, high clarity and corresponding adaptability, further improving the efficiency of counter services, reducing unnecessary repeated operations of users, and avoiding potential property losses.

[0046] Figure 1 The application scenario diagram of super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown.

[0047] like Figure 1As shown, the application scenario 100 according to this embodiment may include a scenario where a user submits image materials. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0048] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0049] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0050] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received image data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0051] It should be noted that the image super-resolution reconstruction method provided in the embodiment of the present disclosure can generally be performed by the server 105. Accordingly, the image super-resolution reconstruction device provided in the embodiment of the present disclosure can generally be set in the server 105. The image super-resolution reconstruction method provided in the embodiment of the present disclosure can also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image super-resolution reconstruction device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0053] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0054] In the process of daily counter business handling, in the scenario of using personal information for automated decision-making, when business processing is performed based on image materials submitted by users, the methods, devices and systems provided by the embodiments of the present disclosure provide users with corresponding operation entrances for users to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating personal behavior habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0055] The following will be based on Figure 1 The scene described by Figure 2~Figure 10 The super-resolution reconstruction method of an image in the disclosed embodiment is described in detail.

[0056] Figure 2 The flowchart of the method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown.

[0057] Figure 3 A branch network model connection diagram of a method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown;

[0058] like Figure 2 , 3 As shown, the image super-resolution reconstruction method of this embodiment includes operations S210 to S240, and the image super-resolution reconstruction method can be automatically executed.

[0059] In operation S210 , based on the image to be reconstructed, a zero-sample method is used to obtain an input image of an initial subnetwork of a super-resolution branch.

[0060] The images to be reconstructed include, but are not limited to, identity images, financial images, asset ownership images, certification document images, contract document images, qualification document images and other image materials. The images to be reconstructed may be one or more.

[0061] Super-resolution technology (SR) refers to reconstructing a high-resolution image from a series of low-resolution images. The super-resolution branch is a branch of the convolutional neural network applicable to the image to solve the super-resolution processing task of the specific image, which is referred to as the SR branch. The SR branch includes self-supervised convolutional neural networks (CNN). The CNN model can gradually optimize the mapping from low resolution to high resolution from pseudo low-resolution images through self-supervised learning. In the process of learning the mapping from low resolution to high resolution, the CNN model automatically extracts some features related to the super-resolution task. These features are called super-resolution features (SR features).

[0062] In an embodiment of the present disclosure, before acquiring the image to be reconstructed, the user's consent or authorization may be obtained. For example, before operation S210, a request to acquire the user's image material information may be issued to the user. When the user agrees or authorizes that the user information may be acquired, operation S210 is performed.

[0063] In the embodiments of the present disclosure, a corresponding operation entry may be provided for the user to choose to agree or reject the automated decision result. That is, before image reconstruction processing / decision is performed on user information, an instruction of agreeing or rejecting processing / decision input by the user through the corresponding operation entry may be obtained. If the user agrees to process / decision, image reconstruction processing / decision is performed on the user information, that is, operation S210 is executed. If the user refuses to process / decision, the expert decision process is entered.

[0064] In operation S220, feature extraction is performed on the input image of the initial subnet using a convolutional layer to obtain shallow features of the super-resolution branch.

[0065] In the SR branch, for the input image, the convolution layer performs convolution operations by sliding the convolution kernel on the image, which can automatically capture the local features of the image, such as edges, textures and other information. These initially extracted features are recorded as shallow features. , which provides basic information for subsequent further feature extraction and super-resolution processing.

[0066] In operation S230, based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through the task converter module to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed through a local implicit normalization flow.

[0067] The Reconstruction (Rec) branch is used for image reconstruction tasks, including the Deep Cascading Network (DnCN). DnCN is a deep learning model that gradually optimizes feature extraction and classification tasks by cascading multiple network modules, and gradually refines features to improve the performance and accuracy of the model. In the Rec branch, clear image features with correct structures are recovered from input images with artifacts, and the local texture of the image is processed by the Locally Implicit Normalizing Flow (LINF) to generate high-quality reconstruction features (Rec features).

[0068] According to an embodiment of the present disclosure, the subnet of the reconstruction branch is connected to the subnet of the super-resolution branch through a dense connection between subnets. Figure 4 The subnet dense connection structure diagram of the image super-resolution reconstruction method according to the embodiment of the present disclosure is schematically shown. Figure 4 As shown in the figure, the deep cascade network structure of the improved reconstruction branch is further added with a dense connection method between subnetworks. The Rec branch subnetwork and the SR branch subnetwork are densely connected across subnetworks, that is, the input in each subnetwork is the union of the output of the layer of all previous subnetworks and the output of the previous layer of the subnetwork. The dense connection is because, on the one hand, relying solely on the SR branch is not enough to restore a high-resolution and artifact-corrected image from a low-resolution (LR) image with artifacts as input. On the other hand, the Rec branch can restore a clear image with the correct structure from an input with artifacts, because it is trained under the supervision of the original image, which means that the reconstruction can effectively remove the artifacts introduced by the undersampled k-space.

[0069] In the embodiments of the present disclosure, the subnetworks of the reconstruction branch and the super-resolution branch are densely connected. Through the dense connection across subnetworks, the reconstruction branch is introduced into the super-resolution branch. The task converter module serves as a new end-to-end task conversion network, which overcomes the problem of texture detail loss caused by traditional neural networks when reconstructing low-definition images, significantly improves the participation of auxiliary information, and ensures the authenticity of the reconstructed image.

[0070] In operation S240 , the artifact-free features and the shallow features are upsampled to obtain a super-resolution reconstructed image.

[0071] Based on artifact-free features , obtain the final super-resolution features , specifically:

[0072]

[0073] represent The Nth residual block of the branch.

[0074] Using sub-pixel convolution As an upsampling module, the generated scale is The output super-resolution reconstructed image .

[0075] According to the embodiments of the present disclosure, zero-sample super-resolution reconstruction based on local implicit normalization flow is implemented to quickly learn a new task with a small number of image samples or only a single image, and good reconstruction results can be achieved after only one gradient update. It is also possible to learn artifact-free features without super-resolution motion artifacts from the reconstruction features of the reconstruction branch, and reconstruct the blurred image materials to be reconstructed submitted by the customer into clear super-resolution reconstructed images with higher resolution, thereby improving the fidelity of the reconstructed images. Due to the design of the task transformation module, the artifact-free features after de-artifacting will be passed to the super-resolution branch to reduce the impact of artifacts. The reconstruction process is supervised to ensure the authenticity of the reconstructed image. During the entire reconstruction process, multi-task learning of subnets focusing on reconstruction and super-resolution can effectively remove artifacts. Even when dealing with different image information uploaded by different users and images to be reconstructed with different texture features and semantic information, no pre-training is required. Instead, each input image needs to be optimized separately to quickly adapt to any given image condition, ensuring that the generated image is consistent with the semantic information contained in the original image, with high fidelity, high clarity and corresponding adaptability, further improving counter service efficiency, reducing unnecessary repeated operations for customers, and avoiding potential property losses.

[0076] Figure 5 A flowchart of a zero-sample method for super-resolution reconstruction of an image according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 5 As shown, according to an embodiment of the present disclosure, in operation S210, based on the image to be reconstructed, a zero-sample method is used to obtain an input image of an initial subnetwork of a super-resolution branch, including operations S510-S520.

[0078] In operation S510, data enhancement is performed on the image to be reconstructed to obtain m pseudo-resolution sample pairs; wherein m is an integer greater than 1.

[0079] In the zero-shot method, the image to be reconstructed is generated into multiple pseudo low-resolution images with different resolutions by data enhancement, that is, the low-resolution version , from which the desired high-definition output super-resolution version can be constructed These low-resolution images and the images to be reconstructed (or high-resolution images restored by other means) can form low-resolution and high-resolution (LR-HR) pseudo-resolution sample pairs, providing more data for self-supervised CNN model training. Data augmentation is to increase the diversity of training data, thereby improving the generalization ability and robustness of the model. Data augmentation is not limited to image downsampling, rotation, flipping, cropping, adding noise, etc.

[0080] It is worth noting that in the scenario where the user submits the image material, there is only one image to be reconstructed in some given usage environment. , there are no external examples available for training, so a CNN suitable for the image is built to solve the SR task for that specific image. Since the training set contains only one instance, some data augmentation methods are needed on the image domain I to obtain more pseudo-resolution sample pairs for training. Data augmentation is to reduce the image to be reconstructed I into many smaller versions. , to generate a low-resolution version of itself , where s is the desired SR scaling factor.

[0081] In operation S520, the image to be reconstructed is input into a trained self-supervised convolutional neural network to obtain an input image of an initial subnet of a super-resolution branch; wherein the trained self-supervised convolutional neural network is obtained by training a convolutional neural network based on m pseudo-resolution sample pairs.

[0082] Training a self-supervised convolutional neural network (CNN) from a low-resolution version Reconstruct the image to be reconstructed I, then apply the trained CNN to the image to be reconstructed I, and the result is used as the input image of the SR branch network It should be noted that the trained CNN is fully convolutional and can be applied to images of different sizes, making it suitable for super-resolution processing tasks in the SR branch.

[0083] According to the embodiments of the present disclosure, a large amount of prior data is not required. A low-resolution version of the original image is generated as a pseudo sample through the image material to be reconstructed provided by the customer, and then a convolutional neural network is trained on these pseudo samples to obtain a trained self-supervised convolutional neural network. Through a zero-sample learning strategy, cross-modal knowledge transfer is achieved with the help of auxiliary information. It can be put into production without providing a large amount of prior knowledge, ensuring execution speed, and having good output effects for different picture information proposed by users.

[0084] Figure 6 The figure schematically shows a reconstruction feature extraction flow chart of the image super-resolution reconstruction method according to an embodiment of the present disclosure.

[0085] like Figure 6 As shown, according to an embodiment of the present disclosure, in operation S230, the image to be reconstructed is processed by the local implicit normalization flow to obtain the reconstruction features corresponding to the subnet of the reconstruction branch, including operations S610-S620.

[0086] In operation S610, the image to be reconstructed is input into the deep cascade network of the reconstruction branch to obtain output features corresponding to the subnetwork of the reconstruction branch.

[0087] In operation S620, the output features corresponding to the sub-network are processed by local implicit normalization flow to extract reconstruction features corresponding to the sub-network of the reconstruction branch.

[0088] According to the embodiments of the present disclosure, by introducing a deep neural network and cascading multiple sub-networks to gradually enhance the image processing capabilities, it is helpful for image denoising and reconstruction. By introducing a local discriminant learning framework and effectively processing the local texture of the image, the reconstruction features of the reconstruction branch can be accurately extracted, and reconstruction is encouraged to be achieved in the direction of high-fidelity optimization, thereby improving the reliability and security of the reconstruction process.

[0089] Figure 7 The figure schematically shows a local implicit normalized flow processing diagram of the image super-resolution reconstruction method according to an embodiment of the present disclosure.

[0090] like Figure 7 As shown, according to an embodiment of the present disclosure, in operation S620, the output features corresponding to the subnet are subjected to local implicit normalization flow processing to extract the reconstruction features corresponding to the subnet of the reconstruction branch, including operations S710-S740.

[0091] In operation S710, the output features corresponding to the sub-network of the reconstruction branch are mapped to the latent space using the normalized flow model.

[0092] The normalized flow model approximates the target distribution between the target space and the latent space by learning a dual-target mapping, specifically:

[0093]

[0094] in, represents the flow model parameterized by θ ,arrive express reversible flow layer, in the local implicit normalized flow, the flow model is approximated by the local texture patch distribution and Gaussian distribution The mapping between:

[0095] ,

[0096] in, is a Gaussian random variable, is the temperature coefficient, represents the latent variable in the transformation process, yes By applying the variable transformation technique, the two distributions and The mapping between can be expressed as:

[0097] ,

[0098] in, yes The logarithm of the absolute Jacobian determinant of . Since in the training phase we can directly obtain and Obtained As well as local texture patches, the flow model can be optimized by minimizing the negative log-likelihood loss. The flow model is constructed by using Transformation Sampling to infer local texture patches.

[0099] The standardized flow model performs similar mapping processing on the output features transmitted from the Rec branch deep cascade network, mapping them to a suitable latent space for subsequent extraction and reconstruction of features. The standardized flow model consists of multiple flow layers, each of which contains a linear layer and an affine injection layer. They process the input features according to their respective operation rules (such as the forward and reverse operation rules of the linear layer, the movement and scaling rules of the affine injection layer for the latent variables, etc.), and gradually adjust the representation of the features.

[0100] As an example, the normalized flow model consists of ten flow layers, where each linear layer k consists of a pair of learnable weight matrices and deviation To parameterize. The forward and inverse operations of the linear layer are:

[0101]

[0102] The Jacobian determinant of the linear layer is the weight matrix The determinant of .

[0103] The affine injection layer injects the two conditional parameters generated by the local implicit module into and The combination of these layers makes the local texture block The distribution depends on , and , these conditional parameters are used to implement the latent variables The element-by-element translation and scaling of is:

[0104]

[0105] in, is the refractive index of the affine injection layer, Represents element-wise multiplication.

[0106] In operation S720, local Fourier features are extracted from the output features corresponding to the sub-network and the image parameters to generate conditional parameters.

[0107] Extract local Fourier features from the output features of the deep cascade network of the Rec branch and image parameters (such as parameters related to image resolution, coordinates, etc.), and then generate conditional parameters, as follows: Use the local implicit module to extract local Fourier features, from , and Extract local Fourier features to generate conditional parameters and , expressed as:

[0108]

[0109] in, Indicate as Implemented parameter generation function, is The center coordinates of the local texture block queried in, is the closest to of coordinate The characteristic vector of Indicates the cell size, Indicates relative coordinates.

[0110] The local implicit module uses a local texture estimator To extract any given The Fourier characteristic of , the function is expressed as:

[0111]

[0112] in, It is the Fourier feature extracted by three different functions:

[0113]

[0114] in, , and They are functions for estimating amplitude, frequency, and phase, respectively. The first two are implemented using convolutional layers, and the latter is implemented using a multi-layer perceptron (MLP). is the frequency to be modeled, and the dimensions of these features are .

[0115] In operation S730, the local Fourier features are processed using a local integration method to obtain a Fourier feature set.

[0116] The local integration method is used to process the Fourier feature set to avoid color discontinuity when two different feature vectors are selected for adjacent pixels. By querying the RGB value from the nearest feature vector and fusing it into the bilinear difference, an optimized feature set is formed. The forward and inverse transform of is expressed as:

[0117]

[0118]

[0119] in, is the set of four nearest eigenvectors, Derived weights for performing bilinear interpolation.

[0120] In the local texture estimator Then the four closest eigenvectors are integrated to form a set:

[0121]

[0122] Each eigenvector is Weighted to allow the model to pay more attention to closer feature vectors.

[0123] To capture the same amount of information and provide the same performance as local ensemble methods, and Only one forward pass is performed, expressed as:

[0124]

[0125] In operation S740, based on the conditional parameters, the Fourier feature set is mapped to the latent space through forward transformation to obtain variables in the latent space, and the reconstruction features corresponding to the subnet of the reconstruction branch are recovered from the variables in the latent space through inverse transformation.

[0126] The reconstruction features are extracted based on the forward and inverse transformations of the trained normalized flow model. Specifically, the features processed above are further mapped to the latent space through the forward transformation of the local implicit normalized flow (LINF) to obtain the variables in the latent space. Then, the inverse transformation is used to recover the local texture patches and other information related to the reconstruction features from the variables in the latent space, that is, the reconstruction features, which will be used in subsequent image reconstruction operations to ultimately achieve the goal of restoring high-resolution and artifact-corrected images from low-definition images with artifacts.

[0127] The local implicit normalization flow adopts a two-stage model training, and the first stage only uses the negative log-likelihood loss Training is performed and fine-tuned in the second stage. Increase Loss, in the flow model The VGG network perceptual loss is added to the predicted patches. The total loss function It can be expressed as:

[0128]

[0129] in, , , is the scaling parameter, is a real local texture patch, , The temperature coefficient and Local texture patches predicted by the local implicit normalized flow when .

[0130] According to the embodiments of the present disclosure, the Fourier feature set is used as an important basis for extracting and reconstructing features. It integrates the information of multiple related feature vectors, so that the extracted reconstructed features can more accurately reflect the actual situation of the image, and help to restore high-quality high-resolution and artifact-corrected images. By using forward transformation and inverse transformation, reconstruction features related to information such as local texture patches are restored from variables in the latent space, which will be used in subsequent image reconstruction operations to ultimately achieve the goal of restoring high-resolution and artifact-corrected images from low-definition images with artifacts.

[0131] According to an embodiment of the present disclosure, in operation S230, based on shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through the task converter module to obtain artifact-free features, including operations S2301-S2304.

[0132] In operation S2301, based on the shallow features, the residual block corresponding to the initial subnet of the super-resolution branch is used to obtain the super-resolution features of the current subnet corresponding to the super-resolution branch.

[0133] The first residual block of the SR branch , the process of obtaining the first super-resolution feature is:

[0134]

[0135] is the first super-resolution feature, Shallow features.

[0136] In operation S2302, based on the super-resolution feature of the current subnet, the reconstructed feature of the current subnet of the reconstruction branch is transferred to the current subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the current subnet.

[0137] To enhance features from different tasks, we use a task transformer module Through three mechanisms, namely, correlation embedding, transfer attention, and soft attention, the representation of artifact-free features is transferred to Branches, in the form of:

[0138]

[0139] in, yes The number of Is from The characteristics of the branch, represent The branch Features.

[0140] In operation S2303, based on the artifact-free feature of the current subnet, the current super-resolution feature is updated to the super-resolution feature of the next subnet corresponding to the super-resolution branch through the residual block corresponding to the next subnet of the super-resolution branch.

[0141] The learned artifact-free features The representation is sent to the residual block to obtain the super-resolution features of the next subnet. The process is:

[0142]

[0143] represent The i+1th feature of the branch, Represents the i+1th residual block.

[0144] In operation S2304, based on the super-resolution feature of the next subnet, the reconstruction feature of the next subnet of the reconstruction branch is transferred to the next subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the next subnet, until the reconstruction feature transfer of all subnets in the reconstruction branch is completed and the artifact-free feature is obtained.

[0145] Operation S2304 is actually to continuously loop operations S2302-S2303 until the reconstruction feature transfer of all subnets in the reconstruction branch is completed to obtain the final artifact-free feature. .

[0146] According to the embodiments of the present disclosure, multi-task learning focuses on the reconstruction branch and the super-resolution branch. The reconstruction branch and the task transformation module used have processed and optimized the situation of artifact input to varying degrees. The reconstruction branch can be specifically used to remove artifacts. Therefore, even if the input image has artifacts, this branch will try to learn the features for removing artifacts. Due to the design of the task transformation module, the features after removing artifacts will be passed to the super-resolution branch, reducing the impact of artifacts and improving the clarity of the super-resolution reconstructed image.

[0147] Figure 8 The task conversion flowchart of the image super-resolution reconstruction method according to the embodiment of the present disclosure is schematically shown.

[0148] like Figure 8 As shown, according to an embodiment of the present disclosure, in operation S2302, based on the super-resolution features of the current subnet, the reconstructed features of the current subnet of the reconstruction branch are transferred to the current subnet of the super-resolution branch through the task converter module to obtain the artifact-free features of the current subnet, including operations S810-S840.

[0149] In operation S810, the reconstructed features of the current subnet are sampled to obtain sampled features.

[0150] Since the Rec branch contains stronger artifact removal capabilities than the SR branch, a task converter module is introduced to guide the SR branch to learn a representation without SR motion artifacts from the Rec branch and the high-resolution image branch generated by the local implicit normalization flow. The task converter consists of three parts: a correlation embedding module for similarity calculation, a transfer attention module for feature transfer, and a soft attention module for feature synthesis. The transfer attention module aims to transfer structural features from the Rec branch to the SR branch based on similarity. The soft attention module is used to integrate the features of the two branches in the model. Reconstructed features inherited from the Rec and SR branches , super-resolution features Expressed as , . For Apply upsampling and downsampling in sequence to make it consistent with Domain consistent sampling features .

[0151] In operation S820, the reconstruction feature of the current subnet and the super-resolution feature of the current subnet are correlated and embedded to obtain the similarity between the reconstruction branch and the super-resolution branch.

[0152] Correlation embedding embeds the relevant information from the Rec branch by estimating the similarity between Q and K. The correlation between the two branches is calculated as:

[0153]

[0154] in, , They are , slices, For similarity.

[0155] In operation S830, attention is transferred on the similarity based on the sampled features to obtain an attention map.

[0156] In operation S840, based on the attention map, soft attention processing is performed on the sampled features and the super-resolution features of the current subnet to obtain artifact-free features of the current subnet.

[0157] According to the embodiment of the present disclosure, sampling, correlation embedding, attention transfer, and soft attention processing are implemented through the task converter module, so that the reconstructed features of the current subnet of the reconstruction branch are transferred to the current subnet of the super-resolution branch. Since artifacts may affect the feature correlation calculation of the task transformation module, through correlation embedding, information related to the structure is retained first, rather than directly relying on the features of the artifact area, which alleviates the artifact transmission problem to a certain extent, accurately obtains artifact-free features, and the artifact-free features will be sent to the super-resolution branch to restore a higher quality super-resolution and motion artifact-free reconstructed image.

[0158] Fig. 9 The following schematically shows an attention transfer flowchart of the image super-resolution reconstruction method according to an embodiment of the present disclosure.

[0159] like Fig. 9 As shown, according to an embodiment of the present disclosure, in operation S830, attention is transferred to the similarity based on the sampling features to obtain an attention map, including operations S910-S920.

[0160] In operation S910, an attention map is obtained from the similarity.

[0161] Will come from In each branch The most relevant position features are passed to Branches, from similarity Get an attention map from ,in express The elements. The present disclosure is different from the traditional attention mechanism and does not The reconstructed features are weighted summed to avoid blurring of the image after reconstruction.

[0162] In operation S920 , based on the attention map, index selection is performed on the sampled features to obtain structural features.

[0163] use The value of The branch most relevant to The branch The location of the position, in order to obtain no Artifact-bearing structural features of branch transfers ,right The expanded patch performs index selection operations to For index: ,in express In the The value of the position is equal to No. locations.

[0164] According to an embodiment of the present disclosure, structural features are transferred from the reconstruction branch to the super-resolution branch. Unlike the traditional attention mechanism, the present disclosure does not use the traditional method of weighted summation of each reconstructed feature for transfer. Only weighted summation will cause the image to become blurred after reconstruction. In order to pass the features from the position most relevant to each reconstructed feature in the reconstruction branch to the super-resolution branch, the accuracy of obtaining artifact-free features is improved, so as to improve the clarity of the reconstructed image.

[0165] Fig.10 The soft attention processing diagram of the image super-resolution reconstruction method according to an embodiment of the present disclosure is schematically shown.

[0166] like Fig.10 As shown, according to an embodiment of the present disclosure, in operation S840, based on the attention mapping, the sampled features and the super-resolution features of the current subnet are soft-attention processed to obtain the artifact-free features of the current subnet, including operations S1010-S1030.

[0167] In operation S1010, the reconstruction features and the structural features of the current subnet are convolved to obtain a synthesized feature.

[0168] Rebuild the feature and structural features Concatenate them and send them to the convolutional layer , and obtain the synthetic characteristics after polymerization .

[0169] In operation S1020, a soft attention confidence is calculated based on the similarity and the structural features.

[0170] From the similarity Compute soft attention map To express The confidence of the structural features transferred at each position in : ,in, It is a soft attention map No. locations.

[0171] In operation S1030 , based on the soft attention confidence, the synthesized features and the super-resolution features of the current subnet are feature fused to obtain artifact-free features of the current subnet.

[0172] To utilize More information about the branch will be synthesized features and The original super-resolution features of the branch Combined, we get the artifact-free features of the final output of the task conversion module , specifically:

[0173]

[0174] According to the embodiments of the present disclosure, a higher quality SR and motion artifact-free image is restored. The structural features are used to enhance the transferred structural information, and more information of the super-resolution branch is used through convolution synthesis and feature fusion. It will be sent to the SR branch in order to recover a higher quality super-resolution and motion artifact-free image.

[0175] Based on the above-mentioned image super-resolution reconstruction method, the present disclosure also provides an image super-resolution reconstruction device. Fig.11 The device is described in detail.

[0176] Fig.11 The structure block diagram of the image super-resolution reconstruction device according to an embodiment of the present disclosure is schematically shown.

[0177] like Fig.11As shown, the image super-resolution reconstruction device 1100 of this embodiment includes a zero sample module 1110 , a shallow feature extraction module 1120 , an artifact-free processing module 1130 and an up-sampling module 1140 .

[0178] The zero sample module 1110 is used to obtain an input image of an initial subnetwork of a super-resolution branch based on the image to be reconstructed using a zero sample method. In one embodiment, the zero sample module 1110 can be used to perform the operation S210 described above, which will not be described in detail herein.

[0179] The shallow feature extraction module 1120 is used to extract features from the input image of the initial subnet using a convolutional layer to obtain shallow features of the super-resolution branch. In one embodiment, the shallow feature extraction module 1120 can be used to perform the operation S220 described above, which will not be repeated here.

[0180] The artifact-free processing module 1130 is used to transfer the reconstruction features corresponding to the subnet of the reconstruction branch to the super-resolution features corresponding to the subnet of the super-resolution branch through the task converter module based on the shallow features, so as to obtain the artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed by the local implicit normalization flow. In one embodiment, the artifact-free processing module 1130 can be used to perform the operation S230 described above, which will not be repeated here.

[0181] The up-sampling module 1140 is used to up-sample the artifact-free features and the shallow features to obtain a super-resolution reconstructed image. In one embodiment, the up-sampling module 1140 can be used to perform the operation S240 described above, which will not be described in detail here.

[0182] According to an embodiment of the present disclosure, the artifact-free processing module 1130 includes: a first processing submodule, which is used to obtain the super-resolution feature of the current subnetwork corresponding to the super-resolution branch by using the residual block corresponding to the initial subnetwork of the super-resolution branch based on the shallow feature; a second processing submodule, which is used to transfer the reconstruction feature of the current subnetwork of the reconstruction branch to the current subnetwork of the super-resolution branch through the task converter module based on the super-resolution feature of the current subnetwork, so as to obtain the artifact-free feature of the current subnetwork; a third processing submodule, which is used to update the current super-resolution feature to the super-resolution feature of the next subnetwork corresponding to the super-resolution branch through the residual block corresponding to the next subnetwork of the super-resolution branch based on the artifact-free feature of the current subnetwork; and a fourth processing submodule, which is used to transfer the reconstruction feature of the next subnetwork of the reconstruction branch to the next subnetwork of the super-resolution branch through the task converter module based on the super-resolution feature of the next subnetwork, so as to obtain the artifact-free feature of the next subnetwork, until the transfer of the reconstruction features of all subnetworks in the reconstruction branch is completed to obtain the artifact-free feature.

[0183] According to an embodiment of the present disclosure, the second processing submodule includes: a first processing unit, which is used to sample the reconstruction features of the current subnet to obtain sampled features; a second processing unit, which is used to perform correlation embedding on the reconstruction features of the current subnet and the super-resolution features of the current subnet to obtain the similarity between the reconstruction branch and the super-resolution branch; a third processing unit, which is used to transfer attention to the similarity based on the sampled features to obtain an attention map; and a fourth processing unit, which is used to perform soft attention processing on the sampled features and the super-resolution features of the current subnet based on the attention map to obtain artifact-free features of the current subnet.

[0184] According to an embodiment of the present disclosure, the third processing unit includes: an attention mapping subunit, used to obtain an attention map from the similarity; and an index selection subunit, used to perform index selection on the sampling features based on the attention map to obtain the structural features.

[0185] According to an embodiment of the present disclosure, the fourth processing unit includes: a convolution synthesis subunit, used to convolute and synthesize the reconstruction features and structural features of the current subnet to obtain synthetic features; a soft attention processing subunit, used to calculate the soft attention confidence based on the similarity and structural features; and a feature fusion subunit, used to fuse the synthetic features and the super-resolution features of the current subnet based on the soft attention confidence to obtain the artifact-free features of the current subnet.

[0186] According to an embodiment of the present disclosure, the artifact-free processing module 1130 also includes: a fifth processing sub-module, used to input the image to be reconstructed into the deep cascade network of the reconstruction branch to obtain the output features corresponding to the subnet of the reconstruction branch; and a sixth processing sub-module, used to perform local implicit normalized flow processing on the output features corresponding to the subnet to extract the reconstruction features corresponding to the subnet of the reconstruction branch.

[0187] According to an embodiment of the present disclosure, the sixth processing submodule includes: a spatial mapping unit, which is used to map the output features corresponding to the subnet of the reconstruction branch to the latent space using a standardized flow model; a parameter generation unit, which is used to extract local Fourier features from the output features corresponding to the subnet and the image parameters to generate conditional parameters; a local integration unit, which is used to process the local Fourier features using a local integration method to obtain a Fourier feature set; and a reconstruction unit, which is used to map the Fourier feature set to the latent space through a forward transformation based on the conditional parameters to obtain variables in the latent space, and to recover the reconstruction features corresponding to the subnet of the reconstruction branch from the variables in the latent space through an inverse transformation.

[0188] According to an embodiment of the present disclosure, the zero-sample module 1110 includes: a first zero-sample submodule, used to perform data enhancement on the image to be reconstructed to obtain m pseudo-resolution sample pairs; wherein m is an integer greater than 1; and a second zero-sample submodule, used to input the image to be reconstructed into a trained self-supervised convolutional neural network to obtain an input image of an initial subnet of a super-resolution branch; wherein the trained self-supervised convolutional neural network is obtained by training a convolutional neural network based on the m pseudo-resolution sample pairs.

[0189] According to an embodiment of the present disclosure, the apparatus 1100 further includes: connecting the subnet of the reconstruction branch and the subnet of the super-resolution branch by dense connection between the subnets.

[0190] According to an embodiment of the present disclosure, any multiple modules of the zero sample module 1110, the shallow feature extraction module 1120, the artifact-free processing module 1130, and the upsampling module 1140 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the zero sample module 1110, the shallow feature extraction module 1120, the artifact-free processing module 1130, and the upsampling module 1140 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the zero sample module 1110 , the shallow feature extraction module 1120 , the artifact-free processing module 1130 , and the upsampling module 1140 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0191] Fig.12 A block diagram of an electronic device suitable for implementing a super-resolution reconstruction method of an image according to an embodiment of the present disclosure is schematically shown.

[0192] like Fig.12As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 to a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0193] In RAM 1203, various programs and data required for the operation of electronic device 1200 are stored. Processor 1201, ROM 1202 and RAM 1203 are connected to each other via bus 1204. Processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1202 and / or RAM 1203. It should be noted that the program can also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0194] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the input / output (I / O) interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output (I / O) interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage portion 1208 as needed.

[0195] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0196] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0197] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image super-resolution reconstruction method provided by the embodiment of the present disclosure.

[0198] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1201. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0199] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0200] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0201] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0202] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0204] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A super-resolution reconstruction method for an image, characterized in that: The method comprises: Based on the image to be reconstructed, a zero-shot method is used to obtain an input image of an initial subnetwork of a super-resolution branch; Using a convolutional layer, extracting features from the input image of the initial subnet to obtain shallow features of the super-resolution branch; Based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed through a local implicit normalization flow; and The artifact-free features and the shallow features are upsampled to obtain a super-resolution reconstructed image.

2. The method according to claim 1, characterized in that Based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch are transferred to the super-resolution features corresponding to the subnet of the super-resolution branch through the task converter module to obtain artifact-free features, including: Based on the shallow features, using the residual block corresponding to the initial subnet of the super-resolution branch to obtain the super-resolution features of the current subnet corresponding to the super-resolution branch; Based on the super-resolution feature of the current subnet, transferring the reconstruction feature of the current subnet of the reconstruction branch to the current subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the current subnet; Based on the artifact-free feature of the current subnet, updating the current super-resolution feature to the super-resolution feature of the next subnet corresponding to the super-resolution branch through the residual block corresponding to the next subnet of the super-resolution branch; and Based on the super-resolution feature of the next subnet, the reconstruction feature of the next subnet of the reconstruction branch is transferred to the next subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the next subnet, until the reconstruction feature transfer of all subnets in the reconstruction branch is completed to obtain the artifact-free feature.

3. The method according to claim 2, characterized in that The super-resolution feature of the current subnet is transferred to the current subnet of the super-resolution branch through the task converter module to obtain the artifact-free feature of the current subnet, including: Sampling the reconstruction features of the current subnet to obtain sampled features; Correlation embedding is performed on the reconstruction features of the current subnet and the super-resolution features of the current subnet to obtain the similarity between the reconstruction branch and the super-resolution branch; Based on the sampled features, performing attention transfer on the similarity to obtain an attention map; and Based on the attention map, soft attention processing is performed on the sampled features and the super-resolution features of the current subnet to obtain artifact-free features of the current subnet.

4. The method according to claim 3, characterized in that The step of performing attention transfer on the similarity based on the sampling feature to obtain an attention map includes: deriving an attention map from the similarities; and Based on the attention map, the sampled features are indexed and selected to obtain the structural features.

5. The method according to claim 3, characterized in that: The step of performing soft attention processing on the sampling features and the super-resolution features of the current subnet based on the attention map to obtain artifact-free features of the current subnet includes: Convolutionally synthesizing the reconstruction features of the current subnet and the structural features to obtain a synthesized feature; Calculating soft attention confidence according to the similarity and the structural feature; and Based on the soft attention confidence, the synthesized features and the super-resolution features of the current subnet are feature fused to obtain the artifact-free features of the current subnet.

6. The method according to claim 1, characterized in that The reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed by a local implicit normalization flow, including: Inputting the image to be reconstructed into the deep cascade network of the reconstruction branch to obtain output features corresponding to the subnetwork of the reconstruction branch; and The output features corresponding to the sub-network are subjected to local implicit normalization flow processing to extract the reconstruction features corresponding to the sub-network of the reconstruction branch.

7. The method according to claim 6, characterized in that The performing local implicit normalized flow processing on the output features corresponding to the subnet to extract the reconstruction features corresponding to the subnet of the reconstruction branch includes: Using a standardized flow model, the output features corresponding to the sub-network of the reconstruction branch are mapped to a latent space; Extracting local Fourier features from the output features corresponding to the subnetwork and the image parameters to generate conditional parameters; Processing the local Fourier features using a local integration method to obtain a Fourier feature set; and Based on the conditional parameters, the Fourier feature set is mapped to the latent space through forward transformation to obtain variables in the latent space, and the reconstruction features corresponding to the subnet of the reconstruction branch are recovered from the variables in the latent space through inverse transformation.

8. The method according to claim 1, characterized in that The method of obtaining an input image of an initial subnet of a super-resolution branch based on the image to be reconstructed by using a zero-sample method includes: Performing data enhancement on the image to be reconstructed to obtain m pseudo-resolution sample pairs; wherein m is an integer greater than 1; and Inputting the image to be reconstructed into a trained self-supervised convolutional neural network to obtain an input image of an initial subnet of the super-resolution branch; The trained self-supervised convolutional neural network is obtained by training a convolutional neural network based on the m pseudo-resolution sample pairs.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The subnet of the reconstruction branch and the subnet of the super-resolution branch are connected by dense connection between subnets.

10. A super-resolution reconstruction device for an image, characterized in that: The device comprises: A zero-sample module, used to obtain an input image of an initial subnetwork of a super-resolution branch using a zero-sample method based on the image to be reconstructed; A shallow feature extraction module, used to extract features from the input image of the initial subnet using a convolutional layer to obtain shallow features of the super-resolution branch; an artifact-free processing module, configured to transfer, based on the shallow features, the reconstruction features corresponding to the subnet of the reconstruction branch to the super-resolution features corresponding to the subnet of the super-resolution branch through a task converter module, so as to obtain artifact-free features; wherein the reconstruction features corresponding to the subnet of the reconstruction branch are obtained by processing the image to be reconstructed by a local implicit normalization flow; and An upsampling module is used to upsample the artifact-free features and the shallow features to obtain a super-resolution reconstructed image.

11. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images