Image processing method and device, training method, and computer-readable storage medium

Abstract

Image processing method and device, training method, and computer-readable storage medium. The present disclosure provides an image processing method, comprising: acquiring an input image and N-1 momentum terms, N being a positive integer and N>2; generating N levels of initial feature images arranged from high to low resolution based on the input image; performing iteration back projection processing of the i-th level based on the initial feature image of the i+1-th level and the momentum term of the i-th level to generate an updated feature image of the i-th level; i=1, 2, …, N-1; and generating an output image based on the updated feature image of the first level. The present disclosure also provides an image processing device, a neural network training method, and a non-transitory computer-readable storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of display technology, specifically to an image processing method and apparatus, a neural network training method, and a non-transient computer-readable storage medium. Background Technology

[0002] Currently, deep learning technology based on artificial neural networks has made significant progress in fields such as image classification, image capture and search, facial recognition, age and speech recognition. The advantage of deep learning lies in its ability to leverage general structures to solve very different technical problems with relatively similar systems. Convolutional Neural Networks (CNNs) are a type of artificial neural network that has developed in recent years and attracted widespread attention. CNNs are a special type of image recognition method, belonging to highly efficient networks with feedforward feedback. Summary of the Invention

[0003] This disclosure presents an image processing method, an image processing apparatus, a neural network training method, and a non-transient computer-readable storage medium.

[0004] In a first aspect, this disclosure provides an image processing method, comprising:

[0005] Obtain the input image and the momentum terms of N-1 levels, where N is a positive integer and N>2;

[0006] Based on the input image, generate initial feature images of N levels with resolution arranged from high to low;

[0007] For the initial feature images of the N levels, based on the initial feature image of the (i+1)th level and the momentum term of the ith level, iterative back projection processing of the ith level is performed to generate the updated feature image of the ith level; i = 1, 2, ..., N-1;

[0008] The output image is generated based on the updated feature image of level 1.

[0009] In some embodiments, the iterative back projection processing at each level includes: downsampling processing, concatenation processing, upsampling processing, first superposition processing, and second superposition processing;

[0010] The downsampling process at level i includes: downsampling the input based on the iterative backprojection process at level i to generate the downsampled output at level i.

[0011] The connection processing of the i-th level includes: performing a connection operation based on the downsampled output of the i-th level and the initial feature image of the (i+1)-th level to generate the joint output of the i-th level;

[0012] The upsampling process at level i includes: generating an upsampled output at level i based on the joint output of level i;

[0013] The first superposition process of the i-th level includes: superimposing the first superposition input of the i-th level with the upsampled output of the i-th level to generate the first superposition output of the i-th level;

[0014] The second superposition process of the i-th level includes: superimposing the input of the iterative back projection process of the i-th level with the first superposition output of the i-th level to generate the output of the iterative back projection process of the i-th level;

[0015] The (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process. The input of the (j+1)th level iterative backprojection process includes the j-th level downsampling output, where j = 1, 2, ..., N-2. At least one level of iterative backprojection process is executed multiple times consecutively, and the input of the subsequent iterative backprojection process includes the output of the previous iterative backprojection process. The first superposition input of the first superposition process in the subsequent iterative backprojection process includes the first superposition output of the first superposition process in the previous iterative backprojection process. The first superposition input of the first iterative backprojection process includes the momentum term of this level. The updated feature image of the first level includes the output of the last iterative backprojection process of the first level.

[0016] In some embodiments, the downsampled output of the i-th level and the initial feature image of the (i+1)-th level are concatenated to generate the joint output of the i-th level, specifically including:

[0017] The downsampled output of the i-th level is used as the input of the iterative backprojection process of the (i+1)-th level to generate the output of the iterative backprojection process of the (i+1)-th level; and the output of the iterative backprojection process of the (i+1)-th level is concatenated with the initial feature image of the (i+1)-th level to generate the joint output of the i-th level.

[0018] In some embodiments, based on the input image, generating initial feature images of N levels arranged from high to low resolution includes:

[0019] The input image is analyzed and processed at N different levels to generate initial feature images at each of the N levels, arranged from high to low resolution.

[0020] In some embodiments, generating an output image based on the updated feature image of the first level includes:

[0021] The updated feature image of the first level is transformed to generate the output image.

[0022] In some embodiments, based on the input image, generating initial feature images of N levels arranged from high to low resolution includes:

[0023] The input image is used as the intermediate input image of the first level. The input image is downsampled to generate intermediate input images of the second to Nth levels with resolutions arranged from high to low.

[0024] The intermediate input images of each level are analyzed and processed to generate the input feature images of each level; wherein, the input feature image of the Nth level is used as the initial feature image of the Nth level.

[0025] For each of the first N-1 levels, the intermediate input image of this level is downsampled and analyzed sequentially to generate an intermediate feature image; the intermediate feature image of this level is concatenated with the initial feature image of the next level, and the concatenated image is upsampled; the upsampled image is superimposed with the momentum term of this level to generate a first momentum term; the first momentum term is superimposed with the input feature image of this level to generate the initial feature image of this level.

[0026] In some embodiments, the iterative backprojection process of each level is executed M times consecutively, where M is an integer greater than 1. Each iterative backprojection process includes downsampling, concatenation, upsampling, first superposition, and second superposition.

[0027] The downsampling process in the m-th iteration of the backprojection process at level i includes: downsampling based on the input of the m-th iteration of the backprojection process at level i to generate the downsampled output of the m-th iteration of the backprojection process at level i; wherein, the initial feature image at level i includes the input of the 1-th iteration of the backprojection process at level i, and the input of each subsequent iteration of the backprojection process at level i includes the output of the previous iteration of the backprojection process.

[0028] The connection process in the m-th iteration back projection process of the i-th level includes: performing a connection operation based on the downsampled output of the m-th iteration back projection process of the i-th level and the output of the m-th iteration back projection process of the (i+1)-th level to generate the m-th compensated feature image of the i-th level.

[0029] The upsampling process in the m-th iteration back projection process of the i-th level includes: upsampling based on the m-th compensated feature image of the i-th level to generate the upsampled output of the m-th iteration back projection process of the i-th level.

[0030] The first superposition process in the first iteration of the back projection process at level i includes: performing a superposition operation based on the upsampled output of the first iteration of the back projection process at level i and the first momentum term of level i to generate the first superposition output of the first iteration of the back projection process at level i; the first superposition process in each subsequent iteration of the back projection process at level i includes: performing a superposition operation based on the upsampled output of the current iteration of the back projection process and the first superposition output of the previous iteration of the back projection process to generate the first superposition output of the current iteration of the back projection process;

[0031] The second overlay process in the first iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the first iteration of the backprojection process at level i and the initial feature image of level i to generate the second overlay output of the first iteration of the backprojection process at level i; the second overlay process in each subsequent iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the current iteration of the backprojection process and the second overlay output of the previous iteration of the backprojection process to generate the second overlay output of the current iteration of the backprojection process, and the second overlay output of each iteration of the backprojection process is used as the output of the current iteration of the backprojection process;

[0032] Where m = 1, 2, ..., M, the second superimposed output of the last iteration of backprojection processing of the first level is used as the updated feature image of the first level.

[0033] In some embodiments, generating an output image based on the updated feature image of the first level includes:

[0034] The updated feature image of the first level is transformed, and the transformed image is superimposed on the input image to generate the output image.

[0035] In some embodiments, based on the input image, generating initial feature images of the N levels arranged in descending resolution includes:

[0036] The input image is concatenated with a random noise image to generate a joint input image;

[0037] The joint input image is analyzed and processed at N different levels to generate initial feature images at the N levels with resolutions arranged from high to low.

[0038] In some embodiments, among the N initial feature images, the first-level initial feature image has the highest resolution, and the resolution of the first-level initial feature image is the same as the resolution of the input image.

[0039] The resolution of the initial feature image of the previous level is an integer multiple of the resolution of the initial feature image of the next level.

[0040] Secondly, this disclosure provides a method for training a neural network, the neural network comprising: an analysis network, an iterative backprojection processing network, and an output network, the training method comprising:

[0041] Obtain the training input image and the preset N-1 momentum terms, where N is a positive integer and N>2;

[0042] The analysis network is used to process the training input image to generate N levels of initial training feature images with resolution arranged from high to low.

[0043] Using the iterative backprojection processing network, based on the initial training feature image of the (i+1)th layer and the momentum term of the ith layer, iterative backprojection processing of the ith layer is performed to generate the training update feature image of the ith layer; i = 1, 2, ..., N-1;

[0044] Using the output network, a training output image is generated based on the training update feature image of the first layer;

[0045] Based on the training output image, the loss value of the neural network is calculated using a loss function, and the parameters of the neural network are corrected according to the loss value.

[0046] In some embodiments, the loss function includes the mean squared error between the training standard image corresponding to the training input image and the training output image.

[0047] Thirdly, this disclosure provides an image processing apparatus, comprising:

[0048] The image acquisition module is configured to acquire the input image and N-1 preset momentum terms, where N is a positive integer;

[0049] The image processing module is configured to generate N initial feature images of varying resolutions from high to low based on the input image, where N is a positive integer and N>2; for the N initial feature images, perform iterative back-projection processing of the i-th level based on the (i+1)-th level initial feature image and the momentum term of the i-th level to generate an updated feature image of the i-th level; i = 1, 2, ..., N-1; and generate an output image based on the updated feature image of the i-th level.

[0050] Fourthly, this disclosure provides an image processing apparatus, comprising:

[0051] The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the image processing method or the training method described in the above embodiments.

[0052] Fifthly, this disclosure provides a non-transient computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method or the training method in the above embodiments. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart of an image processing method provided in some embodiments of this disclosure.

[0055] Figure 2A This is a schematic flowchart illustrating an image processing method provided in some embodiments of this disclosure.

[0056] Figure 2B This is a schematic flowchart illustrating an image processing method provided in some other embodiments of this disclosure.

[0057] Figure 3 This is a schematic flowchart illustrating an image processing method provided in some further embodiments of the present disclosure.

[0058] Figure 4 This is a schematic flowchart of an image processing method as a specific example of this disclosure.

[0059] Figure 5 This is a schematic diagram of each image processing subprocess provided in some embodiments of this disclosure.

[0060] Figure 6 A schematic block diagram of a neural network structure is provided for some embodiments of this disclosure.

[0061] Figure 7 This is a flowchart of a neural network training method provided in some embodiments of this disclosure.

[0062] Figure 8 This is a schematic architecture block diagram of a neural network training method provided in some embodiments of this disclosure.

[0063] Figure 9 This is a schematic block diagram of an image processing apparatus provided for some embodiments of the present disclosure.

[0064] Figure 10A schematic block diagram of an image processing apparatus provided for other embodiments of this disclosure. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0066] Image enhancement is a hot research topic in the field of image processing. Due to various physical limitations during image acquisition (e.g., the small size of image sensors in mobile phone cameras and other software and hardware limitations) and interference from environmental noise, image quality can be significantly degraded. The purpose of image enhancement is to improve the grayscale histogram and contrast of an image through image enhancement techniques, thereby highlighting image details and improving the visual effect of the image.

[0067] Figure 1 This is a flowchart of an image processing method provided in some embodiments of the present disclosure, wherein a trained neural network can be used to perform the image processing method. For example... Figure 1 As shown, the image processing methods include:

[0068] S101. Obtain the input image and the preset momentum terms for N-1 levels, where N is a positive integer and N>2. The momentum terms are features used to accelerate error convergence during the training process of the neural network (i.e., to improve the parameter update speed of the neural network).

[0069] S102. Based on the input image, generate initial feature images of N levels with resolution arranged from high to low.

[0070] S103. For the initial feature images of the N levels, based on the initial feature image of the (i+1)th level and the momentum term of the i-th level, perform iterative back projection processing of the i-th level to generate the updated feature image of the i-th level; i = 1, 2, ..., N-1.

[0071] The iterative backprojection process of the i-th level can be performed by using the initial feature image of the (i+1)-th level to compensate the initial feature image of the i-th level, and using the momentum term of the i-th level during the compensation process.

[0072] S104. Generate the output image based on the updated feature image of the first level.

[0073] In this embodiment, multiple initial feature images with different resolutions are generated based on the input image. Iterative backprojection processing at the i-th layer is then performed based on the (i+1)-th layer's initial feature image and the momentum term at the i-th layer, thereby improving the quality of the output image. Furthermore, the introduction of a momentum term at each layer enables the neural network to form a reversible residual neural network. During training, the momentum term increases the training speed and reduces memory usage. Since a momentum term is introduced at each layer, the training speed of the neural network can be further improved.

[0074] Figure 2A This is a schematic flowchart illustrating the image processing method provided in some embodiments of this disclosure. Figure 2B Here are schematic flowcharts illustrating image processing methods provided in other embodiments of this disclosure, in conjunction with the following... Figures 1 to 2B The image processing method of this disclosure will be described in detail.

[0075] like Figure 1 As shown, the image processing method includes:

[0076] S101. Obtain the input image and the momentum terms of the preset N-1 levels, where N is a positive integer and N>2.

[0077] For example, such as Figure 2A and Figure 2B As shown, the input image is labeled INP. The input image can include photos captured by a smartphone camera, tablet camera, personal computer camera, digital camera lens, surveillance camera, or webcam, etc., and can include images of people, animals, plants, or landscapes, etc., without limitation in this embodiment.

[0078] The input image INP can be a grayscale image or a color image. For example, a color image includes, but is not limited to, a 3-channel RGB image. It should be noted that, in the embodiments of this disclosure, when the input image INP is a grayscale image, the output image OUTP is also a grayscale image; when the input image INP is a color image, the output image OUTP is also a color image.

[0079] The image processing method provided in the embodiments of this disclosure can enhance images, thereby improving image quality.

[0080] It should be noted that all images in the embodiments of this disclosure are represented in matrix form. In some embodiments, momentum terms can be represented in matrix form, and the matrix corresponding to the momentum term at the same level has the same dimension as the matrix corresponding to the input image. The momentum terms at N-1 levels can be preset; for example, each element in the momentum term at each level can be zero.

[0081] S102. Based on the input image, generate initial feature images of N levels with resolution arranged from high to low.

[0082] Among the N initial feature images, the first-level initial feature image has the highest resolution, and the resolution of the first-level initial feature image is the same as that of the input image.

[0083] The resolution of the initial feature image of the previous level (e.g., the i-th level) is an integer multiple of the resolution of the initial feature image of the next level (e.g., the (i+1)-th level). For example, if the resolution of the initial feature image of the previous level is twice that of the initial feature image of the next level, and the resolution of the initial feature image of a certain level is 32*32, then the resolution of the initial feature image of the next level is 16*16.

[0084] In some embodiments, such as Figure 2A As shown, the input image INP can be analyzed and processed at N different levels by an analysis network, thereby generating initial feature images F01 to F0N at N levels with resolutions arranged from high to low (e.g., Figure 2A (as shown in F01 to F05). For example, as... Figure 4 As shown, the analysis network includes N analysis subnetworks ASN, each of which is used to perform the analysis processing at different levels, thereby generating initial feature images F01 to F0N at N levels of resolution arranged from high to low (e.g., Figure 2A (As shown in F01 to F05). For example, each analysis sub-network ASN can be implemented as a convolutional network module including convolutional neural networks (CNN), residual networks (ResNet), dense networks (DenseNet), etc. For example, each analysis sub-network ASN may include convolutional layers, downsampling layers, normalization layers, etc., but is not limited to these. The network parameters of different analysis sub-networks ASN can be different to generate initial feature images F01 to F0N with different resolutions.

[0085] For example, in some embodiments, such as Figure 2B As shown, the input image INP can be concatenated with the random noise image (noise, as shown in the diagram). Figure 2A and Figure 2B(As shown in the circle marked "C"), a joint input image is generated; then, an analysis network performs N different levels of analysis on the joint input image to generate N initial feature images F01 to F0N with resolutions arranged from high to low. For example, the concatenation process can be viewed as stacking the channel images of multiple (e.g., two or more) images to be concatenated, so that the number of channels of the concatenated image is the sum of the number of channels of the multiple images to be concatenated. For example, the channel images of the joint input image are the combination of the channel images of the input image and the channel images of the random noise image. For example, the random noise in the noise image can conform to a Gaussian distribution, but is not limited to this. For example, Figure 2B The specific process and details of the analysis and processing in the illustrated embodiments can be found in [reference]. Figure 2A The descriptions of the analysis and processing in the illustrated embodiments will not be repeated here.

[0086] It should be noted that during image enhancement processing, the detailed features (e.g., hair, lines, etc.) in the output image are often related to noise. When applying neural networks for image enhancement, the amplitude of the input noise is adjusted according to actual needs (whether details need to be highlighted and the degree of highlighting, etc.) so that the output image meets the actual requirements. For example, in some embodiments, the noise amplitude of the random noise image can be 0; for example, in other embodiments, the noise amplitude of the random noise image may not be 0. The embodiments disclosed herein do not impose any limitations on this.

[0087] For example, in Figure 2A and Figure 2B In this system, the order of each level is determined from top to bottom.

[0088] For example, in some embodiments, the resolution of the highest-resolution first-level initial feature image F01 can be the same as the resolution of the input image INP. For example, in some embodiments, the input image is obtained by performing resolution conversion processing (e.g., image super-resolution reconstruction processing) on ​​the original input image. In this case, the resolution of the lowest-resolution Nth-level initial feature image can be the same as the resolution of the original input image. It should be noted that embodiments of this disclosure include, but are not limited to, these.

[0089] It should be noted that, although Figure 2A and Figure 2B The examples show the generation of initial feature images F01 to F05 at five levels (i.e., N = 5), but this should not be considered a limitation of the present disclosure. The value of N can be set according to actual needs.

[0090] S103. Based on the initial feature image of the (i+1)th level and the momentum term of the i-th level, perform iterative back projection processing of the i-th level to generate the updated feature image of the i-th level; i = 1, 2, ..., N-1.

[0091] In this context, the updated feature images at the same level have the same resolution as the initial feature images.

[0092] In some embodiments, the iterative backprojection process for each level includes downsampling, concatenation, upsampling, first superposition, and second superposition. Furthermore, the iterative backprojection processes for levels 1 to N-1 are nested.

[0093] For example, such as Figure 2A and Figure 2B As shown, the downsampling process at level i includes: downsampling based on the input of the iterative backprojection process at level i to generate the downsampled output at level i. The concatenation process at level i includes: concatenating the downsampled output at level i and the initial feature image at level (i+1) to generate the joint output at level i. The upsampling process at level i includes: generating the upsampled output at level i based on the joint output at level i. The first superposition process at level i includes: superimposing the first superposition input at level i with the upsampled output at level i to generate the first superposition output at level i. The second superposition process at level i includes: superimposing the input of the iterative backprojection process at level i with the first superposition output at level i to generate the output of the iterative backprojection process at level i.

[0094] In this process, the (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process. The input of the (j+1)th level iterative backprojection process includes the j-th level downsampling output, where j = 1, 2, ..., N-2. It should be noted that, in this disclosure, "nesting" means that an object includes another object that is similar to or identical to it, and the objects include, but are not limited to, processes or network structures.

[0095] In this process, at least one level of iterative backprojection processing is executed multiple times consecutively, and the input of each subsequent iterative backprojection processing includes the output of the previous iterative backprojection processing; the first superposition input of the first superposition process in the subsequent iterative backprojection processing includes the first superposition output of the first superposition process in the previous iterative backprojection processing, and the first superposition input in the first iterative backprojection processing includes the momentum term of this level. For example, as... Figure 2A and Figure 2B As shown, the iterative backprojection process at each level can be executed twice consecutively. In this case, the quality of the output image can be improved while avoiding the complexity of the network structure.

[0096] Since the (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process, when each level's iterative backprojection process is executed twice consecutively (taking N=5 as an example), during each execution of the 3rd level iterative backprojection process, two 4th level iterative backprojections are performed between the downsampling process and the connection process; during each execution of the 2nd level iterative backprojection process, two 3rd level iterative backprojections are performed between the 2nd level downsampling process and the connection process; and during each execution of the 1st level iterative backprojection process, two 2nd level iterative backprojections are performed between the 3rd level downsampling process and the connection process. In other words, the 4th level iterative backprojection process is executed 16 times, the 3rd level iterative backprojection process is executed 8 times, the 2nd level iterative backprojection process is executed 4 times, and the 1st level iterative backprojection process is executed 2 times. In this scenario, when the fourth-level iterative backprojection process is executed for the first time, the first superposition input of the first superposition process includes the momentum term of the fourth level; when the fourth-level iterative backprojection process is executed for the second time, the first superposition input of the first superposition process includes the first superposition output of the first superposition process in the first iterative backprojection process; when the fourth-level iterative backprojection process is executed for the third time, the first superposition input of the first superposition process includes the first superposition output of the first superposition process in the second iterative backprojection process, and so on. Similarly, when the third-level iterative backprojection process is executed for the first time, the first superposition input of the first superposition process includes the momentum term of the third level; when the third-level iterative backprojection process is executed for the second time, the first superposition input of the first superposition process includes the first superposition output of the first superposition process in the first iterative backprojection process, and so on. For iterative backprojection processing at other levels, the same sampling processing method is used. That is, in each level, the first superposition input of the first superposition processing in the first iterative backprojection processing includes the momentum term of this level; the first superposition input of the first superposition processing in each subsequent iterative backprojection processing includes the first superposition output of the previous iterative backprojection processing at this level.

[0097] It should be noted that the embodiments of this disclosure do not limit the specific number of executions of the iterative backprojection process at each level.

[0098] In some embodiments, such as Figure 2A and Figure 2BAs shown, the connection processing of the i-th level specifically includes: using the downsampled output of the i-th level as the input of the iterative backprojection processing of the i+1-th level to generate the output of the iterative backprojection processing of the i+1-th level; and connecting the output of the iterative backprojection processing of the i+1-th level with the initial feature image of the i+1-th level to generate the joint output of the i-th level.

[0099] Downsampling (DS) is used to reduce the size of feature maps, thereby reducing the amount of data in the feature maps. This can be achieved through downsampling layers, but is not limited to these methods. For example, downsampling layers can employ downsampling methods such as max pooling, average pooling, and strided convolution.

[0100] Upsampling (US) is used to increase the size of feature maps, thereby increasing the amount of data in the feature maps. This can be achieved through upsampling layers, but is not limited to these methods. For example, upsampling layers can employ methods such as strided-transposed convolution and interpolation algorithms. Interpolation algorithms can include, for example, interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. For instance, when using interpolation algorithms for upsampling, both the original pixel values ​​and the interpolated values ​​can be preserved, thus increasing the size of the feature map.

[0101] The first overlay process AD1 has the same input and output dimensions. For example, taking feature images as an example, the first overlay process can directly overlay two feature images, that is, add the values ​​of each row and each column of the two feature images accordingly. Alternatively, the first overlay process can weightedly overlay two feature images, that is, multiply each row and each column value of one feature image by a first weight, multiply each row and each column value of the other feature image by a second weight, and then add the values ​​of the two feature images corresponding to the weighted values. The sum of the first and second weights can be 1. The first overlay process can be implemented using a convolutional network including convolutional layers.

[0102] The second overlay process AD2 is similar to the first overlay process AD1. AD2 can be implemented using a convolutional network that includes convolutional layers. The input and output dimensions of AD2 are the same. For example, using feature images, AD2 can directly overlay two feature images; or, it can weightedly overlay two feature images. AD2 maintains the input of each level's iterative backprojection processing within the output of each level's iterative backprojection processing at a certain ratio.

[0103] It should be noted that in some embodiments of this disclosure, the downsampling factor of downsampling at the same level corresponds to the upsampling factor of upsampling. That is, when the downsampling factor of the downsampling is 1 / y, the upsampling factor of the upsampling is y, where y is a positive integer and is typically equal to or greater than 2. This ensures that the output of upsampling at the same level and the input size of downsampling are the same.

[0104] It should be noted that in some embodiments of this disclosure (not limited to this embodiment), the parameters of downsampling processing at different levels (i.e., the parameters of the network structure corresponding to the downsampling processing) can be the same or different; the parameters of upsampling processing at different levels (i.e., the parameters of the network structure corresponding to the upsampling processing) can be the same or different; the parameters of the first superposition processing at different levels can be the same or different; and the parameters of the second superposition processing at different levels can be the same or different. The embodiments of this disclosure do not impose limitations on these aspects.

[0105] In some embodiments, the updated feature image at the same level has the same resolution as the initial feature image.

[0106] S104. Generate the output image based on the updated feature image of the first level.

[0107] For example, in some embodiments, the updated feature image of the first level can be transformed to generate an output image. For example, when the updated feature image includes multiple channels, such as... Figure 2A and Figure 2B As shown, the updated feature image of the first layer can be synthesized using the MERG synthesis network to generate the output image OUTP. For example, in some embodiments, the MERG synthesis network may include convolutional layers, etc. For example, the output image may include a one-channel grayscale image, or it may include, for example, a three-channel RGB image (i.e., a color image). It should be noted that the embodiments of this disclosure do not limit the structure and parameters of the MERG synthesis network, as long as it can convert the updated feature image of the first layer into the output image OUTP.

[0108] It should be noted that directly using the above image processing method to process high-resolution (e.g., 4k or higher resolution) input images places high demands on the hardware requirements of the image processing device (e.g., video memory). Therefore, in some embodiments, to solve the above problem, the input image can be cropped first to obtain multiple sub-input images with overlapping regions; then, the above image processing method (e.g., steps S101 to S104) is used to process each of the multiple sub-input images to generate corresponding multiple sub-output images; finally, the corresponding multiple sub-output images are stitched together to form the output image.

[0109] Figure 3 Here are schematic flowcharts illustrating the image processing methods provided in some embodiments of this disclosure, in conjunction with the following... Figure 1 and Figure 3 Other image processing methods disclosed herein will be described in detail.

[0110] like Figure 1 As shown, the image processing method includes:

[0111] S101. Obtain the input image and the momentum terms of the preset N-1 levels, where N is a positive integer and N>2.

[0112] like Figure 3 As shown, the input image is labeled INP. The input image can include photos captured by a smartphone camera, tablet camera, personal computer camera, digital camera lens, surveillance camera, or webcam, etc., and can include images of people, animals, plants, or landscapes, etc., without limitation in this embodiment.

[0113] The input image INP can be a grayscale image or a color image. For example, a color image includes, but is not limited to, a 3-channel RGB image. It should be noted that, in the embodiments of this disclosure, when the input image INP is a grayscale image, the output image OUTP is also a grayscale image; when the input image INP is a color image, the output image OUTP is also a color image.

[0114] In some embodiments, the momentum terms for N-1 levels can be preset, and the momentum terms for the same level have the same resolution as the input image. For example, each element in the momentum term for each level can be zero.

[0115] S102. Based on the input image, generate initial feature images of N levels arranged from high to low resolution, such as... Figure 3 F01 to F04 in the series.

[0116] Among the N initial feature images, the first-level initial feature image has the highest resolution, and the resolution of the first-level initial feature image is the same as that of the input image INP.

[0117] The resolution of the initial feature image of the previous level (e.g., the i-th level) is an integer multiple of the resolution of the initial feature image of the next level (e.g., the (i+1)-th level). For example, if the resolution of the initial feature image of the previous level is twice that of the initial feature image of the next level, and the resolution of the initial feature image of a certain level is 32×32, then the resolution of the initial feature image of the next level is 16×16.

[0118] In some embodiments, the input image can be processed to generate N intermediate input images arranged in descending order of resolution, such as... Figure 3 INM1 to INM4.

[0119] Next, the intermediate input images (i.e., INM1 to INM4) of each level are analyzed and processed to generate the input feature image of each level. For example, the intermediate input images (i.e., INM1 to INM4) are analyzed and processed using an analysis subnetwork ASN. The network parameters of different analysis subnetworks ASN can be different. The input feature image of the Nth level is used as the initial feature image of the Nth level. For each of the first N-1 levels, the intermediate input image of this level is sequentially subjected to downsampling processing DS and analysis processing to generate an intermediate feature image. The intermediate feature image of this level is then concatenated with the initial feature image of the next level, and the concatenated image is upsampled US. The upsampled image is then superimposed with the momentum term of this level to generate the first momentum term. The first momentum term is then superimposed with the input feature image of this level to generate the initial feature image of this level.

[0120] For example, the input image INP is used as the intermediate input image for level 1. The input image is downsampled N-1 times to generate intermediate input images for levels 2 to N. Figure 3 Downsampling processing and Figure 2A and Figure 2B The principle of downsampling is the same as that in [the context of the previous sentence]. Figure 3 Upsampling processing and Figure 2A and Figure 2B The principle of upsampling is the same as that in [the context of the previous sentence]. Figure 3 Overlay processing and Figure 2A and Figure 2BThe principle is the same as the first overlay process. The concatenation operation can be viewed as stacking the channel images of multiple (e.g., two or more) images to be concatenated, so that the number of channels of the concatenated image is the sum of the number of channels of the multiple images to be concatenated.

[0121] S103. Based on the initial feature image of level i+1 and the momentum term of level i, perform iterative back projection processing at level i to generate the updated feature image of level i; i = 1, 2, ..., N-1. The updated feature image at the same level has the same resolution as the initial feature image.

[0122] For example, the iterative backprojection process at each level is executed M times consecutively, where M is an integer greater than 1. Each iterative backprojection process includes downsampling, concatenation, upsampling, first superposition, and second superposition.

[0123] The downsampling process in the m-th iteration of the backprojection process at level i includes: downsampling based on the input of the m-th iteration of the backprojection process at level i to generate the downsampled output of the m-th iteration of the backprojection process at level i. The input of the first iteration of the backprojection process at level i is the initial feature image of level i, and the input of each subsequent iteration of the backprojection process at level i is the output of the previous iteration. Here, m takes integer values ​​from [1, M].

[0124] The connection process in the m-th iteration backprojection process of the i-th level includes: performing a connection operation based on the downsampled output of the m-th iteration backprojection process of the i-th level and the output of the m-th iteration backprojection process of the (i+1)-th level to generate the m-th compensated feature image of the i-th level.

[0125] The upsampling process in the m-th iteration of the back projection process of the i-th level includes: upsampling based on the m-th compensated feature image of the i-th level to generate the upsampled output of the m-th iteration of the back projection process of the i-th level.

[0126] The first superposition process in the first iteration of the back projection process of the i-th level includes: performing a superposition operation based on the upsampled output of the first iteration of the back projection process of the i-th level and the first momentum term of the i-th level to generate the first superposition output of the first iteration of the back projection process of the i-th level; the first superposition process in each subsequent iteration of the back projection process of the i-th level includes: performing a superposition operation based on the upsampled output of the current iteration of the back projection process and the first superposition output of the previous iteration of the back projection process to generate the first superposition output of the current iteration of the back projection process.

[0127] The second overlay process in the first iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the first iteration of the backprojection process at level i and the initial feature image of level i to generate the second overlay output of the first iteration of the backprojection process at level i; the second overlay process in each subsequent iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the current iteration of the backprojection process and the second overlay output of the previous iteration of the backprojection process to generate the second overlay output of the current iteration of the backprojection process, and the second overlay output of each iteration of the backprojection process is used as the output of the current iteration of the backprojection process.

[0128] S104. Generate the output image based on the updated feature image UDP of the first level.

[0129] For example, in some embodiments, the updated feature image UDP can be transformed to generate an output image OUTP. For example, when the first-level updated feature image UDP includes multiple channels, such as... Figure 3 As shown, the synthesis network MERG can be used to synthesize the updated feature image UDP of the first layer, and then superimpose the synthesized image onto the input image INP (i.e., the intermediate input image INM1) to generate the output image OUTP. For example, in some embodiments, the synthesis network MERG may include convolutional layers, etc. For example, the output image OUTP may include a grayscale image with one channel, or it may include, for example, an RGB image with three channels (i.e., a color image). It should be noted that the resolution and number of channels of the input image UDP of the synthesis network MERG are the same as those of the input image INP, so that superposition with the input image INP can be achieved. The embodiments of this disclosure do not limit the structure and parameters of the synthesis network MERG, as long as the image it converts can be superimposed with the input image INP to generate the output image OUTP.

[0130] Figure 4 This is a schematic flowchart illustrating an image processing method in a specific example of this disclosure. Figure 4 Taking N=4 and M=3 as an example, let's examine... Figure 3 The image processing method shown is illustrated below. Figure 4 The image processing methods shown will be described in detail.

[0131] For example, based on the input image INP, generate a first intermediate input image INM1, a second intermediate input image INM2, a third intermediate input image INM3, and a fourth intermediate input image INM4, arranged in descending order of resolution. For instance, using the input image as the first intermediate input image INM1, downsampling the input image generates the second intermediate input image INM2; downsampling the second intermediate input image INM2 generates the third intermediate input image INM3; and downsampling the third intermediate input image INM3 generates the fourth intermediate input image INM4.

[0132] The first intermediate input image INM1 is analyzed and processed to generate the first input feature image IF1; the second intermediate input image INM2 is analyzed and processed to generate the second input feature image IF2; the third intermediate input image INM3 is analyzed and processed to generate the third input feature image IF3; and the fourth intermediate input image INM4 is analyzed and processed to generate the fourth input feature image IF4.

[0133] The fourth input feature image IF4 is used as the initial feature image F04 of the fourth level. The third intermediate input image INM3 is downsampled and analyzed sequentially to generate the third intermediate feature image. The third intermediate feature image is concatenated with the initial feature image F04 of the fourth level, and the concatenated image is upsampled. The upsampled image is superimposed with the momentum term Z3 of the third level to generate the first momentum term Z31 of the third level. The first momentum term Z31 of the third level is superimposed with the third input feature image IF3 to generate the initial feature image F03 of the third level.

[0134] The second intermediate input image INM2 is downsampled and analyzed sequentially to generate a second intermediate feature image. This second intermediate feature image is then concatenated with the initial feature image F03 of the third level, and the concatenated image is upsampled. The upsampled image is then superimposed with the momentum term Z2 of the second level to generate the first momentum term Z21 of the second level. The first momentum term Z21 of the second level is then superimposed with the second input feature image IF2 to generate the initial feature image F02 of the second level.

[0135] The first intermediate input image INM1 is downsampled and analyzed sequentially to generate a first intermediate feature image. This first intermediate feature image is concatenated with the second-level initial feature image F02, and the concatenated image is upsampled. The upsampled image is then concatenated with the first-level momentum term Z1 to generate the first-level momentum term Z11. The first-level momentum term Z11 is then superimposed with the first input feature image IF1 to generate the first-level initial feature image F01.

[0136] The initial feature image F03 of level 3 is downsampled, and the downsampled image is concatenated with the initial feature image F04 of level 4 to generate a concatenated image IMC1. The concatenated image IMC1 is upsampled, and the upsampled image is superimposed with the first momentum term Z31 of level 3 to generate the first superimposed output IA11_3 of level 3. The first superimposed output IA11_3 is then superimposed with the initial feature image F03 of level 3 to generate the first superimposed output IA21_3 of level 3.

[0137] The first second superimposed output IA21_3 of the third level is downsampled, and the downsampled image is concatenated with the initial feature image F04 of the fourth level to generate the concatenated image IMC4. The concatenated image IMC4 is upsampled, and the upsampled image is superimposed with the first first superimposed output IA11_3 of the third level to generate the second first superimposed output IA12_3 of the third level. The first superimposed output IA12_3 is then superimposed with the first second superimposed output IA21_3 of the third level to generate the second second superimposed output IA22_3 of the third level.

[0138] The second second superimposed output IA22_3 of the third level is downsampled, and the downsampled image is concatenated with the initial feature image F04 of the fourth level to generate the concatenated image IMC7. The concatenated image IMC7 is upsampled, and the upsampled image is superimposed with the second first superimposed output IA12_3 of the third level to generate the third first superimposed output IA13_3 of the third level. The first superimposed output IA13_3 is then superimposed with the second second superimposed output IA22_3 of the third level to generate the third second superimposed output IA23_3 of the third level.

[0139] The initial feature image F02 of the second level is downsampled, and the downsampled image is concatenated with the first second superimposed output IA21_3 of the third level to generate the concatenated image IMC2. The concatenated image IMC2 is upsampled, and the upsampled image is superimposed with the first momentum term Z21 of the second level to generate the first first superimposed output IA11_2 of the second level. The first superimposed output IA11_2 is then superimposed with the initial feature image F02 of the second level to generate the first second superimposed output IA21_2 of the second level.

[0140] The first second overlay output IA21_2 of the second layer is downsampled, and the downsampled image is concatenated with the second second overlay output IA22_3 of the third layer to generate the concatenated image IMC5. The concatenated image IMC5 is upsampled, and the upsampled image is overlaid with the first first overlay output IA11_2 of the second layer to generate the second first overlay output IA12_2 of the second layer. This first overlay output IA12_2 is then overlaid with the first second overlay output IA21_2 of the second layer to generate the second second overlay output IA22_2 of the second layer.

[0141] The second second overlay output IA22_2 of the second level is downsampled, and the downsampled image is concatenated with the third second overlay output IA23_3 of the third level to generate the concatenated image IMC8. The concatenated image IMC8 is upsampled, and the upsampled image is overlaid with the second first overlay output IA12_2 of the second level to generate the third first overlay output IA13_2 of the second level. This first overlay output IA13_2 is then overlaid with the second second overlay output IA22_3 of the second level to generate the third second overlay output IA23_2 of the second level.

[0142] The initial feature image F01 of the first level is downsampled, and the downsampled image is concatenated with the first second superimposed output IA21_2 of the second level to generate the concatenated image IMC3. The concatenated image IMC3 is upsampled, and the upsampled image is superimposed with the first momentum term Z11 of the first level to generate the first first superimposed output IA11_1 of the first level. The first superimposed output IA11_1 is then superimposed with the initial feature image F01 of the first level to generate the first second superimposed output IA21_1 of the first level.

[0143] The first second overlay output IA21_1 of the first level is downsampled, and the downsampled image is concatenated with the second second overlay output IA22_2 of the second level to generate the concatenated image IMC6. The concatenated image IMC6 is upsampled, and the upsampled image is overlaid with the first first overlay output IA11_1 of the first level to generate the second first overlay output IA12_1 of the first level. This first overlay output IA12_1 is then overlaid with the first second overlay output IA21_1 of the first level to generate the second second overlay output IA22_1 of the first level.

[0144] The second second overlay output IA22_1 of the first level is downsampled, and the downsampled image is concatenated with the third second overlay output IA23_2 of the second level to generate the concatenated image IMC9. The concatenated image IMC9 is upsampled, and the upsampled image is overlaid with the second first overlay output IA12_1 of the first level to generate the third first overlay output IA13_1 of the first level. This first overlay output IA13_1 is then overlaid with the second second overlay output IA22_1 of the first level to generate the third second overlay output IA23_1 of the first level. This third second overlay output IA23_1 is the updated feature image UDP of the first level.

[0145] The synthesis network MERG is used to synthesize the updated feature image UDP (i.e., the second superimposed output IA23_1 mentioned above) of the first level, and the synthesized image is superimposed on the input image to generate the output image OUTP.

[0146] Figure 3 and Figure 4 The image processing process in the text is equivalent to including multiple image processing sub-processes. Figure 5 This is a schematic diagram of each image processing sub-process provided in some embodiments of this disclosure, such as... Figure 5 As shown, each image processing sub-process has 4 inputs (i.e., Figure 5 The inputs are 1 to 4 and there are 4 outputs (i.e., Figure 5 The image processing subprocesses output 1 through 4. Input 1 and Input 2 are superimposed, and the result is output 4. This output 4 is then superimposed with input 3, and the result is output 2. This output 2 is then combined with input 4, and the combined result is output 3. Simultaneously, output 2 is downsampled to generate output 1. It should be noted that some image processing subprocesses may lack one or two inputs, in which case the corresponding operation will be skipped. For example, if only input 1, input 2, input 3, output 1, output 2, and output 4 are present, then no operation is required. Figure 5 The concatenation and upsampling operations are used. For example, when input 1, input 2, output 1, and output 4 are missing, input 3 is directly used as output 2, and input 3 and input 4 are concatenated, with the concatenated result being output 3.

[0147] This disclosure also provides a method for training a neural network. Figure 6 A schematic block diagram of a neural network structure is provided for some embodiments of this disclosure. Figure 7 This is a flowchart of a neural network training method provided in some embodiments of this disclosure. Figure 8This is a schematic architecture block diagram of a neural network training method provided in some embodiments of this disclosure.

[0148] The neural network is used to execute the image processing method provided in the foregoing embodiments.

[0149] For example, such as Figure 6 As shown, the neural network 100 includes an analysis network 110, an iterative backprojection processing network 120, and an output network 130. The analysis network 110 executes step S102 of the image processing method described above, that is, the analysis network 110 processes the input image to generate N initial feature images of varying resolutions from high to low, where N is a positive integer and N>2. The iterative backprojection processing network 120 executes step S130 of the image processing method described above, that is, the iterative backprojection processing network 120 performs iterative backprojection processing at the i-th level based on the initial feature image of the (i+1)-th level and the momentum term of the i-th level to generate an updated feature image at the i-th level; i = 1, 2, ..., N-1. The output network executes step S140 of the image processing method described above, that is, the output network 130 generates an output image based on the updated feature image of the first level.

[0150] The specific structure of the neural network 100 and its corresponding processing procedure can be found in the relevant descriptions in the aforementioned image processing methods, and will not be repeated here.

[0151] like Figure 7 and Figure 8 As shown, the training method for the neural network includes steps S201 to S205.

[0152] Step S201: Obtain the training input image and the momentum terms of the preset N-1 levels. N is a positive integer and N>2.

[0153] For example, similar to the input image in step S101 above, the training input image may also include photos captured by a smartphone camera, tablet camera, personal computer camera, digital camera lens, surveillance camera, or webcam, etc., which may include images of people, animals, plants, or landscapes, etc., and the embodiments of this disclosure are not limited thereto. The training input image can be a grayscale image or a color image. For example, a color image includes, but is not limited to, a 3-channel RGB image.

[0154] Step S202: The training input image is processed using the analysis network 110 to generate initial training feature images arranged from high to low resolution.

[0155] Step S203: Using the iterative backprojection processing network 120, based on the initial training feature image of the (i+1)th layer and the momentum term of the i-th layer, perform iterative backprojection processing of the i-th layer to generate the updated feature image of the i-th layer; i = 1, 2, ..., N-1.

[0156] The process of iterative backprojection processing at the i-th level of the iterative backprojection processing network can be found in the description above, and will not be repeated here.

[0157] Step S204: Using the output network 130, generate a training output image based on the training update feature image of the first layer.

[0158] The specific process by which the output network generates the training output image based on the updated feature image from the first layer can be found in the description of step S104 above. It will not be repeated here.

[0159] Step S205: Based on the training output image, calculate the loss value of the neural network through the loss function, and correct the parameters of the neural network according to the loss value of the neural network.

[0160] The parameters of the neural network can include the parameters of the analysis network, the parameters of the iterative backprojection network, and the parameters of the output network.

[0161] In some embodiments, the loss function Loss may include the mean squared error between the training standard image corresponding to the training input image and the training output image. Specifically, it is shown in the following formula:

[0162] Loss = E[(xy)] 2 ]

[0163] Where x is the training output image, y is the training standard image, and E[] represents the calculation of matrix energy. For example, E[] can be used to calculate the maximum or average value of the elements in the matrix within the brackets. For example, the training standard image and the training input image have the same scene, i.e., their content is the same, and the quality of the training standard image is higher than that of the training input image. For example, the training standard image is equivalent to the target output image of the neural network.

[0164] For example, the training standard image can be a photograph taken by, for example, a digital SLR camera. In some embodiments, the training standard image may be degraded to generate the training input image.

[0165] During the training process of a neural network, the training objective is to minimize the loss value. For example, during the training process, the parameters of the neural network are continuously adjusted so that the training output image output by the neural network after parameter adjustment continuously approaches the training standard image, thereby continuously reducing the loss value. It should be noted that the loss function provided in this embodiment is exemplary, and the embodiments of this disclosure include, but are not limited to, this. For example, the loss function can also be an L1 regularization term between the training standard image and the training output image.

[0166] This disclosure also provides an image processing apparatus. Figure 9 A schematic block diagram of an image processing apparatus provided for some embodiments of this disclosure, such as Figure 9 As shown, the image processing apparatus includes an image acquisition module 301 and an image processing module 302. The image processing apparatus can be used to execute the image processing method described in the above embodiments. The image acquisition module 301 can be used to execute step S101 in the above image processing method. For example, the image acquisition module 301 can be used to acquire an input image and momentum terms at multiple levels. For example, the image acquisition module 301 may include a memory storing the input image and momentum terms at multiple levels; or, the image acquisition module 301 may also include one or more cameras to acquire the input image; it may also include an information input structure to acquire momentum terms.

[0167] The image processing module 302 can be used to perform steps S102 to S104 of the aforementioned image processing method. In some embodiments, the image acquisition module 301 and the image processing module 302 can be implemented as hardware, software, firmware, or any feasible combination thereof.

[0168] Figure 10 A schematic block diagram of an image processing apparatus provided for other embodiments of this disclosure, such as Figure 10 As shown, the image processing apparatus 200 includes a memory 201 and a processor 202. For example, the memory 201 is used to non-temporarily store computer-readable instructions, and the processor 202 is used to execute the computer-readable instructions, which, when executed by the processor, perform the image processing method and / or neural network training method provided in any embodiment of this disclosure.

[0169] The memory 201 and processor 202 can communicate directly or indirectly with each other. For example, in some examples, the image processing device 200 may also include a system bus, through which the memory 201 and processor 202 can communicate, for example, the processor 202 can access the memory through the system bus. In other examples, components such as the memory 201 and processor 202 can communicate via a network connection. The network may include a wireless network, a wired network, and / or any combination of wireless and wired networks. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things (IoT) based on the Internet and / or a telecommunications network, and / or any combination of the above networks. Wired networks may use methods such as twisted-pair cables, coaxial cables, or fiber optic transmission for communication, while wireless networks may use methods such as 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee, or WiFi. This disclosure does not limit the type and function of the network.

[0170] For example, processor 202 can control other components in the image processing device to perform desired functions. The processor can be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), tensor processor (TPU), or graphics processing unit (GPU). The CPU can be based on x86 or ARM architectures, etc. The GPU can be integrated directly onto the motherboard or built into the motherboard's northbridge chip. The GPU can also be built into the CPU.

[0171] For example, memory 201 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0172] For example, one or more computer instructions may be stored on memory 201, and processor 202 may execute the computer instructions to perform various functions. Various application programs and various data may also be stored in the computer-readable storage medium, such as input images, output images, first / second training input images, first / second training output images, first / second training standard images, and various data used and / or generated by the application programs.

[0173] For example, some computer instructions stored in memory 201, when executed by a processor, can perform one or more steps in the image processing method described above. As another example, other computer instructions stored in memory 201, when executed by a processor, can perform one or more steps in the neural network training method described above.

[0174] The image processing apparatus 200 may also include an input interface that allows external devices to communicate with the image processing apparatus. For example, the input interface may be used to receive instructions from an external computer device, a user, or the like. The image processing apparatus may also include an output interface that interconnects the image processing apparatus with one or more external devices. For example, the image processing apparatus may display images through the output interface.

[0175] For a detailed explanation of the image processing method, please refer to the relevant descriptions in the embodiments of the image processing method described above. For a detailed explanation of the neural network training method, please refer to the relevant descriptions in the embodiments of the neural network training method described above. Repeated descriptions will not be repeated here.

[0176] This disclosure also provides a non-transient computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image processing method or neural network training method described in the above embodiments. For example, the computer-readable storage medium may include the storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0177] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. An image processing method, characterized in that, include: Obtain the input image and the momentum terms of N-1 levels, where N is a positive integer and N>2; Based on the input image, generate initial feature images of N levels with resolution arranged from high to low; For the initial feature images of the N levels, based on the initial feature image of the (i+1)th level and the momentum term of the ith level, iterative back projection processing of the ith level is performed to generate the updated feature image of the ith level; i = 1, 2, ..., N-1; The output image is generated based on the updated feature image of the first level; The iterative back projection processing at each level includes: downsampling processing, concatenation processing, upsampling processing, first superposition processing, and second superposition processing. The downsampling process at level i includes: downsampling the input based on the iterative backprojection process at level i to generate the downsampled output at level i. The connection processing of the i-th level includes: performing a connection operation based on the downsampled output of the i-th level and the initial feature image of the (i+1)-th level to generate the joint output of the i-th level; The upsampling process at level i includes: generating an upsampled output at level i based on the joint output of level i; The first superposition process of the i-th level includes: superimposing the first superposition input of the i-th level with the upsampled output of the i-th level to generate the first superposition output of the i-th level; The second superposition process of the i-th level includes: superimposing the input of the iterative back projection process of the i-th level with the first superposition output of the i-th level to generate the output of the iterative back projection process of the i-th level; The (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process. The input of the (j+1)th level iterative backprojection process includes the j-th level downsampling output, where j = 1, 2, ..., N-2. At least one level of iterative backprojection process is executed multiple times consecutively, and the input of the subsequent iterative backprojection process includes the output of the previous iterative backprojection process. The first superposition input of the first superposition process in the subsequent iterative backprojection process includes the first superposition output of the first superposition process in the previous iterative backprojection process. The first superposition input of the first iterative backprojection process includes the momentum term of this level. The updated feature image of the first level includes the output of the last iterative backprojection process of the first level.

2. The image processing method according to claim 1, characterized in that, The downsampled output of the i-th level and the initial feature image of the (i+1)-th level are concatenated to generate the joint output of the i-th level, specifically including: The downsampled output of the i-th level is used as the input of the iterative backprojection process of the (i+1)-th level to generate the output of the iterative backprojection process of the (i+1)-th level; and the output of the iterative backprojection process of the (i+1)-th level is concatenated with the initial feature image of the (i+1)-th level to generate the joint output of the i-th level.

3. The image processing method according to claim 1, characterized in that, Based on the input image, generate initial feature images of N levels arranged from high to low resolution, including: The input image is analyzed and processed at N different levels to generate initial feature images at each of the N levels, arranged from high to low resolution.

4. The image processing method according to claim 1, characterized in that, The output image is generated based on the updated feature image of level 1, including: The updated feature image of the first level is transformed to generate the output image.

5. The image processing method according to claim 1, characterized in that, Based on the input image, generate initial feature images of N levels arranged from high to low resolution, including: The input image is used as the intermediate input image of the first level. The input image is downsampled to generate intermediate input images of the second to Nth levels with resolutions arranged from high to low. The intermediate input images of each level are analyzed and processed to generate the input feature images of each level; wherein, the input feature image of the Nth level is used as the initial feature image of the Nth level. For each of the first N-1 levels, the intermediate input image of this level is downsampled and analyzed sequentially to generate an intermediate feature image; the intermediate feature image of this level is concatenated with the initial feature image of the next level, and the concatenated image is upsampled; the upsampled image is superimposed with the momentum term of this level to generate a first momentum term; the first momentum term is superimposed with the input feature image of this level to generate the initial feature image of this level.

6. The image processing method according to claim 5, characterized in that, Each level of iterative backprojection processing is executed M times consecutively, where M is an integer greater than 1. Each iterative backprojection processing includes downsampling processing, concatenation processing, upsampling processing, first superposition processing, and second superposition processing. The downsampling process in the m-th iteration of the backprojection process at level i includes: downsampling based on the input of the m-th iteration of the backprojection process at level i to generate the downsampled output of the m-th iteration of the backprojection process at level i; wherein, the initial feature image at level i includes the input of the 1-th iteration of the backprojection process at level i, and the input of each subsequent iteration of the backprojection process at level i includes the output of the previous iteration of the backprojection process. The connection process in the m-th iteration back projection process of the i-th level includes: performing a connection operation based on the downsampled output of the m-th iteration back projection process of the i-th level and the output of the m-th iteration back projection process of the (i+1)-th level to generate the m-th compensated feature image of the i-th level. The upsampling process in the m-th iteration back projection process of the i-th level includes: upsampling based on the m-th compensated feature image of the i-th level to generate the upsampled output of the m-th iteration back projection process of the i-th level. The first superposition process in the first iteration of the back projection process at level i includes: performing a superposition operation based on the upsampled output of the first iteration of the back projection process at level i and the first momentum term of level i to generate the first superposition output of the first iteration of the back projection process at level i; the first superposition process in each subsequent iteration of the back projection process at level i includes: performing a superposition operation based on the upsampled output of the current iteration of the back projection process and the first superposition output of the previous iteration of the back projection process to generate the first superposition output of the current iteration of the back projection process; The second overlay process in the first iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the first iteration of the backprojection process at level i and the initial feature image of level i to generate the second overlay output of the first iteration of the backprojection process at level i; the second overlay process in each subsequent iteration of the backprojection process at level i includes: performing an overlay operation based on the first overlay output of the current iteration of the backprojection process and the second overlay output of the previous iteration of the backprojection process to generate the second overlay output of the current iteration of the backprojection process, and the second overlay output of each iteration of the backprojection process is used as the output of the current iteration of the backprojection process; Where m = 1, 2, ..., M, the second superimposed output of the last iteration of backprojection processing of the first level is used as the updated feature image of the first level.

7. The image processing method according to claim 5, characterized in that, The output image is generated based on the updated feature image of level 1, including: The updated feature image of the first level is transformed, and the transformed image is superimposed on the input image to generate the output image.

8. The image processing method according to claim 1 or 5, characterized in that, Based on the input image, initial feature images of the N levels arranged from high to low resolution are generated, including: The input image is concatenated with a random noise image to generate a joint input image; The joint input image is analyzed and processed at N different levels to generate initial feature images at the N levels with resolutions arranged from high to low.

9. The image processing method according to any one of claims 1 to 7, characterized in that, Among the N initial feature images, the first-level initial feature image has the highest resolution, and the resolution of the first-level initial feature image is the same as that of the input image. The resolution of the initial feature image of the previous level is an integer multiple of the resolution of the initial feature image of the next level.

10. A method for training a neural network, characterized in that, The neural network includes: an analysis network, an iterative back-projection processing network, and an output network; the training method includes: Obtain the training input image and the preset N-1 momentum terms, where N is a positive integer and N>2; The analysis network is used to process the training input image to generate N levels of initial training feature images with resolution arranged from high to low. Using the iterative backprojection processing network, based on the initial training feature image of the (i+1)th layer and the momentum term of the ith layer, iterative backprojection processing of the ith layer is performed to generate the training update feature image of the ith layer; i = 1, 2, ..., N-1; Using the output network, a training output image is generated based on the training update feature image of the first layer; Based on the training output image, the loss value of the neural network is calculated using a loss function, and the parameters of the neural network are corrected according to the loss value of the neural network. The iterative back projection processing at each level includes: downsampling processing, concatenation processing, upsampling processing, first superposition processing, and second superposition processing. The downsampling process at level i includes: downsampling the input based on the iterative backprojection process at level i to generate the downsampled output at level i. The connection processing of the i-th level includes: performing a connection operation based on the downsampled output of the i-th level and the initial feature image of the (i+1)-th level to generate the joint output of the i-th level; The upsampling process at level i includes: generating an upsampled output at level i based on the joint output of level i; The first superposition process of the i-th level includes: superimposing the first superposition input of the i-th level with the upsampled output of the i-th level to generate the first superposition output of the i-th level; The second superposition process of the i-th level includes: superimposing the input of the iterative back projection process of the i-th level with the first superposition output of the i-th level to generate the output of the iterative back projection process of the i-th level; The (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process. The input of the (j+1)th level iterative backprojection process includes the j-th level downsampling output, where j = 1, 2, ..., N-2. At least one level of iterative backprojection process is executed multiple times consecutively, and the input of the subsequent iterative backprojection process includes the output of the previous iterative backprojection process. The first superposition input of the first superposition process in the subsequent iterative backprojection process includes the first superposition output of the first superposition process in the previous iterative backprojection process. The first superposition input of the first iterative backprojection process includes the momentum term of this level. The updated feature image of the first level includes the output of the last iterative backprojection process of the first level.

11. The training method according to claim 10, characterized in that, The loss function includes the mean squared error between the training standard image corresponding to the training input image and the training output image.

12. An image processing apparatus, characterized in that, include: The image acquisition module is configured to acquire the input image and N-1 preset momentum terms, where N is a positive integer; The image processing module is configured to generate N initial feature images of varying resolutions from high to low based on the input image, where N is a positive integer and N>2; for the N initial feature images, iterative backprojection processing of the i-th level is performed based on the (i+1)-th level initial feature image and the momentum term of the i-th level to generate an updated feature image of the i-th level; i = 1, 2, ..., N-1; And generate the output image based on the updated feature image of level 1; The iterative back projection processing at each level includes: downsampling processing, concatenation processing, upsampling processing, first superposition processing, and second superposition processing. The downsampling process at level i includes: downsampling the input based on the iterative backprojection process at level i to generate the downsampled output at level i. The connection processing of the i-th level includes: performing a connection operation based on the downsampled output of the i-th level and the initial feature image of the (i+1)-th level to generate the joint output of the i-th level; The upsampling process at level i includes: generating an upsampled output at level i based on the joint output of level i; The first superposition process of the i-th level includes: superimposing the first superposition input of the i-th level with the upsampled output of the i-th level to generate the first superposition output of the i-th level; The second superposition process of the i-th level includes: superimposing the input of the iterative back projection process of the i-th level with the first superposition output of the i-th level to generate the output of the iterative back projection process of the i-th level; The (j+1)th level iterative backprojection process is nested between the j-th level downsampling process and the j-th level connection process. The input of the (j+1)th level iterative backprojection process includes the j-th level downsampling output, where j = 1, 2, ..., N-2. At least one level of iterative backprojection process is executed multiple times consecutively, and the input of the subsequent iterative backprojection process includes the output of the previous iterative backprojection process. The first superposition input of the first superposition process in the subsequent iterative backprojection process includes the first superposition output of the first superposition process in the previous iterative backprojection process. The first superposition input of the first iterative backprojection process includes the momentum term of this level. The updated feature image of the first level includes the output of the last iterative backprojection process of the first level.

13. An image processing apparatus, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 9 or the training method according to any one of claims 10 to 11.

14. A non-transient computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 9 or the training method according to any one of claims 10 to 11.

Images