An image processing method, apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an image processing method and device, electronic equipment and storage medium, relating to the field of image processing, the method of the present disclosure mainly comprises: acquiring an initial image to be processed; performing inference training on the initial image to obtain a stop image reconstruction condition corresponding to the initial image; performing image reconstruction processing on the initial image based on a neural network model, and when an intermediate layer reconstruction image of the neural network model meets the stop image reconstruction condition, stopping the image reconstruction processing on the initial image to obtain a target reconstruction image corresponding to the initial image. The present method predicts the target quality standard of image reconstruction in advance, and monitors the real-time intermediate layer output of the reconstruction model, and when the target quality standard is met, the reconstruction is ended and the final output is obtained. Compared with the previous complete process of image reconstruction, the present method reduces the consumption of computing resources of the model and improves the processing speed of the model while ensuring the quality.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of image processing, deep learning has been widely applied to image reconstruction and other image processing tasks, demonstrating excellent performance in terms of accuracy. However, due to the typically complex structure of existing image reconstruction models, their deployment in real-world scenarios inevitably faces challenges such as high computational resource consumption and long inference times. Therefore, how to better enable existing image processing models to perform image processing quickly and accurately is a problem that needs to be solved. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, electronic device, and storage medium to solve problems in related technologies and improve the training and computation speed of image reconstruction models.

[0004] A first aspect of this disclosure provides an image processing method, comprising: acquiring an initial image to be processed; performing inference training on the initial image to obtain a stopping image reconstruction condition corresponding to the initial image; performing image reconstruction processing on the initial image based on a neural network model; and stopping image reconstruction processing on the initial image when the intermediate layer reconstructed image of the neural network model satisfies the stopping image reconstruction condition, thereby obtaining a target reconstructed image corresponding to the initial image.

[0005] In some embodiments of this disclosure, the inference training of the initial image to obtain the stopping image reconstruction condition for image reconstruction of the initial image includes: inference training of the initial image based on the image quality control discriminant neural network model to obtain the stopping image reconstruction condition corresponding to the initial image.

[0006] In some embodiments of this disclosure, image reconstruction processing is performed on the initial image based on a neural network model. When the reconstructed image of the intermediate layer of the neural network model meets the stopping image reconstruction condition, the image reconstruction processing on the initial image is stopped to obtain the target reconstructed image corresponding to the initial image. This includes: reconstructing the initial image using an image reconstruction processing neural network model layer by layer to obtain the intermediate layer reconstructed image of the neural network; determining whether the intermediate layer reconstructed image of the neural network meets the stopping image reconstruction condition; if the intermediate layer reconstructed image of the neural network meets the stopping image reconstruction condition, stopping the image reconstruction processing of the initial image by other layers of the image reconstruction processing neural network model to obtain the target reconstructed image corresponding to the initial image; if the intermediate layer reconstructed image of the neural network does not meet the stopping image reconstruction condition, continuing the image reconstruction processing on the intermediate layer reconstructed image of the neural network until the intermediate layer reconstructed image of the neural network meets the stopping image reconstruction condition.

[0007] In some embodiments of this disclosure, the stopping image reconstruction condition is the number of image reconstructions. The step of determining whether the image reconstructed by the intermediate layer of the neural network satisfies the stopping image reconstruction condition includes: determining whether the number of times the initial image has been reconstructed has reached the number of image reconstructions; if the number of image reconstructions has been reached, then the image reconstructed by the intermediate layer of the neural network satisfies the stopping image reconstruction condition; if the number of image reconstructions has not been reached, then the image reconstructed by the intermediate layer of the neural network does not satisfy the stopping image reconstruction condition.

[0008] In some embodiments of this disclosure, the stopping image reconstruction condition is a target reconstructed image quality standard. Determining whether the reconstructed image from the intermediate layer of the neural network meets the stopping image reconstruction condition includes: comparing the quality of the reconstructed image from the intermediate layer of the neural network with the target reconstructed image quality standard; if the quality of the reconstructed image from the intermediate layer of the neural network meets the target reconstructed image quality standard, then the reconstructed image from the intermediate layer of the neural network is determined to meet the stopping image reconstruction condition; if the quality of the reconstructed image from the intermediate layer of the neural network does not meet the target reconstructed image quality standard, then the reconstructed image from the intermediate layer of the neural network is determined not to meet the stopping image reconstruction condition.

[0009] In some embodiments of this disclosure, the method further includes: constructing the image quality control discrimination neural network model based on the image reconstruction processing neural network model.

[0010] A second aspect of this disclosure provides an image processing apparatus, comprising: an acquisition unit for acquiring an initial image to be processed; a processing unit for performing inference training on the initial image to obtain a stopping image reconstruction condition corresponding to the initial image; and a judgment unit for performing image reconstruction processing on the initial image based on a neural network model, wherein when the intermediate layer reconstructed image of the neural network model satisfies the stopping image reconstruction condition, the image reconstruction processing on the initial image is stopped, thereby obtaining a target reconstructed image corresponding to the initial image.

[0011] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0012] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0013] A fifth aspect of this disclosure provides a computer program product including a computer program that is executed by a processor using the methods described in the first aspect of this disclosure.

[0014] The image processing method, apparatus, electronic device, and storage medium disclosed herein include: acquiring an initial image to be processed; performing inference training on the initial image to obtain a stopping image reconstruction condition corresponding to the initial image; performing image reconstruction processing on the initial image based on a neural network model; and stopping image reconstruction processing on the initial image when the intermediate layer reconstructed image of the neural network model meets the stopping image reconstruction condition, thereby obtaining a target reconstructed image corresponding to the initial image. This method predicts the target quality standard for image reconstruction in advance during image reconstruction processing and monitors the real-time intermediate layer output of the model during processing. When the real-time intermediate layer output meets the target quality standard, the reconstruction processing ends, and this output is used as the final image output. Compared with previous image reconstruction processes that performed a complete workflow, this method reduces the computational resource consumption of the image processing model and improves the processing speed of the image processing model while ensuring quality.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0017] Figure 1 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0018] Figure 2 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0019] Figure 3 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0020] Figure 4 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0021] Figure 5 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0022] Figure 6 A schematic flowchart of an image reconstruction method provided in an embodiment of this disclosure;

[0023] Figure 7 This is a schematic diagram of the structure of an image discrimination model provided in an embodiment of the present disclosure;

[0024] Figure 8 This is a schematic diagram of the structure of an image reconstruction neural network model provided in an embodiment of the present disclosure;

[0025] Figure 9 This is a block diagram of an image processing apparatus provided in an embodiment of the present disclosure;

[0026] Figure 10 This is a block diagram of an image processing apparatus provided in an embodiment of the present disclosure;

[0027] Figure 11 This is a block diagram of an image processing apparatus provided in an embodiment of the present disclosure;

[0028] Figure 12 A block diagram of an image processing apparatus provided in the disclosed embodiments;

[0029] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0031] Images are inevitably affected by noise during their formation and transmission. Noise in images is often intertwined with signals, blurring details such as boundaries and lines. Meanwhile, due to hardware limitations of devices or sensors, the images we obtain are often low-resolution, while high-resolution images are frequently desired in numerous electronic imaging applications. Therefore, research on image reconstruction denoising and image super-resolution is of great significance. In the field of image processing, deep learning has been widely applied to image reconstruction and other image processing tasks, demonstrating excellent performance in terms of accuracy. However, existing image reconstruction models are typically complex in structure, inevitably facing problems of high computational resource consumption and long inference times when deployed in real-world scenarios. Therefore, how to better enable existing image processing models to perform image processing quickly and accurately is a problem that needs to be solved.

[0032] To address the problems existing in related technologies, this method predicts the target quality standard for image reconstruction in advance and monitors the real-time intermediate layer output of the image reconstruction model during processing. When the real-time intermediate layer output meets the target quality standard, the reconstruction process ends, and this output is used as the final image output. Compared with previous image reconstruction processes that perform a complete workflow, this method reduces the computational resource consumption of the image processing model and improves the processing speed while ensuring quality.

[0033] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this disclosure. Figure 1 As shown, steps 101-103 are included.

[0034] Step 101: Obtain the initial image to be processed.

[0035] In this embodiment of the present disclosure, it should be noted that the initial image to be processed is the image data of the image reconstruction task to be processed. It may be low-noise image data, low-resolution image data, or other image data that needs to be reconstructed. The specific image data type is not limited in this embodiment of the present disclosure.

[0036] Image reconstruction primarily involves processing the initial image using image super-resolution (LR) and image denoising (HR) techniques to obtain the target image. Image super-resolution refers to the technique of using image processing methods and computers to recover a high-resolution (HR) image from a low-resolution (LR) image. A high-resolution image means that the image has a high pixel density, which can provide more image details. Image denoising refers to using image processing methods and computers to reduce the noise of a noisy image to obtain a low-noise image.

[0037] Step 102: Perform inference training on the initial image to obtain the stopping image reconstruction conditions corresponding to the initial image.

[0038] It should be noted in the embodiments of this disclosure that the inference training can be performed by training the input image data through a neural network model to obtain the image reconstruction conditions corresponding to the image data. The specific processing model or the method of obtaining the image reconstruction conditions is not limited in the embodiments of this disclosure.

[0039] The conditions for stopping image reconstruction, that is, the quality standards for image reconstruction corresponding to the image data, may include the number of image reconstructions, or may include commonly used evaluation indicators for image reconstruction as quality standards, such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), or other commonly used evaluation indicators for image reconstruction. This disclosure does not limit the specific conditions for stopping image reconstruction.

[0040] Step 103: Perform image reconstruction processing on the initial image based on the neural network model. When the reconstructed image of the intermediate layer of the neural network model meets the stop image reconstruction condition, stop the image reconstruction processing on the initial image and obtain the target reconstructed image corresponding to the initial image.

[0041] In this embodiment of the present disclosure, it should be noted that the neural network model refers to a trained image reconstruction model that performs image processing on the initial image. This model can be a convolutional neural network, a multilayer perceptron, a graph neural network, other neural network models, or an ensemble of one or more neural network models. The specific model type and network structure are not limited in this embodiment. The intermediate layer of the neural network model refers to the model layer after the input layer, i.e., the hidden layer of the neural network model. This layer can be a convolutional layer or a layer of other neural networks. The specific type of layer is not limited in this embodiment.

[0042] In summary, the image processing method provided in this disclosure includes: acquiring an initial image to be processed; performing inference training on the initial image to obtain a stopping image reconstruction condition corresponding to the initial image; performing image reconstruction processing on the initial image based on a neural network model; and stopping image reconstruction processing on the initial image when the intermediate layer reconstructed image of the neural network model meets the stopping image reconstruction condition, thereby obtaining a target reconstructed image corresponding to the initial image. This method predicts the target quality standard for image reconstruction in advance during image reconstruction processing and monitors the real-time intermediate layer output of the model during processing. When the real-time intermediate layer output meets the target quality standard, the reconstruction processing ends and the output is used as the final image output. Compared with previous image reconstruction processes that performed a complete workflow, this method reduces the computational resource consumption of the image processing model and improves the processing speed of the image processing model while ensuring quality.

[0043] Furthermore, in some embodiments of this disclosure, when performing inference training on the initial image in step 102 to obtain the stopping image reconstruction condition for image reconstruction of the initial image, this can be achieved by, but is not limited to, the following methods, specifically as follows: Figure 2 As shown, it includes the following steps:

[0044] Step 201: Perform inference training on the initial image based on the image quality control discriminative neural network model to obtain the stopping image reconstruction conditions corresponding to the initial image.

[0045] It should be noted here that the image quality control discrimination neural network model used to obtain the predicted stopping image reconstruction conditions can be a convolutional neural network, a deep neural network with multiple convolutional neural layers, multiple perceptron layers and pooling layers, or other types of neural network models. The specific model type and model structure are not limited in the embodiments of this disclosure.

[0046] In one possible implementation, the image quality control discriminative neural network model that judges the input information mainly includes deep neural networks such as convolutional neural networks and multilayer perceptrons. Its final output is the output quality of the intermediate layer used for image reconstruction, determining whether it meets the target quality or the number of inferences performed. Figure 7 The diagram shown illustrates the structure of the image discrimination model in this method. The discrimination neural network M_J in this embodiment is a deep neural network model composed of multiple neural network modules. The image quality control discrimination neural network model, which judges the input information, mainly includes deep neural network modules such as convolutional neural networks, pooling layers, and multilayer perceptrons. Its final output determines whether the output of the image reconstruction processing neural network model meets the output quality requirements or the number of inference iterations of the image reconstruction processing neural network model.

[0047] Convolutional Neural Networks (CNNs) are deep neural networks with convolutional structures. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between a trainable filter and an input image or a convolutional feature map. A CNN can be composed of multiple convolutional neurons, where a convolutional neuron can be a neuron that... s An operation unit with an intercept of 1 as input can be defined as follows:

[0048]

[0049] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For X S The weights are denoted by b, where b is the bias term of the neuron. f is the activation function of the neuron, used to introduce nonlinear characteristics into the convolutional neural network to convert the input signal in the neuron into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be sigmoid, ReLU, or tanh. A neural network is a network formed by connecting many of the above-mentioned individual neurons together; that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neurons.

[0050] A multilayer perceptron is a neural network consisting of fully connected layers, with at least one hidden layer, where the output of each hidden layer is transformed by an activation function. The specific form of this operational unit can be defined as:

[0051]

[0052] Where φ represents the activation function, h represents the hidden variables, and O represents the output. In deep neural networks, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Therefore, a multilayer perceptron can have performance capabilities exceeding those of a single-layer perceptron.

[0053] In one possible implementation, the input x of the image quality control discriminative neural network model that judges the input information is an image. An image is a matrix with channel dimensions, primarily containing pixel values ​​and image features. The pixel value of the image can be a red-green-blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value of 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. Within each color component, the smaller the value, the lower the brightness; the larger the value, the higher the brightness. For grayscale images, the pixel value is simply the grayscale value, not including multiple channels such as RGB.

[0054] Furthermore, in some embodiments of this disclosure, when performing step 103, image reconstruction processing of the initial image based on a neural network model, and stopping image reconstruction processing of the initial image when the intermediate layer reconstructed image of the neural network model meets the stopping image reconstruction condition, to obtain the target reconstructed image corresponding to the initial image, this can be achieved by, but is not limited to, the following methods, specifically as follows: Figure 3 As shown, it includes the following steps:

[0055] Step 301: Using the layers of the neural network model as units, the initial image is reconstructed using the image reconstruction processing neural network model to obtain the reconstructed image of the intermediate layer of the neural network.

[0056] In this embodiment of the present disclosure, it should be noted that the term "layer" refers to the basic unit of a layer in the construction of a neural network, but it can also refer to other basic units of a neural network, and this disclosure does not limit this. Furthermore, the intermediate layer can be the first layer or any of the intermediate layers. When obtaining the output of the intermediate layers of the neural network, the output features of the intermediate layers can be extracted and judged layer by layer, or the extraction and judgment of the output features of the intermediate layers can be performed by skipping layers. The specific method of obtaining the intermediate layers is not limited in the embodiments of this disclosure.

[0057] Regarding the image reconstruction processing neural network model, in one possible implementation, the image reconstruction processing neural network model is a deep neural network with multiple convolutional neural layers, multiple perceptron layers, and pooling layers, whose main purpose is to reconstruct the input image x. For example... Figure 8 The diagram shown is a schematic of the image reconstruction neural network model of this method. The image reconstruction processing neural network model is a deep neural network with multiple convolutional neural layers, multi-layer perceptron layers, and pooling layers that outputs an image. Its main purpose is to reconstruct the input image x.

[0058] Image reconstruction mainly includes image super-resolution and image denoising. Image super-resolution refers to the technique of using image processing methods to recover a high-resolution (HR) image from a low-resolution (LR) image using a computer. A high-resolution image means that the image has a high pixel density, which can provide more image details. Image denoising refers to using image processing methods to reduce the noise of a noisy image using a computer, so as to obtain a low-noise image.

[0059] Specifically, deep learning-based image super-resolution methods utilize deep neural networks to reconstruct images from lower bounds (LR) to higher bounds (HR). These reconstruction-based methods can statistically estimate the maximum posterior probability of a high-resolution image using frequency domain or spatial domain algorithms. Learning-based image super-resolution methods can include two phases: a training phase and a testing phase.

[0060] In image reconstruction for super-resolution tasks, the initial super-resolution model and training set are first established during the training phase. The training set can include multiple low-resolution images and their corresponding high-resolution images. By using the low-resolution images and their corresponding high-resolution images in the training set, the correspondence between the high-resolution and low-resolution images is learned. This allows for the correction of parameter values ​​in the initial super-resolution model, leading to convergence of errors between the high-resolution and reconstructed images, ultimately determining the newly trained super-resolution model. During the testing phase, the newly trained super-resolution model can guide image super-resolution reconstruction.

[0061] One method for obtaining image pairs (a low-resolution image and its corresponding high-resolution image forming an image pair) in the training set is to first obtain a high-resolution image, and then process the high-resolution image through a blurring function to obtain the corresponding low-resolution image, wherein the blurring function is irreversible.

[0062] Specifically, deep learning-based image denoising methods utilize deep neural networks to reconstruct images from high-noise to low-noise images. Reconstruction-based image denoising methods can statistically estimate the low-noise image with the maximum posterior probability using frequency domain or spatial domain algorithms. Learning-based image denoising methods can include two phases: a training phase and a testing phase.

[0063] In image reconstruction after denoising, the initial denoising model and training set are first established during the training phase. The training set can include multiple low-noise images and their corresponding high-noise images. By using the low-noise images and their corresponding high-noise images in the training set, the correspondence between the high-noise and low-noise images is learned. This allows for the correction of the parameter values ​​in the initial hypernoise model, leading to convergence of the error between the high-noise image and the reconstructed image. Finally, the newly trained hypernoise model is determined. During the testing phase, the newly trained hypernoise model can guide the hypernoise reconstruction of the image.

[0064] One method for obtaining image pairs (a low-noise image and its corresponding high-noise image forming an image pair) in the training set is to first obtain a low-noise image, and then to obtain the corresponding high-noise image by adding noise to the low-noise image, wherein the noise addition process is irreversible.

[0065] Furthermore, it should be noted that in one possible implementation, the training of the image quality control discriminative neural network model that judges the input information depends on the image reconstruction processing neural network model:

[0066] First, given the initial input image, the intermediate state variable h is obtained by processing the i-th block of the image reconstruction neural network model. i Compare its PSNR and SSIM with the original image x;

[0067] Secondly, given a clean image I of size m×n and a noisy image K, the peak signal-to-noise ratio (PSNR) is defined as follows:

[0068]

[0069] in Let MSE be the maximum possible pixel value of the image, and MSE be the mean square error between the two images, specifically defined as:

[0070]

[0071] SSIM (Structural Similarity) is used to compare the structural similarity between two images. It includes three comparison measures: luminance, contrast, and structure, and is specifically defined as follows:

[0072]

[0073] Where μ x Let μ be the mean of x. y Let y be the mean. Let x be the variance. Let σ be the variance of y. x σ y Let c1 and c2 be the covariances of x and y, respectively, and both c1 and c2 be constants to avoid computational limitations. SSIM is then:

[0074] SSIM(x, y) = [l(x, y)] α ·c(x, y) β ·s(x, y) γ ] (Formula 6)

[0075] Setting α, β, and γ to 1 yields a simplified formula for SSIM:

[0076]

[0077] Finally, by calculating the intermediate state variable h i We obtain training data by using the difference between the PSNR and SSIM of the original image x. We define the difference as g. i h of each intermediate state variable i The difference between x and the index has a corresponding g.i Defined by accumulating g at each layer. i To obtain training data for the discriminative neural network.

[0078] The image quality control discriminative neural network model, which evaluates the input information, obtains the PSNR and SSIM of the original image x and the target image quality. During inference, it uses the demand difference g. r If it is determined that the i-th layer is required to achieve the target image quality, then the image reconstruction processing neural network model outputs the image when it infers to the i-th block.

[0079] Step 302: Determine whether the reconstructed image from the intermediate layer of the neural network meets the stopping image reconstruction condition. If the reconstructed image from the intermediate layer of the neural network meets the stopping image reconstruction condition, proceed to step 303; if the reconstructed image from the intermediate layer of the neural network does not meet the stopping image reconstruction condition, proceed to step 304.

[0080] It should be noted that, in the embodiments of this disclosure, the judgment can be performed using the image quality control discrimination neural network model constructed in this disclosure, or it can be performed using other external judgment methods, such as external judgment code programs. The specific place where the judgment is executed is not limited in the embodiments of this disclosure.

[0081] Step 303: Stop the other layers of the image reconstruction processing neural network model from performing image reconstruction processing on the initial image to obtain the target reconstructed image corresponding to the initial image.

[0082] The embodiments disclosed herein have been described above and will not be repeated here. The embodiments disclosed herein do not limit the specific stopping method.

[0083] Step 304: Continue image reconstruction processing on the reconstructed image of the intermediate layer of the neural network until the reconstructed image of the intermediate layer of the neural network meets the stop image reconstruction condition.

[0084] The embodiments disclosed herein have been described above and will not be repeated here. The specific implementation methods are not limited in the embodiments disclosed herein.

[0085] Furthermore, in some embodiments of this disclosure, when performing step 302 to determine whether the image reconstructed by the intermediate layer of the neural network satisfies the stopping image reconstruction condition, if the stopping image reconstruction condition is the number of image reconstructions, the determination of whether the image reconstructed by the intermediate layer of the neural network satisfies the stopping image reconstruction condition can be implemented by, but is not limited to, the following methods, specifically as follows: Figure 4 As shown, it includes the following steps:

[0086] Step 401: Determine whether the number of reconstruction processes performed on the initial image has reached the image reconstruction count. If the image reconstruction count has been reached, proceed to step 402; otherwise, proceed to step 403.

[0087] It should be noted here that the number of times the image is reconstructed, that is, the number of times required to achieve the target image reconstruction quality, can be obtained by the image quality control discrimination neural network model of this disclosure, or by other methods. The specific method of acquisition is not limited in the embodiments of this disclosure.

[0088] Step 402: Determine that the reconstructed image from the intermediate layer of the neural network satisfies the stopping image reconstruction condition.

[0089] It should be noted that, in the embodiments of this disclosure, the determination of whether the image reconstruction conditions are met can be made by the image quality control discrimination neural network model of this disclosure or by external execution code. The specific judgment method and judgment location are not limited in the embodiments of this disclosure.

[0090] Step 403: Determine that the image reconstructed by the intermediate layer of the neural network does not meet the stop image reconstruction condition.

[0091] It should be noted that, in the embodiments of this disclosure, the determination of whether the image reconstruction conditions are met can be made by the image quality control discrimination neural network model of this disclosure or by external execution code. The specific judgment method and judgment location are not limited in the embodiments of this disclosure.

[0092] Furthermore, in some embodiments of this disclosure, when performing step 302 to determine whether the reconstructed image of the intermediate layer of the neural network meets the stopping image reconstruction condition, if the stopping image reconstruction condition is the target reconstructed image quality standard, the determination of whether the reconstructed image of the intermediate layer of the neural network meets the stopping image reconstruction condition can be implemented by, but is not limited to, the following methods, specifically as follows: Figure 5 As shown, it includes the following steps:

[0093] Step 501: Compare the quality of the image reconstructed by the intermediate layer of the neural network with the quality standard of the target reconstructed image. If the quality of the image reconstructed by the intermediate layer of the neural network meets the quality standard of the target reconstructed image, proceed to step 502; if the quality of the image reconstructed by the intermediate layer of the neural network does not meet the quality standard of the target reconstructed image, proceed to step 503.

[0094] It should be noted here that the target reconstructed image quality standard can be the indicator standard of the image super-resolution task in the image reconstruction task, or the embodiments of this disclosure do not limit it.

[0095] Step 502: Determine that the reconstructed image from the intermediate layer of the neural network satisfies the stopping image reconstruction condition.

[0096] It should be noted that, in the embodiments of this disclosure, the determination of whether the image reconstruction conditions are met can be made by the image quality control discrimination neural network model of this disclosure or by external execution code. The specific judgment method and judgment location are not limited in the embodiments of this disclosure.

[0097] Step 503: Determine that the image reconstructed by the intermediate layer of the neural network does not meet the stop image reconstruction condition.

[0098] It should be noted that, in the embodiments of this disclosure, the determination of whether the image reconstruction conditions are met can be made by the image quality control discrimination neural network model of this disclosure or by external execution code. The specific judgment method and judgment location are not limited in the embodiments of this disclosure.

[0099] Based on the above method, when performing image reconstruction processing according to the embodiments of the present invention, it is necessary to train and construct an image reconstruction processing neural network model and an image quality control discrimination neural network model. Specifically, the image quality control discrimination neural network model is constructed based on the image reconstruction processing neural network model.

[0100] The image reconstruction processing neural network model of this invention is used to reconstruct images. This model is trained using currently popular image reconstruction processing data. The specific type and structure of the model, and the specific image reconstruction processing data, are not limited in this disclosure. The image quality control discrimination neural network model of this invention is used to obtain the quality standards of the reconstructed image. This model can be trained using, but is not limited to, image quality standard data constructed from a trained image reconstruction processing model. The specific type and structure of the model, and the specific model training data, are not limited in this disclosure.

[0101] In summary, this method predicts the target quality standard for image reconstruction in advance and monitors the real-time intermediate layer output of the image reconstruction model during processing. When the real-time intermediate layer output meets the target quality standard, the reconstruction process ends, and this output is used as the final image output. Compared with previous image reconstruction methods that performed a complete workflow, this method reduces the computational resource consumption of the image processing model and improves the processing speed while maintaining quality.

[0102] Based on the above method, the intermediate layer output of the image reconstruction processing model can be judged according to the predicted template quality conditions, thereby terminating the process early to save computational resources and time. Figure 6 The diagram shows a flowchart of an image reconstruction method based on neural network discriminative adaptive inference technology. The main steps include:

[0103] S1. An image quality control discriminant neural network model for judging input information and a neural network model for image reconstruction processing were constructed.

[0104] S2. Use the image quality control discriminant neural network model to reason about the existing input x, and obtain the stopping reasoning condition (i.e. the stopping image reconstruction condition) required by the image reconstruction processing neural network model.

[0105] S3. Use an image reconstruction processing neural network model to infer the input x, starting from the first block, to obtain the intermediate state variable h. 1 .

[0106] S4. Use an image quality control discriminant neural network model to determine the existing intermediate state variable h. 1 If the image reconstruction processing neural network model does not meet the set quality standards, then inference is performed on the next block to obtain the next intermediate state variable h. 2 .

[0107] S5, in h i If the set quality standard is not met, continue inference using the next block of the image reconstruction processing neural network model to obtain h. (i+1) Using an image quality control discriminant neural network model to determine the existing intermediate state variable h (i+1) Check if it meets the set quality standards; if not, cycle to S5.

[0108] S6, if h i If the set quality standards are met, then output is performed.

[0109] In summary, this scheme monitors the real-time intermediate layer output during image reconstruction processing based on pre-predicted image reconstruction quality conditions. When the intermediate layer meets the output requirements, it is used as the final output, and image reconstruction is stopped. Because the image reconstruction process is stopped based on predicted image quality standards, it avoids the computational overhead of completing the entire process in the past and also reduces the time required for image reconstruction tasks.

[0110] Figure 9 This is a schematic diagram of the structure of an image processing apparatus 900 provided in an embodiment of this disclosure. Figure 9As shown, the image processing apparatus includes:

[0111] Acquisition unit 901 acquires the initial image to be processed;

[0112] Processing unit 902 performs inference training on the initial image to obtain the stopping image reconstruction conditions corresponding to the initial image;

[0113] The judgment unit 903 performs image reconstruction processing on the initial image based on the neural network model. When the reconstructed image of the intermediate layer of the neural network model meets the stop image reconstruction condition, the image reconstruction processing on the initial image is stopped, and the target reconstructed image corresponding to the initial image is obtained.

[0114] In some embodiments of this disclosure, such as Figure 10 As shown, the processing unit 902 includes:

[0115] The inference module 9021 performs inference training on the initial image based on the image quality control discrimination neural network model to obtain the stopping image reconstruction conditions corresponding to the initial image.

[0116] In some embodiments of this disclosure, such as Figure 11 As shown, the judgment unit 903 includes:

[0117] The first processing module 9031 reconstructs the initial image using the image reconstruction processing neural network model, taking the layers of the neural network model as units, to obtain the reconstructed image of the intermediate layer of the neural network.

[0118] The judgment module 9032 determines whether the image reconstructed by the intermediate layer of the neural network meets the stopping image reconstruction condition;

[0119] The second processing module 9033, if the image reconstructed by the intermediate layer of the neural network satisfies the image reconstruction stop condition, stops the other layers of the image reconstruction processing neural network model from performing image reconstruction processing on the initial image, and obtains the target reconstructed image corresponding to the initial image;

[0120] The third processing module 9034, if the reconstructed image of the intermediate layer of the neural network does not meet the stopping image reconstruction condition, continues to perform image reconstruction processing on the reconstructed image of the intermediate layer of the neural network until the reconstructed image of the intermediate layer of the neural network meets the stopping image reconstruction condition.

[0121] In some embodiments of this disclosure, when the stopping image reconstruction condition is the number of image reconstructions, the second processing module 9033 is further configured to:

[0122] Determine whether the number of reconstruction processes performed on the initial image has reached the required number of image reconstructions;

[0123] If the number of image reconstruction attempts has been reached, then the image reconstructed by the intermediate layer of the neural network is determined to meet the stopping image reconstruction condition.

[0124] If the number of image reconstruction attempts is not reached, it is determined that the image reconstruction by the intermediate layer of the neural network does not meet the condition for stopping image reconstruction.

[0125] In some embodiments of this disclosure, when the stopping image reconstruction condition is the target reconstructed image quality standard, the second processing module 9033 is further configured to:

[0126] The quality of the image reconstructed by the intermediate layer of the neural network is compared with the quality standard of the target reconstructed image;

[0127] If the quality of the image reconstructed by the intermediate layer of the neural network meets the quality standard of the target reconstructed image, then the image reconstructed by the intermediate layer of the neural network is determined to meet the stopping image reconstruction condition.

[0128] If the quality of the image reconstructed by the intermediate layer of the neural network does not meet the quality standard of the target reconstructed image, then it is determined that the image reconstructed by the intermediate layer of the neural network does not meet the condition for stopping image reconstruction.

[0129] In some embodiments of this disclosure, such as Figure 12 As shown, the image processing apparatus also includes:

[0130] Training unit 904 constructs the image quality control discrimination neural network model based on the image reconstruction processing neural network model.

[0131] In embodiments of this disclosure, when correcting an image, an initial image to be processed is first acquired; the initial image is then trained through inference to obtain the stopping image reconstruction condition corresponding to the initial image; image reconstruction processing is performed on the initial image based on a neural network model; when the intermediate layer reconstructed image of the neural network model meets the stopping image reconstruction condition, image reconstruction processing on the initial image is stopped, resulting in the target reconstructed image corresponding to the initial image. This method predicts the target quality standard for image reconstruction in advance during image reconstruction processing and monitors the real-time intermediate layer output of the model during processing. When the real-time intermediate layer output meets the target quality standard, the reconstruction processing ends, and this output is used as the final image output. Compared with previous image reconstruction processes that performed a complete workflow, this method reduces the computational resource consumption of the image processing model and improves the processing speed of the image processing model while ensuring quality.

[0132] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include hardware structures and software modules, and may implement the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. One of the above functions may be executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules.

[0133] Figure 13 This is a block diagram illustrating an electronic device 1300 for implementing the above-described image processing method according to an exemplary embodiment. For example, the electronic device 1300 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0134] Reference Figure 13 The electronic device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power supply component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316.

[0135] Processing component 1302 typically controls the overall operation of electronic device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0136] Memory 1304 is configured to store various types of data to support the operation of electronic device 1300. Examples of such data include instructions for any application or method operating on electronic device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0137] Power supply component 1306 provides power to various components of electronic device 1300. Power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1300.

[0138] Multimedia component 1308 includes a screen that provides an output interface between electronic device 1300 and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When electronic device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0139] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when electronic device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0140] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0141] Sensor assembly 1314 includes one or more sensors for providing state assessments of various aspects of electronic device 1300. For example, sensor assembly 1314 may detect the on / off state of electronic device 1300, the relative positioning of components such as the display and keypad of electronic device 1300, changes in position of electronic device 1300 or a component of electronic device 1300, the presence or absence of user contact with electronic device 1300, the orientation or acceleration / deceleration of electronic device 1300, and temperature changes of electronic device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0142] Communication component 1316 is configured to facilitate wired or wireless communication between electronic device 1300 and other devices. Electronic device 1300 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0143] In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0144] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 320 of an electronic device 1300 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0145] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the image processing methods described in the above embodiments of this disclosure.

[0146] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the image processing method described in the above embodiments of this disclosure.

[0147] Embodiments of this disclosure also propose a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the image processing method described in the above embodiments of this disclosure.

[0148] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0149] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0150] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0152] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0153] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0154] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0155] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that, include: Obtain the initial image to be processed; The initial image is subjected to inference training to obtain the stopping image reconstruction condition corresponding to the initial image. The inference training is used to predict the quality standard that the initial image needs to achieve in the image reconstruction process. The quality standard includes at least one of the number of image reconstructions and the target reconstructed image quality standard. The initial image is reconstructed based on a neural network model. When the reconstructed image of the intermediate layer of the neural network model meets the stop image reconstruction condition, the image reconstruction process on the initial image is stopped, and the target reconstructed image corresponding to the initial image is obtained.

2. The method according to claim 1, characterized in that, The step of performing inference training on the initial image to obtain the stopping image reconstruction condition for image reconstruction of the initial image includes: The initial image is trained by inference based on the image quality control discriminant neural network model to obtain the stopping image reconstruction conditions corresponding to the initial image.

3. The method according to claim 2, characterized in that, The image reconstruction process based on the neural network model on the initial image is stopped when the reconstructed image from the intermediate layer of the neural network model meets the stopping image reconstruction condition. The resulting target reconstructed image corresponding to the initial image includes: Using the layers of the neural network model as units, the initial image is reconstructed using the image reconstruction processing neural network model to obtain the reconstructed image of the intermediate layer of the neural network; Determine whether the image reconstructed by the intermediate layer of the neural network meets the stopping image reconstruction condition; If the image reconstructed by the intermediate layer of the neural network satisfies the image reconstruction stop condition, the other layers of the neural network model stop reconstructing the image of the initial image to obtain the target reconstructed image corresponding to the initial image. If the reconstructed image of the intermediate layer of the neural network does not meet the stopping image reconstruction condition, the image reconstruction process continues on the reconstructed image of the intermediate layer of the neural network until the reconstructed image of the intermediate layer of the neural network meets the stopping image reconstruction condition.

4. The method according to claim 3, characterized in that, The stopping image reconstruction condition is the number of image reconstructions. Determining whether the image reconstructed by the intermediate layer of the neural network satisfies the stopping image reconstruction condition includes: Determine whether the number of reconstruction processes performed on the initial image has reached the required number of image reconstructions; If the number of image reconstruction attempts has been reached, then the image reconstructed by the intermediate layer of the neural network is determined to meet the stopping image reconstruction condition. If the number of image reconstruction attempts is not reached, it is determined that the image reconstruction by the intermediate layer of the neural network does not meet the condition for stopping image reconstruction.

5. The method according to claim 3, characterized in that, The stopping image reconstruction condition is the target reconstructed image quality standard. Determining whether the reconstructed image from the intermediate layer of the neural network meets the stopping image reconstruction condition includes: The quality of the image reconstructed by the intermediate layer of the neural network is compared with the quality standard of the target reconstructed image; If the quality of the image reconstructed by the intermediate layer of the neural network meets the quality standard of the target reconstructed image, then the image reconstructed by the intermediate layer of the neural network is determined to meet the stopping image reconstruction condition. If the quality of the image reconstructed by the intermediate layer of the neural network does not meet the quality standard of the target reconstructed image, then it is determined that the image reconstructed by the intermediate layer of the neural network does not meet the condition for stopping image reconstruction.

6. The method according to any one of claims 1-5, characterized in that, Also includes: Based on the neural network model for image reconstruction and processing, an image quality control discrimination neural network model is constructed.

7. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the initial image to be processed; The processing unit is configured to perform inference training on the initial image to obtain the stopping image reconstruction conditions corresponding to the initial image. The inference training is used to predict the quality standard that the initial image needs to achieve in the image reconstruction process. The quality standard includes at least one of the number of image reconstructions and the target reconstructed image quality standard. The judgment unit is used to perform image reconstruction processing on the initial image based on the neural network model. When the reconstructed image of the intermediate layer of the neural network model meets the stop image reconstruction condition, the image reconstruction processing on the initial image is stopped, and the target reconstructed image corresponding to the initial image is obtained.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images