Image recovery method, device, storage medium and program product

Abstract

The embodiment of the present invention provides an image restoration method, device, storage medium and program product, which obtains image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into a pre-trained image restoration model, and the target variable is iteratively updated in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate and output the final restored image data. The image restoration model of the present invention is an interpretable network model based on the deep expansion network structure of the three-operator splitting algorithm, which combines the advantages of both the traditional model optimization image restoration method and the heuristic deep learning image restoration method, and can improve the image restoration quality and robustness.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing, and in particular to an image recovery method, device, storage medium and program product. Background Art

[0002] In the field of image restoration, it specifically includes classic problems such as image denoising, image deblurring, image super-resolution reconstruction and image compressed sensing.

[0003] The existing technologies for solving problems in the field of image restoration can be divided into three categories: image restoration methods based on traditional model optimization, image restoration methods based on heuristic deep learning, and image restoration methods based on interpretable deep learning driven by optimization algorithm models. However, the image quality and robustness of the existing methods are not high. Summary of the invention

[0004] The embodiments of the present invention provide an image restoration method, a device, a storage medium and a program product, which are used to improve the image restoration quality and the robustness.

[0005] In a first aspect, an embodiment of the present invention provides an image restoration method, comprising:

[0006] Obtaining image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data;

[0007] The image data and the initialized target variable are input into a pre-trained image restoration model, and the target variable is iteratively updated in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain a final target variable;

[0008] The final target variable is input into the nonlinear mapping network layer of the image restoration model to generate final restored image data, and the final restored image data is output.

[0009] In a second aspect, an embodiment of the present invention provides an image restoration device, comprising:

[0010] An acquisition module, used for acquiring image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data;

[0011] A three-operator splitting algorithm network module is used to input the image data and the initialized target variable into a pre-trained image restoration model, and iteratively update the target variable in turn through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain a final target variable;

[0012] The nonlinear mapping network module is used to input the final target variable into the nonlinear mapping network layer of the image restoration model, generate the final restored image data, and output the final restored image data.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the method described in the first aspect.

[0018] The image restoration method, device, storage medium and program product provided by the embodiment of the present invention obtain the image data to be restored and the initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into a pre-trained image restoration model, and the target variable is iteratively updated in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate the final restored image data, and the final restored image data is output. The image restoration model provided by the embodiment of the present invention is an interpretable network model based on the deep unfolding network structure of the three-operator splitting algorithm. The structure combines the advantages of both the traditional model optimization image restoration method and the heuristic deep learning image restoration method, and can improve the image restoration quality and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 A schematic diagram of an application scenario of an image restoration method provided by an embodiment of the present invention;

[0021] Figure 2A flow chart of an image recovery method provided by an embodiment of the present invention;

[0022] Figure 3 A network structure diagram of a TOS-Net model provided by an embodiment of the present invention;

[0023] Figure 4 A flow chart of an image recovery method provided by another embodiment of the present invention;

[0024] Figure 5 A flow chart of an image recovery method provided by another embodiment of the present invention;

[0025] Figure 6 A schematic diagram of a visualization of a norm provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the TOS-IDBlock network structure and data flow process provided by an embodiment of the present invention;

[0027] Figure 8 A deep denoising network model H provided by an embodiment of the present invention dip Network structure diagram;

[0028] Fig. 9 A schematic diagram of the TOS-IDBlock network structure and data flow process provided by an embodiment of the present invention;

[0029] Fig.10 A schematic diagram of the structure of an image recovery device provided by an embodiment of the present invention

[0030] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0031] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] In the field of image restoration, it specifically includes classic problems such as image denoising, image deblurring, image super-resolution reconstruction and image compressed sensing.

[0034] The existing technologies for solving problems in the field of image restoration can be mainly divided into three categories: image restoration methods based on traditional model optimization, image restoration methods based on heuristic deep learning, and image restoration methods based on interpretable deep learning driven by optimization algorithm models.

[0035] The image restoration method based on traditional model optimization mainly establishes the corresponding mathematical model based on the maximum a posteriori probability estimation (MAP) of Bayesian theorem, and designs complex image prior constraints or regularization terms based on professional knowledge of image restoration, and uses these image prior constraints and regularization terms to characterize the characteristics of image restoration. p A series of regularization terms such as the norm and the TV (total variation) norm are used to impose sparse constraints on the specific transformation domain of the image while retaining the edge information of the image. After the mathematical model of image restoration is established, the image restoration method based on the traditional optimization model will design different optimization methods according to the characteristics of the model to solve the optimal solution of convergence, that is, the high-quality image to be restored. Classic optimization algorithms such as the alternating direction method of multipliers (ADMM), the projected gradient algorithm (ISTA), and the operator splitting algorithm (TOS) are often used to optimize and solve image restoration models with constraints.

[0036] Image restoration methods based on heuristic deep learning are mainly based on experience to heuristically design deep neural network models. Through end-to-end training of a large number of labeled paired data sets, the designed deep neural network model is allowed to learn a mapping that can restore low-quality / damaged images to high-quality images. The main focus of this type of method is on the design of network structure and loss function. There are currently a series of classic network structures based on heuristic deep learning methods. Classic structures such as convolutional neural networks (CNN) and recurrent neural networks (RNN) and their deformations are used to extract image features of different degrees from different angles. Classic networks such as ResNet, UNet, and ResUNet are currently common image restoration neural network structures. Some differentiable image prior constraints and regularization terms will be added to the loss function of the neural network, which will constrain the output results of the deep neural network.

[0037] Image restoration methods based on model-driven deep learning can be mainly divided into methods based on neural network expansion and methods based on deep neural network embedding. The method based on neural network expansion introduces learnable potential image priors, designs the optimization iterative process of the mathematical model of image restoration into a graph calculation process of the neural network structure, and each layer of the neural network structure is equivalent to each iteration of the optimization process. The network is trained end-to-end to learn the mapping from low-quality / damaged images to high-quality images. The construction of its network structure is more interpretable, and the role of each layer is clearer. Another type of image restoration method based on model-driven deep learning is the method of deep neural network embedding. This method inserts a deep neural network pre-trained based on a specific task into the iterative optimization process of the image restoration method based on traditional model optimization, replacing the sub-steps of its iterative optimization process. In general, the image restoration method based on model-driven deep learning combines the advantages of image restoration methods based on traditional model optimization and image restoration methods based on heuristic deep learning, embeds the image prior designed based on professional domain knowledge into the forward propagation process of the neural network, so that the designed network structure has better interpretability and robustness, can dig out the hidden information of the image restoration problem more deeply, and can restore higher quality images driven by training data and domain knowledge.

[0038] However, the existing technologies for solving problems in the field of image restoration have the following disadvantages:

[0039] 1) The image restoration method based on traditional model optimization is to establish the corresponding mathematical model based on the maximum a posteriori probability estimation (MAP) of Bayesian theorem, and design complex image prior constraints or regularization terms based on professional knowledge of image restoration, and use these image prior constraints and regularization terms to characterize the characteristics of image restoration. The disadvantages of the image restoration method based on traditional model optimization are: constructing image prior regularization terms requires strong mathematical skills and an abstract understanding of the image restoration problem, and in the process of optimizing and solving the traditional model, it is necessary to solve some difficult problems such as the non-differentiability of regularization terms, and the optimization solution process of such methods is usually time-consuming.

[0040] 2) Image restoration methods based on heuristic deep learning are mainly based on experience to heuristically design deep neural network models. Through end-to-end training with a large number of labeled paired data sets, the designed deep neural network model can learn to restore low-quality / damaged images into high-quality image mapping. The disadvantages of image restoration methods based on heuristic deep learning are: they are heavily dependent on a large number of labeled training data sets. If the amount of training data is insufficient, the effect of the trained neural network model is very good. The neural network structure designed for a specific image restoration problem has poor generalization, and the effect of migration to other image restoration problems is usually not good. The design of the neural network structure is usually based on a heuristic black box network structure and lacks interpretability.

[0041] 3) The image restoration method based on model-driven deep learning introduces learnable latent image priors, designs the optimization iterative process of the mathematical model of image restoration into the forward reasoning calculation process of the neural network structure; or inserts the deep neural network pre-trained based on a specific task into the iterative optimization process of the image restoration method based on traditional model optimization, replacing the sub-steps of its iterative optimization process. Generally speaking, the more image priors or regularization terms a model contains, the richer the image prior information it contains, and the better the quality of the restored image. However, the disadvantage of the image restoration method based on model-driven deep learning is that the selected model usually only considers a single image prior or regularization term, and rarely considers an image restoration model that contains two image priors or regularization terms at the same time, resulting in the structure of the expanded deep neural network being unable to process two image prior information at the same time, so the restored image effect is usually suboptimal. In addition, this type of method uses a learnable latent image prior, ignoring the existing classic hand-designed image prior.

[0042] In view of the above technical problems, in order to improve the image restoration quality and robustness, an embodiment of the present invention provides an image restoration method, based on a pre-trained image restoration model, the image restoration model includes multiple three-operator splitting algorithm TOS network layers (TOS-Block) and a nonlinear mapping network layer, by obtaining the image data to be restored and the initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into the pre-trained image restoration model, and the target variable is iteratively updated in turn through the multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate the final restored image data, and the final restored image data is output. The image restoration model provided by the embodiment of the present invention is an interpretable network model based on the deep unfolding network structure of the three-operator splitting algorithm. The structure combines the advantages of both the traditional model optimization image restoration method and the heuristic deep learning image restoration method. The learning process of the structure is driven by domain knowledge and data. The image restoration model can improve the image restoration quality and robustness.

[0043] like Figure 1As shown, the image recovery method provided by the embodiment of the present invention is applicable to electronic devices 101 such as servers and terminal devices (the server is used as an example in the figure), wherein the terminal devices include but are not limited to mobile phones, tablet computers, laptops, desktop computers, personal digital assistants, smart wearable devices, smart home devices, etc., and the servers include but are not limited to independent physical servers, and can also be a server cluster or distributed system composed of multiple physical servers, and can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. In an embodiment of the present invention, the electronic device 101 can obtain the image data to be restored from other electronic devices 102 such as mobile phones, tablet computers, cameras, servers, etc., or can also obtain the image data to be restored from the storage unit of the electronic device 101. Based on the pre-trained image restoration model, the image data to be restored and the initialized target variable are obtained, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into the pre-trained image restoration model, and the target variable is iteratively updated in turn through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate the final restored image data, and the final restored image data is output. The image restoration model can be pre-trained on the electronic device 101, or it can be pre-trained on other devices and then deployed on the electronic device 101. The model training process can adopt conventional training methods, which are not limited here.

[0044] Some terms involved in the embodiments of the present invention are explained below:

[0045] Image prior information / regularization term / constraint term: Image prior information refers to the prior knowledge information obtained from people's experience with natural images. For example, images can usually be represented as a W*H*C matrix, and the value range of each pixel in 8 bits is 0-255. Regularization term: In order to control the parameters of the surface model overfitting. Constraint term: refers to the image must meet the local smoothness constraint, the image patch in the image must meet the self-similarity, etc.

[0046] Operator: Based on the mathematical concept of operator, the mapping of function space is corresponded to the operator function. The computer model established in the problem solving process realizes model deconstruction through problem decomposition and function encapsulation. The model unit formed in this way is called an operator.

[0047] Proximal Mapping Function: When the target function contains non-differentiable functions Then find a z through the proximal mapping function, z makes the function that can make it non-differentiable As small as possible and close to the original non-differentiable point x. The specific formula is as follows.

[0048]

[0049] Loss function: used to evaluate the degree to which the model's predicted value differs from the true value. The better the loss function, the better the model's performance. Loss functions can be divided into empirical risk loss functions and structural risk loss functions. The empirical risk loss function refers to the difference between the predicted structure and the true structure, and the structural risk loss function refers to the empirical risk loss function plus the regularization term.

[0050] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0051] Figure 2 This is a flow chart of an image restoration method provided by an embodiment of the present invention. This embodiment provides an image restoration method, the execution subject of which is an electronic device such as a server, a smart phone, a laptop computer, a desktop computer, a wearable device, etc. The image restoration method of this embodiment can be applied to image denoising, image deblurring, image super-resolution reconstruction or image compression sensing application scenarios, and a corresponding image restoration model can be constructed according to the corresponding model of the specific application scenario. The specific steps of the image restoration method are as follows:

[0052] S201, obtaining image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data.

[0053] In this embodiment, the image data y to be restored can be obtained by any means, which is not limited here; and the initialized target variable can be randomly generated, and the target variable is a Lagrange multiplier with the same dimension (pixel) as the image data, wherein the Lagrange multiplier is a variable parameter based on the Lagrange multiplier method. The basic Lagrange multiplier method is a method for finding the extreme value of the function f(x1, x2, ...) under the constraint condition of g(x1, x2, ...) = 0. The main idea is to introduce a new parameter (i.e., the Lagrange multiplier) to link the constraint function with the original function so that they can be matched to form an equation equal to the number of variables, thereby finding the solution of each variable that obtains the extreme value of the original function.

[0054] S202, inputting the image data and the initialized target variable into a pre-trained image restoration model, and iteratively updating the target variable in turn through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain a final target variable.

[0055] In this embodiment, the image restoration model is a deep unfolding network designed based on the Three Operator Splitting algorithm (TOS), named TOS-Net, where the TOS algorithm is an algorithm in the field of optimization (reference: DAVIS, Damek; YIN, Wotao. A three-operator splitting scheme and its optimization applications. Set-valued and variational analysis, 2017, 25.4: 829-858.). First, in this embodiment, the image restoration problem is modeled as the following optimization problem:

[0056]

[0057] Where f(·) represents the fidelity loss of the image, g(·) and h(·) represent two different prior information of the image. To solve the above image restoration problem, a network structure model with interpretability based on model-driven deep learning is designed, called TOS-Net. Figure 3 As shown in Figure 2, the network consists of n TOS network layers (TOS-Block) and 1 learnable nonlinear mapping network layer Γ g The TOS network layer can iteratively update the target variable in turn to obtain the final target variable. Through n-word iteration, the initial z 0 Get the target variable z after n iterations n , z k-1 Then input to the nonlinear mapping network layer Γ g Perform nonlinear mapping to generate the final restored image data.

[0058] Among them, specifically, Figure 4 As shown, in S202, when the target variable is iteratively updated in sequence by using multiple three-operator splitting algorithm TOS network layers of the image restoration model, it may specifically include:

[0059] S301. For any TOS network layer, input the image data to be restored and the target variable to be updated; for the first TOS network layer, the target variable to be updated is the initialized target variable, and for other TOS network layers, the target variable to be updated is the target variable output by the previous TOS network layer;

[0060] S302, according to the image data to be restored and the target variable to be updated, optimizing the target variable to be updated under the preset dual regularization term constraint to obtain the target variable that satisfies the preset dual regularization term constraint;

[0061] S303: Determine the target variable that satisfies the preset dual regularization term constraint as the updated target variable, and output the updated target variable.

[0062] In this embodiment, the image data y to be restored and the initialized target variable z 0 Input to TOS-Net, specifically, first input the image data y and the initialized target variable z 0 Input to the first TOS-Block, update the target variable, and obtain the updated target variable z that satisfies the preset double regularization term constraint 1 , after the image data y to be restored and the updated target variable z 1 Input to the second TOS-Block, update the target variable, and obtain the updated target variable z that satisfies the preset double regularization term constraint 2 , and so on, the kth TOS-Block will update the multiplier z of the last update k-1 and the image data y to be restored as input, and output the updated target variable z that satisfies the preset double regularization term constraint k , until the nth TOS-Block outputs the updated target variable z that satisfies the preset dual regularization term constraint n , as the final target variable output to the nonlinear mapping network layer Γ g For different application scenarios, corresponding dual regularization constraints (such as image prior information, etc.) can be adopted according to actual needs. There is no restriction here, and TOS-Block can design corresponding network structures according to different dual regularization constraints.

[0063] S203: input the final target variable into the nonlinear mapping network layer of the image restoration model to generate final restored image data, and output the final restored image data.

[0064] In this embodiment, the final target variable z n Input nonlinear mapping network layer Γ g , through the pre-learned nonlinear mapping network layer Γg For the target variable z n By performing nonlinear mapping, the final restored image data can be generated.

[0065] The image restoration method provided in this embodiment obtains the image data to be restored and the initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into a pre-trained image restoration model, and the target variable is iteratively updated in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate the final restored image data, and the final restored image data is output. The image restoration model provided in the embodiment of the present invention is an interpretable network model based on the deep unfolding network structure of the three-operator splitting algorithm. The structure combines the advantages of both the traditional model optimization image restoration method and the heuristic deep learning image restoration method, and can improve the image restoration quality and robustness.

[0066] Based on the above embodiment, optionally, TOS-Block is composed of a learnable first nonlinear mapping sub-network layer Γ g , fidelity sub-network layer F, learnable second nonlinear mapping sub-network layer Γ h And variable update sub-network layer M. Learnable nonlinear mapping module Γ g and Γ h The network structure will be designed according to the proximal mapping function corresponding to different image priors.

[0067] like Figure 5 As shown, in the above embodiment, according to S302 of the embodiment, the target variable to be updated is optimized under the preset dual regularization term constraint according to the image data to be restored and the target variable to be updated, so as to obtain the target variable satisfying the preset dual regularization term constraint, which may specifically include:

[0068] S401, through the first nonlinear mapping sub-network layer Γ g , generating first image data satisfying a first preset regularization term according to the target variable to be updated;

[0069] S402, updating the target variable to be updated according to the target variable to be updated, the image data to be restored, and the first image data through the fidelity sub-network layer F to obtain an intermediate target variable;

[0070] S403, through the second nonlinear mapping sub-network layer Γ h , generating second image data satisfying second preset image prior information or second preset regular term according to the intermediate target variable;

[0071] S404. Update the target variable to be updated through the variable updating subnetwork layer M according to the target variable to be updated, the first image data, and the second image data to obtain an updated target variable, and output the updated target variable.

[0072] In this embodiment, through the first nonlinear mapping sub-network layer Γ g , fidelity sub-network layer F, nonlinear mapping sub-network layer Γ h And the variable updating sub-network layer M can realize the update of the target variable, and the target variable satisfies the preset double regular term constraint. Through continuous iterative updates, the final target variable is continuously optimized.

[0073] Optionally, the first preset regularization item is first preset image prior information; the second preset regularization item is second preset image prior information. Of course, for different application scenarios, the first preset regularization item and the second preset regularization item may also adopt other corresponding regularization item constraints according to actual needs, and no limitation is made here.

[0074] In an optional embodiment, based on the above embodiment, the method is applied to an image compression sensing restoration scenario, and the image data to be restored is sparse observation data; the preset double regularization term constraint can adopt the l1-l2 norm, where the visualization of the norm can be as follows Figure 6 As shown, in this embodiment, the first preset regularization term is l l norm; the second preset regularization term is the -l2 norm. Correspondingly, when designing a deep expansion network in this embodiment, the TOS-Net network structure model can be TOS-CSNet (Three Operator Splitting algorithm based Compress Sensing Network), and the overall network structure framework of TOS-CSNet is similar to Figure 3 The same, wherein the TOS network layer TOS-Block may be a TOS-CSBlock, and its principle and implementation details will be introduced in detail below.

[0075] For the compressed sensing problem, the target image is restored based on the sparse observation data y obtained by random sampling. Usually, the compressed sensing problem mainly restores the original target image x through the following model:

[0076]

[0077] Among them, Φ represents the upsampling matrix, Ψ represents the transformation matrix, and g(·) is the regularization term, which is usually taken as norm to impose sparse constraints on x in other transformation domains, and λ is a penalty parameter used to balance the fidelity term and the regularization term g(Ψx).

[0078] Since Ψ is just a fixed matrix in the original optimization problem, the new image compression sensing model proposed in this embodiment replaces the transformation matrix Ψ with a parameterized learnable nonlinear transformation matrix H, and updates the model of the traditional optimization problem to a module in a deep neural network, turning it into a parameterized mapping. In this way, the deep neural network can be used to learn the transformation matrix, thereby obtaining better results. At the same time, this embodiment adopts sparse constraints to achieve better results. The norm is used as the regularization term of the model. The compressed sensing model we proposed with double regularization constraints is as follows:

[0079]

[0080] in, and is the convolution operation, and ReLu(·) is the activation function. Definition g(·)=λ||Hx||1,h(·)=-λ||Hx||2。 According to the traditional TOS optimization algorithm (reference: Davis D, Yin WA three-operator splitting scheme and its optimization applications. Set-valued and variational analysis. 2017 Dec; 25(4): 829-58.), the calculation process of each module in each iteration is as follows:

[0081]

[0082] The proximal mapping function Substituting the corresponding mathematical formulas and deducing mathematically, the update process of each module is as follows:

[0083]

[0084] Where S γλ (·) is the soft threshold function, It is the left inverse of H, and has a network structure symmetric to H. G is defined as follows:

[0085]

[0086] Where (x) +In network design, the activation function ReLu(x) can be used to implement it, that is, the activation function ReLu(x) is used to implement max(0, x).

[0087] It can be seen that for the first nonlinear mapping subnetwork layer Γ g , the target variable z to be updated k Input the first nonlinear mapping sub-network layer Γ g , according to the parameterized learnable nonlinear transformation matrix calculation model H, soft threshold function model S γλ (·), and the left inverse matrix calculation model of the nonlinear transformation matrix H Get the first image data x that satisfies the l1 norm k ; wherein the nonlinear transformation matrix calculation model H is implemented by convolution operation and activation function;

[0088] For the fidelity sub-network layer F, according to the target variable z to be updated k , the image data y to be restored, and the first image data x k , update the target variable with a specific operation to obtain the intermediate target variable

[0089] For the learnable second nonlinear mapping sub-network layer Γ h , the intermediate target variable Input the second nonlinear mapping sub-network layer Γ h , according to the nonlinear transformation matrix calculation model H, and the left inverse matrix calculation model of the nonlinear transformation matrix H And the activation function model G, obtain the second image data v that satisfies the -l2 norm k .

[0090] For the variable update subnetwork layer M, according to the target variable z to be updated k , the first image data x k , the second image data v k , update the target variable and get the updated target variable z k+1 .

[0091] Based on the above embodiments, the overall network structure framework of TOS-CSNet is as follows: Figure 3 As shown, it should be noted that each TOS-Block in TOS-CSNet is replaced by TOS-CSBlock. TOS-CSBlock is designed based on the TOS optimization expansion process of the image compression sensing model constrained by the double regularization term proposed in this technical solution. The specific network structure and data flow process are as follows Figure 7 As shown. For the k+1th layer TOS-CSBlock, the target variable z updated by the kth layer TOS-CSBlockk The sparse observation data y (the image data to be restored) obtained by sampling is used as the input of the k+1th layer TOS-CSBlock, and the updated target variable z is output. k+1 Specifically, the input target variable z k First enter the network module H, and then pass the soft threshold function S γλ (·), and finally enter another network module Get an image x that satisfies a specific regularization term l1 norm k ; Then update the image x k With the target variable z k Together with the sparse observation data y, it is input into the fidelity module F, which is based on the objective function min x The gradient information of the f(x) function in f(x)+g(x)+h(x) updates the new intermediate target variable Updated intermediate target variable Input to the nonlinear mapping module Γ h Generate an image v that satisfies the regularization term -l2 norm k ; Finally, the variable update module M is based on the target variable z k , image x k and image v k Update the new target variable z k+1 The final target variable z is updated after n TOS-CSBlocks n , the final target variable z n Input to the nonlinear mapping network layer Γ g Generate the final restored image The nonlinear mapping network layer Γ g The final restored image is generated through the following process

[0092] In another optional embodiment, based on the above embodiment, the image restoration method can also be applied to the image denoising scenario, the image data to be restored is noisy image data; the first preset regularization term is the depth image prior information; the second preset regularization term is the l1 norm. Correspondingly, when designing the deep unfolding network in this embodiment, the TOS-Net network structure model can be TOS-IDNet (Three Operator Splitting algorithm based Image Denoising Network), and the overall network structure framework of TOS-IDNet is the same as Figure 3 The same, wherein the TOS network layer TOS-Block can be a TOS-IDBlock, and its principle and implementation details will be introduced in detail below.

[0093] For the image denoising problem, for the input noisy image y, its additive noise can be expressed by the following formula:

[0094] y=s+n

[0095] Among them, s represents an image without noise, n represents a noise term, and the noise may include but is not limited to salt and pepper noise, Gaussian noise, etc. There are different denoising models according to different noises. One of the classic image denoising models is as follows:

[0096]

[0097] Where s represents the sparse mapping matrix of the noise-free image, D is the dictionary matrix, Ds represents the noise-free image we need to restore, g(s) is the regularization term, which acts as a sparse constraint on s, and σ represents the noise level. The model in this embodiment assumes that the image can be represented by a combination of a dictionary and a sparse matrix. In this embodiment, a learnable nonlinear transformation matrix H is adopted. d The dictionary matrix D is replaced, and a deep image prior (DIP) is introduced to learn the potential noise prior. The image denoising model constrained by the dual regularization term proposed in this embodiment is as follows:

[0098]

[0099] where α1=σ 2 λ1, α2 = σ 2 λ2, nonlinear transformation matrix and is the convolution operation, ReLu(·) is the activation function; the first regularization term is the DIP deep image prior information, and the second regularization term is the l1 norm, which is mainly used to solve the problem of image sparsity and recover data from sparse observations. According to the TOS optimization algorithm, through mathematical derivation, the calculation process of each module of the optimization iteration process is as follows:

[0100]

[0101] Among them, S γλ (·) is the soft threshold function, Yes H d Pseudo-inverse, its network structure is similar to H d Symmetrical, H dipThe deep denoising network model, the specific structure is ResUnet (reference: Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 9446-9454).), such as Figure 8 shown.

[0102] It can be seen that for the first nonlinear mapping subnetwork layer Γ g , the target variable z to be updated k Input the first nonlinear mapping sub-network layer Γ g , the model H is calculated based on the parameterized learnable nonlinear transformation matrix d , deep denoising network model H dip , and the nonlinear transformation matrix H d Pseudo-inverse matrix calculation model H e , obtain the first image data s that meets the depth image prior information DIP k ; The nonlinear transformation matrix calculation model H d It is realized through convolution operation and activation function; the deep denoising network model is a deep denoising network model with ResUnet structure;

[0103] For the fidelity sub-network layer F, according to the target variable z to be updated k , the image data y to be restored, and the first image data s k , update the target variable with a specific operation to obtain the intermediate target variable

[0104] For the learnable second nonlinear mapping sub-network layer Γ h , the intermediate target variable Input the second nonlinear mapping sub-network layer Γ h , according to the soft threshold function model S γλ (·) Obtain the second image data v that satisfies the l1 norm k .

[0105] For the variable update subnetwork layer M, according to the target variable z to be updated k , first image data s k , the second image data v k , update the target variable and get the updated target variable z k+1 .

[0106] Based on the above embodiments, the overall network structure framework of TOS-IDNet is as follows: Figure 3 Similarly, we only need to replace TOS-Block with TOS-IDBlock to get TOS-IDNet. TOS-IDBlock is designed based on the TOS optimization expansion process of the image denoising model constrained by the dual regularization term proposed in this technical solution. The specific network structure and data flow process are as follows: Fig. 9 As shown. For the k+1th layer TOS-IDBlock, the output z of the kth layer is k The sampled observation data y is used as the input of the k+1th layer TOS-IDBlock, and the updated target variable z is output k+1 Specifically, the k-th layer input variable z k First enter the network module H d , and then through the deep image prior module H dip (like Figure 8 ResUnet shown), and finally enter another network module Get the initial restored image s that satisfies the depth image prior information k (corresponding to Figure 3 x in k ); then update the image s k With the target variable z k Together with the sparse observation data y, it is input into the fidelity module F to update the new variable Updated intermediate target variable Input to the nonlinear mapping module Г h Generate an image v that satisfies the sparse regularization term l1 norm k ; Finally, the variable update module M is based on the target variable z k , images k and image v k Update the new target variable z k+1 The final target variable z is updated after n TOS-IDBlocks n , the final target variable z n Input to the nonlinear mapping network layer Γ g Generate the final restored denoised image The nonlinear mapping network layer Γ g The final restored image is generated through the following process

[0107] The deep unfolding network structure based on the three-operator splitting algorithm proposed in the above embodiments is an interpretable network structure, which combines the advantages of the traditional model optimization image restoration method and the heuristic deep learning image restoration method. The learning process of the structure is driven by domain knowledge and data. The forward propagation module of the TOS-Net network structure proposed in the above embodiments can process two image priors or regularization terms at the same time, and can restore higher quality images.

[0108] The above implementations can expand TOS-Net into corresponding TOS-CSNet and TOS-IDNet structures according to image compression sensing and image denoising problems respectively, and design specific learnable nonlinear mapping modules according to their specific constraints to replace the proximal mapping operator of the iterative optimization process.

[0109] It should be noted that the TOS-Net proposed in the above embodiments is a universal network structure, which can handle image restoration models with two image priors or regularization terms. According to different image restoration problems, the corresponding TOS-Block can be expanded into different network structures to handle specific image restoration problems. Therefore, each TOS-Block can be replaced with other TOS-Block modules according to different image priors and regularization terms. Replacing the image priors and regularization terms in TOS-Block with other image priors and regularization terms is also within the protection scope of the present invention.

[0110] In the above embodiment, optionally, the image restoration model with two image priors or regularization terms can be processed by an image restoration method based on traditional model optimization. It is sufficient to select an optimization model that can handle three operators. However, the disadvantage of such a method is that the proximal mapping function in the iterative optimization calculation process is difficult to solve, and the calculation process is time-consuming. In addition, the image restoration model with two image priors or regularization terms can also be processed by an image restoration method based on heuristic deep learning. The method is to add the image prior or regularization term to the loss function of training the deep network model. During the training process, the solution output by the network is approximated to the corresponding constraint term. The disadvantage is that the penalty parameter corresponding to the constraint term in the loss function is difficult to adjust, because the effects of different image priors may inhibit each other, which ultimately leads to poor quality of the restored image.

[0111] Fig.10 The structure diagram of the image recovery device provided by the embodiment of the present invention. The image recovery device provided by this embodiment can execute the processing flow provided by the method embodiment, such as Fig.10 As shown, the image restoration device 500 includes an acquisition module 501 , a three-operator splitting algorithm network module 502 , and a nonlinear mapping network module 503 .

[0112] An acquisition module 501 is used to acquire image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data;

[0113] A three-operator splitting algorithm network module 502 is used to input the image data and the initialized target variable into a pre-trained image restoration model, and iteratively update the target variable in turn through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain a final target variable;

[0114] The nonlinear mapping network module 503 is used to input the final target variable into the nonlinear mapping network layer of the image restoration model, generate the final restored image data, and output the final restored image data.

[0115] Based on any of the above embodiments, when the three-operator splitting algorithm network module 502 iteratively updates the target variable in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model, it is used to:

[0116] For any TOS network layer, the image data to be restored and the target variable to be updated are input; wherein for the first TOS network layer, the target variable to be updated is the initialized target variable, and for other TOS network layers, the target variable to be updated is the target variable output by the previous TOS network layer;

[0117] According to the image data to be restored and the target variable to be updated, the target variable to be updated is optimized under a preset dual regularization term constraint to obtain a target variable that satisfies the preset dual regularization term constraint;

[0118] The target variable that satisfies the preset dual regular term constraint is determined as the updated target variable, and the updated target variable is output.

[0119] Based on any of the above embodiments, the TOS network layer includes a first nonlinear mapping subnetwork layer, a fidelity subnetwork layer, a second nonlinear mapping subnetwork layer, and a variable update subnetwork layer;

[0120] The three-operator splitting algorithm network module 502 optimizes the target variable to be updated under the preset dual regularization term constraint according to the image data to be restored and the target variable to be updated, and obtains the target variable that satisfies the preset dual regularization term constraint, and is used to:

[0121] Generate first image data satisfying a first preset regularization term according to the target variable to be updated through a first nonlinear mapping sub-network layer;

[0122] By means of a fidelity sub-network layer, the target variable to be updated is updated according to the target variable to be updated, the image data to be restored, and the first image data to obtain an intermediate target variable;

[0123] Generate second image data satisfying second preset image prior information or second preset regular term according to the intermediate target variable through a second nonlinear mapping sub-network layer;

[0124] Through the variable updating subnetwork layer, the target variable to be updated is updated according to the target variable to be updated, the first image data, and the second image data to obtain an updated target variable, and the updated target variable is output.

[0125] On the basis of any of the above embodiments, the first preset regularization term is first preset image prior information; and the second preset regularization term is second preset image prior information.

[0126] In an optional embodiment, based on any of the above embodiments, the method is applied to a compressed sensing image restoration scenario, and the image data to be restored is sparse observation data;

[0127] The first preset regularization term is the l1 norm; the second preset regularization term is the -l2 norm.

[0128] Furthermore, when the three-operator splitting algorithm network module 502 generates the first image data satisfying the first preset regularization term according to the target variable to be updated through the first nonlinear mapping sub-network layer, it is used to:

[0129] Inputting the target variable to be updated into the first nonlinear mapping sub-network layer, and acquiring first image data satisfying the l1 norm according to a parameterized learnable nonlinear transformation matrix calculation model, a soft threshold function model, and a left inverse matrix calculation model of the nonlinear transformation matrix; wherein the nonlinear transformation matrix calculation model is implemented by a convolution operation and an activation function;

[0130] The three-operator splitting algorithm network module 502 is used to: when generating the second image data satisfying the second preset image prior information or the second preset regular term according to the intermediate target variable through the second nonlinear mapping sub-network layer;

[0131] The intermediate target variable is input into the second nonlinear mapping sub-network layer, and the second image data satisfying the -l2 norm is obtained according to the nonlinear transformation matrix calculation model, the left inverse matrix calculation model of the nonlinear transformation matrix, and the activation function model.

[0132] In an optional embodiment, based on any of the above embodiments, the method is applied to an image denoising scenario, and the image data to be restored is image data with noise;

[0133] The first preset regularization term is depth image prior information; the second preset regularization term is l1 norm.

[0134] Furthermore, when the three-operator splitting algorithm network module 502 generates the first image data satisfying the first preset regularization term according to the target variable to be updated through the first nonlinear mapping sub-network layer, it is used to:

[0135] Input the target variable to be updated into the first nonlinear mapping sub-network layer, and obtain first image data that meets the deep image prior information according to a parameterized learnable nonlinear transformation matrix calculation model, a deep denoising network model, and a pseudo-inverse matrix calculation model of the nonlinear transformation matrix; wherein the nonlinear transformation matrix calculation model is implemented by a convolution operation and an activation function; and the deep denoising network model is a deep denoising network model of a ResUnet structure;

[0136] The three-operator splitting algorithm network module 502 is used to: when generating the second image data satisfying the second preset image prior information or the second preset regular term according to the intermediate target variable through the second nonlinear mapping sub-network layer;

[0137] The intermediate target variable is input into the second nonlinear mapping sub-network layer, and second image data satisfying the l1 norm is obtained according to the soft threshold function model.

[0138] The image recovery device provided in the embodiment of the present invention can be specifically used to perform the above Figure 2 , 4 -5 provides a method embodiment, and the specific functions are not repeated here.

[0139] The image restoration device provided by the embodiment of the present invention obtains the image data to be restored and the initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; the image data and the initialized target variable are input into a pre-trained image restoration model, and the target variable is iteratively updated in sequence through multiple three-operator splitting algorithm TOS network layers of the image restoration model to obtain the final target variable; the final target variable is input into the nonlinear mapping network layer of the image restoration model to generate the final restored image data, and the final restored image data is output. The image restoration model provided by the embodiment of the present invention is an interpretable network model based on the deep unfolding network structure of the three-operator splitting algorithm. The structure combines the advantages of both the traditional model optimization image restoration method and the heuristic deep learning image restoration method, and can improve the image restoration quality and robustness.

[0140] Fig.11 The electronic device provided by the embodiment of the present invention can execute the processing flow provided by the image restoration method embodiment, such as Fig.11 As shown, the electronic device 60 includes a memory 61, a processor 62, and a computer program; wherein the computer program is stored in the memory 61 and is configured so that the processor 62 executes the image restoration method described in the above embodiment. In addition, the electronic device 60 may also have a communication interface 63 for transmitting control instructions and / or transmitting data.

[0141] Fig.11 The electronic device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned image restoration method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0142] In addition, this embodiment further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the image restoration method described in the above embodiment.

[0143] In addition, this embodiment further provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the image restoration method described in the above embodiment.

[0144] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0147] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0148] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0149] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention are described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

[0150] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0151] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for restoring an image, characterized in that: include: Obtaining image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; Input the image data and the initialized target variable into a pre-trained image restoration model, and for any TOS network layer, input the image data to be restored and the target variable to be updated; wherein, for the first TOS network layer, the target variable to be updated is the initialized target variable, and for other TOS network layers, the target variable to be updated is the target variable output by the previous TOS network layer; According to the image data to be restored and the target variable to be updated, the target variable to be updated is optimized under a preset dual regularization term constraint to obtain a target variable that satisfies the preset dual regularization term constraint; Determine the target variable that satisfies the preset dual regularization term constraint as the updated target variable, and output the updated target variable to obtain the final target variable; The final target variable is input into the nonlinear mapping network layer of the image restoration model to generate final restored image data, and the final restored image data is output.

2. The method according to claim 1, characterized in that The TOS network layer includes a first nonlinear mapping subnetwork layer, a fidelity subnetwork layer, a second nonlinear mapping subnetwork layer, and a variable updating subnetwork layer; The step of optimizing the target variable to be updated under a preset dual regularization term constraint according to the image data to be restored and the target variable to be updated to obtain a target variable satisfying the preset dual regularization term constraint includes: Generate first image data satisfying a first preset regularization term according to the target variable to be updated through a first nonlinear mapping sub-network layer; By means of a fidelity sub-network layer, the target variable to be updated is updated according to the target variable to be updated, the image data to be restored, and the first image data to obtain an intermediate target variable; Generate second image data satisfying second preset image prior information or second preset regular term according to the intermediate target variable through a second nonlinear mapping sub-network layer; Through the variable updating subnetwork layer, the target variable to be updated is updated according to the target variable to be updated, the first image data, and the second image data to obtain an updated target variable, and the updated target variable is output.

3. The method according to claim 2, characterized in that The first preset regularization term is first preset image prior information; the second preset regularization term is second preset image prior information.

4. The method according to claim 2, characterized in that: The method is applied to a compressed sensing restoration scenario of an image, wherein the image data to be restored is sparse observation data; The first preset regularization term is the l1 norm; the second preset regularization term is the -l2 norm.

5. The method according to claim 4, characterized in that The method of generating first image data satisfying a first preset regularization term according to a target variable to be updated through a first nonlinear mapping sub-network layer includes: Inputting the target variable to be updated into the first nonlinear mapping sub-network layer, and acquiring first image data satisfying the l1 norm according to a parameterized learnable nonlinear transformation matrix calculation model, a soft threshold function model, and a left inverse matrix calculation model of the nonlinear transformation matrix; wherein the nonlinear transformation matrix calculation model is implemented by a convolution operation and an activation function; The step of generating second image data satisfying second preset image prior information or second preset regularization term according to the intermediate target variable through the second nonlinear mapping sub-network layer includes: The intermediate target variable is input into the second nonlinear mapping sub-network layer, and the second image data satisfying the -l2 norm is obtained according to the nonlinear transformation matrix calculation model, the left inverse matrix calculation model of the nonlinear transformation matrix, and the activation function model.

6. The method according to claim 2, characterized in that The method is applied to an image denoising scenario, wherein the image data to be restored is image data with noise; The first preset regularization term is depth image prior information; the second preset regularization term is l1 norm.

7. The method according to claim 6, characterized in that The first image data satisfying the first preset regularization term is generated according to the target variable to be updated through the first nonlinear mapping sub-network layer; Input the target variable to be updated into the first nonlinear mapping sub-network layer, and obtain first image data that meets the deep image prior information according to a parameterized learnable nonlinear transformation matrix calculation model, a deep denoising network model, and a pseudo-inverse matrix calculation model of the nonlinear transformation matrix; wherein the nonlinear transformation matrix calculation model is implemented by a convolution operation and an activation function; and the deep denoising network model is a deep denoising network model of a ResUnet structure; The step of generating second image data satisfying second preset image prior information or second preset regularization term according to the intermediate target variable through the second nonlinear mapping sub-network layer includes: The intermediate target variable is input into the second nonlinear mapping sub-network layer, and second image data satisfying the l1 norm is obtained according to the soft threshold function model.

8. An image restoration device, characterized in that: include: An acquisition module, used for acquiring image data to be restored and an initialized target variable, wherein the initialized target variable is a randomly generated Lagrange multiplier with the same dimension as the image data; A three-operator splitting algorithm network module is used to input the image data and the initialized target variable into a pre-trained image restoration model, and for any TOS network layer, input the image data to be restored and the target variable to be updated; wherein, for the first TOS network layer, the target variable to be updated is the initialized target variable, and for other TOS network layers, the target variable to be updated is the target variable output by the previous TOS network layer; according to the image data to be restored and the target variable to be updated, the target variable to be updated is optimized under the preset dual regularization term constraint to obtain the target variable that satisfies the preset dual regularization term constraint; the target variable that satisfies the preset dual regularization term constraint is determined as the updated target variable, and the updated target variable is output to obtain the final target variable; The nonlinear mapping network module is used to input the final target variable into the nonlinear mapping network layer of the image restoration model, generate the final restored image data, and output the final restored image data.

9. An electronic device, characterized in that: include: at least one processor; and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images