Image Reconstruction Method, System, Terminal and Storage Medium Based on Hybrid Network Framework

Abstract

The present invention discloses an image reconstruction method, system, terminal and storage medium based on a hybrid network framework. The method includes: obtaining prediction data by predicting an original image, acquiring a quality loss and a physical loss of momentum conservation during the prediction process, and at the same time comparing the prediction data with the original data to obtain a matching value between the two, thereby constructing a total loss function; introducing the NTK matrix to determine an appropriate learning rate, and using the learning rate and the total loss function to train the hybrid network framework, and finally obtaining a DeepONet-NTK hybrid network. The present invention utilizes DeepONet to learn the nonlinear mapping relationship in the multi-dimensional function space, and at the same time combines the NTK theory to optimize the training process, improve the convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining the high-dimensional semantic features.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and particularly to an image reconstruction method, system, terminal and computer-readable storage medium based on a hybrid network framework. Background Art

[0002] An inverse problem is a class of scientific problems that start from observational results or known effects and infer the internal causes, parameters, or system attributes that led to the results in reverse.

[0003] Inverse problems have extensive applications in computational physics, medical imaging, and engineering. For example, locating the position of a source through scattered field data or reconstructing an image from noisy observations.

[0004] For traditional numerical methods (such as iterative algorithms and sampling algorithms), there are problems of high computational complexity for high-dimensional non-linear problems and dependence on a large amount of high-quality data, and their performance significantly degrades under sparse or noisy data. For the current method of embedding partial differential equation constraints into the neural network loss function, it is prone to falling into local optima during the training process, has a slow convergence speed, is sensitive to noise, and has insufficient generalization ability under low signal-to-noise ratio data.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main objective of the present invention is to provide an image reconstruction method, system, terminal and computer-readable storage medium based on a hybrid network framework, aiming to solve the problems in the prior art that when dealing with high-dimensional non-linear inverse problems, there are low computational efficiency, poor physical consistency, sensitivity to noise and sparse data, resulting in high signal-to-noise ratio and inaccurate reconstructed images.

[0007] To achieve the above objective, the present invention provides an image reconstruction method based on a hybrid network framework. The image reconstruction method based on a hybrid network framework includes the following steps:

[0008] Obtain multiple image data in the target image, input all the image data into the hybrid network framework, and output prediction mapping information and a prediction result;

[0009] Construct a physical loss function according to the prediction mapping information, construct a data loss function according to the prediction result and the target image, and construct a quality loss function according to the prediction result;

[0010] Construct the NTK regularization term, construct the objective loss function according to the physical loss function, the data loss function and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain the DeepONet-NTK hybrid network;

[0011] Input the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output the reconstructed image.

[0012] Optionally, in the image reconstruction method based on the hybrid network framework, the image data includes: source parameters and observation point coordinates;

[0013] The obtaining of multiple image data in the target image, inputting all the image data into the hybrid network framework, and outputting prediction mapping information and a prediction result specifically includes:

[0014] Obtain multiple source parameters and multiple observation point coordinates in the target image, and construct a hybrid network framework according to the deep operator network and the neural tangent kernel;

[0015] Input all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework, and output prediction mapping information and a prediction result.

[0016] Optionally, in the image reconstruction method based on the hybrid network framework, the source parameters include source intensity and source position;

[0017] The inputting of all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework, and outputting prediction mapping information and a prediction result specifically includes:

[0018] Input all the source intensities and all the source positions into the branch network in the deep operator network, and output a plurality of high-dimensional feature vectors;

[0019] Input the observation point coordinates into the backbone network in the deep operator network, and output a plurality of feature vectors;

[0020] Construct prediction mapping information and a prediction result according to the observation point coordinates, all the high-dimensional feature vectors and all the feature vectors:

[0021] ;

[0022] ;

[0023] ;

[0024] Wherein, Represents the predicted mapping information, represents the coordinates of the observation point, represents the total number of source parameters, represents the high-dimensional feature vector of the th source parameter, and respectively represent the abscissa and ordinate of represents the set of real numbers, represents the dimension of the space.

[0025] Optionally, in the image reconstruction method based on the hybrid network framework, wherein, constructing a physical loss function according to the predicted mapping information, constructing a data loss function according to the prediction result and the target image, and constructing a quality loss function according to the prediction result, specifically including:

[0026] Constructing a physical loss function according to the predicted mapping information:

[0027] ;

[0028] Wherein, represents the physical loss function, represents the prediction result predicted by the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, represents the total number of steps in the model simulation process, represents the divergence of the vector field;

[0029] Constructing a data loss function according to the prediction result output by the hybrid network framework and the target image, wherein the data loss function represents the matching degree between the prediction result output by the model and the observed result input to the model:

[0030] ;

[0031] Wherein, represents the data loss function, represents the total number of prediction results, represents the th original data of the target image, represents the th prediction result;

[0032] Constructing a quality loss function according to the prediction result:

[0033] ;

[0034] ;

[0035] Among them, represents the quality loss function, represents the index of the total number of steps.

[0036] Optionally, for the image reconstruction method based on the hybrid network framework, wherein, constructing the NTK regularization term, constructing the objective loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the objective loss function and the NTK regularization term to obtain the DeepONet-NTK hybrid network, specifically includes:

[0037] Constructing the NTK regularization term:

[0038] ;

[0039] ;

[0040] Among them, represents the NTK regularization term, represents the eigenvalue of the NTK matrix, represents taking the trace operation on the matrix performing the trace operation, represents the NTK matrix, and respectively represent the pixel points in the input NTK matrix, represents the parameters of the hybrid network framework, represents the transpose, represents the gradient of, represents the gradient of;

[0041] Constructing the objective loss function according to the physical loss function, the data loss function and the quality loss function:

[0042] ;

[0043] Among them, represents the objective loss function, represents the data loss function;

[0044] Adjusting the eigenvalue of the NTK regularization term to the maximum to obtain the corresponding target learning rate:

[0045] ;

[0046] Among them, represents the target learning rate, Represents the maximum eigenvalue;

[0047] Input the target loss function and the target learning rate into the hybrid network framework for training to obtain the DeepONet-NTK hybrid network.

[0048] Optionally, for the image reconstruction method based on the hybrid network framework, where inputting the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstructing the current image according to the pixel values in the final field value prediction, and outputting the reconstructed image, specifically includes:

[0049] Obtain the damaged image data of the current image input by the user, and input the damaged image data into the DeepONet-NTK hybrid network;

[0050] The optimization branch network in the DeepONet-NTK hybrid network outputs the damaged feature vector of the damaged image data;

[0051] Input the damaged feature vector into the optimization backbone network in the DeepONet-NTK hybrid network, and the optimization backbone network encodes to obtain pixel coordinates according to the damaged feature vector, and outputs the position feature according to the pixel coordinates;

[0052] The DeepONet-NTK hybrid network constructs an inner product according to the damaged feature vector and the position feature:

[0053] ;

[0054] Wherein, Represents the inner product, Represents the damaged feature vector, Represents the position feature, And Respectively represent the pixel abscissa and pixel ordinate of the damaged position;

[0055] Reconstruct the target image according to the inner product, and output the reconstructed image.

[0056] Optionally, for the image reconstruction method based on the hybrid network framework, where inputting the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstructing the current image according to the pixel values in the final field value prediction, and outputting the reconstructed image, and then further includes:

[0057] Obtain all target pixel values of the current image and all reconstructed pixel values in the reconstructed image, and calculate the maximum reconstructed pixel value and the mean square error of the reconstructed pixels based on all the reconstructed pixel values;

[0058] Construct the peak signal-to-noise ratio based on the maximum reconstructed pixel value and the mean square error of the reconstructed pixels:

[0059] ;

[0060] wherein, represents the peak signal-to-noise ratio, represents the maximum reconstructed pixel value, represents the mean square error of the reconstructed pixels;

[0061] Construct the structural similarity index based on all the target pixel values and all the reconstructed pixel values:

[0062] ;

[0063] wherein, represents the structural similarity index, represents the mean pixel value of the pixel abscissa in the current image and the reconstructed image, represents the mean pixel value of the pixel ordinate in the current image and the reconstructed image, represents the pixel variance of the pixel abscissa in the current image and the reconstructed image, represents the pixel variance of the pixel ordinate in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and both represent stable constants;

[0064] Judge the accuracy of the reconstructed image according to the peak signal-to-noise ratio and the structural similarity index.

[0065] In addition, to achieve the above object, the present invention also provides an image reconstruction system based on a hybrid network framework, wherein the image reconstruction system based on the hybrid network framework includes:

[0066] An information extraction module, configured to obtain a plurality of image data in a target image, input all the image data into the hybrid network framework, and output prediction mapping information and a prediction result;

[0067] A loss construction module, configured to construct a physical loss function according to the prediction mapping information, construct a data loss function according to the prediction result and the target image, and construct a quality loss function according to the prediction result;

[0068] A model training module, configured to construct an NTK regularization term, construct an objective loss function according to the physical loss function, the data loss function, and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network;

[0069] An image reconstruction module, configured to input a current image into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and reconstruct the current image according to pixel values in the final field value prediction, and output a reconstructed image.

[0070] In addition, to achieve the above object, the present invention further provides a terminal, where the terminal includes: a memory, a processor, and an image reconstruction program based on a hybrid network framework stored on the memory and executable on the processor. When the image reconstruction program based on the hybrid network framework is executed by the processor, the steps of the image reconstruction method based on the hybrid network framework as described above are implemented.

[0071] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores an image reconstruction program based on a hybrid network framework. When the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the image reconstruction method based on the hybrid network framework as described above are implemented.

[0072] In the present invention, a plurality of image data in a target image are obtained, all the image data are input into a hybrid network framework, and prediction mapping information and a prediction result are output; a physical loss function is constructed according to the prediction mapping information, a data loss function is constructed according to the prediction result and the target image, and a quality loss function is constructed according to the prediction result; an NTK regularization term is constructed, an objective loss function is constructed according to the physical loss function, the data loss function, and the quality loss function, and the hybrid network framework is trained according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; a current image is input into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and the current image is reconstructed according to pixel values in the final field value prediction, and a reconstructed image is output. The present invention uses DeepONet to learn the nonlinear mapping relationship in the multi-dimensional function space, and at the same time combines the NTK theory to optimize the training process, improve the convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining the high-dimensional semantic features. Description of the Drawings

[0073] Figure 1 is a flowchart of a preferred embodiment of the image reconstruction method based on a hybrid network framework of the present invention;

[0074] Figure 2 It is a schematic diagram of the network framework of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0075] Figure 3 It is a schematic diagram of the PSNR and SSIM results of the first data set of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0076] Figure 4 It is a schematic diagram of the PSNR and SSIM results of the second data set of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0077] Figure 5 It is a schematic diagram of the PSNR and SSIM results of the third data set of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0078] Figure 6 It is a schematic diagram of the PSNR and SSIM results of the fourth data set of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0079] Figure 7 It is a schematic diagram of the result of image reconstruction of the preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention;

[0080] Figure 8 It is a structural diagram of the preferred embodiment of the image reconstruction system based on the hybrid network framework of the present invention;

[0081] Figure 9 It is a structural diagram of the preferred embodiment of the terminal of the present invention. Detailed implementation manners

[0082] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0083] The image reconstruction method based on the hybrid network framework described in the preferred embodiment of the present invention, as Figure 1 shown, the image reconstruction method based on the hybrid network framework includes the following steps:

[0084] Step S10, obtain multiple image data in the target image, input all the image data into the hybrid network framework, and output prediction mapping information and a prediction result.

[0085] Among them, the image data includes: source parameters and observation point coordinates. The source parameters include source intensity and source position. The source intensity represents the energy or intensity distribution of the point source of the pixel point in space, and the source position represents the coordinates of the point source of the pixel point in space.

[0086] Specifically, obtain multiple of the source parameters and multiple of the observation point coordinates in the target image, and construct a hybrid network framework based on the deep operator network and the neural tangent kernel; input all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework, and output prediction mapping information and a prediction result.

[0087] Among them, input the image data of the target image into the Figure 2 shown hybrid network framework. Through the operator mapping of DeepONet (Deep Operator Network), learn the non-linear mapping relationship in the infinite-dimensional function space, so that in hydrodynamic problems, the generated field satisfies the physical equation constraints, significantly improving the training efficiency and stability to solve the problem in the prior art that using generative models (such as variational autoencoders, diffusion models) to achieve data-driven reconstruction or prediction leads to a lack of physical constraints and the reconstruction results may violate physical laws.

[0088] Furthermore, input all the source intensities and all the source positions into the branch network in the deep operator network, and output multiple high-dimensional feature vectors; input the observation point coordinates (i.e., Figure 2 in and ) into the backbone network in the deep operator network, and output feature vectors (i.e., Figure 2 in and ); construct prediction mapping information and a prediction result according to the observation point coordinates, all the high-dimensional feature vectors and all the feature vectors:

[0089] ;

[0090] Among them, represents the prediction mapping information of , represents the observation point coordinates, represents the total number of source parameters, represents the th high-dimensional feature vector of the source parameter, represents the feature vector.

[0091] Among them, through the branch network in the hybrid network, the source parameters are encoded into high-dimensional feature vectors for subsequent field value prediction. At the same time, by combining the observation point coordinates processed by the backbone network, the initial inner product of the target image (i.e., the predicted mapping information) is constructed to realize the continuous function mapping from the source parameters to the spatial field, making the results predicted by the model in the Navier-Stokes source localization task have smaller errors, thereby improving the accuracy of the model for image reconstruction.

[0092] Step S20: Construct a physical loss function according to the predicted mapping information, construct a data loss function according to the prediction result and the target image, and construct a quality loss function according to the prediction result.

[0093] Among them, by introducing NTK (Neural Tangent Kernel), the convergence speed of the model can be improved, and at the same time, the stability of the training process can be enhanced, avoiding the problem that the model training process falls into local optimality. The introduced NTK theory can effectively solve the problems in the prior art that using DeepONet independently leads to unstable model training dynamics, large influence of the convergence speed on parameter initialization, insufficient robustness to sparse or noisy data, easy to produce overfitting, and lack of theoretical guarantee, making it difficult to explain the generalization behavior in the training process.

[0094] Specifically, construct a physical loss function according to the predicted mapping information:

[0095] ;

[0096] Among them, represents the physical loss function, represents the prediction result predicted by the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, represents the total number of steps in the model simulation process, represents the divergence of the vector field; construct a data loss function according to the prediction result output by the hybrid network framework and the target image, where the data loss function represents the matching degree between the prediction result output by the model and the observed result input to the model:

[0097] ;

[0098] Among them, represents the data loss function, represents the total number of prediction results, represents the th original data of the target image, represents the prediction results;

[0099] Construct a quality loss function according to the prediction results:

[0100] ;

[0101] ;

[0102] where represents the quality loss function, represents the index of the total number of steps.

[0103] During the training process, the physical loss function can ensure that the model is forced to satisfy the momentum conservation equation, and the quality loss function can ensure that the model is forced to satisfy the mass conservation equation. This process can effectively solve the problem of lack of physical constraints caused by using traditional deep learning methods (such as VAE, Variational Autoencoder), and the reconstruction results may violate physical laws.

[0104] Furthermore, by combining the data loss function representing the matching relationship between the predicted value and the observed value, it is ensured that the output result of the model conforms to physical laws and retains high-dimensional semantic features, improving the accuracy and stability when solving inverse problems.

[0105] Step S30: Construct the NTK regularization term, construct the target loss function according to the physical loss function, the data loss function, and the quality loss function, and train the hybrid network framework according to the target loss function and the NTK regularization term to obtain the DeepONet-NTK hybrid network.

[0106] Specifically, construct the NTK regularization term:

[0107] ;

[0108] ;

[0109] where represents the NTK regularization term, represents the eigenvalue of the NTK matrix, represents taking the trace operation on the matrix , represents the NTK matrix, and respectively represent the pixel points in the input NTK matrix, represents the parameters of the hybrid network framework, represents the transpose, represents the gradient of, represents The gradient; construct an objective loss function based on the physical loss function, the data loss function, and the quality loss function:

[0110] ;

[0111] where, represents the objective loss function, represents the data loss function; adjust the eigenvalue of the NTK regularization term to the maximum to obtain the corresponding objective learning rate:

[0112] ;

[0113] where, represents the objective learning rate, represents the maximum eigenvalue; input the objective loss function and the objective learning rate into the hybrid network framework for training to obtain the DeepONet-NTK hybrid network.

[0114] Among them, based on the DeepONet network, NTK regularization is introduced. First, construct the NTK matrix, and then construct the regularization term to dynamically adjust the learning rate so that the NTK matrix reaches the maximum eigenvalue, avoiding the situation of gradient overrun.

[0115] Furthermore, when constructing NTK regularization, the gradient alignment term can also be calculated (as shown in Figure 2 , where represents , represents the model parameters after training), calculate the regularization term loss, where, and both represent the points input into the model, represents the point at the gradient value; then jointly minimize the objective loss function to realize the embedding of feature interaction and physical constraints, which can effectively improve the convergence speed of model training, reduce the sensitivity of the model to parameter initialization, and avoid the model falling into local optimum.

[0116] Step S40: Input the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output the reconstructed image.

[0117] Specifically, obtain the damaged image data of the current image input by the user, and input the damaged image data into the DeepONet-NTK hybrid network; the optimization branch network in the DeepONet-NTK hybrid network outputs the damaged feature vector of the damaged image data; input the damaged feature vector into the optimization backbone network in the DeepONet-NTK hybrid network, and the optimization backbone network encodes to obtain pixel coordinates according to the damaged feature vector, and outputs a position feature according to the pixel coordinates; the DeepONet-NTK hybrid network constructs an inner product according to the damaged feature vector and the position feature:

[0118] ;

[0119] wherein, represents the inner product, represents the damaged feature vector, represents the position feature, and respectively represent the pixel abscissa and pixel ordinate of the damaged position; according to the inner product, reconstruct the target image and output the reconstructed image.

[0120] Among them, image reconstruction is realized according to the optimized and trained DeepONet-NTK hybrid network. In this embodiment, taking the Navier-Stokes inverse problem (the Navier-Stokes equation is the basic equation in fluid mechanics to describe fluid motion) as an example, the finally output reconstruction result strictly satisfies the hydrodynamics constraints:

[0121] ;

[0122] ;

[0123] wherein, represents the convective acceleration term (i.e., the nonlinear term), represents the fluid pressure scalar, represents the Laplacian operator of the velocity field (i.e., the viscous diffusion term), then represents the external volume force (such as gravity or electromagnetic force).

[0124] Furthermore, obtain all the target pixel values of the current image and all the reconstructed pixel values in the reconstructed image, and calculate the maximum reconstructed pixel value and the mean square error of the reconstructed pixels according to all the reconstructed pixel values; construct the peak signal-to-noise ratio according to the maximum reconstructed pixel value and the mean square error of the reconstructed pixels:

[0125] ;

[0126] wherein, represents the peak signal-to-noise ratio, represents the maximum value of the reconstructed pixels, represents the mean squared error of the reconstructed pixels; a structural similarity index is constructed based on all the target pixel values and all the reconstructed pixel values:

[0127] ;

[0128] wherein, represents the structural similarity index, represents the pixel mean value of the pixel abscissa in the current image and the reconstructed image, represents the pixel mean value of the pixel ordinate in the current image and the reconstructed image, represents the pixel variance of the pixel abscissa in the current image and the reconstructed image, represents the pixel variance of the pixel ordinate in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and both represent stability constants; the accuracy of the reconstructed image is judged according to the peak signal-to-noise ratio and the structural similarity index.

[0129] Specifically, in this embodiment, the PSNR (Peak Signal-to-Noise Ratio), SSIM (Structure Similarity Index Measure), and MSE (mean squared error) of multiple methods are evaluated on multiple datasets, and the experimental results are as shown in Figures 3 to 6 the results of:

[0130] wherein, VQVAE (Vector Quantized Variational Autoencoder) represents the vector quantization variational autoencoder; S-IntroVAE (Self-Supervised Introspective Variational Autoencoder) represents the self-supervised introspective variational autoencoder; DDPM (Denoising Diffusion Probabilistic Models) represents the denoising diffusion probability model; Beta-VAE-T (Beta refers to the hyperparameter that weighs the reconstruction loss and the KL divergence in the model, T represents the temperature) represents the beta variational autoencoder with temperature adjustment; Ours represents the DeepONet-NTK hybrid network used in this embodiment.

[0131] wherein, Figure 3Indicates the evaluation results on the CIFAR10 dataset (a classic image classification dataset in the field of computer vision, containing 60,000 color images of 32x32, divided into 10 categories, with 6,000 images in each category, 50,000 in the training set and 10,000 in the test set. These categories include common objects such as airplanes, cars, birds, cats, etc.); Figure 4 Indicates the evaluation results on the CIFAR100 dataset (a classic image classification dataset in the field of computer vision, containing 100 sub - classes (fine - grained labels fine_labels), divided into 20 super - classes (coarse - grained labels coarse_labels), with each super - class containing 5 sub - classes); Figure 5 Indicates the evaluation results on the MINST dataset (a classic handwritten digit classification dataset in the field of computer vision, containing 60,000 training images and 10,000 test images, a total of 70,000 handwritten digit samples); Figure 6 Indicates the evaluation results on the fashion - MINST dataset (a widely used image classification benchmark dataset in the field of computer vision, containing 60,000 training images and 10,000 test images, a total of 70,000 grayscale images of 28×28 pixels).

[0132] Furthermore, the high - performance results of the DeepONet - NTK hybrid network in this embodiment can be clearly felt through the following table:

[0133] Table 1: Evaluation Table of PSNR, SSIM and MSE Results of Multiple Methods for the First Dataset

[0134]

[0135] Table 2: Evaluation Table of PSNR, SSIM and MSE Results of Multiple Methods for the Second Dataset

[0136]

[0137] Table 3: Evaluation Table of PSNR, SSIM and MSE Results of Multiple Methods for the Third Dataset

[0138]

[0139] Table 4: Evaluation Table of PSNR, SSIM and MSE Results of Multiple Methods for the Fourth Dataset

[0140]

[0141] Furthermore, as Figure 7 shown, the reconstructed images output by the DeepONet - NTK hybrid network are presented, where, Figure 7Among them, (a) represents the reconstructed image in the MINST dataset, Figure 7 Among them, (b) represents the reconstructed image in the CIFAR-10 dataset, Figure 7 Among them, (c) represents the reconstructed image in the CIFAR-100 dataset, Figure 7 Among them, (d) represents the reconstructed image in the FashionMNIST dataset; among them, the first row shows the real image, the second row shows the damaged image, and the third row shows the reconstructed image generated by the proposed method; Figure 7 It shows the image reconstruction effect of the DeepONet-NTK hybrid network on different datasets, highlighting how the method proposed in this embodiment can restore the damaged input to be very similar to the real image, demonstrating its robustness and generalization ability on datasets with different complexities.

[0142] The present invention utilizes DeepONet to learn the non-linear mapping relationship in the multi-dimensional function space, and at the same time combines the NTK theory to optimize the training process, improve the convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining the high-dimensional semantic features.

[0143] Furthermore, as Figure 8 shown, based on the above image reconstruction method based on the hybrid network framework, the present invention also correspondingly provides an image reconstruction system based on the hybrid network framework, wherein, the image reconstruction system based on the hybrid network framework includes:

[0144] An information extraction module 51, configured to obtain multiple image data in a target image, input all the image data into the hybrid network framework, and output prediction mapping information.

[0145] A loss construction module 52, configured to construct a physical loss function according to the prediction mapping information, construct a data loss function according to the prediction result output by the hybrid network framework and the target image, construct a quality loss function according to the prediction result, and construct an NTK regularization term;

[0146] A model training module 53, configured to construct an objective loss function according to the physical loss function, the data loss function and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network;

[0147] An image reconstruction module 54, configured to input a current image into the DeepONet-NTK hybrid network, obtain the final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output a reconstructed image.

[0148] Furthermore, as Figure 9As shown, based on the above image reconstruction method and system based on a hybrid network framework, the present invention also correspondingly provides a terminal, which includes a processor 10, a memory 20, and a display 30. Figure 9 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0149] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In some other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as the program code for installing the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, an image reconstruction program 40 based on a hybrid network framework is stored on the memory 20, and this image reconstruction program 40 based on a hybrid network framework can be executed by the processor 10, thereby implementing the image reconstruction method based on a hybrid network framework in the present application.

[0150] In some embodiments, the processor 10 may be a Central Processing Unit (CPU), a microprocessor, or other data processing chips, and is used to run the program code stored in the memory 20 or process data, such as executing the image reconstruction method based on a hybrid network framework, etc.

[0151] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other through a system bus.

[0152] In one embodiment, when the processor 10 executes the image reconstruction program 40 based on a hybrid network framework in the memory 20, the steps of the above-mentioned image reconstruction method based on a hybrid network framework are implemented.

[0153] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image reconstruction program based on a hybrid network framework, and when the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the above-mentioned image reconstruction method based on the hybrid network framework are implemented.

[0154] In summary, the present invention provides an image reconstruction method and related devices based on a hybrid network framework. The method includes: obtaining a plurality of image data in a target image, inputting all the image data into the hybrid network framework, and outputting prediction mapping information and a prediction result; constructing a physical loss function according to the prediction mapping information, constructing a data loss function according to the prediction result and the target image, and constructing a quality loss function according to the prediction result; constructing an NTK regularization term, constructing a target loss function according to the physical loss function, the data loss function, and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; inputting a current image into the DeepONet-NTK hybrid network, obtaining a final field value prediction of the current image, and reconstructing the current image according to the pixel values in the final field value prediction, and outputting a reconstructed image. The present invention uses DeepONet to learn the non-linear mapping relationship in the multi-dimensional function space, and at the same time combines the NTK theory to optimize the training process, improve the convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining the high-dimensional semantic features.

[0155] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or terminal including that element.

[0156] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the above-mentioned method embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes of the above-mentioned method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0157] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or modifications can be made according to the above description, and all such improvements and modifications shall fall within the protection scope of the appended claims of the present invention.

Claims

1. An image reconstruction method based on a hybrid network framework, characterized in that, The image reconstruction method based on the hybrid network framework includes: Obtain multiple image data in the target image, input all the image data into the hybrid network framework, and output prediction mapping information and a prediction result; The image data includes: source parameters and observation point coordinates; The step of obtaining multiple image data in the target image, inputting all the image data into the hybrid network framework, and outputting prediction mapping information and a prediction result specifically includes: Obtain multiple source parameters and multiple observation point coordinates in the target image, and construct a hybrid network framework according to a deep operator network and a neural tangent kernel; Input all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework, and output prediction mapping information and a prediction result; The source parameters include source intensity and source position; The step of inputting all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework, and outputting prediction mapping information and a prediction result specifically includes: Input all the source intensities and all the source positions into a branch network in the deep operator network, and output multiple high-dimensional feature vectors; Input the observation point coordinates into a backbone network in the deep operator network, and output multiple feature vectors; Construct prediction mapping information and a prediction result according to the observation point coordinates, all the high-dimensional feature vectors, and all the feature vectors: ； ； ； Among them, represents the predicted mapping information of represents the coordinates of the observation point, represents the total number of source parameters, represents the -th high-dimensional feature vector of the source parameter, represents the feature vector, and respectively represent the abscissa and ordinate of represents the set of real numbers, represents the dimension of the space; Wherein, the prediction result represents the field value predicted by the hybrid network framework; Construct a physical loss function according to the prediction result, construct a data loss function according to the prediction result and the target image, and construct a quality loss function according to the prediction result; Construct an NTK regularization term, construct an objective loss function according to the physical loss function, the data loss function, and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; Wherein, the physical loss function is used to ensure that the hybrid network framework satisfies the momentum conservation equation, the data loss function is used to ensure that the result output by the hybrid network framework conforms to physical laws and retains high-dimensional semantic features, and the quality loss function is used to ensure that the hybrid network framework satisfies the energy conservation equation; Input the current image into the DeepONet-NTK hybrid network, obtain the final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output a reconstructed image.

2. The image reconstruction method based on a hybrid network framework according to claim 1, wherein The step of constructing a physical loss function according to the prediction result, constructing a data loss function according to the prediction result and the target image, and constructing a quality loss function according to the prediction result specifically includes: Construct a physical loss function according to the prediction result: ； Among them, represents the physical loss function, represents the prediction result predicted by the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, represents the total number of steps in the model simulation process, represents the divergence of the vector field; Construct a data loss function according to the prediction result output by the hybrid network framework and the target image, wherein the data loss function represents the matching degree between the prediction result output by the model and the observation result input to the model; ； Among them, represents the data loss function, and the total number of prediction results is N, represents the th original data of the target image, represents the th prediction result; Construct a quality loss function according to the prediction result: ； ； Among them, represents the quality loss function, represents the index of the total number of steps.

3. The image reconstruction method based on a hybrid network framework according to claim 1, wherein Construct the NTK regularization term, construct an objective loss function according to the physical loss function, the data loss function, and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network, specifically including: Construct the NTK regularization term: ； ； Among them, represents the NTK regularization term, represents the eigenvalues of the NTK matrix, represents the trace operation on the matrix performing the trace operation, represents the NTK matrix, and respectively represent the pixel points in the input NTK matrix, represents the parameters of the hybrid network framework, represents the transpose, represents the gradient of represents the gradient of; Construct an objective loss function according to the physical loss function, the data loss function, and the quality loss function: ； Among them, represents the target loss function, represents the data loss function; Adjust the eigenvalue of the NTK regularization term to the maximum to obtain the corresponding target learning rate: ； Among them, represents the target learning rate, represents the maximum eigenvalue; Input the objective loss function and the target learning rate into the hybrid network framework for training to obtain a DeepONet-NTK hybrid network.

4. The image reconstruction method based on the hybrid network framework according to claim 1, wherein Input the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output the reconstructed image, specifically including: Obtain the damaged image data of the current image input by the user, and input the damaged image data into the DeepONet-NTK hybrid network; The optimization branch network in the DeepONet-NTK hybrid network outputs the damaged feature vector of the damaged image data; Input the damaged feature vector into the optimization backbone network in the DeepONet-NTK hybrid network. The optimization backbone network encodes the damaged feature vector to obtain pixel coordinates and outputs position features according to the pixel coordinates; The DeepONet-NTK hybrid network constructs an inner product according to the damaged feature vector and the position feature; ； Among them, represents the inner product, represents the damaged feature vector, represents the position feature, and respectively represent the pixel abscissa and pixel ordinate of the damaged position; Reconstruct the current image according to the inner product and output the reconstructed image.

5. The image reconstruction method based on a hybrid network framework according to claim 1, wherein After inputting the current image into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and reconstructing the current image according to the pixel values in the final field value prediction, and outputting the reconstructed image, it further includes: Obtain all the target pixel values of the current image and all the reconstructed pixel values in the reconstructed image, and calculate the maximum reconstructed pixel value and the mean square error of the reconstructed pixels according to all the reconstructed pixel values; Construct a peak signal-to-noise ratio according to the maximum reconstructed pixel value and the mean square error of the reconstructed pixels; ； Among them, represents the peak signal-to-noise ratio, represents the maximum value of the reconstructed pixels, represents the mean square error of the reconstructed pixels; Construct a structural similarity index according to all the target pixel values and all the reconstructed pixel values; ； Among them, represents the structural similarity index, represents the pixel mean of the abscissa of the pixels in the current image and the reconstructed image, represents the pixel mean of the ordinate of the pixels in the current image and the reconstructed image, represents the pixel variance of the abscissa of the pixels in the current image and the reconstructed image, represents the pixel variance of the ordinate of the pixels in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and both represent the stability constant; Judge the accuracy of the reconstructed image according to the peak signal-to-noise ratio and the structural similarity index.

6. An image reconstruction system based on a hybrid network framework, characterized in that, The image reconstruction system based on the hybrid network framework is applied to the image reconstruction method based on the hybrid network framework according to any one of claims 1-5. The image reconstruction system based on the hybrid network framework includes: An information extraction module, configured to obtain a plurality of image data in a target image, input all the image data into the hybrid network framework, and output prediction mapping information and a prediction result; A loss construction module, configured to construct a physical loss function according to the prediction mapping information, construct a data loss function according to the prediction result and the target image, and construct a quality loss function according to the prediction result; A model training module, configured to construct an NTK regularization term, construct an objective loss function according to the physical loss function, the data loss function, and the quality loss function, and train the hybrid network framework according to the objective loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; An image reconstruction module, configured to input a current image into the DeepONet-NTK hybrid network, obtain a final field value prediction of the current image, and reconstruct the current image according to the pixel values in the final field value prediction, and output a reconstructed image.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and an image reconstruction program based on a hybrid network framework stored on the memory and executable on the processor. When the image reconstruction program based on the hybrid network framework is executed by the processor, the steps of the image reconstruction method based on the hybrid network framework according to any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image reconstruction program based on a hybrid network framework. When the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the image reconstruction method based on the hybrid network framework according to any one of claims 1-5 are implemented.

Images