Image reconstruction method and system based on hybrid network framework, terminal and storage medium

Abstract

The invention discloses an image reconstruction method and system based on a hybrid network framework, a terminal and a storage medium, and the method comprises the steps: carrying out the prediction of an original image, obtaining the prediction data, obtaining the quality loss and the physical loss of momentum conservation in a prediction process, comparing the prediction data with the original data, and obtaining a prediction result; obtaining a matching value between the two to construct a total loss function; and introducing an NTK matrix to draw up a proper learning rate, training a hybrid network framework by using the learning rate and a total loss function, and finally obtaining a DeepONet-NTK hybrid network. According to the method, DeepONet is used for learning a nonlinear mapping relation in a multi-dimensional function space, and the NTK theory is combined to optimize the training process and improve the convergence speed and stability, so that the signal-to-noise ratio of a reconstructed image is reduced, and high-dimensional semantic features are reserved.

Description

Technical Field

[0001] The present invention relates to the field of image reconstruction technology, and in particular to an image reconstruction method, system, terminal and computer-readable storage medium based on a hybrid network framework. Background Art

[0002] An inverse problem is a type of scientific problem that starts from observed results or known effects and infers the intrinsic causes, parameters or system properties that lead to the results.

[0003] Inverse problems have a wide range of applications in computational physics, medical imaging, and engineering, such as localizing the position of a source from scattered field data or reconstructing an image from noisy observations.

[0004] Traditional numerical methods (such as iterative algorithms and sampling algorithms) have high computational complexity for high-dimensional nonlinear problems, and rely on a large amount of high-quality data, and their performance is significantly reduced under sparse or noisy data. As for the current method of embedding partial differential equation constraints into the neural network loss function, it is easy to fall into local optimality during training, has slow convergence speed, is sensitive to noise, and has insufficient generalization ability under low signal-to-noise ratio data.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide an image reconstruction method, system, terminal and computer-readable storage medium based on a hybrid network framework, aiming to solve the problems of low computational efficiency, poor physical consistency, sensitivity to noise and sparse data, resulting in high signal-to-noise ratio and inaccuracy of the reconstructed image in the prior art when dealing with high-dimensional nonlinear inverse problems.

[0007] To achieve the above object, the present invention provides an image reconstruction method based on a hybrid network framework, and the image reconstruction method based on the hybrid network framework comprises the following steps: Acquire multiple image data in the target image, input all of the image data into the hybrid network framework, and output prediction mapping information and prediction results; constructing a physical loss function according to the predicted mapping information, constructing a data loss function according to the predicted result and the target image, and constructing a quality loss function according to the predicted result; Constructing an NTK regularization term, constructing a target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; The current image is input into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and the current image is reconstructed according to the pixel values ​​in the final field value prediction, and the reconstructed image is output.

[0008] Optionally, in the image reconstruction method based on the hybrid network framework, the image data includes: source parameters and observation point coordinates; The acquiring of multiple image data in the target image, inputting all the image data into the hybrid network framework, and outputting prediction mapping information and prediction results specifically includes: Acquire a plurality of said source parameters and a plurality of said observation point coordinates in the target image, and construct a hybrid network framework according to a deep operator network and a neural tangent kernel; All the source parameters and the observation point coordinates are input into the deep operator network in the hybrid network framework, and the prediction mapping information and the prediction result are output.

[0009] Optionally, in the image reconstruction method based on the hybrid network framework, the source parameters include source intensity and source position; The inputting of all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework and outputting the prediction mapping information and the prediction result specifically includes: Inputting all the source intensities and all the source positions into a branch network in the deep operator network, and outputting a plurality of high-dimensional feature vectors; Inputting the observation point coordinates into the backbone network in the deep operator network, and outputting a plurality of feature vectors; Construct prediction mapping information and prediction results according to the observation point coordinates, all the high-dimensional feature vectors and all the feature vectors: ; ; ; in, express The prediction mapping information, represents the coordinates of the observation point, Indicates the total number of source parameters, Indicates A high-dimensional feature vector of source parameters, represents the feature vector, and Respectively The horizontal and vertical coordinates of represents the set of real numbers, Represents the dimension of the space.

[0010] Optionally, the image reconstruction method based on the hybrid network framework, wherein the step of constructing a physical loss function according to the predicted mapping information, constructing a data loss function according to the predicted result and the target image, and constructing a quality loss function according to the predicted result, specifically includes: Construct a physical loss function based on the predicted mapping information: ; in, represents the physical loss function, represents the prediction result of the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, Represents the total number of steps in the model simulation process, represents the divergence of the vector field; A data loss function is constructed according to the prediction result output by the hybrid network framework and the target image, wherein the data loss function represents the matching degree between the prediction result output by the model and the observation result input by the model: ; in, represents the data loss function, Represents the total number of prediction results, The target image The original data, Indicates prediction results; Construct a quality loss function based on the prediction results: ; ; in, represents the quality loss function, An index indicating the total number of steps.

[0011] Optionally, the image reconstruction method based on the hybrid network framework, wherein the constructing of the NTK regularization term, constructing the target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain the DeepONet-NTK hybrid network, specifically includes: Construct the NTK regularization term: ; ; in, represents the NTK regularization term, represents the eigenvalue of the NTK matrix, Represents the matrix Perform trace operations, represents the NTK matrix, and Respectively represent the pixel points in the input NTK matrix, Represents the parameters of the hybrid network framework, represents transpose, express The gradient of express The gradient of Construct a target loss function according to the physical loss function, the data loss function and the quality loss function: ; in, represents the target loss function, represents the data loss function; The eigenvalue of the NTK regularization term is adjusted to the maximum limit to obtain the corresponding target learning rate: ; in, represents the target learning rate, represents the maximum eigenvalue; The target loss function and the target learning rate are input into the hybrid network framework for training to obtain a DeepONet-NTK hybrid network.

[0012] Optionally, the image reconstruction method based on the hybrid network framework, wherein the current image is input into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and the current image is reconstructed according to the pixel values ​​in the final field value prediction, and the reconstructed image is output, specifically comprising: Obtaining damaged image data of the current image input by the user, and inputting the damaged image data into the DeepONet-NTK hybrid network; The optimized branch network in the DeepONet-NTK hybrid network outputs a damaged feature vector of the damaged image data; Input the damaged feature vector into the optimized backbone network in the DeepONet-NTK hybrid network, the optimized backbone network encodes the damaged feature vector to obtain pixel coordinates, and outputs position features according to the pixel coordinates; The DeepONet-NTK hybrid network constructs the inner product based on the damaged feature vector and the position feature: ; in, represents the inner product, represents the broken feature vector, Represents location characteristics, and The pixel abscissa and pixel ordinate represent the damage position respectively; The target image is reconstructed according to the inner product, and a reconstructed image is output.

[0013] Optionally, the image reconstruction method based on the hybrid network framework, wherein the current image is input into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and the current image is reconstructed according to the pixel values ​​in the final field value prediction, and the reconstructed image is output, and then the method further includes: Acquire all target pixel values ​​of the current image and all reconstructed pixel values ​​of the reconstructed image, and calculate a reconstructed pixel maximum value and a reconstructed pixel mean square error according to all the reconstructed pixel values; The peak signal-to-noise ratio is constructed according to the reconstructed pixel maximum value and the reconstructed pixel mean square error: ; in, represents the peak signal-to-noise ratio, represents the maximum value of the reconstructed pixel, represents the mean square error of reconstructed pixels; Constructing a structural similarity index based on all the target pixel values ​​and all the reconstructed pixel values: ; in, represents the structural similarity index, represents the pixel mean of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel mean of the ordinate of the pixel in the current image and the reconstructed image, represents the pixel variance of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel variance of the pixel ordinate in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and All represent stability constants; The accuracy of the reconstructed image is determined according to the peak signal-to-noise ratio and the structural similarity index.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides an image reconstruction system based on a hybrid network framework, wherein the image reconstruction system based on the hybrid network framework comprises: An information extraction module, used for acquiring multiple image data in a target image, inputting all of the image data into a hybrid network framework, and outputting prediction mapping information and prediction results; A loss construction module, configured to construct a physical loss function according to the predicted mapping information, construct a data loss function according to the predicted result and the target image, and construct a quality loss function according to the predicted result; A model training module, used to construct an NTK regularization term, construct a target loss function according to the physical loss function, the data loss function and the quality loss function, and train the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; The image reconstruction module is used to input the current image into the DeepONet-NTK hybrid network, obtain the final field value prediction of the current image, reconstruct the current image according to the pixel values ​​in the final field value prediction, and output the reconstructed image.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image reconstruction program based on a hybrid network framework stored in the memory and executable on the processor, wherein the image reconstruction program based on the hybrid network framework, when executed by the processor, implements the steps of the image reconstruction method based on the hybrid network framework as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image reconstruction program based on a hybrid network framework, and when the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the image reconstruction method based on the hybrid network framework as described above are implemented.

[0017] In the present invention, multiple image data in the target image are obtained, all the image data are input into the hybrid network framework, and prediction mapping information and prediction results are output; a physical loss function is constructed according to the prediction mapping information, a data loss function is constructed according to the prediction result and the target image, and a quality loss function is constructed according to the prediction result; an NTK regularization term is constructed, a target loss function is constructed according to the physical loss function, the data loss function and the quality loss function, and the hybrid network framework is trained according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; the current image is input into the DeepONet-NTK hybrid network to obtain the final field value prediction of the current image, and the current image is reconstructed according to the pixel values ​​in the final field value prediction, and the reconstructed image is output. The present invention uses DeepONet to learn nonlinear mapping relationships in a multidimensional function space, and combines NTK theory to optimize the training process, improve convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining high-dimensional semantic features. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 2 Schematic diagram of a network framework of a preferred embodiment of an image reconstruction method based on a hybrid network framework of the present invention; Figure 3 It is a schematic diagram of PSNR and SSIM results of the first data set of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 4 It is a schematic diagram of PSNR and SSIM results of the second data set of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 5 It is a schematic diagram of PSNR and SSIM results of the third data set of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 6 It is a schematic diagram of PSNR and SSIM results of the fourth data set of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 7 It is a schematic diagram of the image reconstruction result of a preferred embodiment of the image reconstruction method based on the hybrid network framework of the present invention; Figure 8 It is a structural diagram of a preferred embodiment of the image reconstruction system based on a hybrid network framework of the present invention; Fig. 9 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] The image reconstruction method based on the hybrid network framework described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the image reconstruction method based on the hybrid network framework includes the following steps: Step S10: Acquire multiple image data in the target image, input all of the image data into the hybrid network framework, and output prediction mapping information and prediction results.

[0021] The image data includes: source parameters and observation point coordinates. The source parameters include source intensity and source position. The source intensity represents the energy or intensity distribution of the point source of the pixel point in space, and the source position represents the coordinates of the point source of the pixel point in space.

[0022] Specifically, a plurality of the source parameters and a plurality of the observation point coordinates in the target image are obtained, and a hybrid network framework is constructed according to a deep operator network and a neural tangent kernel; all the source parameters and the observation point coordinates are input into the deep operator network in the hybrid network framework, and prediction mapping information and prediction results are output.

[0023] The image data of the target image is input into Figure 2 In the hybrid network framework shown, the operator mapping of DeepONet (Deep Operator Network) is used to learn the nonlinear mapping relationship in the infinite-dimensional function space, so that in fluid dynamics problems, the generated field satisfies the constraints of the physical equations, and the training efficiency and stability are significantly improved. This is to solve the problem in the existing technology that data-driven reconstruction or prediction is achieved using generative models (such as variational autoencoders and diffusion models), which leads to a lack of physical constraints and the reconstruction results may violate physical laws.

[0024] Further, all the source intensities and all the source positions are input into the branch network in the deep operator network, and a plurality of high-dimensional feature vectors are output; the observation point coordinates (i.e. Figure 2 In and ) is input into the backbone network in the deep operator network, and the feature vector (i.e. Figure 2 In and ); construct prediction mapping information and prediction results according to the observation point coordinates, all the high-dimensional feature vectors and all the feature vectors: ; in, express The prediction mapping information, represents the coordinates of the observation point, Indicates the total number of source parameters, Indicates A high-dimensional feature vector of source parameters, represents a feature vector.

[0025] Among them, the source parameters are encoded into high-dimensional feature vectors through the branch network in the hybrid network for subsequent field value prediction. At the same time, combined with the observation point coordinates processed by the backbone network, the initial inner product of the target image (i.e., the prediction mapping information) is constructed to realize the continuous function mapping from source parameters to spatial fields, so that the model's predicted results in the Navier-Stokes source localization task have smaller errors, thereby improving the accuracy of the model's image reconstruction.

[0026] Step S20: construct a physical loss function according to the predicted mapping information, construct a data loss function according to the predicted result and the target image, and construct a quality loss function according to the predicted result.

[0027] Among them, by introducing NTK (Neural Tangent Kernel), the model convergence speed can be improved, and at the same time the stability of the training process can be improved, and the problem of the model training process falling into the local optimal state can be avoided; and the introduced NTK theory can effectively solve the problems in the existing technology that the independent use of DeepONet leads to dynamic instability of model training, the convergence speed is greatly affected by parameter initialization, the robustness to sparse or noisy data is insufficient, overfitting is easy to occur, and there is a lack of theoretical guarantee, and it is difficult to explain the generalization behavior during the training process.

[0028] Specifically, a physical loss function is constructed according to the prediction mapping information: ; in, represents the physical loss function, represents the prediction result of the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, Represents the total number of steps in the model simulation process, Represents the divergence of the vector field; constructs a data loss function according to the prediction result output by the hybrid network framework and the target image, wherein the data loss function represents the matching degree between the prediction result output by the model and the observation result input by the model: ; in, represents the data loss function, Represents the total number of prediction results, The target image The original data, Indicates prediction results; Construct a quality loss function based on the prediction results: ; ; in, represents the quality loss function, An index indicating the total number of steps.

[0029] Among them, during the training process, the physical loss function can ensure that the model is forced to satisfy the momentum conservation equation, and the mass loss function can ensure that the model is forced to satisfy the mass conservation equation. This process can effectively solve the problem of lack of physical constraints caused by the use of traditional deep learning methods (such as VAE, Variational Autoencoder, Variational Autoencoder), and the reconstruction results may violate the laws of physics.

[0030] Furthermore, combined with the data loss function that represents the matching relationship between the predicted value and the observed value, it ensures that the output of the model conforms to the physical laws and retains high-dimensional semantic features, thereby improving the accuracy and stability when solving inverse problems.

[0031] Step S30, constructing an NTK regularization term, constructing a target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network.

[0032] Specifically, construct the NTK regularization term: ; ; in, represents the NTK regularization term, represents the eigenvalue of the NTK matrix, Represents the matrix Perform trace operations, represents the NTK matrix, and Respectively represent the pixel points in the input NTK matrix, Represents the parameters of the hybrid network framework, represents transpose, express The gradient of express The gradient of ; constructing the target loss function according to the physical loss function, the data loss function and the quality loss function: ; in, represents the target loss function, Represents the data loss function; the eigenvalue of the NTK regularization term is adjusted to the maximum limit to obtain the corresponding target learning rate: ; in, represents the target learning rate, Represents the maximum eigenvalue; inputting the target loss function and the target learning rate into the hybrid network framework for training to obtain a DeepONet-NTK hybrid network.

[0033] Among them, on the basis of DeepONet network, NTK regularization is introduced. First, the NTK matrix is ​​constructed, and then the regularization term is constructed to dynamically adjust the learning rate so that the NTK matrix reaches the maximum eigenvalue and avoids the gradient exceeding the limit.

[0034] Furthermore, when constructing NTK regularization, we can also calculate the gradient alignment term (such as Figure 2 As shown, That is to say , represents the model parameters after training) , calculate the regularization term loss, where and are points that are input to the model. Indicate point The gradient value at the point where the gradient is calculated; and then the objective loss function is minimized to achieve the embedding of feature interaction and physical constraints, which can effectively improve the convergence speed of model training, reduce the sensitivity of the model to parameter initialization, and avoid the model falling into the local optimum.

[0035] Step S40: input the current image into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, reconstruct the current image according to the pixel values ​​in the final field value prediction, and output the reconstructed image.

[0036] Specifically, the damaged image data of the current image input by the user is obtained, and the damaged image data is input into the DeepONet-NTK hybrid network; the optimized branch network in the DeepONet-NTK hybrid network outputs a damaged feature vector of the damaged image data; the damaged feature vector is input into the optimized backbone network in the DeepONet-NTK hybrid network, the optimized backbone network encodes the damaged feature vector to obtain pixel coordinates, and outputs position features according to the pixel coordinates; the DeepONet-NTK hybrid network constructs an inner product according to the damaged feature vector and the position feature: ; in, represents the inner product, represents the broken feature vector, Represents location characteristics, and Respectively represent the pixel horizontal coordinate and the pixel vertical coordinate of the damaged position; reconstruct the target image according to the inner product and output the reconstructed image.

[0037] Among them, image reconstruction is achieved according to the optimized and trained DeepONet-NTK hybrid network. In this embodiment, taking the Navier-Stokes inverse problem (the Navier-Stokes equation is a basic equation in fluid mechanics that describes fluid motion) as an example, the reconstruction result finally output strictly satisfies the fluid dynamics constraints: ; ; in, represents the convection acceleration term (i.e., nonlinear term), represents the fluid pressure scalar, represents the Laplace operator of the velocity field (i.e., the viscous diffusion term), represents an external body force (such as gravity or electromagnetic force).

[0038] Furthermore, all target pixel values ​​of the current image and all reconstructed pixel values ​​in the reconstructed image are obtained, and a maximum reconstructed pixel value and a mean square error of the reconstructed pixel value are calculated according to all the reconstructed pixel values; a peak signal-to-noise ratio is constructed according to the maximum reconstructed pixel value and the mean square error of the reconstructed pixel: ; in, represents the peak signal-to-noise ratio, represents the maximum value of the reconstructed pixel, Represents the reconstructed pixel mean square error; constructs a structural similarity index based on all the target pixel values ​​and all the reconstructed pixel values: ; in, represents the structural similarity index, represents the pixel mean of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel mean of the ordinate of the pixel in the current image and the reconstructed image, represents the pixel variance of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel variance of the pixel ordinate in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and Both represent stability constants; the accuracy of the reconstructed image is judged according to the peak signal-to-noise ratio and the structural similarity index.

[0039] Specifically, this embodiment evaluates the PSNR (Peak Signal-to-Noise Ratio), SSIM (Structure Similarity Index Measure), and MSE (Mean Square Error) of various methods on multiple data sets. The experimental results are as follows: Figures 3 to 6 The results are shown: Among them, VQVAE (Vector Quantized Variational Autoencoder) represents vector quantized variational autoencoder; S-IntroVAE (Self-Supervised Introspective Variational Autoencoder) represents self-supervised variational autoencoder; DDPM (Denoising Diffusion Probabilistic Models) represents denoising diffusion probability model; Beta-VAE-T (Beta refers to the hyperparameter for the trade-off between reconstruction loss and KL divergence in the model, and T represents temperature) represents beta variational autoencoder with temperature adjustment; Ours represents the DeepONet-NTK hybrid network used in this embodiment. ‌‌‌

[0040] in, Figure 3 Indicates the evaluation results on the CIFAR10 dataset (a classic image classification dataset in the field of computer vision, containing 60,000 32x32 color images, divided into 10 categories, each with 6,000 images, 50,000 training sets, and 10,000 test sets. These categories include common objects such as airplanes, cars, birds, and cats); Figure 4Indicates the evaluation results on the CIFAR100 dataset (a classic image classification dataset in the field of computer vision, containing 100 small categories (fine-grained labels fine_labels), divided into 20 large categories (coarse-grained labels coarse_labels), each large category contains 5 small categories); Figure 5 Represents the evaluation results on the MINST dataset (a classic handwritten digit classification dataset in the field of computer vision, containing 60,000 training images and 10,000 test images, a total of 70,000 handwritten digit samples); Figure 6 Represents the evaluation results on the fashion-MINST dataset (a widely used image classification benchmark dataset in the field of computer vision, containing 60,000 training images and 10,000 test images, totaling 70,000 28×28 pixel grayscale images).

[0041] Furthermore, the high performance results of the DeepONet-NTK hybrid network in this embodiment can be clearly felt through the following table: Table 1: Multi-method PSNR, SSIM and MSE results evaluation table of the first dataset

[0042] Table 2: Multi-method PSNR, SSIM and MSE results evaluation table of the second dataset

[0043] Table 3: Evaluation table of multi-method PSNR, SSIM and MSE results of the third dataset

[0044] Table 4: Multi-method PSNR, SSIM and MSE result evaluation table of the fourth data set

[0045] Furthermore, if Figure 7 As shown, the reconstructed image output by the DeepONet-NTK hybrid network is shown, where Figure 7 (a) in the figure represents the reconstructed image in the MINST dataset. Figure 7 (b) in the figure represents the reconstructed image in the CIFAR-10 dataset. Figure 7 (c) in the figure represents the reconstructed image in the CIFAR-100 dataset. Figure 7 (d) in the figure represents the reconstructed images in the FashionMNIST dataset; the first row shows the real images, the second row shows the damaged images, and the third row shows the reconstructed images generated by the proposed method; Figure 7The image reconstruction effect of the DeepONet-NTK hybrid network on different datasets is demonstrated, highlighting how the method proposed in this embodiment can restore the damaged input to be very similar to the real image, demonstrating its robustness and generalization ability on datasets of different complexity.

[0046] The present invention utilizes DeepONet to learn the nonlinear mapping relationship in the multidimensional function space, and combines the NTK theory to optimize the training process, improve the convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining the high-dimensional semantic features.

[0047] Furthermore, if Figure 8 As shown, based on the above-mentioned image reconstruction method based on the hybrid network framework, the present invention also provides an image reconstruction system based on the hybrid network framework, wherein the image reconstruction system based on the hybrid network framework includes: The information extraction module 51 is used to obtain multiple image data in the target image, input all the image data into the hybrid network framework, and output prediction mapping information.

[0048] A loss construction module 52, configured to construct a physical loss function according to the prediction mapping information, construct a data loss function according to the prediction result output by the hybrid network framework and the target image, construct a quality loss function according to the prediction result, and construct an NTK regularization term; A model training module 53 is used to construct a target loss function according to the physical loss function, the data loss function and the quality loss function, and train the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; The image reconstruction module 54 is used to input the current image into the DeepONet-NTK hybrid network, obtain the final field value prediction of the current image, reconstruct the current image according to the pixel values ​​in the final field value prediction, and output the reconstructed image.

[0049] Furthermore, if Fig. 9 As shown, based on the above-mentioned image reconstruction method and system based on the hybrid network framework, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig. 9 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0050] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image reconstruction program 40 based on a hybrid network framework is stored on the memory 20, and the image reconstruction program 40 based on the hybrid network framework can be executed by the processor 10, thereby realizing the image reconstruction method based on the hybrid network framework in the present application.

[0051] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image reconstruction method based on the hybrid network framework.

[0052] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0053] In one embodiment, when the processor 10 executes the image reconstruction program 40 based on the hybrid network framework in the memory 20, the steps of the image reconstruction method based on the hybrid network framework as described above are implemented.

[0054] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image reconstruction program based on a hybrid network framework, and when the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the image reconstruction method based on the hybrid network framework as described above are implemented.

[0055] In summary, the present invention provides an image reconstruction method and related equipment based on a hybrid network framework, the method comprising: obtaining multiple image data in a target image, inputting all the image data into a hybrid network framework, outputting prediction mapping information and prediction results; constructing a physical loss function according to the prediction mapping information, constructing a data loss function according to the prediction results and the target image, and constructing a quality loss function according to the prediction results; constructing an NTK regularization term, constructing a target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; inputting a current image into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and reconstructing the current image according to the pixel values ​​in the final field value prediction, and outputting a reconstructed image. The present invention utilizes DeepONet to learn nonlinear mapping relationships in a multidimensional function space, and combines NTK theory to optimize the training process, improve convergence speed and stability, thereby reducing the signal-to-noise ratio of the reconstructed image and retaining high-dimensional semantic features.

[0056] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0057] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0058] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image reconstruction method based on a hybrid network framework, characterized in that: The image reconstruction method based on the hybrid network framework includes: Acquire multiple image data in the target image, input all of the image data into the hybrid network framework, and output prediction mapping information and prediction results; constructing a physical loss function according to the predicted mapping information, constructing a data loss function according to the predicted result and the target image, and constructing a quality loss function according to the predicted result; Constructing an NTK regularization term, constructing a target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; The current image is input into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, and the current image is reconstructed according to the pixel values ​​in the final field value prediction, and the reconstructed image is output.

2. The image reconstruction method based on the hybrid network framework according to claim 1, characterized in that: The image data includes: source parameters and observation point coordinates; The acquiring of multiple image data in the target image, inputting all the image data into the hybrid network framework, and outputting prediction mapping information and prediction results specifically includes: Acquire a plurality of said source parameters and a plurality of said observation point coordinates in the target image, and construct a hybrid network framework according to a deep operator network and a neural tangent kernel; All the source parameters and the observation point coordinates are input into the deep operator network in the hybrid network framework, and the prediction mapping information and the prediction result are output.

3. The image reconstruction method based on the hybrid network framework according to claim 2 is characterized in that: The source parameters include source intensity and source position; The inputting of all the source parameters and the observation point coordinates into the deep operator network in the hybrid network framework and outputting the prediction mapping information and the prediction result specifically includes: Inputting all the source intensities and all the source positions into a branch network in the deep operator network, and outputting a plurality of high-dimensional feature vectors; Inputting the observation point coordinates into the backbone network in the deep operator network, and outputting a plurality of feature vectors; Construct prediction mapping information and prediction results according to the observation point coordinates, all the high-dimensional feature vectors and all the feature vectors: ； ； ； in, express The prediction mapping information, represents the coordinates of the observation point, Indicates the total number of source parameters, Indicates A high-dimensional feature vector of source parameters, represents the feature vector, and Respectively The horizontal and vertical coordinates of represents the set of real numbers, Represents the dimension of the space.

4. The image reconstruction method based on the hybrid network framework according to claim 3 is characterized in that: The step of constructing a physical loss function according to the predicted mapping information, constructing a data loss function according to the predicted result and the target image, and constructing a quality loss function according to the predicted result specifically includes: Construct a physical loss function based on the predicted mapping information: ； in, represents the physical loss function, represents the prediction result of the hybrid network framework, represents the pressure field predicted by the model, represents the kinematic viscosity predicted by the model, represents the external source term, Represents the total number of steps in the model simulation process, represents the divergence of the vector field; A data loss function is constructed according to the prediction result output by the hybrid network framework and the target image, wherein the data loss function represents the matching degree between the prediction result output by the model and the observation result input by the model: ； in, represents the data loss function, Represents the total number of prediction results, The target image The original data, Indicates prediction results; Construct a quality loss function based on the prediction results: ； ； in, represents the quality loss function, An index indicating the total number of steps.

5. The image reconstruction method based on the hybrid network framework according to claim 4 is characterized in that: The constructing of the NTK regularization term, constructing the target loss function according to the physical loss function, the data loss function and the quality loss function, and training the hybrid network framework according to the target loss function and the NTK regularization term to obtain the DeepONet-NTK hybrid network specifically includes: Construct the NTK regularization term: ； ； in, represents the NTK regularization term, represents the eigenvalue of the NTK matrix, Represents the matrix Perform trace operations, represents the NTK matrix, and Respectively represent the pixel points in the input NTK matrix, Represents the parameters of the hybrid network framework, represents transpose, express The gradient of express The gradient of Construct a target loss function according to the physical loss function, the data loss function and the quality loss function: ； in, represents the target loss function, represents the data loss function; The eigenvalue of the NTK regularization term is adjusted to the maximum limit to obtain the corresponding target learning rate: ； in, represents the target learning rate, represents the maximum eigenvalue; The target loss function and the target learning rate are input into the hybrid network framework for training to obtain a DeepONet-NTK hybrid network.

6. The image reconstruction method based on the hybrid network framework according to claim 2, characterized in that: The step of inputting the current image into the DeepONet-NTK hybrid network, obtaining a final field value prediction of the current image, reconstructing the current image according to pixel values ​​in the final field value prediction, and outputting a reconstructed image specifically includes: Obtaining damaged image data of the current image input by the user, and inputting the damaged image data into the DeepONet-NTK hybrid network; The optimized branch network in the DeepONet-NTK hybrid network outputs a damaged feature vector of the damaged image data; Input the damaged feature vector into the optimized backbone network in the DeepONet-NTK hybrid network, the optimized backbone network encodes the damaged feature vector to obtain pixel coordinates, and outputs position features according to the pixel coordinates; The DeepONet-NTK hybrid network constructs the inner product based on the damaged feature vector and the position feature: ； in, represents the inner product, represents the broken feature vector, Represents location characteristics, and The pixel abscissa and pixel ordinate represent the damage position respectively; The current image is reconstructed according to the inner product, and a reconstructed image is output.

7. The image reconstruction method based on the hybrid network framework according to claim 1, characterized in that: The step of inputting the current image into the DeepONet-NTK hybrid network to obtain a final field value prediction of the current image, reconstructing the current image according to the pixel values ​​in the final field value prediction, and outputting the reconstructed image further comprises: Acquire all target pixel values ​​of the current image and all reconstructed pixel values ​​of the reconstructed image, and calculate a reconstructed pixel maximum value and a reconstructed pixel mean square error according to all the reconstructed pixel values; The peak signal-to-noise ratio is constructed according to the reconstructed pixel maximum value and the reconstructed pixel mean square error: ； in, represents the peak signal-to-noise ratio, represents the maximum value of the reconstructed pixel, represents the mean square error of reconstructed pixels; Constructing a structural similarity index based on all the target pixel values ​​and all the reconstructed pixel values: ； in, represents the structural similarity index, represents the pixel mean of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel mean of the ordinate of the pixel in the current image and the reconstructed image, represents the pixel variance of the horizontal coordinates of the pixels in the current image and the reconstructed image, represents the pixel variance of the pixel ordinate in the current image and the reconstructed image, represents the pixel covariance of the current image and the reconstructed image, and All represent stability constants; The accuracy of the reconstructed image is determined according to the peak signal-to-noise ratio and the structural similarity index.

8. An image reconstruction system based on a hybrid network framework, characterized in that: The image reconstruction system based on the hybrid network framework is applied to the image reconstruction method based on the hybrid network framework according to any one of claims 1 to 7, and the image reconstruction system based on the hybrid network framework comprises: An information extraction module, used for acquiring multiple image data in a target image, inputting all of the image data into a hybrid network framework, and outputting prediction mapping information and prediction results; A loss construction module, configured to construct a physical loss function according to the predicted mapping information, construct a data loss function according to the predicted result and the target image, and construct a quality loss function according to the predicted result; A model training module, used to construct an NTK regularization term, construct a target loss function according to the physical loss function, the data loss function and the quality loss function, and train the hybrid network framework according to the target loss function and the NTK regularization term to obtain a DeepONet-NTK hybrid network; The image reconstruction module is used to input the current image into the DeepONet-NTK hybrid network, obtain the final field value prediction of the current image, reconstruct the current image according to the pixel values ​​in the final field value prediction, and output the reconstructed image.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an image reconstruction program based on a hybrid network framework stored in the memory and executable on the processor. When the image reconstruction program based on the hybrid network framework is executed by the processor, the steps of the image reconstruction method based on the hybrid network framework as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image reconstruction program based on a hybrid network framework, and when the image reconstruction program based on the hybrid network framework is executed by a processor, the steps of the image reconstruction method based on the hybrid network framework as described in any one of claims 1 to 7 are implemented.

Images