A medical spectrum reconstruction method and device

Abstract

The present invention relates to the field of image processing technology, and in particular to a method and device for medical spectrum reconstruction, the method comprising: firstly, performing dimensionality reduction and feature extraction processing on an original hyperspectral image, and generating a first image space; then adding forward noise to the first image space to generate a noisy image, and using the noisy image and the first RGB image as joint training data, inputting the noisy image into a residual denoising network model to be trained for training; then, based on the trained residual denoising network model, and in combination with the spectral information of a second RGB image, performing reverse iterative denoising processing on a target noise image to generate a second image space, and finally performing data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image. The present invention can better simulate the distribution and changes of hyperspectral data, achieve resolution balance, and improve the accuracy and reliability of reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a medical spectrum reconstruction method and device. Background Art

[0002] Hyperspectral imaging (HSI) is a novel optical imaging technology with unique spectral resolution capabilities. It can generate image data containing rich spectral information by capturing the reflection or emission characteristics of an object in a continuous spectral range.

[0003] However, although hyperspectral imaging technology has broad application prospects in the medical field, there are still some practical bottlenecks that need to be broken through. First, although snapshot hyperspectral imaging technology can obtain information from all spectral bands at the same time, it relies on expensive hardware equipment, and due to technical limitations, it is often difficult to meet the requirements of spatial and spectral resolution at the same time. Secondly, although push-broom hyperspectral imaging technology can provide higher spectral resolution, it sacrifices temporal resolution and may not be able to obtain key information in a timely manner when rapid detection or real-time monitoring is required. In addition, push-broom hyperspectral imaging technology has strict requirements on equipment and application scenarios, requiring absolute stillness, which greatly limits its application in real-time detection of dynamic organs during surgery. Summary of the invention

[0004] In view of this, the present invention provides a medical spectral reconstruction method and device to solve the problem of insufficient accuracy and reliability of hyperspectral image reconstruction in snapshot hyperspectral imaging technology and push-broom hyperspectral imaging technology.

[0005] In a first aspect, the present invention provides a medical spectrum reconstruction method, the method comprising:

[0006] Obtaining original hyperspectral images;

[0007] Performing dimension reduction and feature extraction processing on the original hyperspectral image to generate a first image space;

[0008] Adding forward noise to the first image space to generate a noisy image;

[0009] The noisy image and the first RGB image are input as joint training data into a residual denoising network model to be trained for training, and the trained residual denoising network model is obtained;

[0010] Based on the trained residual denoising network model and in combination with the spectral information of the second RGB image, a reverse iterative denoising process is performed on the target noise image to generate a second image space;

[0011] Data reconstruction processing is performed on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image.

[0012] In an optional implementation, the performing dimensionality reduction and feature extraction processing on the original hyperspectral image and generating a first image space includes:

[0013] Calculating a covariance matrix corresponding to the original hyperspectral image, and performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors corresponding to the eigenvalues;

[0014] According to the size of the eigenvalue, extract the target number of eigenvectors to construct a transformation matrix to extract the principal component information;

[0015] Based on the transformation matrix, projecting the original hyperspectral image to the direction of the principal component information, and obtaining a data representation after the projection;

[0016] Based on the projected data representation, the first image space is constructed; each pixel point in the first image space is represented by the principal component information.

[0017] In an optional implementation, adding forward noise to the first image space to generate a noisy image includes:

[0018] Acquire forward noise; the forward noise has the same dimension as the first image space;

[0019] The forward noise is added element-wise to the principal component representation of each pixel in the first image space to generate the noisy image with noise.

[0020] In an optional implementation, the step of taking the noisy image and the first RGB image as joint training data and inputting them into a residual denoising network model to be trained for training includes:

[0021] Acquire a first RGB image spatially aligned with the original hyperspectral image;

[0022] Jointly processing the noisy image and the first RGB image to obtain the joint training data;

[0023] Inputting the joint training data into the residual denoising network model to be trained to perform nonlinear activation function processing, and obtaining a nonlinear activation function processing result;

[0024] Extracting local features of the processing result of the nonlinear activation function to obtain spatial structure information;

[0025] Embed the time feature as an additional channel into the extracted local feature, and perform feature extraction again on the embedding result;

[0026] Adding the extracted local features to the result of feature extraction again to construct a residual connection;

[0027] The residual connection result is input into the attention module of the residual denoising network model to be trained for weighted processing, and a predicted noise image is output.

[0028] In an optional embodiment, the method further includes:

[0029] During training, regularization is performed on the model parameters in the residual denoising network model in training.

[0030] In an optional implementation, the target noise image is subjected to reverse iterative denoising based on the trained residual denoising network model and combined with spectral information of the second RGB image to generate a second image space, including:

[0031] Acquire a target noise image to be processed and a second RGB image spatially aligned with the target noise image;

[0032] Based on the target number of iterations, the target noise image is used as the initial input for reverse iterative denoising, and the second image space finally generated after the iteration is obtained. The following steps are performed in each iteration:

[0033] Inputting the current noise image into the trained residual denoising network model, and combining the spectral information of the second RGB image, outputting the predicted noise corresponding to the current noise image;

[0034] The current noise image is updated according to the predicted noise corresponding to the current noise image, and an updated noise image is obtained.

[0035] In an optional implementation, performing data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image includes:

[0036] Obtaining an inverse transformation matrix corresponding to the transformation matrix;

[0037] Extracting data representation from the second image space, and restoring the data representation in the second image space through the inverse transformation matrix to obtain a reconstructed hyperspectral image data representation;

[0038] Based on the reconstructed hyperspectral image data representation, a reconstructed hyperspectral image corresponding to the target noise image is obtained.

[0039] In a second aspect, the present invention provides a medical spectrum reconstruction device, the device comprising:

[0040] An original hyperspectral image acquisition module is used to acquire an original hyperspectral image;

[0041] A first image space acquisition module, used for performing dimension reduction and feature extraction processing on the original hyperspectral image and generating a first image space;

[0042] A noisy image acquisition module, used for adding forward noise to the first image space to generate a noisy image;

[0043] A spectral reconstruction training module, used for inputting the noisy image and the first RGB image as joint training data into a residual denoising network model to be trained for training, and obtaining the residual denoising network model after training;

[0044] A spectral reconstruction inference module, used to perform reverse iterative denoising processing on the target noise image based on the trained residual denoising network model and in combination with the spectral information of the second RGB image to generate a second image space;

[0045] The reconstructed hyperspectral image acquisition module is used to perform data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image.

[0046] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute a medical spectrum reconstruction method according to the first aspect or any corresponding embodiment thereof.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a medical spectrum reconstruction method according to the first aspect or any corresponding embodiment thereof.

[0048] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute a medical spectrum reconstruction method according to the first aspect or any corresponding embodiment thereof.

[0049] The technical solution provided by the present invention may include the following beneficial effects:

[0050] The present invention firstly adopts the dimension reduction and feature extraction processing mode for the original hyperspectral image, which can effectively reduce the computational complexity and the demand for computing resources of the model during the training process, so that the reconstruction training of hyperspectral data can be carried out under limited computing resources, simplifying the training process and helping to shorten the training time. In addition, the present invention introduces the residual denoising network model (ARDNet) containing the attention mechanism during the diffusion model training, which can more accurately extract the key features and details of the image, help to improve the image quality of the reconstructed image, and can provide a clearer and more accurate hyperspectral image, so that users can better observe and analyze the spectral features. In addition, the present invention applies the diffusion model to the hyperspectral data reconstruction task, first defines a process of gradually transforming from data distribution to noise distribution, and then uses the residual denoising network (ARDNet) containing the attention mechanism to learn how to restore the original hyperspectral image from the noise data, and gradually generates a clear and complete hyperspectral image from the noise data through iterative denoising. The application of the diffusion model can better simulate the distribution and change of hyperspectral data, achieve resolution balance, and improve the accuracy and reliability of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 is a flow chart of a medical spectrum reconstruction method according to an embodiment of the present invention;

[0053] Figure 2 is a flow chart of another medical spectrum reconstruction method according to an embodiment of the present invention;

[0054] Figure 3 is a flow chart of spectral reconstruction training according to an embodiment of the present invention;

[0055] Figure 4 is a flow chart of spectral reconstruction reasoning according to an embodiment of the present invention;

[0056] Figure 5 is a calculation flow chart of a residual denoising network model according to an embodiment of the present invention;

[0057] Figure 6 is a structural block diagram of a medical spectrum reconstruction device according to an embodiment of the present invention;

[0058] Figure 7It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] According to an embodiment of the present invention, an embodiment of a medical spectral reconstruction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0061] In this embodiment, a medical spectrum reconstruction method is provided. Figure 1 is a flow chart of a medical spectrum reconstruction method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0062] Step S101, obtaining an original hyperspectral image.

[0063] Furthermore, in this embodiment, an original hyperspectral image for subsequent processing is first obtained through a specific medical imaging device or from an existing data set. The hyperspectral image contains rich spectral information.

[0064] Step S102: performing dimension reduction and feature extraction processing on the original hyperspectral image to generate a first image space.

[0065] Furthermore, this embodiment can project the high-dimensional data of the original hyperspectral image into a lower-dimensional principal component space through dimensionality reduction and feature extraction, while retaining the main features of the data as much as possible. In this step, this embodiment performs dimensionality reduction and feature extraction on the original hyperspectral image to extract the main components in the image and generate a lower-dimensional first image space. The image data in the first image space is more concise than the original hyperspectral image, which is conducive to subsequent noise addition and denoising processing.

[0066] Step S103: adding forward noise to the first image space to generate a noisy image.

[0067] Furthermore, in order to simulate the noise environment, this embodiment adds a certain degree of forward noise to the image in the first image space. The forward noise may be Gaussian noise, etc., and its specific type depends on the actual application scenario and noise characteristics. The image after adding the forward noise forms a noisy image, which will be used for the subsequent training of the residual denoising network model.

[0068] Step S104: The noisy image and the first RGB image are used as joint training data and input into a residual denoising network model to be trained for training, and the trained residual denoising network model is obtained.

[0069] Furthermore, this embodiment uses the noisy image and the first RGB image corresponding thereto (the first RGB image is closely related to the original hyperspectral image in time or space) as joint training data, and the joint training data is input into the residual denoising network model to be trained. The residual denoising network model analyzes the joint training data, identifies the characteristics and patterns of the noise, and attempts to learn the noise distribution in the noisy image. After training, the residual denoising network model outputs a prediction of the noise in the noisy image. The predicted noise is an estimate generated by the model based on the learned noise distribution and the characteristics of the input image. By subtracting the predicted noise from the noisy image, the noise can be gradually removed and a signal close to the original hyperspectral image can be restored.

[0070] Step S105, based on the trained residual denoising network model and in combination with the spectral information of the second RGB image, reverse iterative denoising is performed on the target noise image to generate a second image space.

[0071] Furthermore, this embodiment uses the trained residual denoising network model to perform reverse iterative denoising on the target noise image (the target noise image is a noisy image that needs to be denoised to obtain a reconstructed hyperspectral image), combined with the spectral information of the second RGB image (the second RGB image is closely related to the target noise image in time or space and can provide additional spectral information), which can provide more references and constraints for the denoising process, making the denoising result more accurate. Through continuous iterative denoising, the noise in the target noise image is gradually removed to generate a second image space, and the image in the second image space is closer to the original hyperspectral image.

[0072] Step S106: performing data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image.

[0073] Furthermore, this embodiment performs data reconstruction processing on the image in the second image space, and can restore the lower dimensional image data to the original high dimensional space. Through the data reconstruction processing, the system obtains the reconstructed hyperspectral image corresponding to the target noise image. The reconstructed hyperspectral image not only removes the noise, but also retains the main spectral information and characteristics of the original image.

[0074] In summary, the present embodiment firstly adopts the dimension reduction and feature extraction processing method for the original hyperspectral image, which can effectively reduce the computational complexity and the demand for computing resources of the model during the training process, so that the reconstruction training of hyperspectral data can be carried out under limited computing resources, simplifying the training process and helping to shorten the training time. In addition, the present embodiment introduces the residual denoising network model (ARDNet) containing the attention mechanism during the diffusion model training, which can more accurately extract the key features and details of the image, help to improve the image quality of the reconstructed image, and can provide a clearer and more accurate hyperspectral image, so that users can better observe and analyze the spectral features. In addition, the present embodiment applies the diffusion model to the hyperspectral data reconstruction task, first defines a process of gradually transforming from data distribution to noise distribution, and then uses the residual denoising network (ARDNet) containing the attention mechanism to learn how to restore the original hyperspectral image from the noise data, and gradually generates a clear and complete hyperspectral image from the noise data through iterative denoising. The application of the diffusion model can better simulate the distribution and change of hyperspectral data, achieve resolution balance, and improve the accuracy and reliability of reconstruction.

[0075] In this embodiment, another medical spectrum reconstruction method is provided. Figure 2 is a flow chart of another medical spectrum reconstruction method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0076] Step S201, obtain the original hyperspectral image. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0077] Furthermore, this embodiment can realize the reconstruction of hyperspectral images from traditional camera RGB images using the principal component latent diffusion model (PCLDM), and achieve some breakthroughs in the medical application of hyperspectral imaging technology. This embodiment applies the diffusion model to the task of hyperspectral data reconstruction, and proposes a novel method, that is, to simplify the training process by means of principal component analysis (PCA), perform the diffusion process to achieve the purpose of spectral reconstruction, and at the same time, introduce the residual denoising network model (ARDNet) containing the attention mechanism to extract the key features and details of the image, thereby improving the image quality of the reconstructed image. The residual denoising network model (ARDNet) is used to learn the global noise distribution of different noise images and temporal feature embeddings, and by taking the RGB image condition and the noisy image as the joint input, the noise is deeply learned and analyzed to obtain the predicted noise.

[0078] Step S202: Dimension reduction and feature extraction are performed on the original hyperspectral image to generate a first image space. The dimension reduction and feature extraction may be principal component analysis.

[0079] In an optional implementation, the step S202 includes:

[0080] Calculate the covariance matrix corresponding to the original hyperspectral image, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors corresponding to the eigenvalues;

[0081] According to the size of the eigenvalue, extract the target number of eigenvectors to construct a transformation matrix to extract the principal component information;

[0082] Based on the transformation matrix, the original hyperspectral image is projected to the direction of the principal component information, and a data representation after the projection is obtained;

[0083] Based on the projected data representation, the first image space is constructed; each pixel point in the first image space is represented by the principal component information.

[0084] Further, when the principal component analysis is performed on the original hyperspectral image, the present embodiment first calculates the covariance matrix corresponding to the original hyperspectral image, and performs eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​reflect the variance of the data, and the eigenvectors represent the changes in the data in different directions. Then, according to the size of the eigenvalues, the eigenvectors of the previous target number are extracted to construct a transformation matrix, so that the principal component information can be extracted. The eigenvectors corresponding to the larger eigenvalues ​​usually contain more data change information. By selecting the eigenvectors of the previous target number, the main information components can be retained. Then, based on the transformation matrix, the original hyperspectral image is projected in the direction of the principal component information to obtain the data representation after projection. This process can convert the high-dimensional original hyperspectral image data into a low-dimensional space composed of the principal component information. Finally, based on the data representation after projection, the first image space is constructed. In this space, each pixel is represented by the principal component information, which reduces the redundancy of the data and facilitates subsequent processing.

[0085] Furthermore, this embodiment is mainly divided into two parts: spectral reconstruction training and spectral reconstruction reasoning. The first step of spectral reconstruction training is to extract the principal component information of hyperspectral data by using the principal component analysis (PCA) method, which can effectively reduce the model's demand for computing resources and memory during the training and sampling stages. Figure 3 The flowchart of spectral reconstruction training is shown. In this embodiment, the original hyperspectral image is converted into a set of refined principal component information through the principal component analysis (PCA) method to generate a first image space, thereby reducing the redundancy of hyperspectral data and enhancing the feature expression capability during spectral reconstruction reasoning to facilitate subsequent training processes.

[0086] Step S203: adding forward noise to the first image space to generate a noisy image.

[0087] In an optional implementation, the step S203 includes:

[0088] Acquire forward noise; the forward noise has the same dimension as the first image space;

[0089] The forward noise is added element-by-element to the principal component representation of each pixel in the first image space to generate the noisy image with noise.

[0090] Furthermore, this embodiment first obtains forward noise with the same dimension as the first image space to ensure that the noise can operate on the same dimension as the image data, and then adds the forward noise to the principal component representation of each pixel in the first image space element by element, so that the image data is interfered by the noise, generating a noisy image with noise, and providing noisy samples for subsequent model training.

[0091] Step S204: The noisy image and the first RGB image are used as joint training data, input into a residual denoising network model to be trained for training, and the trained residual denoising network model is obtained.

[0092] In an optional implementation manner, before step S204, this embodiment includes:

[0093] Based on the target network structure and target loss function, a residual denoising network model to be trained is constructed.

[0094] Furthermore, the present embodiment first obtains a first RGB image that is spatially aligned with the original hyperspectral image, ensuring that the first RGB image corresponds to the original hyperspectral image in spatial position, so as to provide more spectral information for the model. The first RGB image and the noisy image are jointly processed to obtain joint training data, so that two different types of image information can be combined, allowing the model to learn how to use the information of the first RGB image to predict the noise in the original hyperspectral image. Then, based on the target network structure and the target loss function, a residual denoising network model to be trained is constructed. The target network structure may include components such as convolutional layers, pooling layers, and attention mechanisms. The target loss function is used to measure the difference between the model output and the true value, and guide the model to be optimized.

[0095] In an optional implementation, the step S204 includes:

[0096] Acquire a first RGB image spatially aligned with the original hyperspectral image;

[0097] Jointly processing the first RGB image and the noisy image to obtain the joint training data;

[0098] The joint training data is input into the residual denoising network model to be trained to perform nonlinear activation function processing, and a nonlinear activation function processing result is obtained;

[0099] Extract local features of the processing result of the nonlinear activation function to obtain spatial structure information;

[0100] The temporal feature is embedded into the extracted local feature as an additional channel, and feature extraction is performed again on the embedded result;

[0101] The extracted local feature is added to the result of feature extraction again to construct a residual connection;

[0102] The residual connection result is input into the attention module of the residual denoising network model to be trained for weighted processing, and the predicted noise image is output.

[0103] Furthermore, in this embodiment, the joint training data is first input into the residual denoising network model, and nonlinear activation function processing is performed through a series of convolutional layers or fully connected layers. The nonlinear activation functions include ReLU (Rectified Linear Unit), Sigmoid and Tanh, etc. These activation functions can introduce nonlinear characteristics, so that the model can learn complex mapping relationships. After being processed by the nonlinear activation function, the model will extract local features of the data, which correspond to spatial structural information such as edges, textures, shapes, etc. in the image. In the residual denoising network, these local features are crucial for removing noise and retaining image details. In this embodiment, time features (for example, time series images or video frames) are also embedded into the extracted local features as additional channels, which can make full use of time information and improve the model's processing ability for dynamic data. After combining the time features with the local features, the model will extract these features again, which usually involves deeper convolutional layers or more complex network structures to further explore the potential features of the data. In order to improve the training effect and performance of the model, the residual denoising network model in this embodiment adds the extracted local features to the results of feature extraction again to construct a residual connection, which helps the model learn the residual (i.e., difference) between the input and output, so as to more easily capture the change information in the image. Finally, the residual connection result is input into the attention module of the residual denoising network model for weighted processing. The attention module can dynamically adjust the importance of different features, thereby enhancing the model's attention to key information. Through weighted processing, the model can more accurately predict the noise part of the noisy image and output the predicted noise image.

[0104] During the training process, the residual denoising network model calculates the loss function based on the difference between the predicted noise and the real noise (or the forward noise mentioned above), and updates the parameters of the residual denoising network model through the back-propagation algorithm. This process is repeated many times until the model achieves satisfactory performance on the validation set.

[0105] Further, such as Figure 3As shown, the first image space generates a noisy image with noise by adding forward noise. In this embodiment, the first RGB image and the noisy image are used as joint training data, and are jointly input into the residual denoising network model (ARDNet) of the attention mechanism to characterize the noise. Compared with the original U-Net deep learning architecture, the residual denoising network model (ARDNet) can significantly improve the computational efficiency by introducing residual connections and attention mechanisms. In this embodiment, the residual denoising network model (ARDNet) learns the global noise distribution of different noise images and temporal feature embeddings. This structure can effectively learn the local texture features at a high-resolution scale and the global structural features at a low-resolution scale after downsampling to obtain predicted noise. In addition, after obtaining the predicted noise, this implementation also performs feedback network calculation loss processing based on the difference between the forward noise and the predicted noise to further optimize the training results of the residual denoising network model.

[0106] Step S205: during training, regularization is performed on the model parameters in the residual denoising network model being trained.

[0107] Furthermore, regularization can prevent model overfitting and improve the generalization ability of the residual denoising network model. By constraining the model parameters, the model is made simpler and avoids learning too much noise and irrelevant features. When updating the model parameters in each iteration, the regularization term is added to the objective loss function and optimized together. In this way, the residual denoising network model will not only learn how to remove noise during training, but will also be constrained by the regularization term, thereby avoiding overfitting.

[0108] Step S206, obtaining a target noise image to be processed and a second RGB image spatially aligned with the target noise image; performing reverse iterative denoising processing with the target noise image as initial input based on the target iteration number, and obtaining the second image space finally generated after the iteration, and performing the following steps in each iteration:

[0109] Input the current noise image into the trained residual denoising network model, and combine the spectral information of the second RGB image to output the predicted noise corresponding to the current noise image;

[0110] The current noise image is updated according to the predicted noise corresponding to the current noise image, and an updated noise image is obtained.

[0111] Further, in this embodiment, the target noise image is used as the starting point of the denoising process. This initial target noise image can be the image data with noise obtained in practical applications. A target number of iterations is set, which determines the fineness of the denoising process. Generally, the more iterations, the better the denoising effect may be, but it will also increase the calculation time. In each iteration, the current noise image is input into the trained residual denoising network model. The model has been trained by joint training data in step S204 and has the ability to predict noise from the noisy image. In the process of the residual denoising network model processing the current noise image, the spectral information of the second RGB image is combined. The spectral information in the second RGB image can provide additional references and constraints for denoising, helping the model to more accurately predict the noise in the current noise image. After being processed by the residual denoising network model, the predicted noise corresponding to the current noise image is output, and the predicted noise is the noise component in the current noise image estimated by the model based on the information of the input image and the second RGB image. According to the predicted noise corresponding to the current noise image, the current noise image is updated. Specifically, the predicted noise is subtracted from the current noise image to obtain an updated noise image, in which the noise content of the updated image is less than that of the image in the previous iteration. By continuously repeating this iterative process, the current noise image is updated each time, the noise is gradually removed, and finally a second image space is generated. The image data in the second image space has removed most of the noise, while retaining the main information and features of the original target noise image, which will be used as the input for the inverse principal component analysis in the subsequent steps to obtain the reconstructed hyperspectral image corresponding to the target noise image.

[0112] For further information, see Figure 4 The flowchart of spectral reconstruction reasoning is shown. This embodiment innovatively uses the DDPM (Denoising Diffusion Probabilistic Models) generation model in the spectral reconstruction task. The DDPM generation model uses an iterative denoising method to learn the distribution of target data. In the spectral reconstruction reasoning of this embodiment, the most critical theoretical idea is the diffusion model (Denoising Module), that is, this embodiment performs reverse iterative denoising processing based on the diffusion model, and the trained residual denoising network model is applied in the diffusion model). The DDPM generation model approximates the target data distribution through the process of gradually removing noise. In the spectral reconstruction task, this means that it can gradually iteratively denoise from the noisy target noise image and finally generate a high-quality reconstructed spectrum. The diffusion model is its core part, through the residual denoising network model (that is Figure 4The denoising network in the denoising process) realizes the denoising process and continuously updates the state of the current noisy image (such as the state of the noisy image from time step t Keep updating to status ,state ,,, until the final second image space), by continuously adjusting the model parameters, the model can better learn the characteristics and distribution laws of spectral data, thereby achieving accurate spectral reconstruction.

[0113] Furthermore, the diffusion model combines the first RGB image with the noisy image to generate joint training data. Then, the global noise distribution of different noise images and time embeddings is learned through the residual denoising network model to be trained (ARDNet), and the predicted noise is calculated. Figure 4 As shown, in this embodiment, the predicted noise is subtracted from the current noisy image to obtain a new noisy image. This process is then repeated in this way. The purpose of this process is to recover a result closer to the original image from the current noisy image by gradually denoising. By learning the noise distribution using the residual denoising network model (ARDNet), noise can be predicted and removed more accurately. With each iteration, the noise in the noisy image is gradually reduced, and it is expected that a high-quality reconstructed image or an image closer to the original state will be obtained in the end.

[0114] This embodiment provides richer spectral information by using the second RGB image. Initially, the diffusion model is used to gradually transform the Gaussian white noise into the second image space through the reverse denoising process and the spectral information provided by the second RGB image. After the spectral reconstruction inference of this embodiment generates the second image space through denoising, it undergoes the inverse principal component analysis process to obtain the reconstructed hyperspectral image generated after super-resolution. Gaussian white noise In x T A representation of Gaussian white noise at the initial stage in the reverse denoising process of the diffusion model. It means that it follows a Gaussian distribution with a mean of 0 and a variance of 1.

[0115] Step S207: Perform data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image. The data reconstruction processing may be an inverse principal component analysis processing.

[0116] In an optional implementation, step S207 includes:

[0117] Get the inverse transformation matrix corresponding to the transformation matrix;

[0118] Extracting data representation from the second image space, and restoring the data representation in the second image space through the inverse transformation matrix to obtain a reconstructed hyperspectral image data representation;

[0119] Based on the reconstructed hyperspectral image data representation, a reconstructed hyperspectral image corresponding to the target noise image is obtained.

[0120] Furthermore, with regard to the overall implementation process of this embodiment, this embodiment applies the diffusion model to the task of hyperspectral data reconstruction, and proposes a novel method, namely, simplifying the training process by means of principal component analysis (PCA), and performing the diffusion process to achieve the purpose of spectral reconstruction. At the same time, the residual denoising network model (ARDNet) containing the attention mechanism is introduced to extract the key features and details of the image, thereby improving the image quality of the reconstructed image. This embodiment is mainly divided into two parts: spectral reconstruction training and spectral reconstruction reasoning.

[0121] For the spectral reconstruction training part, such as Figure 3 As shown, in this embodiment, the original hyperspectral image is converted into a set of refined principal component information through the principal component analysis processing method, and then the first image space is generated. In the principal component analysis processing, the principal component information is obtained through the following formula:

[0122] ;

[0123] in, represents the original hyperspectral image matrix corresponding to the original hyperspectral image, Represents each pixel in the original hyperspectral image matrix, and its dimension is , represents the total number of pixels in the original hyperspectral image, H is the length, is the width, Represents the number of hyperspectral bands, that is, the number of features of the original image in the spectral dimension. is the extracted principal component information, Represents each principal component information in a set of principal component information, is the transformation matrix, with dimension .

[0124] Furthermore, this embodiment obtains the forward probability distribution of the forward noise through the following formula:

[0125] ;

[0126] in, represents the forward probability distribution, from the state of the noise image at time step t-1 To the noisy image state at time step t The probability transfer relationship. , represents the time step, represents a Gaussian distribution, represents the hyperparameter sequence of Gaussian noise variance, and I is the identity matrix.

[0127] The first image space generated after the principal component analysis is processed is subjected to forward noise to generate a noisy image.

[0128] In this embodiment, the first RGB image and the noisy image are jointly input to the residual denoising network model ARDNet of the attention mechanism to characterize the noise. Figure 5 The calculation flow chart of the residual denoising network model is shown as Figure 5 As shown. The core component of the residual denoising network model (ARDNet) is the attention-based residual block (ARB). Each residual convolution interface ARB based on the attention mechanism first applies regularization technology to enhance the generalization ability of the model, thereby reducing the risk of overfitting. Secondly, the input data is processed through a nonlinear activation function, that is, the training data is jointly trained, so as to achieve the effect of improving the expression ability of the model. Third, the convolution layer is used to extract the local features of the input data and capture the spatial structure information. Fourth, after embedding the temporal features, a second model feature extraction is performed to further refine the feature representation. Finally, a residual connection is constructed by adding a number, and the result is input into the attention module.

[0129] This embodiment also learns the global noise distribution of different noisy images and time feature embedding through the residual denoising network model (ARDNet), and obtains the predicted noise output by the residual denoising network model (ARDNet) through the following formula:

[0130] ;

[0131] in, represents the noise prediction model based on the residual denoising network model (ARDNet), Represents the state of the noise image at time step t during the diffusion model training process The prediction noise contained in , the time embedding layer introduced in this embodiment is to improve the accuracy of noise prediction, and at the same time, it is combined with the first RGB image Z1 by splicing to provide a basis for spectral reconstruction reasoning, Indicates the direction of prediction. For the spectrum reconstruction reasoning part, such as Figure 4 As shown, this embodiment provides richer spectral information by using the second RGB image, from the Gaussian white noise At the beginning, the reverse denoising process is used, and the spectral information provided by the second RGB image is combined to gradually transform the Gaussian white noise into the final result in the second image space. Specifically, the reverse denoising process requires the inverse probability distribution The distribution of is iteratively sampled, the range is ,in Represents the target number of iterations for reverse iteration. This distribution is obtained by learning the residual denoising network model. According to Bayes' theorem, the inverse probability distribution can be obtained by the following formula:

[0132] ;

[0133] in, Represents the inverse probability distribution The mean of the residual denoising network model is Build, Represents the inverse probability distribution The best way to parameterize the residual denoising network model is to predict the predicted noise added to the current noisy image, that is, to add the predicted noise obtained in the spectral reconstruction training. .

[0134] The mean is obtained by the following formula :

[0135] ;

[0136] The variance is obtained by the following formula :

[0137] ;

[0138] in, and These are different parameters related to the residual denoising network model, which affect the sampling of the noisy image and the final denoising effect during the reverse denoising process.

[0139] This embodiment is based on the inverse probability distribution formula, and the state of the noise image at time step t is calculated by the following formula: The state of the noise image at time step t-1 under the condition To sample:

[0140] ;

[0141] from By iteratively running the above denoising process, the final result in the second image space can be obtained. .

[0142] In this embodiment, after denoising to generate the principal component information, the spectral reconstruction inference is processed through the inverse principal component analysis process to obtain the reconstructed hyperspectral image generated after super-resolution. , the reconstructed hyperspectral image can be obtained by the following formula :

[0143] ;

[0144] in, Represents the transformation matrix The transposed matrix of , K represents the number of principal components retained during the inverse principal component analysis process.

[0145] In summary, the present embodiment firstly adopts the principal component analysis processing method for the original hyperspectral image, which can effectively reduce the computational complexity and the demand for computing resources of the model during the training process, so that the reconstruction training of hyperspectral data can be carried out under limited computing resources, simplifying the training process and helping to shorten the training time. In addition, the present embodiment introduces the residual denoising network model (ARDNet) containing the attention mechanism during the diffusion model training, which can more accurately extract the key features and details of the image, help to improve the image quality of the reconstructed image, and can provide clearer and more accurate hyperspectral images, so that users can better observe and analyze spectral features. In addition, the present embodiment applies the diffusion model to the hyperspectral data reconstruction task, first defines a process of gradually transforming from data distribution to noise distribution, and then uses the residual denoising network (ARDNet) containing the attention mechanism to learn how to restore the original hyperspectral image from the noise data, and gradually generates a clear and complete hyperspectral image from the noise data through iterative denoising. The application of the diffusion model can better simulate the distribution and changes of hyperspectral data, achieve resolution balance, and improve the accuracy and reliability of reconstruction.

[0146] In this embodiment, a medical spectrum reconstruction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0147] This embodiment provides a medical spectrum reconstruction device, such as Figure 6 As shown, including:

[0148] The original hyperspectral image acquisition module 601 is used to acquire the original hyperspectral image;

[0149] A first image space acquisition module 602 is used to perform dimension reduction and feature extraction processing on the original hyperspectral image and generate a first image space;

[0150] A noisy image acquisition module 603 is used to add forward noise to the first image space to generate a noisy image;

[0151] The spectral reconstruction training module 604 is used to input the noisy image and the first RGB image as joint training data into the residual denoising network model to be trained for training, and obtain the trained residual denoising network model;

[0152] The spectral reconstruction inference module 605 is used to perform reverse iterative denoising processing on the target noise image based on the trained residual denoising network model and in combination with the spectral information of the second RGB image to generate a second image space;

[0153] The reconstructed hyperspectral image acquisition module 606 is used to perform data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image.

[0154] In some optional implementations, the first image space acquisition module 602 is further used to:

[0155] Calculate the covariance matrix corresponding to the original hyperspectral image, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors corresponding to the eigenvalues;

[0156] According to the size of the eigenvalue, extract the target number of eigenvectors to construct a transformation matrix to extract the principal component information;

[0157] Based on the transformation matrix, the original hyperspectral image is projected to the direction of the principal component information, and a data representation after the projection is obtained;

[0158] Based on the projected data representation, the first image space is constructed; each pixel point in the first image space is represented by the principal component information.

[0159] In some optional implementations, the noise-added image acquisition module 603 is further used to:

[0160] Acquire forward noise; the forward noise has the same dimension as the first image space;

[0161] The forward noise is added element-by-element to the principal component representation of each pixel in the first image space to generate the noisy image with noise.

[0162] In some optional implementations, the spectrum reconstruction training module 604 is further used to:

[0163] Acquire a first RGB image spatially aligned with the original hyperspectral image;

[0164] Jointly processing the noisy image and the first RGB image to obtain the joint training data;

[0165] The joint training data is input into the residual denoising network model to be trained to perform nonlinear activation function processing, and a nonlinear activation function processing result is obtained;

[0166] Extract local features of the processing result of the nonlinear activation function to obtain spatial structure information;

[0167] The temporal feature is embedded into the extracted local feature as an additional channel, and feature extraction is performed again on the embedded result;

[0168] The extracted local feature is added to the result of feature extraction again to construct a residual connection;

[0169] The residual connection result is input into the attention module of the residual denoising network model to be trained for weighted processing, and the predicted noise image is output.

[0170] In some optional embodiments, the device is also used for:

[0171] During training, the model parameters in the residual denoising network model under training are regularized.

[0172] In some optional implementations, the spectrum reconstruction reasoning module 605 is further used to:

[0173] Acquire a target noise image to be processed and a second RGB image spatially aligned with the target noise image;

[0174] Based on the target number of iterations, the target noise image is used as the initial input for reverse iterative denoising, and the second image space finally generated after the iteration is obtained. The following steps are performed in each iteration:

[0175] Input the current noise image into the trained residual denoising network model, and combine the spectral information of the second RGB image to output the predicted noise corresponding to the current noise image;

[0176] The current noise image is updated according to the predicted noise corresponding to the current noise image, and an updated noise image is obtained.

[0177] In some optional implementations, the reconstructed hyperspectral image acquisition module 606 is further used to:

[0178] Get the inverse transformation matrix corresponding to the transformation matrix;

[0179] Extracting data representation from the second image space, and restoring the data representation in the second image space through the inverse transformation matrix to obtain a reconstructed hyperspectral image data representation;

[0180] Based on the reconstructed hyperspectral image data representation, a reconstructed hyperspectral image corresponding to the target noise image is obtained.

[0181] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0182] In summary, the present embodiment firstly adopts the principal component analysis processing method for the original hyperspectral image, which can effectively reduce the computational complexity and the demand for computing resources of the model during the training process, so that the reconstruction training of hyperspectral data can be carried out under limited computing resources, simplifying the training process and helping to shorten the training time. In addition, the present embodiment introduces the residual denoising network model (ARDNet) containing the attention mechanism during the diffusion model training, which can more accurately extract the key features and details of the image, help to improve the image quality of the reconstructed image, and can provide clearer and more accurate hyperspectral images, so that users can better observe and analyze spectral features. In addition, the present embodiment applies the diffusion model to the hyperspectral data reconstruction task, first defines a process of gradually transforming from data distribution to noise distribution, and then uses the residual denoising network (ARDNet) containing the attention mechanism to learn how to restore the original hyperspectral image from the noise data, and gradually generates a clear and complete hyperspectral image from the noise data through iterative denoising. The application of the diffusion model can better simulate the distribution and changes of hyperspectral data, achieve resolution balance, and improve the accuracy and reliability of reconstruction.

[0183] The embodiment of the present invention also provides a computer device, see Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0184] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0185] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0186] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0188] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0189] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0190] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0191] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A medical spectrum reconstruction method, characterized in that: The method comprises: Obtaining original hyperspectral images; Performing dimension reduction and feature extraction processing on the original hyperspectral image to generate a first image space; Adding forward noise to the first image space to generate a noisy image; The noisy image and the first RGB image are input as joint training data into a residual denoising network model to be trained for training, and the trained residual denoising network model is obtained; Based on the trained residual denoising network model and in combination with the spectral information of the second RGB image, a reverse iterative denoising process is performed on the target noise image to generate a second image space; Performing data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image; The step of performing dimension reduction and feature extraction processing on the original hyperspectral image and generating a first image space includes: Calculating a covariance matrix corresponding to the original hyperspectral image, and performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors corresponding to the eigenvalues; According to the size of the eigenvalue, extract the target number of eigenvectors to construct a transformation matrix to extract the principal component information; Based on the transformation matrix, projecting the original hyperspectral image to the direction of the principal component information, and obtaining a data representation after the projection; Based on the projected data representation, construct the first image space; each pixel in the first image space is represented by the principal component information; The method of performing reverse iterative denoising on the target noise image based on the trained residual denoising network model and combining the spectral information of the second RGB image to generate a second image space includes: Acquire a target noise image to be processed and a second RGB image spatially aligned with the target noise image; Based on the target number of iterations, the target noise image is used as the initial input for reverse iterative denoising, and the second image space finally generated after the iteration is obtained. The following steps are performed in each iteration: Inputting the current noise image into the trained residual denoising network model, and combining the spectral information of the second RGB image, outputting the predicted noise corresponding to the current noise image; The current noise image is updated according to the predicted noise corresponding to the current noise image, and an updated noise image is obtained.

2. The method according to claim 1, characterized in that Adding forward noise to the first image space to generate a noisy image includes: Acquire forward noise; the forward noise has the same dimension as the first image space; The forward noise is added element-wise to the principal component representation of each pixel in the first image space to generate the noisy image with noise.

3. The method according to claim 1, characterized in that The step of taking the noisy image and the first RGB image as joint training data and inputting them into the residual denoising network model to be trained for training includes: Acquire a first RGB image spatially aligned with the original hyperspectral image; Jointly processing the noisy image and the first RGB image to obtain the joint training data; Inputting the joint training data into the residual denoising network model to be trained to perform nonlinear activation function processing, and obtaining a nonlinear activation function processing result; Extracting local features of the processing result of the nonlinear activation function to obtain spatial structure information; Embed the time feature as an additional channel into the extracted local feature, and perform feature extraction again on the embedding result; Adding the extracted local features to the result of feature extraction again to construct a residual connection; The residual connection result is input into the attention module of the residual denoising network model to be trained for weighted processing, and a predicted noise image is output.

4. The method according to claim 3, characterized in that The method further comprises: During training, regularization is performed on the model parameters in the residual denoising network model in training.

5. The method according to claim 1, characterized in that The performing data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image includes: Obtaining an inverse transformation matrix corresponding to the transformation matrix; Extracting data representation from the second image space, and restoring the data representation in the second image space through the inverse transformation matrix to obtain a reconstructed hyperspectral image data representation; Based on the reconstructed hyperspectral image data representation, a reconstructed hyperspectral image corresponding to the target noise image is obtained.

6. A medical spectrum reconstruction device, characterized in that: The device comprises: An original hyperspectral image acquisition module is used to acquire an original hyperspectral image; A first image space acquisition module, used for performing dimension reduction and feature extraction processing on the original hyperspectral image and generating a first image space; A noisy image acquisition module, used for adding forward noise to the first image space to generate a noisy image; A spectral reconstruction training module, used for inputting the noisy image and the first RGB image as joint training data into a residual denoising network model to be trained for training, and obtaining the residual denoising network model after training; A spectral reconstruction inference module, used to perform reverse iterative denoising processing on the target noise image based on the trained residual denoising network model and in combination with the spectral information of the second RGB image to generate a second image space; A reconstructed hyperspectral image acquisition module is used to perform data reconstruction processing on the second image space to obtain a reconstructed hyperspectral image corresponding to the target noise image; The first image space acquisition module is further used for: Calculating a covariance matrix corresponding to the original hyperspectral image, and performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors corresponding to the eigenvalues; According to the size of the eigenvalue, extract the target number of eigenvectors to construct a transformation matrix to extract the principal component information; Based on the transformation matrix, projecting the original hyperspectral image to the direction of the principal component information, and obtaining a data representation after the projection; Based on the projected data representation, construct the first image space; each pixel in the first image space is represented by the principal component information; The second image space acquisition module is further used for: Acquire a target noise image to be processed and a second RGB image spatially aligned with the target noise image; Based on the target number of iterations, the target noise image is used as the initial input for reverse iterative denoising, and the second image space finally generated after the iteration is obtained. The following steps are performed in each iteration: Inputting the current noise image into the trained residual denoising network model, and combining the spectral information of the second RGB image, outputting the predicted noise corresponding to the current noise image; The current noise image is updated according to the predicted noise corresponding to the current noise image, and an updated noise image is obtained.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the medical spectrum reconstruction method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute a medical spectrum reconstruction method according to any one of claims 1 to 5.

Images