Single photon reconstruction method and device and electronic equipment
Denoising and reconstruction of single-photon data through deep learning network model based on attention mechanism, the problem of low imaging efficiency in the existing technology is solved, high-resolution and high-precision rapid imaging is achieved, and computing speed is improved.
Patent Information
- Application Number
- CN202411770859.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-18
AI Technical Summary
Among the existing single-photon imaging methods, sparse Poisson reconstruction method requires setting convolution kernel hyperparameters and running time is long, making it difficult to achieve efficient and fast imaging.
By collecting the true photon values, the training set is generated by simulation, and the photon simulation values are input into the denoising module and reconstruction module in the deep learning network model constructed based on the attention mechanism for training, the reconstruction point cloud is obtained. Then, the point cloud input assisted training model will be reconstructed, the loss will be calculated and the network parameters will be adjusted to obtain the trained deep learning network model.
It achieves faster imaging with higher resolution and higher accuracy, significantly improving computing speed.
Smart Images
Figure CN119648767A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of single-photon imaging technology, and in particular to a single-photon reconstruction method, device and electronic equipment. Background Art
[0002] Typically, single-photon data has the characteristics of a large number of noise photons and sparse signal photons. Existing single-photon imaging methods mainly include the traditional sparse Poisson reconstruction method, which, however, requires setting convolution kernel hyperparameters for the scene and takes a long time to run. Summary of the invention
[0003] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a single-photon reconstruction method, device and electronic device.
[0004] In a first aspect, an embodiment of the present application provides a single-photon reconstruction method, the single-photon reconstruction method comprising:
[0005] Collecting real photon values, and simulating and generating a training set based on the real photon values, wherein the training set includes photon simulation values;
[0006] Inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud;
[0007] Inputting the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain a photon prediction value;
[0008] Based on the predicted photon value and the actual photon value, the loss of the deep learning network model is calculated and the network parameters of the deep learning network model are adjusted to obtain a trained deep learning network model.
[0009] In a possible implementation, the step of inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud includes:
[0010] Inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in multiple deep learning network models for training to obtain a reconstructed point cloud;
[0011] The step of calculating the loss of the deep learning network model and adjusting the network parameters of the deep learning network model based on the predicted photon value and the real photon value to obtain a trained deep learning network model comprises:
[0012] Based on the predicted photon value and the actual photon value, the losses of the multiple deep learning network models are calculated and the network parameters of the multiple deep learning network models are adjusted to obtain multiple trained deep learning network models.
[0013] In a possible implementation, the method further includes:
[0014] Get the validation set and test set;
[0015] After the step of calculating the losses of the plurality of deep learning network models and adjusting the network parameters of the plurality of deep learning network models based on the predicted photon values and the true photon values, thereby obtaining a plurality of trained deep learning network models, the method further comprises:
[0016] Using the validation set, selecting one of the trained deep learning models as a deep learning network model to be evaluated;
[0017] The deep learning network model to be evaluated is evaluated using the test set to obtain an evaluated deep learning network model.
[0018] In a possible implementation, the step of calculating the loss of the deep learning network model based on the predicted photon value and the real photon value includes:
[0019] Based on the predicted value of the photon and the true value of the photon, the loss of the deep learning network model is calculated using a negative Poisson log-likelihood function as a loss function.
[0020] In one possible implementation, the negative Poisson log-likelihood function is:
[0021]
[0022] in, represents the predicted value of photons, and y represents the actual value of photons.
[0023] In a possible implementation, the method further includes:
[0024] The hyperparameters of the deep learning network model are adjusted through a learning rate algorithm to obtain a trained deep learning network model.
[0025] In a possible implementation, the step of adjusting the hyperparameters of the deep learning network model by using a learning rate algorithm includes:
[0026] The hyperparameters of the deep learning network model are adjusted by a cosine annealing algorithm.
[0027] In a possible implementation, after the step of calculating the loss of the deep learning network model based on the predicted photon value and the true photon value and adjusting the network parameters of the deep learning network model to obtain a trained deep learning network model, the method further includes:
[0028] Get the point cloud to be processed at a specific distance;
[0029] The point cloud to be processed is input into the trained deep learning network model to obtain a target point cloud, and a depth image and an intensity image are generated based on the target point cloud.
[0030] In a second aspect, the present application also provides a single-photon reconstruction device, including:
[0031] A collection module, used for collecting real photon values, and simulating and generating a training set according to the real photon values, wherein the training set includes simulated photon values;
[0032] A training module, used for inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud;
[0033] An auxiliary training module, used for inputting the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain a photon prediction value;
[0034] An adjustment module is used to calculate the loss of the deep learning network model and adjust the network parameters of the deep learning network model based on the predicted value of the photon and the true value of the photon, so as to obtain a trained deep learning network model.
[0035] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method described in any of the above aspects is implemented.
[0036] Based on any of the above aspects, the single-photon reconstruction method, device and electronic device provided in the embodiments of the present application can achieve higher resolution, higher precision and rapid imaging by training the denoising module and reconstruction module of the deep learning network model, and perform long-distance auxiliary training on the deep learning network model through the auxiliary training module, thereby greatly improving the computing speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0038] Figure 1 One of the flow charts of the single photon reconstruction method provided in the embodiment of the present application;
[0039] Figure 2 A schematic diagram of the structure of a deep learning network model provided in an embodiment of the present application;
[0040] Figure 3 The second flowchart of the single photon reconstruction method provided in the embodiment of the present application;
[0041] Figure 4 The third flowchart of the single photon reconstruction method provided in the embodiment of the present application;
[0042] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0043] Figure 6 A schematic diagram of the structure of a single-photon reconstruction device provided in an embodiment of the present application.
[0044] Icon: 100 - denoising module; 200 - reconstruction module; 800 - electronic device; 810 - processor; 820 - machine-readable storage medium; 830 - single-photon reconstruction device; 831 - acquisition module; 832 - training module; 833 - auxiliary training module; 834 - adjustment module. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0046] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0047] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0048] In the description of this application, it should be noted that the terms "upper", "lower", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the application is usually placed when in use, which is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0049] It should be noted that, in the absence of conflict, different features in the embodiments of the present application may be combined with each other.
[0050] The specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings.
[0051] Please refer to Figure 1 , Figure 1 A schematic flow chart of a single-photon reconstruction method provided by the present embodiment is given as an example. The method may include the following steps.
[0052] Step S110, collecting real photon values, and simulating and generating a training set based on the real photon values, wherein the training set includes photon simulation values.
[0053] In this embodiment, the training set can be obtained by simulating the NYU V2 dataset through a single-photon physical imaging model. The training set can be a single-photon dataset, and the training set can generate a point cloud with a spatial resolution of 256×256.
[0054] Step S120, input the photon simulation value into the denoising module 100 and the reconstruction module 200 constructed based on the attention mechanism in the deep learning network model for training to obtain a reconstructed point cloud.
[0055] In this embodiment, the photon simulation value can be input into the denoising module 100 to remove most of the noise photons to obtain sparse signal photons, and then the sparse signal photons can be input into the reconstruction module 200 to reconstruct the signal photons to obtain a reconstructed point cloud.
[0056] Specifically, the denoising module 100 and the reconstruction module 200 can be constructed by the attention mechanism, please refer to Figure 2The denoising module 100 and the reconstruction module 200 may include a basic unit SKunit, which applies different attention mechanisms to convolution kernels of different sizes to achieve different receptive fields for data with different signal-to-noise ratios, thereby achieving better denoising and reconstruction functions. The deep learning network model may combine multiple basic units SKunit through residual connections.
[0057] Step S130, inputting the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain a photon prediction value.
[0058] In this embodiment, the reconstructed point cloud obtained in step S120 can be input into the auxiliary training model of the deep learning network model to obtain the photon prediction value.
[0059] Specifically, the auxiliary training module can pre-design convolution kernel parameters to implement deconvolution for a specific distance during training, so as to be used for long-distance auxiliary training.
[0060] Step S140, based on the predicted photon value and the actual photon value, calculate the loss of the deep learning network model and adjust the network parameters of the deep learning network model to obtain a trained deep learning network model.
[0061] In this embodiment, the loss of the deep learning network model can be calculated based on the predicted value of the photon obtained in step S130 and the actual value of the photon obtained in step S110, and the network parameters of the deep learning network model can be adjusted to gradually reduce the loss function until the network converges, thereby obtaining a trained deep learning network model.
[0062] In the above design, by training the denoising module 100 and the reconstruction module 200 of the deep learning network model and performing remote auxiliary training on the deep learning network model through the auxiliary training module, higher resolution and higher precision rapid imaging can be achieved, thereby greatly improving the computing speed.
[0063] In one possible implementation, when the photon simulation value is input into the denoising module 100 and the reconstruction module 200 constructed based on the attention mechanism in the deep learning network model for training to obtain the reconstructed point cloud, the photon simulation value can be input into the denoising module 100 and the reconstruction module 200 constructed based on the attention mechanism in multiple deep learning network models for training to obtain the reconstructed point cloud.
[0064] When the loss of the deep learning network model is calculated and the network parameters of the deep learning network model are adjusted based on the predicted photon value and the true photon value to obtain a trained deep learning network model, the losses of multiple deep learning network models can be calculated and the network parameters of multiple deep learning network models can be adjusted based on the predicted photon value and the true photon value to obtain multiple trained deep learning network models.
[0065] In this embodiment, when the deep learning network model is trained using the training set, the parameters of multiple deep learning models can be trained using the training set.
[0066] In a possible implementation, in step S110, a validation set and a test set may also be obtained.
[0067] In this embodiment, the NYU V2 dataset can be simulated by a single-photon physical imaging model to obtain the training set and the validation set. At the same time, the Middlebury dataset can be simulated by a single-photon physical imaging model to obtain the test set, wherein the training set, the validation set and the test set can all generate a point cloud with a spatial resolution of 256×256.
[0068] Please refer to Figure 3 , after calculating the losses of multiple deep learning network models and adjusting the network parameters of multiple deep learning network models based on the photon prediction value and the photon true value, thereby obtaining multiple trained deep learning network models, the single photon reconstruction method may also include the following steps.
[0069] Step S210: Use the validation set to select one of the trained deep learning models as the deep learning network model to be evaluated.
[0070] Step S220: Use the test set to evaluate the deep learning network model to be evaluated to obtain an evaluated deep learning network model.
[0071] In this embodiment, the validation set can be used to select a trained deep learning model with good convergence effect as the deep learning network model to be evaluated, and the test set can be used to evaluate the deep learning network model to be evaluated to evaluate the generalization ability of the model's single-photon reconstruction, thereby obtaining the evaluated deep learning network model.
[0072] In one possible implementation, when calculating the loss of the deep learning network model based on the predicted photon value and the true photon value, the loss of the deep learning network model can be calculated based on the predicted photon value and the true photon value using a negative Poisson log-likelihood function as a loss function.
[0073] Specifically, since both signal photons and noise conform to Poisson distribution, the negative Poisson log-likelihood function can be used as the loss function to calculate the loss of the deep learning network model.
[0074] In one possible implementation, the negative Poisson log-likelihood function can be:
[0075]
[0076] in, represents the predicted value of photons, and y represents the actual value of photons.
[0077] In a possible implementation, in step S120, when the deep learning network model is trained using the training set, the hyperparameters of the deep learning network model may also be adjusted through a learning rate algorithm to obtain a trained deep learning network model.
[0078] Exemplarily, when the hyperparameters of the deep learning network model are adjusted by a learning rate algorithm, the learning rate of the deep learning network model can be adjusted.
[0079] In a possible implementation, when the hyperparameters of the deep learning network model are adjusted by a learning rate algorithm, the hyperparameters of the deep learning network model may be adjusted by a cosine annealing algorithm.
[0080] In this embodiment, when training the deep learning network model, the optimizer can use AdamW (Adam with Weight Decay), the learning rate can be dynamically adjusted using the cosine annealing method, the initial value of the learning rate can be 1e-5, and the training graphics processor (GPU) can use A100.
[0081] In one possible implementation, see Figure 4 , after calculating the loss of the deep learning network model and adjusting the network parameters of the deep learning network model based on the predicted value of the photon and the true value of the photon to obtain the trained deep learning network model, the single photon reconstruction method may further include the following steps.
[0082] Step S310, obtaining a point cloud to be processed at a specific distance.
[0083] Step S320: input the point cloud to be processed into the trained deep learning network model to obtain a target point cloud, and generate a depth image and an intensity image based on the target point cloud.
[0084] In this embodiment, the point cloud to be processed at a specific distance can be collected, and the point cloud to be processed can be input into the denoising module 100 and the reconstruction module 200 of the trained deep learning network model to obtain the target point cloud, thereby generating the depth image and the intensity image with more accurate details based on the target point cloud.
[0085] Exemplarily, the point cloud to be processed may be collected at a distance of 1.5 km.
[0086] This embodiment also provides an electronic device 800, please refer to Figure 5 , Figure 5 A block diagram of the electronic device 800 is shown as an example. The electronic device includes a processor 810, a machine-readable storage medium 820 and a single-photon reconstruction device 830. The machine-readable storage medium 820 and the processor 810 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The single-photon reconstruction device 830 includes a plurality of software function modules that can be stored in the machine-readable storage medium 820 in the form of software or firmware or solidified in the operating system (OS) of the single-photon reconstruction device 830. The processor 810 is used to execute executable modules stored in the machine-readable storage medium 820, such as software function modules and computer programs included in the single-photon reconstruction device 830.
[0087] The machine-readable storage medium 820 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium 820 is used to store a program, and the processor 810 executes the program after receiving an execution instruction.
[0088] The processor 810 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 810 may be a general-purpose processor 810, including a central processing unit 810 (CPU), a network processor 810 (NP), etc.; it may also be a digital signal processor 810 (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor 810 may be a microprocessor 810 or the processor 810 may also be any conventional processor 810, etc.
[0089] Please refer to Figure 6 The present application also provides a single photon reconstruction device 830, which includes a plurality of functional modules that can be stored in a machine-readable storage medium 820 in the form of software. Functionally, the single photon reconstruction device 830 may include an acquisition module 831, a training module 832, an auxiliary training module 833, and an adjustment module 834. Among them:
[0090] The acquisition module 831 is used to acquire the real value of photons, and simulate and generate a training set according to the real value of photons, wherein the training set includes the simulated value of photons.
[0091] In this embodiment, the acquisition module 831 can be used to perform Figure 1 As shown in step S110, for the specific description of the acquisition module 831, reference may be made to the description of the step S110.
[0092] The training module 832 is used to input the photon simulation value into the denoising module 100 and the reconstruction module 200 constructed based on the attention mechanism in the deep learning network model for training to obtain a reconstructed point cloud.
[0093] In this embodiment, the training module 832 can be used to perform Figure 1 As shown in step S120, for the specific description of the training module 832, please refer to the description of the step S120.
[0094] The auxiliary training module 833 is used to input the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain the photon prediction value.
[0095] In this embodiment, the auxiliary training module 833 can be used to perform Figure 1As shown in step S130, for the detailed description of the auxiliary training module 833, reference may be made to the description of step S130.
[0096] The adjustment module 834 is used to calculate the loss of the deep learning network model and adjust the network parameters of the deep learning network model based on the predicted photon value and the actual photon value, so as to obtain a trained deep learning network model.
[0097] In this embodiment, the adjustment module 834 can be used to perform Figure 1 As shown in step S140, for the detailed description of the adjustment module 834, please refer to the description of step S140.
[0098] In summary, the present application provides a single-photon reconstruction method, device and electronic device. By training the denoising module and reconstruction module of the deep learning network model, and performing long-distance auxiliary training of the deep learning network model through the auxiliary training module, higher resolution and higher precision rapid imaging can be achieved, and the computing speed can be greatly improved.
[0099] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0100] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A single photon reconstruction method, characterized in that: The method comprises: Collecting real photon values, and simulating and generating a training set based on the real photon values, wherein the training set includes photon simulation values; Inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud; Inputting the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain a photon prediction value; Based on the predicted photon value and the actual photon value, the loss of the deep learning network model is calculated and the network parameters of the deep learning network model are adjusted to obtain a trained deep learning network model.
2. The single photon reconstruction method according to claim 1, characterized in that: The step of inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud includes: Inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in multiple deep learning network models for training to obtain a reconstructed point cloud; The step of calculating the loss of the deep learning network model and adjusting the network parameters of the deep learning network model based on the predicted photon value and the real photon value to obtain a trained deep learning network model comprises: Based on the predicted photon value and the actual photon value, the losses of the multiple deep learning network models are calculated and the network parameters of the multiple deep learning network models are adjusted to obtain multiple trained deep learning network models.
3. The single photon reconstruction method according to claim 2, characterized in that: The method further comprises: Get the validation set and test set; After the step of calculating the losses of the plurality of deep learning network models and adjusting the network parameters of the plurality of deep learning network models based on the predicted photon values and the true photon values, thereby obtaining a plurality of trained deep learning network models, the method further comprises: Using the validation set, selecting one of the trained deep learning models as a deep learning network model to be evaluated; The deep learning network model to be evaluated is evaluated using the test set to obtain an evaluated deep learning network model.
4. The single photon reconstruction method according to claim 1, characterized in that: The step of calculating the loss of the deep learning network model based on the predicted photon value and the real photon value comprises: Based on the predicted value of the photon and the true value of the photon, the loss of the deep learning network model is calculated using a negative Poisson log-likelihood function as a loss function.
5. The single photon reconstruction method according to claim 4, characterized in that: The negative Poisson log-likelihood function is: in, represents the predicted value of photons, and y represents the actual value of photons.
6. The single photon reconstruction method according to claim 1, characterized in that: The method further comprises: The hyperparameters of the deep learning network model are adjusted through a learning rate algorithm to obtain a trained deep learning network model.
7. The single photon reconstruction method according to claim 6, characterized in that: The step of adjusting the hyperparameters of the deep learning network model by using a learning rate algorithm comprises: The hyperparameters of the deep learning network model are adjusted by a cosine annealing algorithm.
8. The single photon reconstruction method according to claim 1, characterized in that: After the step of calculating the loss of the deep learning network model based on the predicted photon value and the true photon value and adjusting the network parameters of the deep learning network model to obtain a trained deep learning network model, the method further includes: Get the point cloud to be processed at a specific distance; The point cloud to be processed is input into the trained deep learning network model to obtain a target point cloud, and a depth image and an intensity image are generated based on the target point cloud.
9. A single photon reconstruction device, characterized in that: include: A collection module, used for collecting real photon values, and simulating and generating a training set according to the real photon values, wherein the training set includes simulated photon values; A training module, used for inputting the photon simulation value into a denoising module and a reconstruction module constructed based on an attention mechanism in a deep learning network model for training to obtain a reconstructed point cloud; An auxiliary training module, used for inputting the reconstructed point cloud into the auxiliary training model of the deep learning network model to obtain a photon prediction value; An adjustment module is used to calculate the loss of the deep learning network model and adjust the network parameters of the deep learning network model based on the predicted value of the photon and the true value of the photon, so as to obtain a trained deep learning network model.
10. An electronic device, characterized in that: The electronic device includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method described in any one of claims 1 to 8 is implemented.