Scanning light field self-supervised network reconstruction method and apparatus, electronic device, and medium

Abstract

The present application relates to the technical field of computational imaging, and in particular to a scanning light field self-supervised network reconstruction method and apparatus, an electronic device, and a medium. The method comprises: acquiring scanning light field data and a point spread function of an optical system capturing the scanning light field data; preprocessing the scanning light field data to obtain preprocessed data; and constructing a self-supervised reconstruction network on the basis of the preprocessed data and the point spread function until a preset iteration stop condition is reached, to obtain a scanning light field self-supervised network reconstruction result. Thus, the present application solves the problems in the related art that the need to provide high-resolution ground truth images leads to increased costs, longer times consumed, and reduced reconstruction efficiency, and the existence of serious reconstruction artifacts at the original object plane easily leads to loss or distortion of details on object surfaces and reduced quality and accuracy of reconstruction results.

Description

Scanning light field self-supervised network reconstruction method, device, electronic device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410074899.5 and application date on January 18, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of computational imaging technology, and in particular to a method, device, electronic device, and medium for self-supervised network reconstruction of a scanned light field. Background Art

[0004] Light fields, or scanned light fields, have attracted considerable attention for their ability to achieve rapid, large-scale 3D imaging using a single or small number of captured images. They are widely used in biological applications, including 3D calcium imaging. Reconstruction is a crucial step in scanned light field imaging and a major factor affecting image quality. Traditional deconvolution methods based on light propagation models can be used to restore 3D structures from captured images based on the point spread function of the optical system and the light propagation model. Alternatively, deep learning-based network reconstruction methods can be used, using light field or scanned light field data as input and high-resolution ground truth images as supervision, enabling the network to learn the reconstruction process.

[0005] However, in related technologies, the need to provide high-resolution true-value images leads to the need to use additional microscopy methods to photograph the same set of samples, which increases costs, takes a long time, and reduces reconstruction efficiency. In addition, due to problems such as serious reconstruction artifacts at the original object plane, it is easy to cause details on the object surface to be lost or distorted, thereby affecting the quality and accuracy of the reconstruction results.

[0006] Summary of the Invention

[0007] The present application provides a method, device, electronic device, and storage medium for self-supervised network reconstruction of a scanned light field to address the problems in related technologies such as the need to provide high-resolution true-value images, which increases costs, takes a long time, and reduces reconstruction efficiency. Furthermore, due to the presence of severe reconstruction artifacts at the original object plane, details on the object surface are easily lost or distorted, reducing the quality and accuracy of the reconstruction results.

[0008] A first aspect of the present application provides a method for self-supervised network reconstruction of a scanned light field, comprising the following steps: obtaining scanned light field data and a point spread function of an optical system that captures the scanned light field data; preprocessing the scanned light field data to obtain preprocessed data; and constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

[0009] Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, including: rearranging the pixel points of the scanned light field data based on their positions behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

[0010] Optionally, in one embodiment of the present application, preprocessing the scanned light field data to obtain preprocessed data further includes: performing a linear transformation on the multi-angle data so that the pixel values ​​are within a preset range to obtain transformed data.

[0011] Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a self-supervised network reconstruction result of the scanned light field, including: selecting any image processing network, inputting multi-angle data and forwarding it to obtain network output; randomly selecting several angles of the point spread function, and forward projecting the network output.

[0012] Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a self-supervised network reconstruction result of the scanned light field, and further includes: calculating the mean square error between the network output and the corresponding angle of the forward projection; calculating the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; weighting the mean square error, the second norm and the axial continuity constraint, calculating the self-supervised loss function of the self-supervised reconstruction network, and returning the updated network parameters to obtain the self-supervised network reconstruction result of the scanned light field.

[0013] A second aspect of the present application provides a scanning light field self-supervised network reconstruction device, including: an acquisition module for acquiring scanning light field data and a point spread function of an optical system that captures the scanning light field data; a processing module for preprocessing the scanning light field data to obtain preprocessed data; and a reconstruction module for constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanning light field self-supervised network reconstruction result.

[0014] Optionally, in one embodiment of the present application, the processing module includes: a generation unit, used to rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

[0015] Optionally, in one embodiment of the present application, the processing module further includes: a transformation unit, configured to perform a linear transformation on the multi-angle data so that the pixel values ​​are within a preset range, thereby obtaining transformed data.

[0016] Optionally, in one embodiment of the present application, the reconstruction module includes: a forward transmission unit, used to select any image processing network, input the multi-angle data and forward transmit it to obtain the network output; a projection unit, used to randomly select several angles of the point spread function and obtain forward projection of the network output.

[0017] Optionally, in one embodiment of the present application, the reconstruction module further includes: a first calculation unit, used to calculate the mean square error between the network output and the corresponding angle of the forward projection; a second calculation unit, used to calculate the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; a weighting unit, used to weight the mean square error, the second norm and the axial continuity constraint, calculate the self-supervised loss function of the self-supervised reconstruction network, and feed back the updated network parameters to obtain the scanned light field self-supervised network reconstruction result.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field self-supervised network reconstruction method as described in the above embodiment.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned scanning light field self-supervised network reconstruction method.

[0020] The embodiments of the present application can pre-process the scanned light field data by obtaining the point spread function of the optical system that captured the scanned light field data, and construct a self-supervised reconstruction network based on the pre-processed data and the point spread function until a preset iterative stop condition is reached, thereby obtaining the reconstruction results of the scanned light field self-supervised network. The network can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, reducing the need for manual intervention, and avoiding the influence of subjective factors and operational errors. This solves the problems in the related art such as the need to provide high-resolution true value images, which leads to increased costs, longer time consumption, and reduced reconstruction efficiency, and the presence of severe reconstruction artifacts at the original object plane, which easily leads to loss or distortion of surface details of the object, thereby reducing the quality and accuracy of the reconstruction results.

[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] FIG1 is a schematic diagram of the structure of a scanning light field self-supervised network reconstruction method according to one embodiment of the present application;

[0024] FIG2 is a flow chart of a scanning light field self-supervised network reconstruction method provided according to an embodiment of the present application;

[0025] FIG3 is a schematic diagram showing a comparison between an original image of a scanned light field and an image reconstructed through a network according to an embodiment of the present application;

[0026] FIG4 is a schematic diagram illustrating the principle of random multi-angle forward transmission and self-supervised loss function calculation process according to one embodiment of the present application;

[0027] FIG5 is a schematic diagram of the structure of a scanning light field self-supervisory network reconstruction device provided according to an embodiment of the present application;

[0028] FIG6 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the scanning light field self-supervised network reconstruction method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the related technologies mentioned in the background technology center above, due to the need to provide high-resolution true value images, the cost is increased, the time is long, the reconstruction efficiency is reduced, and due to the existence of serious reconstruction artifacts at the original object plane, the details of the object surface are easily lost or distorted, and the quality and accuracy of the reconstruction results are reduced. The present application provides a scanning light field self-supervised network reconstruction method, in which the scanning light field data can be pre-processed by obtaining the point spread function of the optical system that shoots the scanning light field data, and a self-supervised reconstruction network is constructed based on the pre-processed data and the point spread function until a preset iterative stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result, which can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, and reducing the need for manual intervention, avoiding the influence of subjective factors and operational errors. This solves the problems in related technologies, such as the need to provide high-resolution true-value images, which increases costs, takes a long time, and reduces reconstruction efficiency. In addition, the existence of serious reconstruction artifacts at the original object plane easily leads to loss or distortion of object surface details, thereby reducing the quality and accuracy of the reconstruction results.

[0031] Before explaining the scanning light field self-supervised network reconstruction method provided in the embodiment of the present application, the structure of the scanning light field self-supervised network reconstruction method involved in the embodiment of the present application is first explained.

[0032] As shown in FIG1 , the structure of the scanning light field self-supervised network reconstruction method includes: a scanning light field data acquisition unit, a data preprocessing unit, a self-supervised reconstruction network training unit, and a self-supervised reconstruction network testing unit.

[0033] Among them, a scanning light field data unit is obtained to provide data for network training and testing, and the point spread function of the optical system that captured the data is passed to the random multi-angle forward transmission module trained by the self-supervised reconstruction network, wherein the data and the corresponding point spread function can be captured by a scanning optical microscope or downloaded from a public dataset;

[0034] The data preprocessing unit includes data rearrangement function, data normalization function and data amplification function.

[0035] Among them, the data rearrangement function is used to rearrange the pixels of the scanned light field data according to their positions behind the microlens. All pixels at the same position behind the microlens are arranged in spatial order to form 3D multi-angle data. The data normalization function is used to perform linear transformation on the multi-angle data so that the pixel values ​​are between 0 and 1 to obtain normalized data. The data augmentation function is used to perform random angle rotation, flipping, cropping and other operations on the normalized data to obtain the final amplified data. The final data will be used for self-supervised reconstruction network training or self-supervised reconstruction network testing.

[0036] The self-supervised reconstruction network training unit includes the network input forward transmission function, the random multi-angle forward transmission function and the self-supervised loss function return transmission function. Completing the above three steps is called an iteration.

[0037] Among them, the network input forward transmission function receives the final data output by the data preprocessing unit through the input layer, and passes it through the network to obtain the network output through the output layer; the random multi-angle forward transmission function is used to randomly select several angles of the point spread function, convolve the point spread function of the corresponding angle with the network output to obtain the forward projection, and adjust the size of the forward projection according to the scanning magnification and the number of pixels after the microlens; the self-supervised loss function return function is used to first calculate the self-supervised loss function value, and then perform gradient return to update the network parameters, wherein the self-supervised loss function value is formed by the mean square error between the adjusted forward projection and the corresponding angle of the network input, the second norm of the second-order derivative of the network output, and the axial continuity constraint of the network output. Usually, a threshold number of iterations is set. When it does not exceed this value, it is considered that the network training has not ended. When the network training has not ended, it enters the next iteration, that is, the next batch of data undergoes network input forward transmission, random multi-angle forward transmission, and self-supervised loss function return;

[0038] The self-supervised reconstruction network testing unit is performed after the self-supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

[0039] Among them, the selected data often does not overlap with the data used by the self-supervised reconstruction network training unit, and the size is not necessarily the same. If the size is different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the network input layer. Let the network predict separately and then splice the prediction results.

[0040] Next, the scanning light field self-supervised network reconstruction method of the embodiment of the present application is described in detail.

[0041] Specifically, FIG2 is a flow chart of a scanning light field self-supervised network reconstruction method provided in an embodiment of the present application.

[0042] As shown in Figure 2, the scanned light field self-supervised network reconstruction method includes the following steps:

[0043] In step S201 , scanning light field data and a point spread function of an optical system for capturing the scanning light field data are obtained.

[0044] It can be understood that the light field refers to the spatial and temporal distribution of information such as light intensity, direction, and phase, and scanned light field data refers to a series of images or measurements obtained by sampling the light field from different angles and positions.

[0045] Specifically, the embodiments of the present application can obtain scanning light field data and the point spread function of the optical system that shoots the scanning light field data through a scanning optical microscope. For example, through a scanning optical microscope, image information of cells at different angles and positions can be obtained, including light source brightness, light direction, etc., and the point spread function of the optical system that shoots the scanning light field data can be obtained through calculation and measurement.

[0046] The embodiment of the present application improves the comprehensiveness of image information by acquiring scanning light field data and the point spread function of the optical system that captures the scanning light field data, and provides a data basis for subsequent processing.

[0047] In step S202 , the scanned light field data is preprocessed to obtain preprocessed data.

[0048] It can be understood that preprocessing includes data rearrangement, data normalization, data amplification, etc.

[0049] Specifically, the embodiment of the present application can rearrange the pixel points of the scanned light field data according to their positions behind the microlens, and all the pixel points at the same position behind the microlens are arranged in sequence according to spatial order to form 3D multi-angle data. The multi-angle data can be linearly transformed so that the pixel values ​​are between 0 and 1 to obtain normalized data. The normalized data can be subjected to random angle rotation, flipping, cropping and other operations to obtain the final amplified data.

[0050] The embodiments of the present application can obtain preprocessed data by preprocessing the scanned light field data, such as data rearrangement, data normalization, data amplification, etc., thereby improving the precision and accuracy of the data, improving the clarity of the image, and further improving the authenticity of the reconstruction results.

[0051] Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, including: rearranging the pixel points of the scanned light field data based on their positions behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

[0052] Specifically, the embodiment of the present application can rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system, and all the pixel points at the same position behind the microlens are arranged in sequence in spatial order, thereby forming 3D multi-angle data. For example, the scanned light field data includes 5 perspectives scanned from left to right in the horizontal direction, each perspective has 10×10 pixels. For the first perspective, after rearrangement, the 10 pixel points in the first row will correspond to a specific spatial position. Similarly, the 10 pixel points in the second row will correspond to the next spatial position, and so on. Subsequently, for other perspectives, the pixel points will be arranged in sequence according to the same spatial position.

[0053] The embodiments of the present application can rearrange the pixel points of the scanned light field data based on their positions behind the microlens of the optical system to generate multi-angle data, and can obtain 3D multi-angle data based on the rearranged positions behind the microlens, thereby providing more comprehensive and accurate information, reducing artifacts, improving detail restoration, and ensuring the accuracy of the reconstruction results.

[0054] Optionally, in one embodiment of the present application, preprocessing the scanned light field data to obtain preprocessed data further includes: performing a linear transformation on the multi-angle data so that the pixel values ​​are within a preset range to obtain transformed data.

[0055] It is understandable that the preset range refers to a preset pixel value range of data, such as scaling the pixel value to between 0 and 1.

[0056] Specifically, the embodiments of the present application can perform a linear transformation on multi-angle data so that the pixel values ​​are within a preset range to obtain the changed data. For example, if the pixel values ​​need to be scaled to between 0 and 1, a linear transformation can be performed to map them from the current pixel value range to the target range to obtain the changed data.

[0057] The embodiment of the present application performs a linear transformation on multi-angle data so that the pixel values ​​are within a preset range to obtain the changed data, which can effectively improve the contrast and display effect of the image, help identify and analyze details in the image, and reduce visual inconsistency.

[0058] In step S203, a self-supervised reconstruction network is constructed according to the pre-processed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

[0059] It is understandable that the preset iteration stopping condition may be that the number of iterations reaches a preset iteration threshold, such as the number of iterations reaches 50,000 times.

[0060] Specifically, the embodiments of the present application can utilize preprocessed data for supervised reconstruction network training or supervised reconstruction network testing, wherein self-supervised reconstruction network training includes network input forward transmission, random multi-angle forward transmission and self-supervised loss function return transmission. Completion of the above three steps is called an iteration. When the network training is not completed, the next iteration is entered, that is, the next batch of data passes through the network, random multi-angle forward transmission and self-supervised loss function return transmission; self-supervised reconstruction network testing is performed after the self-supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

[0061] For example, in combination with FIG3 , the embodiment of the present application can construct a self-supervised reconstruction network based on the preprocessed data and the point spread function until the preset iteration stop condition is reached to obtain the scanning light field self-supervised network reconstruction result, wherein the image reconstructed by the network is a 3D image, but is displayed as a projection along the z-axis. Specifically, the embodiment of the present application can use a scanning light field instrument (scanning magnification is 3, the number of pixels after the microlens is 13×13) to shoot zebrafish embryo data, and then rearrange, normalize and amplify the data to generate 1,000 multi-angle images of size 60×60×169 for self-supervised reconstruction network training; use the Keras deep learning framework based on Tensorflow and the Python programming language to build a self-supervised reconstruction network, specifically, use bilinear interpolation after the input layer to adjust the network input to the target size, and then pass through a U-Net, and further, through the training network, wherein the initial learning rate is 1×10 -4 The training batch size is 3, and the Adam optimizer is used for back-propagation iterative optimization. A total of 50,000 iterations are trained, and the learning rate is reduced to half of the original every 10,000 iterations. This allows the multi-angle test images to be input into the trained self-supervised reconstruction network to obtain the reconstructed images.

[0062] The embodiments of the present application can utilize preprocessed data and point spread functions to gradually improve the ability to reconstruct light field images during the training process. Through iterative training, the adaptability and generalization ability to different samples and scenes can be improved, thereby improving the accuracy and efficiency of reconstruction.

[0063] Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a self-supervised network reconstruction result of the scanned light field, including: selecting any image processing network, inputting multi-angle data and forwarding it to obtain network output; randomly selecting several angles of the point spread function, and forward projecting the network output.

[0064] During the actual implementation process, the embodiment of the present application can randomly select two point spread function angles for forward projection, convolve the point spread function of the corresponding angle with the network output to obtain the forward projection, and can adjust the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens.

[0065] Specifically, as shown in FIG4 , the embodiment of the present application adds a random multi-angle forward transmission module and uses information of the point spread function to simulate the optical forward transmission process of the network output. The specific steps are as follows:

[0066] Step S1: Randomly select an angle.

[0067] It is understandable that the core of the light field or scanning light field microscopy system is the microlens array in front of the camera. There are Nnum×Nnum camera photosensitive pixels behind each microlens, also known as angles. Because pixels at different positions correspond to different angles, and the angle information at the edge of the microlens is small and the light intensity is low, only the angles within the circle with a radius of r and the microlens optical center as the center are used. Among them, r is generally one or two pixels less than half of Nnum. It can also be specified based on computing resources and actual needs, and N angles are selected from the angles within the circle for subsequent calculations. The selection method can be as follows:

[0068] (1) N angles are randomly selected in each iteration;

[0069] (2) Each iteration has M fixed angles that do not change with each iteration, and NM randomly selected angles that do not overlap with the fixed M angles. For example, M = 1, the angle where the optical center of the microlens is fixed, and N-1 randomly selected angles other than the angle where the optical center of the microlens is located;

[0070] (3) NM angles are randomly selected in each iteration, and another M angles are randomly selected from the N angles selected in the previous iteration, and they are guaranteed not to overlap with the NM angles selected in this iteration;

[0071] (4) Divide the circle formed by arranging all the selectable angles into N equal parts, and randomly select an angle from each of the N parts in each iteration;

[0072] It is worth noting that the above selection method may cause slightly different network convergence speeds in different embodiments, but the final effect is theoretically similar, and the purpose of selecting some angles for calculation is to save memory and video memory overhead and avoid memory or video memory overflow during network training. After a sufficient number of training iterations, due to the randomness of angle selection, the network optimization result should theoretically be the same as the result of selecting all angles for calculation each time.

[0073] Step S2: Convolve the point spread function of the corresponding angle with the network output.

[0074] Among them, for each angle selected in step S1, the part of the point spread function corresponding to the angle, that is, a 3D vector, is convolved with the 3D network output to obtain the forward projection. In the actual implementation process, the frequency domain operation can be used instead of the spatial domain convolution, which can be shown as follows: Proj i =iFFTshift{iFFT[FFTshift(FFT(Output))·FFTshift(FFT(PSF i ))]}

[0075] Among them, Proj i is the i-th angle component of the forward projection, which is a 2D image; PSF i is the i-th angular component of the point spread function; Output is the network output; FFT is the fast Fourier transform; FFTshift is the operation of shifting the spectrum so that the zero frequency is in the center; iFFT is the inverse operation of FFT; iFFTshift is the inverse operation of FFTshift.

[0076] Then, Proj1,...,Proj N Splice them into a 3D vector Proj_tmp and get the forward projection of N angles.

[0077] Step S3: adjusting the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens.

[0078] Specifically, the relationship between the adjusted forward projection and the forward projection obtained by splicing in step S2 can be determined by the following expression: Proj = ImResize (Proj_tmp, scanning / Nnum)

[0079] Where Proj is the adjusted forward projection; Proj_tmp is the forward projection obtained by splicing in step S2; ImResize(Img,scale) is a method for adjusting the image size, such as bilinear interpolation or nearest neighbor interpolation, which adjusts Img to scale times the original size; scanning is the scanning magnification; Nnum is the number of pixels in a column or row behind the microlens.

[0080] It is worth noting that in steps S1 to S3, the embodiment of the present application completes the optical forward transmission simulation of the network output Output. If the parts corresponding to N angles are selected from the network input and recorded as Input, then Input and Proj have the same size.

[0081] The embodiment of the present application randomly selects the point spread function angle for forward projection and adjusts the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens, which can improve the comprehensiveness and diversity of the projection results and thus improve the applicability of the scanning light field technology.

[0082] Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a self-supervised network reconstruction result of the scanned light field, and further includes: calculating the mean square error between the network output and the corresponding angle of the forward projection; calculating the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; weighting the mean square error, the second norm and the axial continuity constraint, calculating the self-supervised loss function of the self-supervised reconstruction network, and returning the updated network parameters to obtain the self-supervised network reconstruction result of the scanned light field.

[0083] Specifically, in conjunction with FIG4 , the embodiment of the present application can calculate the random multi-angle forward transmission and self-supervised loss function, and the specific steps are as follows:

[0084] Step S4: Calculate the self-supervised loss function.

[0085] Specifically, the function value can be determined by the following expression: L(Proj,Input,Output)=αMSE(Proj,Input)+βHess(Output)+γCont(Output)

[0086] Among them, Proj is the adjusted forward projection, Input is the part of the network input corresponding to the angle, Output is the network output, L is the self-supervised loss function, MSE is the mean square error between the forward projection and the network input, Hess is the second norm of the second derivative of the network output, Cont is the axial continuity constraint of the network output, α is the weight of the mean square error, β is the weight of the second norm of the second derivative, and γ is the weight of the continuity constraint.

[0087] The axial continuity constraint of the network output can be determined by the following expression:

[0088] Wherein, Output is the network output, which is a three-dimensional vector, the third dimension represents the axial direction, Cont is the axial continuity constraint of the network output, N is the number of axial pixels output by the network, and Sum(·) is the sum of each component of the two-dimensional vector.

[0089] The embodiments of the present application can quantify the error between the network reconstruction result and the real light field data by calculating the mean square error between the network output and the forward projection, and can gradually reduce the mean square error by optimizing the self-supervised loss function, thereby improving the accuracy of the reconstruction result. By considering the second norm of the second-order derivative of the network output and the axial continuity constraint, it helps to suppress noise and artifacts in the reconstruction result and improve the smoothness and stability of the reconstruction result.

[0090] According to the scanning light field self-supervised network reconstruction method proposed in the embodiment of the present application, the scanning light field data can be pre-processed by obtaining the point spread function of the scanning light field data and the optical system that captured the scanning light field data, and a self-supervised reconstruction network can be constructed based on the pre-processed data and the point spread function until a preset iterative stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result. The method can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, reducing the need for manual intervention, and avoiding the influence of subjective factors and operational errors. This solves the problems in the related art such as the need to provide high-resolution true value images, which leads to increased costs, longer time consumption, and reduced reconstruction efficiency, and the presence of severe reconstruction artifacts at the original object plane, which easily leads to loss or distortion of surface details of the object, thereby reducing the quality and accuracy of the reconstruction results.

[0091] Next, the scanning light field self-supervised network reconstruction device proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0092] FIG5 is a schematic structural diagram of a scanning light field self-supervisory network reconstruction device according to an embodiment of the present application.

[0093] As shown in FIG5 , the scanning light field self-supervised network reconstruction device 10 includes: an acquisition module 100 , a processing module 200 and a reconstruction module 300 .

[0094] Specifically, the acquisition module 100 is used to acquire the scanning light field data and the point spread function of the optical system that captures the scanning light field data.

[0095] The processing module 200 is used to pre-process the scanned light field data to obtain pre-processed data.

[0096] The reconstruction module 300 is used to construct a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

[0097] Optionally, in one embodiment of the present application, the processing module 200 includes: a generating unit.

[0098] The generating unit is used to rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order.

[0099] Optionally, in one embodiment of the present application, the processing module 200 further includes: a transformation unit.

[0100] The transformation unit is used to perform linear transformation on the multi-angle data so that the pixel values ​​are within a preset range to obtain the transformed data.

[0101] Optionally, in one embodiment of the present application, the reconstruction module 300 includes: a forward transmission unit and a projection unit.

[0102] The forward transmission unit is used to select any image processing network, input multi-angle data and forward it to obtain network output;

[0103] The projection unit is used to randomly select several angles of the point spread function and obtain the forward projection of the network output.

[0104] Optionally, in one embodiment of the present application, the reconstruction module 300 further includes: a first calculation unit, a second calculation unit and a weighting unit.

[0105] The first calculation unit is used to calculate the mean square error between the network output and the corresponding angle of the forward projection;

[0106] A second calculation unit is used to calculate the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output;

[0107] The weighting unit is used to weight the mean square error, the second norm and the axial continuity constraint, calculate the self-supervised loss function of the self-supervised reconstruction network, and transmit back the updated network parameters to obtain the scanned light field self-supervised network reconstruction result.

[0108] It should be noted that the above explanation of the embodiment of the scanning light field self-supervised network reconstruction method is also applicable to the scanning light field self-supervised network reconstruction device of this embodiment, and will not be repeated here.

[0109] According to the scanning light field self-supervised network reconstruction device proposed in the embodiment of the present application, the scanning light field data can be pre-processed by obtaining the point spread function of the optical system that captured the scanning light field data and constructing a self-supervised reconstruction network based on the pre-processed data and the point spread function until a preset iterative stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result. The device can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, reducing the need for manual intervention, and avoiding the influence of subjective factors and operational errors. This solves the problems in the related art such as the need to provide high-resolution true value images, which leads to increased costs, longer time consumption, and reduced reconstruction efficiency, and the presence of severe reconstruction artifacts at the original object plane, which easily leads to loss or distortion of details on the object surface, reducing the quality and accuracy of the reconstruction results.

[0110] FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0111] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0112] When the processor 602 executes the program, the scanning light field self-supervised network reconstruction method provided in the above embodiment is implemented.

[0113] Furthermore, the electronic device further includes:

[0114] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0115] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0116] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0117] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, FIG6 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0118] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0119] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0120] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned scanning light field self-supervised network reconstruction method.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0124] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0125] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0126] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0127] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for reconstructing a scanned light field self-supervised network, characterized in that It includes the following steps: Obtain the scanned light field data and the point spread function of the optical system that captures the scanned light field data; Preprocess the scanned light field data to obtain preprocessed data; And Construct a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached, and obtain the self-supervised network reconstruction result of the scanned light field.

2. The method for reconstructing a scanned light field self-supervised network according to claim 1, wherein The preprocessing the scanned light field data to obtain preprocessed data includes: Rearrange the pixel points of the scanned light field data based on their positions behind the microlens of the optical system to generate multi-angle data, where all pixel points at the same position behind the microlens are arranged in spatial order.

3. The method for reconstructing a scanned light field self-supervised network according to claim 2, wherein The preprocessing the scanned light field data to obtain preprocessed data further includes: Perform a linear transformation on the multi-angle data to make the pixel values fall within a preset range, and obtain transformed data.

4. The method for reconstructing a scanned light field self-supervised network according to claim 2, wherein The constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached, and obtaining the self-supervised network reconstruction result of the scanned light field includes: Select any image processing network, input the multi-angle data and forward-propagate it to obtain a network output; Randomly select several angles of the point spread function to obtain a forward projection of the network output.

5. The method for reconstructing a scanned light field self-supervised network according to claim 4, wherein The constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached, and obtaining the self-supervised network reconstruction result of the scanned light field further includes: Calculate the mean square error between the corresponding angles of the network output and the forward projection; Calculate the second-order derivative two-norm of the network output and the axial continuity constraint of the network output; Weight the mean square error, the two-norm and the axial continuity constraint, calculate the self-supervised loss function of the self-supervised reconstruction network, and backpropagate to update the network parameters to obtain the self-supervised network reconstruction result of the scanned light field.

6. A scanning light field self-supervised network reconstruction device, characterized in that, It includes: An acquisition module for obtaining the scanned light field data and the point spread function of the optical system that captures the scanned light field data; A processing module for preprocessing the scanned light field data to obtain preprocessed data; And A reconstruction module for constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached, and obtaining the self-supervised network reconstruction result of the scanned light field.

7. The scanning light field self-supervised network reconstruction device according to claim 6, characterized in that, The processing module includes: A generation unit for rearranging the pixel points of the scanned light field data based on their positions behind the microlens of the optical system to generate multi-angle data, where all pixel points at the same position behind the microlens are arranged in spatial order.

8. The scanning light field self-supervised network reconstruction device according to claim 7, characterized in that, The processing module further includes: A transformation unit for performing a linear transformation on the multi-angle data to make the pixel values fall within a preset range, and obtaining transformed data.

9. The scanning light field self-supervised network reconstruction device according to claim 7, characterized in that, The reconstruction module includes: A forward-propagation unit for selecting any image processing network, inputting the multi-angle data and forward-propagating it to obtain a network output; A projection unit for randomly selecting several angles of the point spread function to obtain a forward projection of the network output.

10. The scanning light field self-supervised network reconstruction device according to claim 9, characterized in that, The reconstruction module further includes: A first calculation unit for calculating the mean square error between the network output and the corresponding angle of the forward projection; A second calculation unit for calculating the two-norm of the second derivative of the network output and the axial continuity constraint of the network output; A weighting unit for weighting the mean square error, the two-norm, and the axial continuity constraint, calculating the self-supervised loss function of the self-supervised reconstruction network, and backpropagating to update network parameters to obtain the reconstruction result of the scanned light field self-supervised network.

11. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the scanned light field self-supervised network reconstruction method according to any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor for implementing the scanned light field self-supervised network reconstruction method according to any one of claims 1-5.

Images