Scanning light field multispectral reconstruction network method, device, electronic device and medium

Abstract

The present invention relates to the field of computer imaging technology, and more particularly to a method, apparatus, electronic device, and medium for reconstructing a multispectral network for a scanned light field. The method comprises: acquiring scanned light field data, generating a training set and a test set based on the scanned light field data; using the training set to perform self-supervised iterative training on the multispectral reconstruction network, and testing the trained multispectral reconstruction network using the test set; using the tested multispectral reconstruction network to generate light field angles and perform spectral splitting, and reconstructing full-angle multispectral data based on the generated light field angles and split spectra. This method solves the problems of the related art, such as the long time required for multispectral network reconstruction, the inability to train large amounts of data, the incomplete spectral splitting, and the lack of coherence in light field information.

Description

Technical Field

[0001] The present invention relates to the field of computational imaging technology, and in particular to a scanning light field multispectral reconstruction network method, device, electronic equipment and medium. Background Art

[0002] Scanning light fields can rapidly store three-dimensional light data using high-resolution two-dimensional information, enabling fast, high-definition, and large-scale three-dimensional imaging. They are widely used in biological applications such as neuroscience and immune interactions. Multispectral imaging is also crucial for biological observations. For traditional scanning light fields, time-division multiplexing reduces the imaging rate exponentially, but can also produce motion artifacts if the sample moves rapidly.

[0003] There are currently two main methods for reconstructing multispectral images from RGB (Red-Green-Blue, three-channel) images: a mathematical method based on sparse matrix decomposition and inverse matrix solution, and a multispectral network reconstruction method based on traditional deep learning.

[0004] Mathematical methods based on sparse matrix decomposition and inverse matrix solutions, including linear decomposition and clarification, dictionary learning, and least squares methods and K-SVD (K-Singular Value Decomposotion, a dictionary training algorithm), are used to restore the original matrix solution and split it into multiple spectra. However, these methods are time-consuming, cannot train large amounts of data, and suffer from problems such as incomplete spectral splitting.

[0005] Traditional deep learning-based network reconstruction methods use RGB data as input and multispectral data as supervision, allowing the network to fit the mathematical process of spectral decomposition. However, for scanned light field data, the data acquired by the Bayer color array results in missing light field angles. Traditional reconstruction networks that learn spectral features cannot fully learn the physical relationship between light field angles and spatial information. While the spectrum can be decomposed, the coherence of the light field information is lost. Furthermore, traditional deep learning is not suitable for learning from highly diverse microscopic data, making it difficult to learn spectral decomposition while maintaining its original generalization properties. Summary of the Invention

[0006] The present invention provides a scanning light field multispectral reconstruction network method, device, electronic device and medium to solve the problems in related technologies such as long multispectral network reconstruction time, inability to train large amounts of data, unclean spectral splitting, and lack of light field information coherence.

[0007] A first aspect of the present invention provides a method for a multispectral reconstruction network of a scanned light field, comprising the following steps: acquiring scanned light field data, generating a training set and a test set based on the scanned light field data; performing self-supervised iterative training on a multispectral reconstruction network using the training set, and testing the trained multispectral reconstruction network using the test set; generating light field angles and performing spectrum splitting using the multispectral reconstruction network that has passed the test, and reconstructing full-angle multispectral data based on the generated light field angles and the split spectra.

[0008] Optionally, the multi-spectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract the spatial features of the light field from the input data; the second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial features, the spatial and spectral features and the angular features to obtain a fused feature, split the spectrum and generate a new light field angle according to the fused feature, use the new light field angle to fill the missing angle, and during training, calculate the training loss based on the true value of the new light field angle and the missing angle, and use the training loss to update the network parameters.

[0009] Optionally, the third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of the angle dimension from the features after data transformation.

[0010] Optionally, before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, the method further includes: performing one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set, wherein the data rearrangement processing includes: rearranging the pixel points of the scanned light field data according to their positions behind the microlens, and arranging all the pixel points at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and using the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0011] A second aspect of the present invention provides a scanning light field multispectral reconstruction network device, including: a generation module for acquiring scanning light field data and generating a training set and a test set based on the scanning light field data; a testing module for performing self-supervised iterative training on the multispectral reconstruction network using the training set, and testing the trained multispectral reconstruction network using the test set; a reconstruction module for generating light field angles and performing spectrum splitting using the tested multispectral reconstruction network, and reconstructing full-angle multispectral data based on the generated light field angles and split spectra.

[0012] Optionally, the multi-spectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract the spatial features of the light field from the input data; the second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial features, the spatial and spectral features and the angular features to obtain a fused feature, split the spectrum and generate a new light field angle according to the fused feature, use the new light field angle to fill the missing angle, and during training, calculate the training loss based on the true value of the new light field angle and the missing angle, and use the training loss to update the network parameters.

[0013] Optionally, the third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of the angle dimension from the features after data transformation.

[0014] Optionally, it also includes: a processing module for performing one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, wherein the data rearrangement processing includes: rearranging the pixel points of the scanned light field data according to their positions behind the microlens, and arranging all the pixel points at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0015] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field multispectral reconstruction network method as described in the above embodiment.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the scanning light field multispectral reconstruction network method as described in the above embodiments.

[0017] Therefore, the present invention has at least the following beneficial effects:

[0018] The present invention implements a multispectral reconstruction network based on deep learning supervision, rapidly performing spectral splitting and light field angle generation, and obtaining robust, fast multispectral reconstruction of full-angle light field data with short processing times and good noise robustness. This solves technical issues in related technologies, such as the long time required for multispectral network reconstruction, the inability to train large amounts of data, the incomplete spectral splitting, and the lack of coherence in light field information.

[0019] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 Flowchart of a scanning light field multi-spectral reconstruction network method according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a calculation process of a second extraction module provided according to an embodiment of the present invention;

[0023] Figure 3 A diagram showing the specific composition of a scanning light field multi-spectral reconstruction network method provided according to an embodiment of the present invention;

[0024] Figure 4 A comparison diagram of an original image of a scanned light field and an image reconstructed through a multispectral network according to a specific embodiment of the present invention;

[0025] Figure 5 This is an example diagram of a scanning light field multi-spectral reconstruction network device provided according to an embodiment of the present invention;

[0026] Figure 6 A schematic structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0028] The following describes, with reference to the accompanying drawings, a scanning light field multispectral reconstruction network method, apparatus, electronic device, and medium according to embodiments of the present invention. In response to the aforementioned background art issues with current multispectral reconstruction methods, such as long time consumption, incomplete spectral splitting, and a lack of coherence in light field information, the present invention provides a scanning light field multispectral reconstruction network method. This method utilizes scanned light field data to train and test the multispectral network. This method addresses the related art issues of long multispectral network reconstruction time consumption, inability to train large amounts of data, incomplete spectral splitting, and a lack of coherence in light field information.

[0029] Specifically, Figure 1 A flowchart of a scanning light field multispectral reconstruction network method provided by an embodiment of the present invention.

[0030] like Figure 1 As shown, the scanning light field multispectral reconstruction network method includes the following steps:

[0031] In step S101 , scanned light field data is acquired, and a training set and a test set are generated according to the scanned light field data.

[0032] It is understandable that the embodiments of the present invention can acquire scanned light field data and generate a training set and a test set based on the scanned light field data.

[0033] In step S102, the multispectral reconstruction network is iteratively trained in a self-supervised manner using the training set, and the trained multispectral reconstruction network is tested using the test set.

[0034] It is understandable that the embodiment of the present invention can use the training set to perform self-supervised iterative training on the multispectral reconstruction network. After the training is completed, the trained multispectral reconstruction network can be tested using the test set to test the final performance of the multispectral reconstruction network.

[0035] It should be noted that since the selected data often do not overlap with the data used for multispectral reconstruction network training, and the sizes are not necessarily the same, if the sizes are different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the multispectral reconstruction network input layer. The multispectral reconstruction network is allowed to predict them separately, and then the prediction results are spliced.

[0036] In an embodiment of the present invention, a multispectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract the spatial features of the light field from the input data; the second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss based on the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0037] Specifically, the first extraction module processes any input data to extract spatial features from the input data, input the spatial features and forward them; the second extraction module is used to extract spectral features of the light field from the input data, and can extract spectral features through a three-dimensional U-shaped network structure to increase the learning ability of the spectral reconstruction network, and generate spatial and spectral features based on the spatial features and spectral features; the third extraction module is used to extract the extracted spatial and spectral features to obtain the angular features of the light field; the backpropagation module is used to supervise the loss backpropagation, perform spectral splitting on the features extracted by all extraction modules, and perform loss function backpropagation with the supervision data to update the network. Specifically, it can obtain fused features by fusing spatial features, spatial and spectral features and angular features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss according to the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0038] The process of extracting spectral features by the second extraction module is as follows: Figure 2 As shown, the following steps are included:

[0039] Step 1: The data passes through a 3D residual group, which consists of a three-layer structure: a three-dimensional convolution kernel, an activation layer, and a three-dimensional convolution layer. For example, data A passes through the first layer, which is a three-dimensional convolution kernel with a kernel size of 3×3×3, and then passes through the second layer, which is a LeakyReLU activation layer. Finally, it passes through the third layer, which is a three-dimensional convolution kernel with a kernel size of 3×3×3. The third layer outputs data B. Data A plus data B obtain the final output data C.

[0040] Step 2: The network undergoes convolution to reduce the data size to half of its original size and double the number of channels, and then passes through a 3D residual group;

[0041] Step 3: The network undergoes convolution to reduce the data size to one-third of the original size and triple the number of channels.

[0042] Step 4: The network passes through a 3D residual group and then a layer of 3D convolution, which reduces the data size to half of the initial size and the number of channels to twice the initial size. The output is added to the output of step 2.

[0043] Step 5: The network passes through a 3D residual group and a layer of 3D convolution to change the data size to the initial size, the number of channels to the initial size, and the output is added to the output of step 1;

[0044] Step 6: The network passes through a 3D residual group to obtain the final output result.

[0045] The relationship between the root mean square error of the true value data and the final output of the multispectral reconstruction network is as follows:

[0046] Loss(GT,Output)=MSE(GT,Output);

[0047] Among them, GT represents the supervised true value data required by the multispectral reconstruction network, Output represents the output of the multispectral reconstruction network, Loss represents the supervised loss function, and MSE represents the mean square error between the true value data and the final output of the multispectral reconstruction network.

[0048] It should be noted that the embodiments of the present invention usually set a threshold for the number of iterations. When the number of iterations is not exceeded, the multispectral reconstruction network training is considered not to be completed. When the multispectral reconstruction network training is not completed, the next iteration is entered, that is, the next batch of data enters feature extraction and return.

[0049] Specifically, the embodiment of the present invention is based on a multi-module deep learning supervised network for training. The original image is subjected to three extraction modules to extract the features of the light field, and a feedback module is used to train a multispectral reconstruction network for fast spectrum splitting and under-angle interpolation. It has strong generalization for different data and is relatively robust to noise. It learns light field data through multiple modules, and learns the light field physical information and multispectral fusion process. In addition, the multispectral reconstruction network has a simple structure and a wide range of applications.

[0050] In an embodiment of the present invention, the third extraction module is further configured to perform data dimension transformation on the spatial and spectral features, and extract angle features of the angle dimension from the features after the data transformation.

[0051] It can be understood that the third extraction module of the embodiment of the present invention can perform data dimension transformation on the spatial and spectral features, extract the features of the angle dimension after the transformation, and obtain the full data features. That is, after the data conversion of SAIs and MacPIs, the two-dimensional convolution structure is used to further separate and extract the angle features, and all data features are extracted.

[0052] Among them, the expression for data dimension transformation of spatial domain and spectral features is as follows:

[0053] ;

[0054] Among them, SAIs (sub-aperture image) indicates that the data is stored in the form of SAIs, which is a four-dimensional vector. The third and fourth dimensions represent the light field angle coordinates. MacPIs (macro pixel image) indicates that the data is stored in the form of MacPIs, which is a four-dimensional vector. The third and fourth dimensions represent the light field spatial coordinates. H represents the light field spatial horizontal coordinate, W represents the light field spatial vertical coordinate, U represents the light field angle horizontal coordinate, and V represents the light field angle vertical coordinate.

[0055] In an embodiment of the present invention, before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, the method further includes: performing one or more of data rearrangement processing, data interpolation processing, data normalization processing, and data augmentation processing on the training set and the test set, wherein the data rearrangement processing includes: rearranging the pixels of the scanned light field data according to their positions behind the microlens, and arranging all the pixels at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and using the interpolated angles as the true values ​​of the missing angles; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0056] It is understood that, before the embodiment of the present invention uses the training set to perform self-supervised iterative training on the multispectral reconstruction network, the data in the training set and the test set are preprocessed, including data rearrangement, data interpolation, data normalization and data augmentation.

[0057] Specifically, (1) data rearrangement: used to rearrange the pixel points of the scanned light field data according to their positions behind the microlens, and all the pixel points at the same position behind the microlens are arranged in spatial order to form multi-angle data; (2) data interpolation: used to interpolate the angles of the missing angles in the captured scanned light field data, and use the interpolated angles as the true values ​​of the missing angles, and use them as the true value data in the training set for subsequent training and testing of the multispectral reconstruction network; (3) data normalization: used to perform linear transformation on the multi-angle data, specifically, to perform maximum and minimum linear transformation on the multi-angle data so that the pixel values ​​are between 0 and 1; (4) data augmentation: used to randomly and evenly divide the training set into small data matching pairs in order to amplify the training set and enhance the network effect.

[0058] In step S103, the multispectral reconstruction network that has passed the test is used to generate light field angles and split spectra, and full-angle multispectral data is reconstructed based on the generated light field angles and split spectra.

[0059] It can be understood that the embodiments of the present invention can use a multispectral reconstruction network to generate light field angles and perform spectral splitting, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra, thereby utilizing the trained multispectral reconstruction network to quickly perform spectral splitting and generate light field angles, and then restore full-angle multispectral data from a single under-angle light field data for subsequent three-dimensional reconstruction.

[0060] The following will explain the scanning light field multispectral reconstruction network method through specific implementation, such as Figure 3 As shown, the following steps are included:

[0061] (1) Obtain RGB light field data and generate high-resolution training and test sets required by the network.

[0062] (2) Data preprocessing, including: data rearrangement, data interpolation, data normalization and data augmentation modules, specifically: Data rearrangement: rearrange the pixels of the scanned light field data according to their positions behind the microlens, and all pixels at the same position behind the microlens are arranged in spatial order to form multi-angle data; Data interpolation: perform angle interpolation on the captured multi-spectral under-angle scanned light field data, and use it as the true value data in the training set for network training and testing; Data normalization: perform maximum and minimum linear transformation on the multi-angle data, so that the pixel value is between 0 and 1; Data augmentation: randomly and evenly divide the network training set into small data matching pairs, which are used to amplify the training set and enhance the network effect.

[0063] (3) Supervised multispectral network training includes: spatial feature extraction, three-dimensional scanning spectral feature extraction, angle feature extraction and supervised loss function feedback. Completing the above four steps is called an iteration.

[0064] The final data output from data preprocessing is received through the input layer and extracted to obtain the RGB light field spatial features. The spatial features extracted from the spatial features are further used to extract spectral features using a three-dimensional U-shaped network structure to obtain spatial-spectral features. The spatial-spectral features extracted from the three-dimensional scanning spectral features are then converted through SAIs and MacPIs, and the angular features are further separated and extracted using a two-dimensional convolutional structure. At this point, all data features are extracted. The three extracted full-data features are combined to split the required spectrum, simulate and generate new light field angles to fill in the missing angles, and finally the mean square error is calculated with the true value to calculate the loss function, and the loss is back-propagated for iterative update. A threshold for the number of iterations is usually set. If this value is not exceeded, the network training is considered not to have completed. If the network training is not completed, the next iteration is entered, that is, the next batch of data is subjected to spatial feature extraction, three-dimensional scanning spectral feature extraction, angular feature extraction, and the supervision loss function is back-propagated.

[0065] (4) Supervised multispectral reconstruction network testing: This is done after the supervised multispectral reconstruction network training is completed to test the final performance of the selected data on the network. The selected data often do not overlap with the data used for supervised multispectral reconstruction network training, and the size is not necessarily the same; if the size is different, the selected data should be cropped with overlap, and cropped into several images that meet the size requirements of the network input layer. The network is then allowed to predict separately, and the prediction results are then spliced ​​together.

[0066] According to the scanning light field multispectral reconstruction network method proposed in an embodiment of the present invention, a deep learning supervised multispectral reconstruction network based on multiple modules can be trained and tested, and the tested multispectral reconstruction network can be used to quickly perform spectral splitting and generate light field angles, and further obtain more robust fast multispectral reconstruction of full-angle light field data, which is time-saving and has good noise robustness.

[0067] The following example uses a specific example to illustrate the comparison between the original RGB scanned light field image and the image reconstructed by the multispectral network. All data in the following examples are displayed in SAIs format. Figure 4 As shown in the figure, the original image of the under-angle RGB scanning light field is split by the network to obtain the full-angle multispectral scanning light field image. The specific steps are as follows:

[0068] Step 1: BSC-1 cells were imaged using a scanning light field instrument with a scanning magnification of 3 and a pixel count of 14×14 behind the microlens. The data was then rearranged, interpolated, normalized, and amplified to generate 400 multi-angle images with a size of 108×108×144×4 for supervised multispectral reconstruction network training.

[0069] Step 2: Use the Pytorch deep learning framework and Python programming language to build a supervised multispectral reconstruction network. Specifically, after the input layer, a 3×3 convolution is used to fuse the RGB images, expanding the channels to 64. Then, a 3DU-Net is used to extract features. After data conversion, it enters a 2D convolution sequence. Finally, all the extracted data are fused and output through a 1×1 convolution layer.

[0070] Step 3: Train the network with an initial learning rate of 1×10^(-4) and a training set size of 400. Use the Adam optimizer for backpropagation iterative optimization for a total of 125 iterations, with the learning rate reduced to half every 25 iterations.

[0071] Step 4: Input the RGB under-angle microscopic light field data into the trained supervised multispectral reconstruction network to obtain a multispectral image.

[0072] In summary, the embodiments of the present invention have the function of under-angle interpolation of RGB light field data, which can quickly split the RGB under-angle microscopic light field data collected by the Bayer color array into multispectral full-angle microscopic light field data. While ensuring high resolution, it improves the temporal resolution by at least four times, and is fast, concise, and suitable for three-dimensional living cell microscopic observation.

[0073] Next, a scanning light field multi-spectral reconstruction network device according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0074] Figure 5 4 is a block diagram of a scanning light field multi-spectral reconstruction network device according to an embodiment of the present invention.

[0075] like Figure 5 As shown, the scanning light field multi-spectral reconstruction network device 10 includes: a generation module 100 , a testing module 200 and a reconstruction module 300 .

[0076] Among them, the generation module 100 is used to obtain scanned light field data and generate a training set and a test set based on the scanned light field data; the testing module 200 is used to use the training set to perform self-supervised iterative training of the multispectral reconstruction network, and use the test set to test the trained multispectral reconstruction network; the reconstruction module 300 is used to use the tested multispectral reconstruction network to generate light field angles and split spectra, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra.

[0077] In an embodiment of the present invention, a multispectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract the spatial features of the light field from the input data; the second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss based on the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0078] In an embodiment of the present invention, the third extraction module is further configured to perform data dimension transformation on the spatial and spectral features, and extract angle features of the angle dimension from the features after the data transformation.

[0079] In the embodiment of the present invention, the scanning light field multispectral reconstruction network device 10 of the embodiment of the present invention further includes: a processing module.

[0080] Among them, the processing module is used to perform one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network. Among them, the data rearrangement processing includes: rearranging the pixel points of the scanned light field data according to their positions behind the microlens, and all the pixel points at the same position behind the microlens are arranged in spatial order to form multi-angle data; the data interpolation processing includes: interpolating the data of the missing angles in the scanned light field data, and using the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0081] It should be noted that the above explanations of the embodiment of the scanning light field multi-spectral reconstruction network method are also applicable to the scanning light field multi-spectral reconstruction network device of this embodiment, and will not be repeated here.

[0082] The scanning light field multispectral reconstruction network device proposed in an embodiment of the present invention can quickly perform spectral splitting and generate light field angles based on a multispectral reconstruction network supervised by deep learning, and obtain relatively robust fast multispectral reconstruction of full-angle light field data, which is time-saving and has good noise robustness.

[0083] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0084] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0085] When the processor 602 executes the program, the scanning light field multi-spectral reconstruction network method provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further includes:

[0087] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0088] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0089] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0090] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0091] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0092] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0093] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned scanning light field multispectral reconstruction network method.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0097] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0098] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0099] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A scanning light field multispectral reconstruction network method, characterized in that: The following steps are involved: Acquire scanned light field data, and generate a training set and a test set based on the scanned light field data; The multispectral reconstruction network is self-supervised iteratively trained using the training set, and the trained multispectral reconstruction network is tested using the test set; the multispectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract the spatial domain features of the light field from the input data; the second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial domain features and the spectral features; the third extraction module is used to extract the angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial domain features, the spatial and spectral features, and the angular features to obtain a fused feature, split the spectrum and generate a new light field angle according to the fused feature, and use the new light field angle to fill the missing angle, and during training, calculate the training loss according to the true value of the new light field angle and the missing angle, and use the training loss to update the network parameters; The tested multispectral reconstruction network is used to generate light field angles and split spectra, and full-angle multispectral data is reconstructed based on the generated light field angles and split spectra.

2. The scanning light field multispectral reconstruction network method according to claim 1, characterized in that: The third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of angle dimension from the features after data transformation.

3. The scanning light field multispectral reconstruction network method according to claim 1, characterized in that: Before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, the method further includes: The training set and the test set are subjected to one or more of data rearrangement processing, data interpolation processing, data normalization processing, and data augmentation processing, wherein: The data rearrangement processing includes: rearranging the pixel points of the scanned light field data according to the position behind the microlens, and arranging all the pixel points at the same position behind the microlens in sequence according to the spatial order to form multi-angle data; The data interpolation processing includes: performing angle interpolation on the data of the missing angle in the scanned light field data, and using the interpolated angle as the true value of the missing angle; The data normalization process includes: performing linear transformation on the multi-angle data; The data augmentation process includes: evenly dividing the training set into small data matching pairs.

4. A scanning light field multispectral reconstruction network device, characterized in that: include: A generation module, configured to obtain scanned light field data and generate a training set and a test set based on the scanned light field data; A testing module is used to perform self-supervised iterative training on a multispectral reconstruction network using the training set, and to test the trained multispectral reconstruction network using the test set; the multispectral reconstruction network includes first to third extraction modules and a return module, wherein the first extraction module is used to extract spatial features of the light field from the input data; the second extraction module is used to extract spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract angular features of the light field based on the spatial and spectral features; the return module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, and use the new light field angles to fill in missing angles, and during training, calculate the training loss based on the true values ​​of the new light field angles and the missing angles, and use the training loss to update network parameters; The reconstruction module is used to generate light field angles and split spectra using a tested multispectral reconstruction network, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra.

5. The scanning light field multispectral reconstruction network device according to claim 4, characterized in that: The third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of angle dimension from the features after data transformation.

6. The scanning light field multispectral reconstruction network device according to claim 4, characterized in that: Also includes: A processing module is used to perform one or more of data rearrangement processing, data interpolation processing, data normalization processing, and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, wherein the data rearrangement processing includes: rearranging the pixels of the scanned light field data according to their positions behind the microlens, and arranging all pixels at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of missing angles in the scanned light field data, and using the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field multispectral reconstruction network method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the scanning light field multispectral reconstruction network method according to any one of claims 1 to 3.

Images