Scanning light field multispectral network reconstruction method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of computer imaging, in particular to a scanning light field multispectral network reconstruction method and device, electronic equipment and a medium, and the method comprises the steps: obtaining scanning light field data, and generating a training set and a test set according to the scanning light field data; performing self-supervised iterative training on the multispectral reconstruction network by using the training set, and testing the trained multispectral reconstruction network by using the test set; and performing light field angle generation and spectrum splitting by using the multispectral reconstruction network passing the test, and reconstructing full-angle multispectral data based on the generated light field angle and the split spectrum. Therefore, the problems that multispectral network reconstruction consumes long time, a large amount of data cannot be trained, spectrum splitting is not clean, and light field information coherence is lacked in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computational imaging technology, and in particular to a scanning light field multi-spectral reconstruction network method, device, electronic equipment and medium. Background Art

[0002] Scanning light fields can use high-resolution two-dimensional information to quickly store three-dimensional light data, thus enabling fast, high-definition and large-volume three-dimensional imaging, which is widely used in biological applications such as brain neuroscience and immune interactions. At the same time, multispectral imaging is of great significance for biological observations. For traditional scanning light fields, time-division multiplexing imaging methods reduce the imaging rate exponentially, but if the sample moves faster, motion artifacts will be generated.

[0003] There are currently two main methods for reconstructing multispectral images from RGB (Red-Green-Blue, three-channel) images: a mathematical method based on sparse matrix decomposition and inverse matrix solution, and a multispectral network reconstruction method based on traditional deep learning.

[0004] Mathematical methods based on sparse matrix decomposition and inverse matrix solution, including linear decomposition and clarification, dictionary learning, etc., restore the original matrix solution and split it into multiple spectra according to the least squares method and K-SVD (K-Singular Value Decomposotion, a dictionary training algorithm). However, these methods are time-consuming and have problems such as being unable to train large amounts of data and unclean spectral splitting.

[0005] The network reconstruction method based on traditional deep learning uses RGB data as input and multi-spectral data as supervision to let the network fit the mathematical process of spectral splitting. However, for scanned light field data, the data obtained by the Bayer color array will result in the loss of light field angles. The traditional reconstruction network that learns spectral features cannot fully learn the physical expression between light field angles and spatial information. Although the spectrum can be split, the coherence of the light field information is missing. In addition, traditional deep learning is not suitable for learning microscopic data with great differences, and it is difficult to maintain the original generalization while learning spectral splitting. Summary of the invention

[0006] The present invention provides a scanning light field multi-spectral reconstruction network method, device, electronic device and medium to solve the problems in the related art that multi-spectral network reconstruction takes a long time, cannot train a large amount of data, spectrum splitting is not clean, and the light field information continuity is missing.

[0007] The first aspect of the present invention provides a scanning light field multispectral reconstruction network method, comprising the following steps: acquiring scanning light field data, generating a training set and a test set according to the scanning light field data; using the training set to perform self-supervised iterative training on the multispectral reconstruction network, and using the test set to test the trained multispectral reconstruction network; using the multispectral reconstruction network that has passed the test to generate light field angles and perform spectral splitting, and reconstructing full-angle multispectral data based on the generated light field angles and the split spectra.

[0008] Optionally, the multi-spectral reconstruction network includes first to third extraction modules and a feedback module, wherein the first extraction module is used to extract spatial features of the light field from the input data; the second extraction module is used to extract spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract angular features of the light field based on the spatial and spectral features; the feedback module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss based on the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0009] Optionally, the third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of angle dimension from the features after data transformation.

[0010] Optionally, before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, it also includes: performing one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set, wherein the data rearrangement processing includes: rearranging the pixels of the scanned light field data according to their positions behind the microlens, and arranging all the pixels at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0011] A second aspect of the present invention provides a scanning light field multispectral reconstruction network device, including: a generation module, used to obtain scanning light field data, and generate a training set and a test set according to the scanning light field data; a testing module, used to use the training set to perform self-supervised iterative training on the multispectral reconstruction network, and use the test set to test the trained multispectral reconstruction network; a reconstruction module, used to use the tested multispectral reconstruction network to generate light field angles and perform spectral splitting, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra.

[0012] Optionally, the multi-spectral reconstruction network includes first to third extraction modules and a feedback module, wherein the first extraction module is used to extract spatial features of the light field from the input data; the second extraction module is used to extract spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract angular features of the light field based on the spatial and spectral features; the feedback module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss based on the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0013] Optionally, the third extraction module is further used to perform data dimension transformation on the spatial and spectral features, and extract angle features of angle dimension from the features after data transformation.

[0014] Optionally, it also includes: a processing module, which is used to perform one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, wherein the data rearrangement processing includes: rearranging the pixel points of the scanned light field data according to the position behind the microlens, and all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0015] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field multi-spectral reconstruction network method as described in the above embodiment.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the scanning light field multi-spectral reconstruction network method as described in the above embodiments.

[0017] Therefore, the present invention has at least the following beneficial effects: The present invention can implement a multi-spectral reconstruction network based on deep learning supervision, quickly perform spectrum splitting and generate light field angles, obtain relatively robust fast multi-spectral reconstruction of full-angle light field data, and achieve the beneficial effects of short time consumption and good noise robustness. Thus, the technical problems in the related art such as long time consumption for multi-spectral network reconstruction, inability to train a large amount of data, unclean spectrum splitting, and lack of light field information coherence are solved.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a scanning light field multi-spectral reconstruction network method provided according to an embodiment of the present invention; Figure 2 A schematic diagram of a calculation process of a second extraction module provided according to an embodiment of the present invention; Figure 3 A specific composition diagram of a scanning light field multi-spectral reconstruction network method provided according to an embodiment of the present invention; Figure 4 A comparison diagram of an original image of a scanned light field provided according to a specific embodiment of the present invention and an image reconstructed through a multispectral network; Figure 5 An exemplary diagram of a scanning light field multi-spectral reconstruction network device provided according to an embodiment of the present invention; Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0021] The following describes the scanning light field multi-spectral reconstruction network method, device, electronic device and medium of the embodiment of the present invention with reference to the accompanying drawings. In view of the problems mentioned in the above background technology that the current method for reconstructing multi-spectral has a long time consumption, the spectrum splitting is not clean, and the continuity of light field information is missing, the present invention provides a scanning light field multi-spectral reconstruction network method, in which the scanning light field data is used to train and test the multi-spectral network. Thereby, the problems of the related art that the reconstruction of the multi-spectral network takes a long time, a large amount of data cannot be trained, the spectrum splitting is not clean, and the continuity of light field information is missing are solved.

[0022] Specifically, Figure 1 A schematic diagram of a flow chart of a scanning light field multi-spectral reconstruction network method provided by an embodiment of the present invention.

[0023] like Figure 1 As shown, the scanning light field multi-spectral reconstruction network method includes the following steps: In step S101, scanned light field data is acquired, and a training set and a test set are generated according to the scanned light field data.

[0024] It is understandable that the embodiments of the present invention can acquire scanned light field data, and generate a training set and a test set according to the scanned light field data.

[0025] In step S102, the multispectral reconstruction network is iteratively trained in a self-supervised manner using the training set, and the trained multispectral reconstruction network is tested using the test set.

[0026] It is understandable that the embodiments of the present invention can use the training set to perform self-supervised iterative training on the multispectral reconstruction network. After the training is completed, the trained multispectral reconstruction network can be tested using the test set to test the final performance of the multispectral reconstruction network.

[0027] It should be noted that since the selected data often do not overlap with the data used for multispectral reconstruction network training, and the sizes are not necessarily the same, if the sizes are different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the multispectral reconstruction network input layer, allowing the multispectral reconstruction network to predict separately and then splicing the prediction results.

[0028] In an embodiment of the present invention, a multi-spectral reconstruction network includes first to third extraction modules and a feedback module, wherein the first extraction module is used to extract spatial features of the light field from the input data; the second extraction module is used to extract spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field according to the spatial and spectral features; the feedback module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, fill in the missing angles with the new light field angles, and during training, calculate the training loss according to the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0029] Specifically, the first extraction module processes any input data to extract spatial features from the input data, inputs the spatial features and transmits them forward; the second extraction module is used to extract spectral features of the light field from the input data, and can extract spectral features through a three-dimensional U-shaped network structure, increase the learning ability of the spectral reconstruction network, and generate spatial and spectral features based on the spatial features and spectral features; the third extraction module is used to extract the extracted spatial and spectral features to obtain the angular features of the light field; the feedback module is used for supervised loss feedback, spectrally decomposes the features extracted by all extraction modules, performs loss function back-propagation with the supervised data, and updates the network. Specifically, it can obtain fused features by fusing spatial features, spatial and spectral features and angular features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss according to the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0030] The process of extracting spectral features by the second extraction module is as follows: Figure 2 As shown, the following steps are included: Step 1: The data passes through a 3D residual group, which includes a three-layer structure: a three-dimensional convolution kernel, an activation layer, and a three-dimensional convolution layer. For example, data A passes through the first layer, that is, a three-dimensional convolution kernel with a kernel size of 3×3×3, and then passes through the second layer, that is, a LeakyReLU activation layer, and finally passes through the third layer, that is, a three-dimensional convolution kernel with a kernel size of 3×3×3. The third layer outputs data B. Data A plus data B obtain the final output data C; Step 2: The network undergoes convolution to reduce the data size to half of the original size, double the number of channels, and then passes through a 3D residual group; Step 3: After convolution, the network reduces the data size to one third of the original size and triples the number of channels. Step 4: The network passes through a 3D residual group and then a layer of 3D convolution, which reduces the data size to half of the initial size and the number of channels to twice the initial size. The output is added to the output of step 2. Step 5: The network passes through a 3D residual group and a layer of 3D convolution to change the data size to the initial size, the number of channels to the initial size, and the output is added to the output of step 1; Step 6: The network passes through a 3D residual group to obtain the final output result.

[0031] The relationship between the root mean square error of the true value data and the final output of the multispectral reconstruction network is as follows: Loss(GT,Output)=MSE(GT,Output); Among them, GT represents the supervised true value data required by the multispectral reconstruction network, Output represents the output of the multispectral reconstruction network, Loss represents the supervised loss function, and MSE represents the mean square error between the true value data and the final output of the multispectral reconstruction network.

[0032] It should be noted that the embodiments of the present invention usually set an iteration number threshold. When the number does not exceed this value, it is considered that the multi-spectral reconstruction network training has not been completed. When the multi-spectral reconstruction network training has not been completed, it enters the next iteration, that is, the next batch of data enters feature extraction and feedback.

[0033] Specifically, the embodiment of the present invention is based on a multi-module deep learning supervised network for training. The original image is subjected to three extraction modules to extract the features of the light field, and a feedback module can be trained to obtain a multi-spectral reconstruction network for fast spectrum splitting and under-angle interpolation. It has strong generalization ability for different data and strong noise robustness. It learns light field data through multiple modules, and learns the physical information of the light field and the multi-spectral fusion process. In addition, the multi-spectral reconstruction network has a simple structure and a wide range of applications.

[0034] In the embodiment of the present invention, the third extraction module is further used to perform data dimension transformation on the spatial domain and spectral features, and extract angle features of the angle dimension from the features after the data transformation.

[0035] It can be understood that the third extraction module of the embodiment of the present invention can perform data dimension transformation on the spatial and spectral features, extract the features of the angle dimension after the transformation, and obtain the full data features, that is, after the data conversion of SAIs and MacPIs, the two-dimensional convolution structure is used to further separate and extract the angle features, and all data features are extracted.

[0036] Among them, the expression for data dimension transformation of spatial domain and spectral features is as follows: ; Among them, SAIs (sub-aperture image) means that the data is stored in the form of SAIs, which is a four-dimensional vector. The third and fourth dimensions represent the light field angle coordinates. MacPIs (macro pixel image) means that the data is stored in the form of MacPIs, which is a four-dimensional vector. The third and fourth dimensions represent the light field spatial coordinates. H represents the light field spatial horizontal coordinate, W represents the light field spatial vertical coordinate, U represents the light field angle horizontal coordinate, and V represents the light field angle vertical coordinate.

[0037] In an embodiment of the present invention, before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, it also includes: performing one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set, wherein the data rearrangement processing includes: rearranging the pixels of the scanned light field data according to the position behind the microlens, and all the pixels at the same position behind the microlens are arranged in sequence according to the spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0038] It is understandable that, before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, the embodiment of the present invention preprocesses the data in the training set and the test set, including data rearrangement, data interpolation, data normalization and data amplification. Specifically, (1) data rearrangement: used to rearrange the pixel points of the scanned light field data according to their positions behind the microlens, so that all the pixel points at the same position behind the microlens are arranged in spatial order to form multi-angle data; (2) data interpolation: used to interpolate the angles of the missing angles in the captured scanned light field data, and use the interpolated angles as the true values ​​of the missing angles, which are used as the true value data in the training set for subsequent training and testing of the multi-spectral reconstruction network; (3) data normalization: used to perform linear transformation on the multi-angle data, specifically, to perform maximum and minimum linear transformation on the multi-angle data so that the pixel values ​​are between 0 and 1; (4) data augmentation: used to randomly and evenly divide the training set into small data matching pairs in order to augment the training set and enhance the network effect.

[0039] In step S103, the multi-spectral reconstruction network that has passed the test is used to generate light field angles and split spectra, and full-angle multi-spectral data is reconstructed based on the generated light field angles and split spectra.

[0040] It can be understood that the embodiments of the present invention can use a multi-spectral reconstruction network to generate light field angles and perform spectral splitting, and reconstruct full-angle multi-spectral data based on the generated light field angles and split spectra, thereby utilizing the trained multi-spectral reconstruction network to quickly perform spectral splitting and generate light field angles, and furthermore, restore full-angle multi-spectral data from a single under-angle light field data for subsequent three-dimensional reconstruction.

[0041] The following will explain the scanning light field multi-spectral reconstruction network method through specific implementation, such as Figure 3 As shown, the following steps are included: (1) Obtain RGB light field data and generate high-resolution training and test sets required by the network.

[0042] (2) Data preprocessing, including: data rearrangement, data interpolation, data normalization and data augmentation modules, specifically: Data rearrangement: rearrange the pixels of the scanned light field data according to their positions behind the microlens, and arrange all the pixels at the same position behind the microlens in spatial order to form multi-angle data; Data interpolation: perform angle interpolation on the captured multi-spectral under-angle scanned light field data, and use it as the true value data in the training set for network training and testing; Data normalization: perform maximum and minimum linear transformation on the multi-angle data so that the pixel value is between 0 and 1; Data augmentation: randomly and evenly divide the network training set into small data matching pairs, which are used to augment the training set and enhance the network effect.

[0043] (3) Supervised multispectral network training includes: spatial feature extraction, three-dimensional scanning spectral feature extraction, angle feature extraction and supervised loss function feedback. Completing the above four steps is called an iteration.

[0044] The final data output by the data preprocessing is received through the input layer, and the RGB light field spatial features are obtained after extraction; the spatial features extracted by the spatial features are further extracted by using the three-dimensional U-shaped network structure to extract the spectral features to obtain the spatial-spectral features; the spatial-spectral features are extracted by the three-dimensional scanning spectral features, and after the data conversion of SAIs and MacPIs, the angle features are further separated and extracted by using the two-dimensional convolution structure, and all data features are extracted at this point; the above three extracted full data features are integrated, the required spectrum is split, and the new light field angle is simulated to fill the missing angle, and finally the mean square error is calculated with the true value. The loss function is then back-transmitted and iteratively updated. Among them, a threshold of the number of iterations is usually set. When it does not exceed this value, it is considered that the network training has not ended. When the network training has not ended, it enters the next iteration, that is, the next batch of data is subjected to spatial feature extraction, three-dimensional scanning spectral feature extraction, angle feature extraction, and supervision loss function back-transmission.

[0045] (4) Test of supervised multispectral reconstruction network: This is done after the supervised multispectral reconstruction network training is completed to test the final performance of the selected data on the network. The selected data often do not overlap with the data used for supervised multispectral reconstruction network training, and the size is not necessarily the same; if the size is different, the selected data should be cropped with overlap, and cropped into several images that meet the size requirements of the network input layer, and the network is allowed to predict separately, and then the prediction results are spliced.

[0046] According to the scanning light field multispectral reconstruction network method proposed in an embodiment of the present invention, a multispectral reconstruction network can be trained and tested based on a deep learning supervision of multiple modules, and the multispectral reconstruction network that has passed the test can be used to quickly perform spectral splitting and generate light field angles, and further obtain more robust fast multispectral reconstruction of full-angle light field data, which takes a short time and has good noise robustness.

[0047] The following is a specific example to illustrate the comparison between the original RGB scanned light field image and the image reconstructed by the multispectral network. All data in the following examples are displayed in SAIs format. Figure 4 As shown in the figure, the original image of the under-angle RGB scanning light field is split by the network to obtain the full-angle multi-spectral scanning light field image. The specific steps are as follows: Step 1: BSC-1 cells were photographed using a scanning light field instrument with a scanning magnification of 3 and a pixel number of 14×14 behind the microlens. The data was then rearranged, interpolated, normalized, and amplified to generate 400 multi-angle images with a size of 108×108×144×4 for supervised multispectral reconstruction network training; Step 2: Use the Pytorch deep learning framework and Python programming language to build a supervised multispectral reconstruction network. Specifically, use a 3×3 convolution to fuse the RGB image after the input layer, expand the channels to 64, and then extract features through a 3DU-Net. After data conversion, enter the 2D convolution sequence, and finally fuse all the extracted data and output it through a 1×1 convolution layer; Step 3: Train the network with an initial learning rate of 1×10^(-4) and a training set size of 400. Use the Adam optimizer for back-propagation iterative optimization. Train for a total of 125 iterations, and reduce the learning rate to half of the original value every 25 iterations. Step 4: Input the RGB under-angle microscopic light field data into the trained supervised multispectral reconstruction network to obtain a multispectral image.

[0048] In summary, the embodiment of the present invention has the function of under-angle interpolation of RGB light field data, and realizes the rapid splitting of RGB under-angle microscopic light field data collected by the Bayer color array into multi-spectral full-angle microscopic light field data. While ensuring high resolution, the time resolution is improved by at least four times, and it is fast, concise, and suitable for three-dimensional living cell microscopic observation.

[0049] Next, a scanning light field multi-spectral reconstruction network device according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0050] Figure 5 4 is a block diagram of a scanning light field multi-spectral reconstruction network device according to an embodiment of the present invention.

[0051] like Figure 5 As shown, the scanning light field multi-spectral reconstruction network device 10 includes: a generation module 100, a test module 200 and a reconstruction module 300.

[0052] Among them, the generation module 100 is used to obtain the scanned light field data, and generate a training set and a test set according to the scanned light field data; the testing module 200 is used to use the training set to perform self-supervised iterative training on the multispectral reconstruction network, and use the test set to test the trained multispectral reconstruction network; the reconstruction module 300 is used to use the tested multispectral reconstruction network to generate light field angles and perform spectral splitting, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra.

[0053] In an embodiment of the present invention, a multi-spectral reconstruction network includes first to third extraction modules and a feedback module, wherein the first extraction module is used to extract spatial features of the light field from the input data; the second extraction module is used to extract spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; the third extraction module is used to extract the angular features of the light field according to the spatial and spectral features; the feedback module is used to fuse the spatial features, the spatial and spectral features, and the angular features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, fill in the missing angles with the new light field angles, and during training, calculate the training loss according to the true values ​​of the new light field angles and the missing angles, and use the training loss to update the network parameters.

[0054] In the embodiment of the present invention, the third extraction module is further used to perform data dimension transformation on the spatial domain and spectral features, and extract angle features of the angle dimension from the features after the data transformation.

[0055] In the embodiment of the present invention, the scanning light field multi-spectral reconstruction network device 10 of the embodiment of the present invention further includes: a processing module.

[0056] The processing module is used to perform one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network. The data rearrangement processing includes: rearranging the pixels of the scanned light field data according to their positions behind the microlens, and arranging all the pixels at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

[0057] It should be noted that the above explanation of the embodiment of the scanning light field multi-spectral reconstruction network method is also applicable to the scanning light field multi-spectral reconstruction network device of this embodiment, and will not be repeated here.

[0058] The scanning light field multispectral reconstruction network device proposed in an embodiment of the present invention can quickly perform spectral splitting and light field angle generation based on a multispectral reconstruction network supervised by deep learning, and obtain relatively robust fast multispectral reconstruction of full-angle light field data, which takes a short time and has good noise robustness.

[0059] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0060] When the processor 602 executes the program, the scanning light field multi-spectral reconstruction network method provided in the above embodiment is implemented.

[0061] Furthermore, the electronic device further comprises: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0062] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0063] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0064] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0065] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0066] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0067] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned scanning light field multi-spectral reconstruction network method.

[0068] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0069] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0070] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0071] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0072] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0073] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A scanning light field multi-spectral reconstruction network method, characterized in that: The following steps are involved: Acquire scanned light field data, and generate a training set and a test set according to the scanned light field data; Using the training set to perform self-supervised iterative training on the multispectral reconstruction network, and using the test set to test the trained multispectral reconstruction network; The tested multi-spectral reconstruction network is used to generate light field angles and split spectra, and full-angle multi-spectral data is reconstructed based on the generated light field angles and split spectra.

2. The scanning light field multi-spectral reconstruction network method according to claim 1 is characterized in that: The multi-spectral reconstruction network includes first to third extraction modules and a return module, wherein: The first extraction module is used to extract the spatial domain features of the light field from the input data; The second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; The third extraction module is used to extract the angle characteristics of the light field according to the spatial domain and spectral characteristics; The feedback module is used to fuse the spatial domain features, the spatial domain and spectral features, and the angle features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss according to the new light field angles and the true values ​​of the missing angles, and use the training loss to update the network parameters.

3. The scanning light field multi-spectral reconstruction network method according to claim 2 is characterized in that: The third extraction module is further used to perform data dimension transformation on the spatial domain and spectral features, and extract angle features of angle dimension from the features after data transformation.

4. The scanning light field multi-spectral reconstruction network method according to claim 1 is characterized in that: Before using the training set to perform self-supervised iterative training on the multi-spectral reconstruction network, the method further includes: The training set and the test set are subjected to one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing, wherein: The data rearrangement process includes: rearranging the pixel points of the scanned light field data according to the positions behind the microlens, and arranging all the pixel points at the same position behind the microlens in sequence according to the spatial order to form multi-angle data; The data interpolation processing includes: performing angle interpolation on the data of the missing angle in the scanning light field data, and taking the interpolated angle as the true value of the missing angle; The data normalization process includes: performing a linear transformation on the multi-angle data; The data augmentation process includes: evenly dividing the training set into small data matching pairs.

5. A scanning light field multi-spectral reconstruction network device, characterized in that: include: A generation module, used to obtain scanned light field data, and generate a training set and a test set according to the scanned light field data; A testing module, used to perform self-supervised iterative training on the multispectral reconstruction network using the training set, and to test the trained multispectral reconstruction network using the test set; The reconstruction module is used to generate light field angles and split spectra using a tested multispectral reconstruction network, and reconstruct full-angle multispectral data based on the generated light field angles and split spectra.

6. The scanning light field multi-spectral reconstruction network device according to claim 5, characterized in that: The multi-spectral reconstruction network includes first to third extraction modules and a return module, wherein: The first extraction module is used to extract the spatial domain features of the light field from the input data; The second extraction module is used to extract the spectral features of the light field from the input data, and generate spatial and spectral features based on the spatial features and the spectral features; The third extraction module is used to extract the angle characteristics of the light field according to the spatial domain and spectral characteristics; The feedback module is used to fuse the spatial domain features, the spatial domain and spectral features, and the angle features to obtain fused features, split the spectrum and generate new light field angles according to the fused features, use the new light field angles to fill in the missing angles, and during training, calculate the training loss according to the new light field angles and the true values ​​of the missing angles, and use the training loss to update the network parameters.

7. The scanning light field multi-spectral reconstruction network device according to claim 6, characterized in that: The third extraction module is further used to perform data dimension transformation on the spatial domain and spectral features, and extract angle features of angle dimension from the features after data transformation.

8. The scanning light field multi-spectral reconstruction network device according to claim 5, characterized in that: Also includes: A processing module is used to perform one or more of data rearrangement processing, data interpolation processing, data normalization processing and data augmentation processing on the training set and the test set before using the training set to perform self-supervised iterative training on the multispectral reconstruction network, wherein the data rearrangement processing includes: rearranging the pixels of the scanned light field data according to their positions behind the microlens, and arranging all the pixels at the same position behind the microlens in sequence according to spatial order to form multi-angle data; the data interpolation processing includes: performing angle interpolation on the data of the missing angles in the scanned light field data, and taking the interpolated angle as the true value of the missing angle; the data normalization processing includes: performing linear transformation on the multi-angle data; and the data augmentation processing includes: evenly dividing the training set into small data matching pairs.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field multi-spectral reconstruction network method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the scanning light field multi-spectral reconstruction network method as described in any one of claims 1 to 4.

Images