Scanning light field self-supervised network denoising method and apparatus, electronic device, and medium

Abstract

The present application relates to the technical field of computational imaging, and in particular to a scanning light field self-supervised network denoising method and apparatus, an electronic device, and a storage medium. The method comprises: obtaining scanning light field data; preprocessing the scanning light field data to obtain preprocessed data; and constructing a self-supervised denoising network on the basis of the preprocessed data until a preset iteration stop condition is reached, and thereby obtaining a final self-supervised denoising network that is used to perform scanning light field self-supervised network denoising. In this way, the invention solves problems in the prior art such as limited data availability due to reliance on paired high-quality data having the same or similar content, which in turn increases data acquisition difficulty and leads to increased cost; and the inability to perform denoising with single-frame light field images due to reliance on temporal information and the need for information from multiple frames, which in turn leads to poor responsiveness and reduced adaptability in practical applications.

Description

Scanning light field self-supervised network denoising method, device, electronic device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410032352.9 and application date on January 9, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of computational imaging technology, and in particular to a method, device, electronic device, and medium for self-supervised network denoising of a scanned light field. Background Art

[0004] Light field or scanned light field has attracted much attention due to its ability to achieve large-scale rapid 3D imaging with a single or a small number of images. It is widely used in biological applications including 3D calcium imaging. Among them, the signal-to-noise ratio of the light field image is an important factor affecting the final 3D imaging quality. The detection noise, mainly photon shot noise, exacerbates the measurement uncertainty and may change the morphology and functional interpretation of the underlying structure. Through traditional filter denoising methods based on spatial and transform domains, including median filters and BM3D denoising algorithms, it is possible to filter through a sliding window according to the spatial domain or transform domain of the light field image to obtain a denoised image, but this is time-consuming and there is a serious loss of structural details.

[0005] In related technologies, a supervised deep learning network denoising method can be used to take light field or scanned light field data as input and high-resolution true value images as supervision, so that the network can learn the denoising process. Alternatively, a time-series-based deep learning network denoising method can be used to take a series of time-continuous light field or scanned light field data as input and use two imaging results of adjacent frames at similar times as input and supervision, thereby realizing an unsupervised network learning denoising process.

[0006] However, in related technologies, due to the need to provide high-resolution true-value images and reliance on paired high-quality data with identical or similar content, the availability of data is easily limited, thereby increasing the difficulty of data acquisition and leading to increased costs. Since a set of light field images is required for a long time and reliance on multi-frame information, single-frame light field images cannot be used for denoising, which can easily lead to poor response and reduced adaptability in practical applications, affecting the denoising effect, and urgently needs to be improved.

[0007] Summary of the Invention

[0008] The present application provides a scanned light field self-supervised network denoising method, device, electronic device and storage medium to address the problems in the related art, such as reliance on paired high-quality data with identical or similar content, which easily limits data availability, thereby increasing the difficulty of data acquisition and leading to increased costs. In addition, due to reliance on temporal information, multiple frames of information are required, and denoising cannot be performed using a single frame of light field image, which easily leads to poor response and reduced adaptability in practical applications.

[0009] A first aspect of the present application provides a method for self-supervised network denoising of a scanned light field, comprising the following steps: acquiring scanned light field data; preprocessing the scanned light field data to obtain preprocessed data; and constructing a self-supervised denoising network based on the preprocessed data until a preset iterative stop condition is reached, thereby obtaining a final self-supervised denoising network for performing self-supervised network denoising of the scanned light field.

[0010] Optionally, in one embodiment of the present application, preprocessing the scanned light field data to obtain preprocessed data includes: combining multi-angle data in the scanned light field data according to a plurality of angle scanning arrangement sequences to generate a plurality of light field image data after different arrangements; and / or rotating, flipping and / or cropping the scanned light field data to obtain enhanced data.

[0011] Optionally, in one embodiment of the present application, the preprocessing of the scanned light field data to obtain preprocessed data further includes: segmenting the plurality of differently arranged light field image data to obtain multiple pairs of segmented image data with the same dimensions, so that each pair of segmented image data is used for the multi-channel network input forward transmission and the multi-channel network target or fusion target in a single iteration, and for the self-supervised total loss function backhaul.

[0012] Optionally, in one embodiment of the present application, the self-supervised denoising network is constructed according to the preprocessed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network, including: obtaining multiple branch network outputs according to the preprocessed data; fusing the multiple branch network outputs to obtain a fused network output; respectively calculating the first mean square error and the first absolute value error between the segmented image data and the corresponding branch network output and the second mean square error and the second absolute value error between the segmented image data and the fused network output; weighting the first mean square error and the first absolute value, and the second mean square error and the second absolute value error, to calculate the self-supervised total loss function of the self-supervised denoising network.

[0013] Optionally, in one embodiment of the present application, before obtaining the final self-supervised denoising network, it also includes: inputting a test set obtained from the preprocessed data into the trained self-supervised denoising network, and outputting a network result, so as to output the final self-supervised denoising network when the network result meets a preset test condition, wherein the test set does not overlap with the data of the training set for constructing the self-supervised denoising network and has a different size.

[0014] A second aspect of the present application provides a self-supervised network denoising device for a scanned light field, comprising: an acquisition module for acquiring scanned light field data; a processing module for preprocessing the scanned light field data to obtain preprocessed data; and a denoising module for constructing a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached, thereby obtaining a final self-supervised denoising network for performing self-supervised network denoising on the scanned light field.

[0015] Optionally, in one embodiment of the present application, the processing module includes: a generation unit, used to combine multi-angle data in the scanned light field data according to a plurality of angle scanning arrangement sequences to generate light field image data after a plurality of different arrangements; and / or, an enhancement unit, used to rotate, flip and / or crop the scanned light field data to obtain enhanced data.

[0016] Optionally, in one embodiment of the present application, the processing module further includes: a segmentation module, used to segment the light field image data after the multiple different arrangements to obtain multiple pairs of segmented image data with the same dimension, so that each pair of segmented image data is used for the multi-channel network input forward transmission and the multi-channel network target or fusion target in a single iteration, and is used for the self-supervised total loss function backhaul.

[0017] Optionally, in one embodiment of the present application, the denoising module includes: an acquisition unit for obtaining multiple branch network outputs based on the preprocessed data; a fusion unit for fusing the multiple branch network outputs to obtain a fused network output; a calculation unit for respectively calculating the first mean square error and the first absolute value error between the segmented image data and the corresponding branch network output and the second mean square error and the second absolute value error between the segmented image data and the fused network output; a weighting unit for weighting the first mean square error and the first absolute value, and the second mean square error and the second absolute value error to calculate the self-supervised total loss function of the self-supervised denoising network.

[0018] Optionally, in one embodiment of the present application, the denoising module is further used to input a test set obtained from the preprocessed data into the trained self-supervised denoising network before obtaining the final self-supervised denoising network, and output a network result, so as to output the final self-supervised denoising network when the network result meets a preset test condition, wherein the test set does not overlap with the data of the training set used to construct the self-supervised denoising network and has a different size.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field self-supervised network denoising method as described in the above embodiment.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned scanning light field self-supervised network denoising method.

[0021] The embodiments of the present application can obtain scanned light field data, pre-process the scanned light field data, and construct a self-supervised denoising network based on the pre-processed data until a preset iterative stop condition is reached to obtain a final self-supervised denoising network, thereby performing scanned light field self-supervised network denoising, reducing dependence on paired data, improving the scope of application and performance, and using single-frame images for denoising, which can improve the flexibility and robustness of denoising under different structures and signal-to-noise ratios. This solves the problems in related technologies that, due to reliance on paired high-quality data with identical or similar content, easily limit data availability, thereby increasing the difficulty of data acquisition and leading to increased costs, and due to reliance on time information, multiple frames of information are required, making it impossible to use single-frame light field images for denoising, which easily leads to poor response and reduced adaptability in practical applications.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] FIG1 is a schematic diagram of the structure of a scanned light field self-supervised network denoising method according to one embodiment of the present application;

[0025] FIG2 is a flow chart of a scanning light field self-supervised network denoising method provided according to an embodiment of the present application;

[0026] FIG3 is a schematic diagram showing the principle of the data training process of the segmentation module and the self-supervised denoising network according to one embodiment of the present application;

[0027] FIG4 is a schematic diagram showing a comparison between an original image of a scanned light field and an image denoised by a network according to an embodiment of the present application;

[0028] FIG5 is a schematic diagram of the structure of a scanning light field self-supervised network denoising device provided according to an embodiment of the present application;

[0029] FIG6 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes a scanning light field self-supervised network denoising method, device, electronic device and storage medium according to an embodiment of the present application with reference to the accompanying drawings. In view of the related technologies mentioned in the above background technology center, due to the reliance on paired high-quality data with the same or similar content, the availability of data is easily limited, thereby increasing the difficulty of data acquisition and increasing costs. Moreover, due to the reliance on time information, multiple frames of information are required, and a single frame of light field image cannot be used for denoising, which easily leads to poor response and reduced adaptability in practical applications. The present application provides a scanning light field self-supervised network denoising method, in which scanning light field data can be obtained, pre-processed, and a self-supervised denoising network is constructed based on the pre-processed data until a preset iterative stop condition is reached to obtain a final self-supervised denoising network, thereby performing scanning light field self-supervised network denoising, reducing the reliance on paired data, improving the scope of application and performance, and using a single frame image for denoising, which can improve the flexibility and robustness of denoising under different structures and signal-to-noise ratios. This solves the problem in related technologies that, due to reliance on paired high-quality data with identical or similar content, data availability is easily limited, which in turn increases the difficulty of data acquisition and leads to increased costs. In addition, due to reliance on time information, multiple frames of information are required, and single-frame light field images cannot be used for denoising, which easily leads to poor response and reduced adaptability in practical applications.

[0032] Before explaining the scanning light field self-supervised network denoising method provided in the embodiment of the present application, the structure of the scanning light field self-supervised network denoising method involved in the embodiment of the present application is first explained.

[0033] As shown in FIG1 , the structure of the scanned light field self-supervised network denoising method includes: a scanned light field data acquisition unit, a data preprocessing unit, a self-supervised denoising network training unit, and a self-supervised denoising network testing unit.

[0034] wherein, a scanning light field data unit is obtained to provide data for training and testing the network, wherein the data can be captured by a scanning optical microscope or downloaded from a public dataset;

[0035] The data preprocessing unit includes data rearrangement, data augmentation, and data training pair splitting functions. The final data will be used for self-supervised denoising network training or self-supervised denoising network testing.

[0036] Among them, the data rearrangement function is used to arrange multi-angle data in multiple angle orders and combine them into arranged image data; the data enhancement module function is used to perform image transformations such as cropping, rotating, and flipping on image data to form input data for the data training pair segmentation module; the data training pair segmentation function is used to segment multiple arranged image data to obtain multiple pairs of segmented image data with the same dimension. Each pair of data is used for the multi-channel network input forward transmission and multi-channel network target or fusion target in a single iteration, and is finally used for the self-supervised total loss function backtransmission.

[0037] The self-supervised denoising network training unit includes the multi-channel network input forward transmission function, the multi-channel output fusion module function and the self-supervised total loss function return function. Completing the above three steps is called an iteration.

[0038] Among them, the multi-channel network input forward transmission function receives any arranged image data output by the data pre-processing unit through the input layer, and passes it through the branch network to obtain the branch network output; the multi-channel output fusion module function is used to input all the image data output by the multi-channel network into the fusion network, and pass it forward through the fusion network to obtain the fusion network output; the self-supervised total loss function return function is used to calculate the mean square error and absolute error between the segmented image data and the branch network output, and the mean square error and absolute error between the fusion network output, weighted to form a self-supervised total loss function, and return it to update the network parameters. Usually, a threshold value for the number of iterations is set. When it does not exceed this value, it is considered that the network training has not ended. When the network training has not ended, it enters the next iteration, that is, the next batch of data is returned through the multi-channel network input forward transmission, the multi-channel output fusion module and the self-supervised total loss function;

[0039] The self-supervised denoising network testing unit is performed after the self-supervised denoising network training is completed, and is used to test the final performance of the selected data on the network.

[0040] Among them, the selected data often does not overlap with the data used by the self-supervised denoising network training unit, and the size is not necessarily the same. If the size is different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the network input layer. Let the network predict separately and then splice the prediction results.

[0041] Next, the scanning light field self-supervised network denoising method of the embodiment of the present application is described in detail.

[0042] Specifically, Figure 2 is a flow chart of a scanning light field self-supervised network denoising method provided in an embodiment of the present application.

[0043] As shown in Figure 2, the scanned light field self-supervised network denoising method includes the following steps:

[0044] In step S201 , scanning light field data is acquired.

[0045] It is understood that the scanned light field data refers to a series of images or measurement values ​​obtained by sampling the light field at different angles and positions.

[0046] Specifically, the embodiments of the present application can be photographed by a scanning optical microscope to obtain scanning light field data. For example, by photographing by a scanning optical microscope, a series of light field images of zebrafish embryos at different perspectives can be obtained, including information such as the direction, intensity, and phase of light on the surface or inside the embryo.

[0047] The embodiments of the present application can be photographed by a scanning optical microscope to obtain scanning light field data, which is beneficial to improving the comprehensiveness and accuracy of the data and providing an accurate basis for subsequent operations.

[0048] In step S202 , the scanned light field data is preprocessed to obtain preprocessed data.

[0049] It can be understood that preprocessing includes data rearrangement, data enhancement, etc.

[0050] The embodiments of the present application preprocess the scanned light field data, such as data rearrangement and data enhancement, to obtain preprocessed data, which helps to correct the scanned light field data and improve data accuracy, thereby effectively improving data clarity, signal-to-noise ratio, and detail visibility, making the data more suitable for subsequent analysis and processing.

[0051] Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, including: combining multi-angle data in the scanned light field data according to multiple angle scanning arrangement sequences to generate light field image data after multiple different arrangements; and / or rotating, flipping and / or cropping the scanned light field data to obtain enhanced data.

[0052] It is understandable that multi-angle data refers to light field image information scanned from different angles or positions, including scene depth, texture details, etc.

[0053] Specifically, the embodiments of the present application can scan immune cells from different angles or positions, and combine them according to different arrangement orders to obtain light field image data. For example, immune cells can be scanned from different angles such as top, bottom, left, and right, and combined according to different arrangement methods (such as ABC, ACB, BAC, etc.) to generate a variety of light field image data after different arrangements; the scanned image of the immune cells can be rotated and / or flipped. For example, the immune cell image can be rotated 90 degrees or 180 degrees clockwise or counterclockwise, or flipped horizontally or vertically, etc., to generate enhanced data with different angles and mirror symmetry; according to the size and position of the immune cells, an area can be selected in the scanned image for cropping, so that the local area of ​​the immune cells can be cropped out to provide enhanced data of different sizes and positions.

[0054] The embodiments of the present application can combine multi-angle data in the scanned light field data according to a plurality of angle scanning arrangement sequences to generate light field image data after a plurality of different arrangements, and rotate, flip and / or crop the scanned light field data to obtain enhanced data. Thus, by combining the multi-angle scanning arrangements and rotating, flipping and cropping the scanned light field data, it is possible to expand the training data set, increase the diversity and richness of the data, improve the robustness of image processing, and further improve the accuracy and reliability of the data.

[0055] Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, and the method further includes: segmenting the light field image data after a plurality of different arrangements to obtain multiple pairs of segmented image data with the same dimensions, so that each pair of segmented image data is used for the multi-channel network input forward transmission and the multi-channel network target or fusion target in a single iteration, and is used for the self-supervised total loss function backhaul.

[0056] During the actual implementation process, the embodiment of the present application can obtain multiple pairs of segmented image data with the same dimension by segmenting a variety of arranged image data, wherein each pair of data is used for the multi-channel network input forward transmission and the multi-channel network target or fusion target in a single iteration, and is finally used for the self-supervised total loss function backtransmission, and the multi-channel network input forward transmission can select any image processing network, input any arranged image data into the multi-channel network, and obtain the multi-channel network output forward.

[0057] Specifically, in conjunction with FIG3 , the embodiment of the present application can add a data training segmentation module, using a single noisy light field image to simulate the results after multiple imaging. The specific steps are as follows:

[0058] Step S301: Select a segmentation dimension.

[0059] It is understandable that the core of a light field or scanning light field microscopy system is the microlens array in front of the camera. The microscopy system can ultimately form four-dimensional image information with two angular dimensions and two spatial dimensions. After passing through the data rearrangement module, the two angular dimensions are merged into the S dimension, and the two spatial dimensions are retained as the X and Y dimensions. In order to simulate the results of multiple imaging under the same noise model, specific dimensions can be selected for segmentation to obtain multiple sets of image results. The selection method can be as follows:

[0060] (1) The same spatial dimension X is selected in each iteration;

[0061] (2) The same spatial dimension Y is selected in each iteration;

[0062] (3) Each iteration randomly selects any spatial dimension X or Y;

[0063] (4) Each iteration selects both spatial dimensions X and Y;

[0064] (5) Each iteration selects the angle dimension S;

[0065] The purpose of using different segmentation dimensions is to obtain multiple sets of segmented images in subsequent steps, which are used to provide multiple corresponding training targets for multi-channel network inputs. The segmented images of a selected segmentation dimension will also be used as the fusion network target for the self-supervised total loss function feedback.

[0066] Step S302: Obtain a segmented image.

[0067] It is understandable that in order to obtain multiple sets of image results, it is necessary to segment the selected dimensions. For the arranged image I, after selecting the segmentation dimension D, different image segmentation methods can be selected according to the characteristics of different image data, wherein the segmentation method can be as follows:

[0068] (1) Odd-even division;

[0069] (2) Sliding window random segmentation;

[0070] (3) average sequential segmentation;

[0071] The result of odd-even splitting can be determined by the following expression: odd =I([D1,D3,…,D 2k-1 ]), O even =I([D2,D4,…,D 2k ]), k=Len(D) / 2, O={O odd , O even},

[0072] Among them, I is the input image to be segmented, D is the selected segmentation dimension, O odd is the odd image after odd-even segmentation, O even is the even image after odd-even segmentation, Len is the length of the given dimension, and O is the training pair obtained after segmentation.

[0073] The result of sliding window random segmentation can be determined by the following expression: w j =shuffle({n∈N * |wsize×(j-1) <n≤wsize×j}), j∈{1,2,…,k}, O={O1,O2,…,O i},

[0074] Among them, i is the number of the element in a single sliding window, O i is the i-th randomly segmented image, I is the input image to be segmented, is the kth sliding window W in dimension D k The element corresponding to the i-th dimension number in , Len(D) is the length of the given dimension D, wsize is the sliding window size, k is the total number of sliding windows required, shuffle is to disrupt the order of the given set, j is the sliding window number, n is the dimension number, and O is the training pair obtained after segmentation.

[0075] Since average sequential segmentation refers to dividing the input image I into the required number of equal parts in sequence according to a specific dimension D, which together constitute the training pair O, and cannot be used for self-supervised denoising network training, but only for self-supervised denoising network testing, after selecting N dimensions in step S301, the same number of training pairs will be obtained after step S302, which can be understood as N training pairs O, among which several images in one training pair will be used as the fusion target in step S305, and the denoising of the remaining N-1 training pairs can be the input of step S303.

[0076] Step S303: Multi-channel network input forward transmission.

[0077] Among them, for the N-1 training pairs obtained in step S302, a random segmented image of each training pair will be used as input, and the remaining segmented images will be used as targets. It can be understood that N-1 multi-channel network inputs and N-1 groups of multi-channel network targets are obtained. The obtained N-1 multi-channel network inputs are input into the same number of N-1 networks for forward denoising processing, and N-1 multi-channel network outputs can be obtained.

[0078] Step S304: merging multiple network outputs.

[0079] In steps S301 and S302, selecting a specific dimension and performing image segmentation may cause information corruption in the original light field image along the selected dimension. By selecting multiple dimensions for segmentation and cross-referencing the unsegmented modules, the information corruption caused by segmentation can be effectively compensated. This embodiment of the present application utilizes a multi-channel output fusion module and a deep learning image fusion network to fuse the N-1 multi-channel network outputs in step S303 into a single fused network output.

[0080] Step S305: Calculate the self-supervised total loss function.

[0081] Specifically, the function value can be determined by the following expression:

[0082] Among them, L is the total loss function of self-supervision, α is the weight of multi-path loss, n is the number of branches, Output i For the i-th multi-channel network output, Target i is the i-th group of multi-path network targets, L i is a multi-path loss function, which can be understood as the average of the L1 and L2 loss functions of the multi-path network output and the multi-path network target, β is the weight of the fusion loss, Output is the fusion network output, Target is the fusion target, L f is the fusion loss function, which can be understood as the average of the L1 and L2 loss functions of the fusion network output and the fusion target, γ is the weight of the multi-path similarity loss, and S is the similarity loss function, which can be understood as the average of the L1 and L2 loss functions of the fusion network output and the multi-path network output.

[0083] The embodiments of the present application utilize a multi-slicing approach to introduce multi-dimensional information of light field images, and combine it with the noise-on-noise deep learning training theory to achieve self-supervised scanning light field network denoising. This eliminates the need for true value images for supervision during training, reduces acquisition time, and improves the robustness of light field image data.

[0084] In step S203, a self-supervised denoising network is constructed based on the preprocessed data until a preset iteration stop condition is reached, thereby obtaining a final self-supervised denoising network for performing scanned light field self-supervised network denoising.

[0085] It is understandable that the preset iteration stopping condition may be that the number of iterations reaches a preset iteration threshold, such as the number of iterations reaches 1000 times.

[0086] The embodiment of the present application can construct a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached, thereby obtaining a final self-supervised denoising network, which can effectively reduce image distortion, retain the details and texture information of the original image, and further improve image quality.

[0087] Optionally, in one embodiment of the present application, a self-supervised denoising network is constructed based on the preprocessed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network, including: obtaining multiple branch network outputs based on the preprocessed data; fusing multiple branch network outputs to obtain a fused network output; respectively calculating the first mean square error and the first absolute value error between the segmented image data and the corresponding branch network output and the second mean square error and the second absolute value error between the segmented image data and the fused network output; weighting the first mean square error and the first absolute value, and the second mean square error and the second absolute value error, to calculate the self-supervised total loss function of the self-supervised denoising network.

[0088] Specifically, in combination with what is shown in FIG3 , the embodiment of the present application can feedback the self-supervised total loss function, calculate the weighted sum of the first mean square error and the first absolute value error between the segmented image data and the branch network output as the multi-path loss, and then calculate the weighted sum of the second mean square error and the second absolute value error between the fusion network outputs as the fusion loss. Finally, the weighted sum of the multi-path loss and the fusion loss forms a self-supervised total loss function, and feedback is performed to update the network parameters. By selecting any image fusion network, all the multi-path network outputs can be input into the fusion network, and the fusion network output can be obtained forward.

[0089] Among them, the self-supervised total loss function can be expressed as follows:

[0090] Among them, L is the total loss function of self-supervision, α is the weight of multi-path loss, n is the number of branches, Output i For the i-th multi-channel network output, Target i is the i-th group of multi-path network targets, L i is a multi-path loss function, which can be understood as the average of the L1 and L2 loss functions of the multi-path network output and the multi-path network target, β is the weight of the fusion loss, Output is the fusion network output, Target is the fusion target, L f is the fusion loss function, which can be understood as the average of the L1 and L2 loss functions of the fusion network output and the fusion target, γ is the weight of the multi-path similarity loss, and S is the similarity loss function, which can be understood as the average of the L1 and L2 loss functions of the fusion network output and the multi-path network output.

[0091] The embodiment of the present application can use the preprocessed data to obtain multiple branch network outputs to obtain the fusion network output, and by calculating the error between the segmented image data and the corresponding branch network output and the error between the segmented image data and the fusion network output, construct a self-supervised total loss function of the self-supervised denoising network, thereby increasing the robustness of the network output, and optimizing the performance of the self-supervised denoising network, and improving the accuracy and precision of denoising.

[0092] Optionally, in one embodiment of the present application, before obtaining the final self-supervised denoising network, it also includes: inputting a test set obtained from the preprocessed data into the trained self-supervised denoising network, and outputting the network results, so as to output the final self-supervised denoising network when the network results meet the preset test conditions, wherein the test set does not overlap with the data of the training set for constructing the self-supervised denoising network and has a different size.

[0093] It is understandable that the preset test conditions refer to standards for evaluating the self-supervised denoising network, which can be image quality evaluation indicators, etc.

[0094] During the actual execution process, combined with Figure 4, the embodiment of the present application can construct a self-supervised denoising network based on the preprocessed data until the preset iteration stop condition is reached to obtain the final self-supervised denoising network, wherein the upper two side images are, from left to right, the original light field images of the center 81 angles and the denoised light field images of the center 81 angles, respectively, and the lower two side images are, from left to right, the original central perspective image and the denoised central perspective image, respectively.

[0095] Specifically, the embodiment of the present application can use a scanning light field instrument (scanning magnification of 3, the number of pixels behind the microlens is 13×13) to shoot zebrafish embryo data, and then rearrange and enhance the data to generate 4,900 multi-angle images with a size of 9×9×128×128 for self-supervised denoising network training; use the PyTorch deep learning framework and Python programming language to build a self-supervised denoising network, specifically, use bilinear interpolation after the input layer to adjust the network input to the target size, and then pass it through a U-Net, and further, through the training network, where the initial learning rate is 1×10 -5 The training batch size is 1, and the back-propagation optimization is performed using the Adam optimizer for a total of 98,000 iterations. This allows the multi-angle test images to be fed into the trained self-supervised denoising network to obtain the denoised images.

[0096] In the embodiment of the present application, the test set obtained from the preprocessed data can be input into the trained self-supervised denoising network, and the network results can be output. When the network results meet the preset test conditions, the final self-supervised denoising network can be output, thereby effectively removing noise and retaining image details, and improving the stability and adaptability of denoising.

[0097] According to the scanning light field self-supervised network denoising method proposed in the embodiment of the present application, it is possible to obtain scanning light field data, pre-process the scanning light field data, and construct a self-supervised denoising network based on the pre-processed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network, thereby performing scanning light field self-supervised network denoising, reducing dependence on paired data, and improving the scope of application and performance. It can also use a single frame image for denoising, and can improve the flexibility and robustness of denoising under different structures and signal-to-noise ratios. This solves the problems in the related art that, due to reliance on paired high-quality data with identical or similar content, data availability is easily limited, thereby increasing the difficulty of data acquisition and leading to increased costs. Moreover, due to reliance on time information, multiple frames of information are required, and single-frame light field images cannot be used for denoising, which easily leads to poor response and reduced adaptability in practical applications.

[0098] Next, a scanning light field self-supervised network denoising device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0099] FIG5 is a schematic structural diagram of a scanning light field self-supervised network denoising device according to an embodiment of the present application.

[0100] As shown in FIG5 , the scanning light field self-supervised network denoising device 10 includes: an acquisition module 100 , a processing module 200 and a denoising module 300 .

[0101] Specifically, the acquisition module 100 is used to acquire scanning light field data.

[0102] The processing module 200 is used to pre-process the scanned light field data to obtain pre-processed data.

[0103] The denoising module 300 is used to construct a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network for use in scanning light field self-supervised network denoising.

[0104] Optionally, in one embodiment of the present application, the processing module 200 includes: a generation unit and / or an enhancement unit.

[0105] The generating unit is used to combine the multi-angle data in the scanned light field data according to a plurality of angle scanning arrangement sequences to generate a plurality of light field image data after different arrangements;

[0106] And / or, an enhancement unit, configured to rotate, flip and / or crop the scanned light field data to obtain enhanced data.

[0107] Optionally, in one embodiment of the present application, the processing module 200 further includes: a segmentation module.

[0108] Among them, the segmentation module is used to segment light field image data after a variety of different arrangements to obtain multiple pairs of segmented image data with the same dimension, so that each pair of segmented image data can be used for multi-channel network input forward transmission and multi-channel network target or fusion target in a single iteration, and for self-supervised total loss function backhaul.

[0109] Optionally, in one embodiment of the present application, the denoising module 300 includes: an acquisition unit, a fusion unit, a calculation unit, and a weighting unit.

[0110] Wherein, the acquisition unit is used to obtain multiple branch network outputs according to the preprocessed data;

[0111] A fusion unit, used for fusing multiple branch network outputs to obtain a fused network output;

[0112] a calculation unit, configured to respectively calculate a first mean square error and a first absolute value error between the segmented image data and the corresponding branch network output, and a second mean square error and a second absolute value error between the segmented image data and the fusion network output;

[0113] The weighting unit is used to weight the first mean square error and the first absolute value, and the second mean square error and the second absolute value error, and calculate the self-supervised total loss function of the self-supervised denoising network.

[0114] Optionally, in one embodiment of the present application, the denoising module is also used to input a test set obtained from the preprocessed data into the trained self-supervised denoising network before obtaining the final self-supervised denoising network, and output the network result, so as to output the final self-supervised denoising network when the network result meets the preset test conditions, wherein the test set does not overlap with the data of the training set for constructing the self-supervised denoising network and has a different size.

[0115] It should be noted that the aforementioned explanation of the embodiment of the scanning light field self-supervised network denoising method is also applicable to the scanning light field self-supervised network denoising device of this embodiment, and will not be repeated here.

[0116] According to the scanning light field self-supervised network denoising device proposed in the embodiment of the present application, it is possible to obtain scanning light field data, pre-process the scanning light field data, and construct a self-supervised denoising network based on the pre-processed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network, thereby performing scanning light field self-supervised network denoising, reducing dependence on paired data, improving the scope of application and performance, and being able to use a single frame image for denoising, and being able to improve the flexibility and robustness of denoising under different structures and signal-to-noise ratios. This solves the problems in the related art that, due to reliance on paired high-quality data with identical or similar content, data availability is easily limited, thereby increasing the difficulty of data acquisition and leading to increased costs, and due to reliance on time information, multiple frames of information are required, and single-frame light field images cannot be used for denoising, which easily leads to poor response and reduced adaptability in practical applications.

[0117] FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0118] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0119] When the processor 602 executes the program, the scanning light field self-supervised network denoising method provided in the above embodiment is implemented.

[0120] Furthermore, the electronic device further includes:

[0121] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0122] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0123] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0124] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, FIG6 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0125] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0126] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0127] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned scanning light field self-supervised network denoising method.

[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0129] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0130] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0131] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0132] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0134] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0135] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for denoising a scanned light field self-supervised network, characterized in that, Including the following steps: Obtain scanning light field data; Preprocess the scanning light field data to obtain preprocessed data; And Construct a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached, to obtain a final self-supervised denoising network for performing self-supervised network denoising of the scanning light field.

2. The denoising method of the scanning light field self-supervised network according to claim 1, wherein The preprocessing the scanning light field data to obtain preprocessed data includes: Combining multi-angle data in the scanning light field data according to a scanning arrangement order of multiple angles to generate multiple differently arranged light field image data; And / or, rotating, flipping, and / or cropping the scanning light field data to obtain enhanced data.

3. The self-supervised network denoising method for a scanned light field according to claim 2, wherein The preprocessing the scanning light field data to obtain preprocessed data further includes: Segment the multiple differently arranged light field image data to obtain multiple pairs of segmented image data with the same dimension, so that each pair of segmented image data is respectively used for the forward propagation of the multi-path network input and the multi-path network target or the fusion target in a single iteration, and is used for the backpropagation of the self-supervised total loss function.

4. The self-supervised network denoising method for a scanned light field according to claim 3, wherein The constructing a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network includes: Obtain multiple branch network outputs according to the preprocessed data; Fuse the multiple branch network outputs to obtain a fused network output; Calculate the first mean square error and the sum of the first absolute value errors between the segmented image data and the corresponding branch network outputs and the second mean square error and the second absolute value error between the fused network output respectively; Weight the first mean square error and the first absolute value, and the second mean square error and the second absolute value error to calculate the self-supervised total loss function of the self-supervised denoising network.

5. The denoising method of the scanning light field self-supervised network according to claim 1, wherein Before obtaining the final self-supervised denoising network, it further includes: Input a test set obtained from the preprocessed data into the trained self-supervised denoising network, and output a network result. If the network result meets a preset test condition, output the final self-supervised denoising network, where the test set does not overlap with the data of the training set for constructing the self-supervised denoising network and has a different size.

6. A scanning light field self-supervised network denoising device, characterized in that, Including: An acquisition module for acquiring scanning light field data; A processing module for preprocessing the scanning light field data to obtain preprocessed data; And A denoising module for constructing a self-supervised denoising network based on the preprocessed data until a preset iteration stop condition is reached to obtain a final self-supervised denoising network for performing self-supervised network denoising of the scanning light field.

7. The denoising device for a scanned light field self-supervised network according to claim 6, characterized in that, The processing module includes: A generating unit for combining multi-angle data in the scanning light field data according to a scanning arrangement order of multiple angles to generate multiple differently arranged light field image data; And / or, an enhancing unit for rotating, flipping, and / or cropping the scanning light field data to obtain enhanced data.

8. The denoising device for a scanned light field self-supervised network according to claim 7, characterized in that, The processing module further includes: A splitting module, configured to split the light field image data after the multiple different arrangements to obtain multiple pairs of split image data with the same dimensions, so that each pair of split image data is respectively used for the multi-channel network input forward propagation and the multi-channel network target or fusion target in a single iteration, and is used for the backpropagation of the self-supervised total loss function.

9. The denoising device for a scanned light field self-supervised network according to claim 8, wherein The denoising module includes: An acquisition unit, configured to obtain multiple sub-network outputs according to the preprocessed data; A fusion unit, configured to fuse the multiple sub-network outputs to obtain a fused network output; A calculation unit, configured to calculate the first mean square error and the sum of the first absolute value error between the split image data and the corresponding sub-network output, and the second mean square error and the second absolute value error between the split image data and the fused network output; A weighting unit, configured to weight the first mean square error and the first absolute value, and the second mean square error and the second absolute value error, and calculate the self-supervised total loss function of the self-supervised denoising network.

10. The scanning light field self-supervised network denoising device according to claim 6, wherein, Before obtaining the final self-supervised denoising network, the denoising module is further configured to input a test set obtained from the preprocessed data into the trained self-supervised denoising network, and output a network result. When the network result meets a preset test condition, the final self-supervised denoising network is output, where the test set does not overlap with the data of the training set for constructing the self-supervised denoising network and has a different size.

11. An electronic device, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the scanning light field self-supervised network denoising method according to any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used to implement the scanning light field self-supervised network denoising method according to any one of claims 1-5.

Images