Construction unit of machine learning model for image denoising and system and method using the same

Abstract

This application relates to building units for machine learning models for denoising images and systems and methods using the same. In some instances, a machine learning model can be trained to denoise an image. In some instances, the machine learning model can identify noise in an image of a sequence based at least in part on at least one other image of the sequence. In some instances, the machine learning model can include a recurrent neural network. In some instances, the machine learning model can have a modular architecture including one or more building units. In some instances, the machine learning model can have a multi-branch architecture. In some instances, the noise can be identified and removed from the image through an iterative process.

Description

[0001] Cross-reference to related applications

[0002] This application claims Provisional Application No. 63 / 163,678, filed March 19, 2021, entitled "Modular Machine Learning Models for Denoising Images and Systems and Methods for Using the Same"; Provisional Application No. 63 / 163,682, filed March 19, 2021, entitled "Building Units for Machine Learning Models for Denoising Images and Systems and Methods for Using the Same"; and Provisional Application No. 63 / 163,682, filed March 19, 2021, entitled "Systems and Methods for Training Machine Learning Models for Denoising Images". Priority is claimed in Provisional Application No. 63 / 163,688 of “IMAGES”. The aforementioned application is incorporated herein by reference in its entirety for any purpose. Technical Field

[0003] This application relates to imaging technology, and more specifically, to building blocks of machine learning models for image denoising, and systems and methods for using them. Background Technology

[0004] For example, advances in imaging techniques such as focused ion beam scanning electron microscopy (FIB-SEM) have enabled unprecedented observation and analysis of biological cells and interactions between them. Using FIB-SEM, images can achieve a resolution of approximately 4 nm per pixel, allowing observation of submicron-level structures, such as organelles within cells. However, with increasing resolution, noise also increases. Limitations in imaging modalities can make it difficult or impossible to obtain noise-free images at high resolutions. Returning to the FIB-SEM example, for nanoscale resolution, it may not be possible to completely control the effects of optical properties, beam variations, and / or thermal effects to eliminate noise from the image. Therefore, alternative techniques for reducing noise in high-resolution images may be needed. Summary of the Invention

[0005] According to one aspect of this application, an apparatus is provided. The apparatus includes: a logic circuit configured to receive a first input and a second input, and further configured to provide a first output, a second output, and a noise portion of an image; a convolutional circuit configured to convolve the first input to produce a first output; a synthesizer configured to receive the first input and the second input to produce a second output; a noise concern block configured to provide a noise portion at least partially based on the second output; and an adder configured to combine the noise portion with the image.

[0006] According to another aspect of this application, a system is provided. The system includes: at least one processor; and at least one non-transitory medium accessible to the processor, the at least one non-transitory medium being encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model includes: a plurality of building units, wherein individual of the building units is configured to receive a first input based on a second image and at least partially based on the image to receive a second input, and is further configured to provide a first output, a second output, and a noise portion; and a plurality of adders configured to combine the noise portion provided by the plurality of building units into the image to provide an output image. Attached Figure Description

[0007] Figure 1A This section illustrates an example of a machine learning model used for image denoising.

[0008] Figure 1B This illustrates another example of a machine learning model used for image denoising.

[0009] Figure 2 This describes a machine learning model for image denoising based on at least one example of the present disclosure.

[0010] Figure 3 A sequence of images illustrating at least one instance of this disclosure.

[0011] Figure 4 This is an illustrative description of a computing system configured according to an example of this disclosure.

[0012] Figure 5 A functional block diagram of a machine learning model based on an example of this disclosure.

[0013] Figure 6 This is a functional block diagram of a building unit based on an example of this disclosure.

[0014] Figure 7 A functional block diagram of a machine learning model based on an example of this disclosure.

[0015] Figure 8 This is a functional block diagram of a building unit based on an example of this disclosure.

[0016] Figure 9 This is a functional block diagram of a noise concern block according to an example of this disclosure.

[0017] Figure 10 This is a functional block diagram of a feedback block according to an example of this disclosure.

[0018] Figure 11A A flowchart of a method according to an example of this disclosure.

[0019] Figure 11B For examples based on this disclosure Figure 11A The flowchart shown in the document illustrates the implementation of identification and removal.

[0020] Figure 11C For examples based on this disclosure Figure 11A The flowchart shown in the document illustrates the implementation of identification and removal.

[0021] Figure 11D For examples based on this disclosure Figure 11A The flowchart shown in the document illustrates the implementation of identification and removal.

[0022] Figure 12 A diagram illustrating the process of training a machine learning model for image denoising according to an example of this disclosure.

[0023] Figure 13 This is a flowchart of a method for training a machine learning model for image denoising, according to an example of this disclosure.

[0024] Figure 14 An example image from a verification volume is shown, representing an example of an embodiment of this disclosure.

[0025] Figure 15 Examples of FIB-SEM images denoised using various techniques and corresponding noise curves are shown.

[0026] Figure 16 Examples of FIB-SEM images denoised using various techniques and corresponding noise curves are shown.

[0027] Figure 17 The scatter plot of the input signal across the resin and the layers of the machine learning model is shown according to an example of this disclosure.

[0028] Figure 18 This illustrates the application of examples based on this disclosure. Detailed Implementation

[0029] Image denoising is the process of taking an image x containing noise n (e.g., a noisy image) and separating the noise from the "real" signal s in the image (e.g., x = s + n) to obtain an image with reduced or no noise (e.g., a clean image). In many applications, the noise is random and / or independent (e.g., noise at one pixel does not depend on noise at another pixel). Existing techniques attempt to model the noise in an image to remove it. However, modeling noise can be difficult when multiple sources contribute to noise with different characteristics (e.g., Gaussian noise, Poisson noise, etc.). Artificial intelligence (AI) techniques, such as machine learning, have been utilized for image denoising. For example, machine learning models can be trained to make inferences (e.g., predictions, estimates) that can be used to remove noise from an image and output a "clean" image with reduced or no noise.

[0030] Typically, training a machine learning model involves providing a training dataset. Training can be supervised, unsupervised, or semi-supervised. The training dataset is used to determine the architecture and / or other parameters of the machine learning model. For example, the training dataset can be used to determine the number of layers in the model and / or the weights applied to the model's feature vectors. The trained model can then be used to analyze other datasets to perform one or more tasks, such as providing a denoised output image in response to an input image.

[0031] Figure 1A This describes an example of a machine learning model that can be used for image denoising. Machine learning model 104 can provide a training dataset containing labeled pairs of images. One image in the pair can be designated as input to an image 100 that is "known to be noisy," and the other image in the pair can be designated as the desired output to an image 102 that is "known to be clean" (e.g., noise-free or with reduced noise compared to image 100). Except for the presence of noise, images 100 and 102 can be identical (e.g., from the same spatial and / or temporal location). The known clean image 102 used to train machine learning model 104 can also be referred to as a ground truth image.

[0032] From the training set, machine learning model 104 can learn to identify “real” signals in noisy images. By “identifying,” machine learning model 104 is trained to make inferences about predictions and / or estimates that provide the desired output (here, the real signal). The real signal can then be extracted from the noisy image to produce a clean image. During training, acceptable parameters (e.g., network coefficients, weights applied to the feature matrix) of machine learning model 104 are determined based on how closely the parameter-based output of machine learning model 104 resembles a known clean image 102 of a known noisy image 100. Parameters may include values ​​of matrix and / or vector weights. Once parameters are found that provide output based on the known noisy image 100 that most closely resembles the known clean image 102, machine learning model 104 can be considered trained. Once trained (e.g., the values ​​of the parameters have been determined), machine learning model 104 can provide a clean output image 108 in response to an input noisy image 106 that is not part of the training dataset.

[0033] refer to Figure 1A The described technique utilizes ground-based images to train machine learning models. However, for high-resolution images, such as those at the nanoscale, noise can be ubiquitous. Therefore, suitable ground-based images for training machine learning models may not be available.

[0034] Figure 1B Another example of a machine learning model used for image denoising is illustrated. Similar to machine learning model 104, machine learning model 114 provides a training set containing pairs of images. However, one image 110 in the pair contains noise of a first type, and the other image 112 in the pair contains noise of a second type. Except for the type of noise, images 110 and 112 may be identical (e.g., from the same spatial and / or temporal location). Based on the training dataset containing image pairs with different noise levels, machine learning model 114 can learn to identify noise in the images. The identified noise can then be removed from the images.

[0035] Similar to Figure 1A During training, acceptable parameters for the machine learning model 114 can be determined. However, unlike... Figure 1AThe machine learning model 104 is trained in a manner that, during training, may not receive ground truth images but only noisy image pairs 110 and 112. Alternatively, the machine learning model 114 may output a clean image 116 for each of the image pairs. The clean image 116 can be used to compute a loss function. The loss function value may be higher when the machine learning model 114 makes a poor prediction (e.g., the clean image 116 still contains a lot of noise) and lower when the machine learning model 114 makes a good prediction (e.g., the clean image 116 contains less noise). Parameters may be adjusted during training. The machine learning model 114 may be considered trained when parameters that provide a minimum of the loss function are found (e.g., loss function minimization). Once trained, the machine learning model 114 may provide a clean output image 122 from the input pairs of noisy images 118 and 120, which may have less noise than the input images.

[0036] Although reference Figure 1B The described techniques offer a possible solution for training machine learning models when ground truth images are lacking, but image pairs that differ only in noise content may not be usable for training a trained model or as input to a trained model. Images acquired through temporal processes (e.g., mitosis) and / or through destructive processes may not provide such "matching" image pairs. For some imaging modalities, acquiring identical image pairs except for noise content may be impractical, for example, due to the time spent acquiring the images. Therefore, a machine learning model that can be trained to denoise images when ground truth or matching images are unavailable may be needed.

[0037] Figure 2 This invention describes at least one example of a machine learning model that can be used for image denoising. During training, the machine learning model 202 can provide a training set containing noisy images X. i-1 X i and X i+1 Multiple sequences 200, where i is the image number in the sequence. Although Figure 2 The example shows a sequence containing three images, but in other instances, the sequence may contain two or more images. Based on the training set of sequence 200, machine learning model 202 can learn to recognize image X in the sequence. i At least some noise in the image. The identified noise can then be removed to produce a noisy image X corresponding to the sequence provided as input. i The clean output image. For example, machine learning model 202 can learn by comparing different pairs of images in a sequence—e.g., X. i With X i-1 The difference between them and X i With Xi+1 The differences between them.

[0038] like Figure 3 As shown in the figure, in some instances, the image sequence 300 provided to the machine learning model 202 can correspond to images X acquired at different spatial locations. i-N To X i+M The number. For example, images X can be acquired at different image planes within a volume. i-N To X i+M In some instances, an image may correspond to spatially distinct, non-overlapping image planes. In some instances, non-overlapping image planes may be adjacent to each other. In other instances, image planes may be spaced apart (e.g., there are volume portions between unimage planes). In some instances, an image may correspond to partially overlapping image planes (e.g., existing in image X). i-1 A portion of the volume in image X also exists. i (in Chinese). In some instances, the thickness of the image planes within the volume and / or the spacing between the image planes may be based at least in part on the resolution and / or other parameters of the imaging technique used.

[0039] Figure 3 The document also demonstrates that, in some instances, the image sequence 302 provided to the machine learning model 202 can correspond to images X acquired at different times. t-N To X t+M The number of images. That is, the images can be spaced out in time. The time interval can be at least partially based on the frame rate of the imaging technique used in some instances. In some instances, images spaced out in time can be acquired at the same imaging plane. However, in other instances, the images can be spaced out in both space and time. In some instances, the spatial and / or temporal intervals of the images in sequences 300 and 302 can make the differences between two sequential images in the sequence (e.g., without a sequence of intervening images between the two images) smaller.

[0040] Return to Figure 2 Machine learning model 202 may include noisy images X trained based on sequence 200. i and one or more other images from sequence 200 (e.g., X) i+1 and / or X i-1 Output clean image X i 204. One or more machine learning models. In some instances, for example... Figure 2 In the example shown, the image provided to the machine learning model 202 as sequence 200 can be about image X. i Sequential images (e.g., directly in image X in sequence 200) i (The previous one or the next one).

[0041] Similar to machine learning model 114, machine learning model 202 may not receive ground truth images during training, but only noisy images from sequence 200 and output clean images 204 for individual sequences 200. However, unlike the noisy image pairs 110, 112 provided to machine learning model 114, the images in sequence 200 may not come from the same spatial and / or temporal locations. Clean images 204 can be used to compute the value of a loss function. Machine learning model 202 can be considered trained when the parameters that provide the minimum value of the loss function are found. Once trained, machine learning model 202 can provide clean output images 208 based on the input sequence 206 of noisy images.

[0042] Machine learning model 202 may contain one or more neural networks. A neural network may have one or more layers. A layer is one or more nodes (e.g., computations) that operate on the same input (which may contain one or more values) and provide an output (which may contain one or more values) as input to another layer or as output to the neural network. Various architectures can be used for neural networks. For example, a neural network may have a UNet architecture, a fully convolutional architecture, or a recurrent neural network (RNN) architecture.

[0043] In some applications, RNNs can be adapted to make inferences from spatial and / or temporal series. An RNN can determine what to "remember" and what to "forget" about the images in a sequence in order to make inferences about one or more images in the sequence. For example, an RNN can infer noise in images in a sequence based on images in the sequence and one or more other images. In some instances, an RNN may contain one or more gated recurrent units and / or long short-term memory networks. In some instances, the machine learning model 202 may have a branching architecture. Each branch may receive the same noisy image from sequences 200, 206, different noisy images from sequences 200, 206, or a combination thereof (e.g., some images received by the branch are the same while others are different) as input. The architecture of the branches may be the same or different. The outputs of the branches can be combined to provide a clean image 204 from the machine learning model 202.

[0044] In some instances, the machine learning model 202 may be partially or fully modular. For example, the machine learning model 202 may include a neural network, one or more layers of the neural network, and / or other components forming modules called building units, and any number of building units may be added to or removed from the machine learning model 202. In some instances, individual building units may identify corresponding portions of noise in the image to be removed. In some applications, the number of building units included in the machine learning model 202 may be based at least in part on the magnitude (e.g., intensity) of noise in the images of sequences 200, 202. In instances, the magnitude of noise may be indicated by the signal-to-noise ratio (SNR) of the image. In some instances, the magnitude of noise may be indicated by the type of noise in the image (e.g., Poisson vs. Gaussian). In some instances, the magnitude of noise may be at least in part attributable to the number of different types of noise contained in the image.

[0045] In some instances, the clean / denoised image output by machine learning model 202 can be provided to another machine learning model and / or image processing technique for further analysis. For example, the machine learning model can be trained to detect cancer cells, intercellular interactions, and / or other features of interest (e.g., potential drug targets, indicators of therapeutic efficacy) within the clean image. In other instances, detection or other processing of the clean image can be performed by other image processing techniques such as edge detection, histogram analysis, and speckle analysis. In some instances, the detection of cancer cells and / or other features can be used for diagnosis. In some instances, another machine learning model and / or other techniques can be used for diagnosis.

[0046] While the examples provided herein relate to medical and biomedical images, the machine learning model 202 and other models, devices, systems, and methods disclosed herein are not limited to medical applications. For example, images of semiconductor devices, nanomachines, microfluidic devices, or materials can be fed into the machine learning model 202, and clean / denoised images can be analyzed for quality control and / or other purposes.

[0047] In some instances, the machine learning model 202 may be implemented in hardware and / or software components. For example, a software-based machine learning model 202 may be implemented using a processor (e.g., a single or multi-core central processing unit, a single graphics processing unit (GPU) or a cluster of GPUs, or multiple processors arranged for parallel processing), the processor being configured to execute instructions that can be stored in a computer-readable medium, and which, when executed, cause the processor to implement the machine learning model. In another instance, the hardware-based machine learning model 202 may be implemented using one or more application-specific integrated circuits (ASICs).

[0048] Figure 4 This is an illustrative description of a computing system configured according to an example of this disclosure. The computing system 400 may be used to implement one or more machine learning models, such as machine learning model 202. The computing system 400 may include one or more processors 402, one or more computer-readable media 404, a memory controller 410, a memory 412, and one or more interfaces 414. In some instances, the computing system 400 may include a display 416.

[0049] Computer-readable media 404 is accessible by processor 402. Computer-readable media 404 may be encoded by executable instructions 408. Executable instructions 408 may include executable instructions for implementing a machine learning model to denoise an image (e.g., to generate a clean image from a noisy image). Executable instructions 408 may be executed by processor 402. In some instances, executable instructions 408 may also include instructions for generating or processing a training dataset and / or training a machine learning model. Alternatively or additionally, in some instances, the machine learning model or a portion thereof may be implemented in hardware that includes computer-readable media 404 and / or processor 402, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs).

[0050] Computer-readable media 404 may store data 406. In some instances, data 406 may contain one or more training datasets, such as training dataset 418. Training dataset 418 may contain one or more image sequences. In some instances, training dataset 418 may be received from another computing system (e.g., imaging system 422, cloud computing system). In other instances, training dataset 418 may be generated by computing system 400. In some instances, the training dataset may be used to train one or more machine learning models. In some instances, data 406 may contain data used in the machine learning model (e.g., weights, connections between nodes). In some instances, data 406 may contain other data, such as new data 420. New data 420 may contain one or more image sequences not included in training dataset 418. In some instances, the new data may be analyzed by a trained machine learning model to provide clean (e.g., denoised) images. In some instances, data 406 may contain the output (e.g., clean images) generated by one or more machine learning models implemented by computing system 400. Computer-readable media 404 can be implemented using any media containing non-transitory computer-readable media. Examples include memory, random access memory (RAM), read-only memory (ROM), volatile or non-volatile memory, hard disk drives, solid-state drives, or other storage devices. Although Figure 4 A single media may be displayed, but multiple media may be used to implement a computer-readable media 404.

[0051] In some instances, processor 402 may be implemented using one or more central processing units (CPUs), graphics processing units (GPUs), ASICs, FPGAs, or other processor circuitry systems. In some instances, processor 402 may execute some or all of the instructions 408. In some instances, processor 402 may communicate with memory 412 via memory controller 410. In some instances, memory 412 may be volatile memory, such as dynamic random access memory (DRAM). In some instances, memory 412 may provide information to and / or receive information from processor 402 and / or computer-readable medium 404 via memory controller 410. Although a single memory 412 and a single memory controller 410 are shown, any number may be used. In some instances, memory controller 410 may be integrated with processor 402.

[0052] In some instances, interface 414 may provide a communication interface to another device (e.g., imaging system 422), a user, and / or a network (e.g., LAN, WAN, Internet). Interface 414 may be implemented using wired and / or wireless interfaces (e.g., Wi-Fi, Bluetooth, HDMI, USB, etc.). In some instances, interface 414 may include user interface components that can receive input from a user. Examples of user interface components include a keyboard, mouse, touchpad, touchscreen, and microphone. In some instances, interface 414 may transmit information, including user input, data 406, training dataset 418, and / or new data 420, between an external device (e.g., imaging system 422) and one or more components of computing system 400 (e.g., processor 402 and computer-readable media 404).

[0053] In some instances, the computing system 400 may communicate with a display 416 as a separate component (e.g., using wired and / or wireless connections), or the display 416 may be integrated with the computing system. In some instances, the display 416 may display data 406, such as the output generated by one or more machine learning models implemented by the computing system 400. Any number or variety of displays may be present, including one or more LED, LCD, plasma, or other display devices.

[0054] In some instances, training dataset 418 and / or new data 420 may be provided to computing system 400 via interface 414. Optionally, in some instances, some or all of the training dataset 418 and / or new data 420 may be provided to computing system 400 by imaging system 422. In some instances, imaging system 422 may be a FIB-SEM imaging system, magnetic resonance imaging system, computed tomography system, ultrasound imaging system, or a combination thereof. In some instances, computing system 400 may provide the results of inferences made, for example, by a machine learning application to imaging system 422. In some instances, computing system 400 is included within imaging system 422.

[0055] Figure 5 This is a functional block diagram of a machine learning model according to an example of the present disclosure. In some instances, machine learning model 500 may be included in, used to implement, and / or implemented by machine learning model 202. In some instances, machine learning model 500 may be implemented by a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium encoded with instructions that, when executed by a processor of the computing system, implement machine learning model 500. In some instances, machine learning model 500 may be implemented wholly or partially by hardware (e.g., circuit systems such as ASICs, programmable FPGAs, etc.).

[0056] Machine learning model 500 can be trained (e.g., configured) to denoise images. Machine learning model 500 can receive noisy images X from image sequence 506. i-1 X i and X i+1 Image X, which serves as input and outputs image sequence 506, is used as input. i Clean image X iclean 526. Cleaning means that X has been removed. i At least a portion of the noise. In some instances, image sequence 506 may contain images from different imaging planes (e.g., different planes within a volume). In some instances, image sequence 506 may contain images from the same imaging plane acquired at different times. In some instances, image sequence 506 may contain images from both different imaging planes and different times.

[0057] Machine learning model 500 contains two branches, 502 and 504. Branch 502 can receive image X. i and X i+1 As input, and branch 504 can receive image X i and X i-1 As input. Figure 5As shown in the examples, one branch can receive a set of two images from the image sequence, and a second branch can receive a set of two images from the image sequence. In some instances, more than two images can be received by a branch. In other instances, only one image can be received by a branch. The second branch can receive different sets of images from the image sequence. In some instances, for example... Figure 5 The example shown can provide the same image to two branches (e.g., X). i However, in other instances, all images provided to the two branches are different. In some instances, the images provided to each branch may be consecutive images in a sequence or may be one or more images in a sequence of images (e.g., X). i and X i+2 Images spaced apart. Images provided to the branches can be evenly spaced in the sequence, or can have unequal spacing (e.g., X). i and X i+2 Available to branches and X i and X i-3 (It can be provided to another branch). In some applications, such as when one or more images in a sequence are unavailable (e.g., corrupted files), non-uniform (e.g., non-uniform) spacing may be required.

[0058] Branch 502 can output an image X that has at least some noise removed. i '520, and branch 504 can output at least some noise image X with at least some noise removed.' i "522. The outputs of branches 502 and 504 can be combined to provide a clean image X." iclean 526. Outputs can be combined in various ways (e.g., mixed, average, weighted average). In Figure 5 In the example shown, the average value of the output is determined to be 524. In the example shown, branches 502 and 504 have the same architecture: RNN. Therefore, for the sake of brevity, this paper will only describe the components and operations of branch 502. However, in other examples, branches 502 and 504 may have different architectures. Branch 502 may contain a convolutional / linear rectifier (ConvReLU) layer 508. The ConvReLU layer 508 convolutionally and linearly corrects the input image X. i and X i+1 In the example shown, the 508 ConvReLU layers apply a 3x3 convolutional kernel; other kernel sizes can be used in other examples. In some examples, the convolution may contain 64 channels. In some examples, the channels may be contained within the feature vector and / or matrix.

[0059] Branch 502 may contain one or more additional layers, referred to as building units 510. In the example shown, branch 502 contains D+1 building units 510, where D is a natural number. In some instances, the number of building units 510 may be at least partially based on image X. i The amount of noise in the image. In some instances, the building units 510 can be serialized. That is, individual building units 510 can receive the output of the previous layer of branch 502 as input. For example, building unit 0 receives the output of ConvReLU layer 508, and each subsequent building unit 510 (building units 1 to D) receives the output of the previous building unit 510 as input. ConvRevLU layer 508 can provide two outputs to building unit 0: "memory" and "new input". In some instances, the memory output can contain the image X. i+1 Relevant information, and new inputs and outputs may include information related to image X. i Related information. In some instances, the ConRevLU layer 508 can provide information related to the image X. i+1 and X i A single output of information related to both, but providing the output to two different parts of building unit 0. Similarly, the memory output of building unit 510 can be provided to two different parts of subsequent building units 510. Each building unit 510 can recognize image X, at least in part, based on the input. i At least a portion of the noise in the image. In some instances, the portion of noise identified by individual building units 510 may differ. The portion of noise (e.g., noise components) may be provided as the output of building unit 510 via noise interest block 512. In some instances, each building unit 510 may identify image X. i Different parts of the noise.

[0060] Image X can be provided along data path 528 i The image X i The noise is repeatedly updated to remove it. The arithmetic block 514 along data path 528 can receive the identified noise from the counterpart in building unit 510, and from image X... i Remove identified noise to provide image X i The updated version, in which the noisy portion is removed along data path 528 to the next arithmetic block 514. After the first arithmetic block 514 associated with building unit 0, subsequent arithmetic blocks 514 can be derived from the updated image X provided by the previous arithmetic block 514. i Remove a portion of the noise.

[0061] The updated image X generated by arithmetic block 514 can be used. iThe feedback is provided to feedback block 516. Feedback block 516 can further receive the output as input from the previous building unit 510. Based on the updated image X i In conjunction with the output from the previous building unit, feedback block 516 can provide output to the current building unit 510. For example, feedback block 516 associated with building unit 1 can receive the updated image X. i The output from building unit 0 is used to provide the output to building unit 1. In some instances, the additional feedback block 530 may be the same as feedback block 516; however, instead of providing the output to building unit 510, feedback block 530 provides the output to the linear rectifier / convolutional (ReLUConv) layer 518. Although shown separately from building unit 510 in some instances, in some instances, feedback block 516 and / or feedback block 530 may contain building unit 510.

[0062] The ReLUConv layer 518 linearly corrects and feeds the output of the convolutional feedback block 530 to the noise concern block 532. In the example shown, the ReLUConv layer 518 applies a 3x3 convolutional kernel, but other kernel sizes can be used in other examples. In some examples, the noise concern block 532 can be the same as the noise concern block 512. The noise concern block 532 can provide a portion of the identified noise to the final arithmetic block 534. The final arithmetic block 534 can take the updated image X received from the previous arithmetic block 514. i Remove noise to provide image X i '520.

[0063] In operation, multiple layers of the machine learning model (e.g., ConvReLU 508, building block 510, ReLUConv518) can generate an image X from each layer. i Noise components removed. Each building unit 510 accumulates images from sequence 506 (e.g., image X). i and image X i+1 Or image X i and image X i-1 The machine learning model 500 generates two outputs. One output, provided via noise concern block 512, generates a noise component. The other output contains information related to the image provided as "memory" from sequence 506 to the next building block 510 and / or feedback blocks 516, 530 of the machine learning model 500. Therefore, the machine learning model 500 can perform analysis on image X. i The iterative process of noise reduction.

[0064] Although building blocks 510 can be serialized, in some instances the operations performed by branches 502 and 504 can be performed in parallel. In other instances, the operations performed by one branch can be performed before the operations of another branch are performed. Although machine learning model 500 contains two branches, in other instances, machine learning model 500 may contain only one branch. In these instances, the output of one branch, such as X, is... i '520 or X i "522 can be used as X" iclean Output 526. In some instances, only two images from the sequence, instead of three, can be fed to the machine learning model 500. In some instances, only image X can be... i This is provided to machine learning model 500. In other instances, machine learning model 500 may contain more than two branches, and the outputs of multiple branches may be combined to provide X. iclean 526. In some instances, more than three images from a sequence may be available.

[0065] Figure 6 This is a functional block diagram of a building unit according to an example of this disclosure. In some instances, building unit 600 may be one of several building units in an RNN and / or other machine learning model. In some instances, building unit 600 may be used to implement... Figure 5 One or more of the building units 510 shown are illustrated. In some instances, building unit 600 may be implemented in hardware and / or software. In some instances, building unit 600 may be implemented via a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium encoded with instructions that implement building unit 600 when executed by the processor of the computing system. Building unit 600 may be at least partially based on image X in a sequence. i and at least one other image (e.g., image X) i-1 and / or image X i+1 Identify image X in the sequence i At least a portion of the noise in it.

[0066] In some instances, building unit 600 can act as an accumulator. Building unit 600 receives past information or "memory" and new information (e.g., new input). When building unit 600 is the first building unit (e.g., Figure 5 When constructing unit 0 in the model, input can be received from a ConvReLU layer (e.g., ConvReLU 508). The memory input and new input provided by the ConvReLU layer may contain images (e.g., image X) provided to branches of the machine learning model (e.g., branches 502 and / or 504 of machine learning model 500). i and image X i+1or X i-1 Information related to ) . In some instances, the memory input from the ConvReLU layer may contain information related to at least one other image in the sequence (e.g., X i+1 or X i-1 Information related to image X, and new input may include information related to image X. i Relevant information. When building unit 600 is a subsequent building unit (e.g., Figure 5 When building units 1 to D are constructed, memory input (M) is used. t-1 ) can be the output of the previous building block, and the new input (FM) t This can be the output of a feedback block (e.g., feedback block 516). In some instances, the new input provided by the feedback block may contain information related to the image X. i The relevant information includes a portion of the noise identified by the previous building block, which has been removed. In some instances, the new input provided by the feedback block may further include information related to the memory output of the previous building block.

[0067] The building unit 600 can provide a memory output and a noise portion output. In some instances, the memory output may contain information related to at least one other image in the sequence. In some instances, the memory output of the building unit 600 may further include information related to image X before removing noise identified by one or more previous building units. i Relevant information. The noise component can indicate the image X identified by the building unit 600. i This is a portion of the noise in the image. In some instances, the noise portion of the output can be provided to arithmetic block 614. In some instances, the information may include feature matrices and / or vectors, which may contain values ​​of features extracted from the image.

[0068] In some instances, arithmetic block 614 may contain arithmetic block 514. In some instances, arithmetic block 614 may be an adder (e.g., performing addition operations). See reference... Figure 5 As discussed, the arithmetic block 614 can receive image Xi (X). iprevious The previous "version" of the image X, which may be provided by the ConvReLU layer. i The image X may be at least a portion of the noise removed when provided by the previous arithmetic block. i Arithmetic block 614 can be derived from image X. iprevious Remove the noise received from building unit 600 and output the updated image X. iupdated Therefore, in some instances, image X iupdated It can have an image X that is not required. iprevious Less noise. In some instances, image X can be... iupdated Provided to the next arithmetic block and / or feedback block.

[0069] In some instances, build unit 600 may resemble a GRU unit with two gates: update gate 648 and reset gate 650. Update gate 648 determines the amount of input that is retained (e.g., included) in the memory output of build unit 600 by a previous layer. Reset gate 650 determines the amount of input received from a previous layer to be omitted (e.g., excluded) from the memory output. In other words, update gate 648 determines what information is "remembered," and reset gate 650 determines what information is "forgotten."

[0070] In some instances, building unit 600 may include a nonlinear block 636 that can extract features from memory inputs. In some instances, the output of nonlinear block 636... It can be represented as:

[0071]

[0072] For t = 1…D+1, where D is, for example, the number of building blocks 600 in a machine learning model 500, and The variable t represents the Hadamard product. Although the variable t is used as an exponent of the building blocks (e.g., layers in a machine learning model), t may not indicate time. More precisely, in some instances, t may indicate the layers in a machine learning model.

[0073] Arithmetic block 638 may receive the output (e.g., extracted features) from nonlinear block 636 and combine it with a new input. In some instances, arithmetic block 638 may sum the new input with the output of nonlinear block 636. The output of arithmetic block 638 may be provided to update gate 648 and reset gate 650. Update gate 648 may contain convolution block 640 and activation block 644, and reset gate 650 may contain convolution block 642 and activation block 646. Both convolution blocks 640 and 642 may convolve the output of arithmetic block 638. Figure 6 In the example shown, convolution can utilize a 3x3 kernel, but other filter sizes can be used in other instances. The output of convolution block 640 is provided to activation block 644, and the output of convolution block 642 is provided to activation block 646. In some instances, activation blocks 644 and / or 646 can apply a sigmoid function to the outputs of convolution blocks 640 and 642.

[0074] The output of activation block 644 can be used as the output of update gate 648. t As previously mentioned, update gate 648 determines what information to remember. In some instances, the operations performed by update gate 648 can be provided through the following:

[0075]

[0076] Among them W u For the weight matrix of the new input, and V u This is a matrix of weights applied to the memory input, and * denotes the convolution operator. In some instances, the weights can be determined during the training of a machine learning model containing 600 building blocks.

[0077] The output of activation block 646 can be used as the output of reset gate 650. t The operation of resetting gate 650 to determine what information has been forgotten can be described by the following equation:

[0078]

[0079] Among them W r The weight matrix is ​​applied to the new input, and V r This is the weight matrix applied to the memory input. In some instances, the weights can be determined during the training of a machine learning model containing 600 building blocks.

[0080] The output r of reset gate 650 t Outputs can be provided up to composable outputs and nonlinear block 636. Arithmetic block 654. In some instances, arithmetic block 654 can perform element-wise multiplication to produce a combined output. The output of arithmetic block 654 and the new input can be provided to arithmetic block 656. In some instances, arithmetic block 656 can sum the new input with the output of arithmetic block 654 to provide a combined output to current unit 662.

[0081] Current unit 662 may include convolution block 658 and activation block 660. Convolution block 658 convolves the output of arithmetic block 656. In some instances, such as the one shown, the convolution may use a 3x3 kernel, but other sizes may be used in other instances. The output of convolution block 658 is provided to activation block 660. In some instances, activation block 660 may contain a hyperbolic tangent function. The output of activation block 660 is provided as the current output C of current unit 662. t The operation of unit 662 can be represented as:

[0082]

[0083] Among them W c The weight matrix is ​​applied to the new input, and V c This is the weight matrix applied to the memory input. In some instances, the weights can be determined during the training of the machine learning model containing building blocks 600. The output of convolutional block 658 can be further provided to noise concern block 612, which provides the noisy portion of the output of building blocks 600.

[0084] Return to update gate 648 and output u. t The output of nonlinear block 636 can be provided to arithmetic block 652 and combined. In some instances, arithmetic block 652 can combine the inputs by performing element-wise multiplication functions. Output u t and output C t The outputs can be provided to arithmetic block 664. In some instances, arithmetic block 664 can combine the inputs by performing an element-wise multiplication function. The outputs of arithmetic blocks 652 and 664 can be provided to arithmetic block 666. Arithmetic block 666 can combine the two outputs to provide the memory output M of building unit 600. t In some instances, arithmetic block 666 can sum the outputs of arithmetic blocks 652 and 664. As previously mentioned, the output M... t It can be provided to subsequent building blocks and / or feedback blocks. It generates a memory output M. t It can be provided by the following equation:

[0085]

[0086] In some instances, the first building unit receives input from the ConvReLU layer for equations 1 through 5, M0 = ReLU(W0*(X i+1 or X i-1 ) and FM1 = ReLu(W0*X i W0 is the weight matrix applied to the image. The weights can be determined during the training of the machine learning model containing the building blocks 600. In some instances, some or all of the functions of the building blocks 600 can be applied to the 64 channels that can be arranged in the feature matrix of the vector. However, in other instances, more or fewer channels can be used.

[0087] Figure 7 This is a functional block diagram of an example machine learning model according to the present disclosure. In some instances, machine learning model 700 may be included in machine learning model 202 for implementing machine learning model 202 and / or implemented by machine learning model 202. In some instances, machine learning model 700 may be implemented by a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium with instructions encoded that implement the machine learning model when executed by the processor of the computing system. In some instances, machine learning model 700 may be implemented wholly or partially by hardware (e.g., circuit systems such as ASICs, programmable FPGAs, etc.).

[0088] Machine learning model 700 can be trained (e.g., configured) to denoise images. Machine learning model 700 can receive noisy images X from image sequence 706. j-1 X j and X j+1Image X, which serves as input and outputs image sequence 706, is used as input. j Clean image X jclean 726. Cleanliness means X j At least a portion of the noise has been removed. In some instances, image sequence 706 may contain images from different imaging planes (e.g., different planes within a volume). In some instances, image sequence 706 may contain images acquired at different times from the same imaging plane. In some instances, image sequence 706 may contain images from both different imaging planes and at different times.

[0089] Machine learning model 700 contains two branches, 702 and 704. Branch 702 can receive image X. j and X j+1 As input, branch 704 can receive image X j and X j-1 As input. Figure 7 As shown in the examples, one branch can receive a set of two images from the image sequence, and a second branch can receive a set of two images from the image sequence. In some instances, more than two images can be received by a branch. In other instances, only one image can be received by a branch. The second branch can receive different sets of images from the image sequence. In some instances, for example... Figure 7 The example shown can provide the same image to two branches (e.g., X). j However, in other instances, all images provided to the two branches are different. In some instances, the images provided to each branch may be consecutive images in a sequence or may be one or more images in a sequence of images (e.g., X). j and X j+2 Images spaced apart. Images provided to the branches can be evenly spaced in the sequence, or can have unequal spacing (e.g., X). j and X j+2 Available to branches and X j and X j-3 (It can be provided to another branch). In some applications, such as when one or more images in a sequence are unavailable (e.g., corrupted files), non-uniform (e.g., non-uniform) spacing may be required.

[0090] Branch 702 can output an image X that has at least some noise removed. i '720, and branch 704 can output an image X that has at least some noise removed.' i "722. The outputs of branches 702 and 704 can be combined to provide a clean image X." jclean 726. Outputs can be combined in various ways (e.g., mixed, average, weighted average). In Figure 7In the example shown, the average value of the output is determined to be 724. In the example shown, branches 702 and 704 have the same architecture: RNN. Therefore, for the sake of brevity, this paper will only describe the components and operations of branch 702. However, in other examples, branches 702 and 704 may have different architectures. Branch 702 may contain a Convolutional / Corrected Linear Unit (ConvReLU) layer 708. The ConvReLU layer 708 can perform the convolution operation as indicated by block 780 and the linear correction activation operation as indicated by block 782. Therefore, the ConvReLU layer 708 can convolutionally and linearly correct the input image X. j and X j+1 ConvReLU layer 708 can provide image X j+1 Output and image X j Output

[0091] Branch 702 may contain one or more additional layers, referred to as building units 710. In the example shown, branch 702 contains L+1 building units 710, where L is a natural number. In some instances, the number of building units 710 may be at least partially based on image X. j The amount of noise in the process. Each building block 710 may include an upper block 784 and a lower block 786. The upper block 784 may be referred to as a convolver 784, and the lower block 786 may be referred to as a synthesizer 786.

[0092] The upper block 784 (building unit 0) of the initial building unit 710 can receive output from the ConvReLU layer 708. The output is provided to synthesizer 786 and the upper block 784 (building unit 1) of subsequent building unit 710. The upper block 784 of subsequent building unit 710 can receive the output of the upper block 784 of previous building unit 710. This produces output. The upper block 784 (building block L) of the final building block can provide the output to the synthesizer 786.

[0093] The synthesizer 786 of the initial building block 710 can receive output from the ConvReLU layer 708. The output is then provided to noise concern block 712 and / or feedback block 716. The synthesizer 786 of the subsequent building unit 710 can receive the output of feedback block 716. In some instances, the upper block 784 can primarily learn from image x. j+1 The synthesizer 786 primarily learns information from the preceding building block 710 and / or the ConvReLU layer 708. In some instances, the upper block 784 and / or the synthesizer 786 may learn additional information from other sources.

[0094] At least in part based on the input, each building unit 710 can recognize image X j At least a portion of the noise in the noise. In some instances, the portion of noise identified by individual building units 710 may differ. The portion of the noise (e.g., noise components) may be provided via noise concern block 712. In some instances, the portion of the noise may be provided as a feature vector. Although the main Figure 7 While shown separately, in some instances, the noise concern block 712 may be included within the building unit 710, similar to... Figure 5 The building block 510 shown in the figure.

[0095] Image X can be provided along data path 728 j The image X j The noise is repeatedly updated to remove it. The arithmetic block 714 along data path 728 receives the identified noise from its counterpart in noise concern block 712, and from image X... j Remove identified noise to provide image X' l j The updated version (e.g., x') 0 j 、x' 1 j …x' L j The noisy portion is removed along data path 728 before being passed to the next arithmetic block 714. Following the first arithmetic block 714 associated with building unit 0, subsequent arithmetic blocks 714 can be processed from the updated image X' provided by the previous arithmetic block 714. l j removes a portion of the noise. For example, the arithmetic block 714 associated with building unit 0 can remove noise from the updated image x'. 0 j An arithmetic block 714 associated with construction unit 1 is provided, which can be obtained from the updated image x' 0 j Remove a portion of the noise to provide the updated image x' 1 j .

[0096] The updated image X' generated by arithmetic block 714 can be used. l j The feedback is provided to feedback block 716. Feedback block 716 can further receive the output as input from the previous building unit 710. Based on the updated image X' l j The feedback block 716 can output the output from the previous building block 710. The input is provided to the current building unit 710. For example, the feedback block 716 of building unit 1 can receive the updated image X'. 0 j and the output from building unit 0 to output The output is provided to building unit 1. In some instances, the additional feedback block 730 may be the same as feedback block 716; however, instead of providing the output to building unit 710, feedback block 730 provides the output to ConvReLU layer 718. Although shown separately from building unit 710 in some instances, in some instances, feedback block 716 and / or feedback block 730 may contain building unit 710.

[0097] In some instances, ConvReLU layer 718 may be identical to ConvReLU layer 708. ConvReLU layer 718 may provide feedback to the output of feedback block 730. Convolutional and linearly corrected activations are performed to provide the output to noise concern block 732. In some instances, ConvReLU layer 708 and / or ConReLU layer 718 may apply a 3x3 convolutional kernel with 64 channel feature vectors, but other kernel sizes and / or number of channels may be used in other instances. In some instances, noise concern block 732 may be the same as noise concern block 712. Noise concern block 732 may provide a portion of the identified noise to final arithmetic block 734. Final arithmetic block 734 may receive the updated image X' from the previous arithmetic block 714. L j Remove noise to provide image X j '720.

[0098] In operation, multiple layers of the machine learning model (e.g., ConvReLU 708, 718, building block 710) can generate an image X from each layer. j Noise components removed. Each building unit 710 accumulates images from sequence 706 (e.g., image X). j and image X j+1 Or image X j and image X j-1 The machine learning model 700 processes the image X and generates two outputs. One output is provided by synthesizer 786 to noise attention block 712 to generate noise components and is provided to feedback block 716. The other output is provided by upper block 784 to subsequent building unit 710. Therefore, machine learning model 700 can process the image X. j The iterative process of noise reduction.

[0099] In some instances, the operations performed by branches 702 and 704 can be performed in parallel. In other instances, the operations performed by each branch can be performed sequentially (e.g., the result of branch 702 is produced before the result of branch 704 is produced). Although machine learning model 700 contains two branches, in other instances, machine learning model 700 may contain only one branch. In these instances, the output of one branch, such as X... j '720 or X j "722 can be output as X" jclean 726. In some instances, only two images from a sequence, instead of three, can be fed to the machine learning model 700. In some instances, only image X can be fed... j This is provided to machine learning model 700. In other instances, machine learning model 700 may contain more than two branches, and the outputs of multiple branches may be combined to provide X. jclean 726. In some instances, more than three images from a sequence may be available.

[0100] Figure 8 This is a functional block diagram of a building unit according to an example of this disclosure. In some instances, building unit 800 and / or building unit 802 may be two of several (more than two) building units in an RNN and / or other machine learning model (e.g., machine learning model 202 and / or machine learning model 700). Although Figure 8 The example shows two building blocks, 800 and 802, but in some instances, a machine learning model may contain only one building block (e.g., 800 or 802). In some instances, building block 800 may be used for implementation. Figure 7 One or more of the building units 710 shown are illustrated. In some instances, building units 800 and / or 802 may be implemented in hardware and / or software. In some instances, building units 800 and / or 802 may be implemented via a computing system, such as computing system 400. For example, the computing system may contain non-transitory computer-readable media encoded with instructions that, when executed by the processor of the computing system, implement one or more building units. Building units 800 and / or 802 may be at least partially based on images X in a sequence. j and at least one other image (e.g., image X) j-1 and / or image X j+1 Identify image X in the sequence j At least a portion of the noise in. In some instances, for example Figure 8The examples shown herein, such as building units 800 and 802, may be identical or substantially similar. To avoid confusion in this disclosure, only individual components of building unit 800 are equipped with reference numerals, and component groups that are functional groups (e.g., doors) are labeled in building unit 802. For brevity, only the operation of building unit 800 is described in detail, and building unit 802 may operate in the same or substantially the same manner as building unit 800.

[0101] In some instances, building unit 800 can act as an accumulator. Building unit 800 receives two inputs: one at the upper block 884, called the convolutional unit. and at the lower block 886, which is called the synthesizer When building unit 800 is the first building unit (e.g., Figure 7 When constructing unit 0 in the model, input can be received from a ConvReLU layer (e.g., ConvReLU 708). The input provided by the ConvReLU layer may include images (e.g., image X) that are provided to branches of the machine learning model (e.g., branches 702 and / or 704 of machine learning model 700). j and image X j+1 or X j-1 Information related to this. In some instances, the input from the ConvReLU layer... It may include at least one other image in the sequence (e.g., X). j+1 or X j-1 Please enter the relevant information. It can contain images X j Related information. When building block 800 is a subsequent building block (e.g., building block 802), the input to the upper block 884 can be the output of the previous building block, and the input to the synthesizer 886 can be the output of the feedback block (e.g., feedback block 716).

[0102] Building block 800 can provide two outputs and Output can be This is provided to the upper block 884 of subsequent building units (e.g., building unit 802). If building unit 800 is the final building unit, then in some instances, it may not provide any output. Output can be Provided to the noise concern block and feedback block (not shown). For example, it can be... Provided to noise concern block 712, and can Provided to feedback block 716. In some instances, the feedback block may replace one channel in a multi-channel (e.g., 64-channel) feature vector; therefore, for simplicity, in Figure 8The outputs of building block 800 and feedback block use the same labels.

[0103] The upper block 884 may contain a convolution block 804 and an activation block 806. The convolution block 804 performs a convolution operation on the input and provides the output to the activation block 806. The activation block 806 applies an activation function to the output of the convolution block 804 to produce an output. In some instances, activation block 806 may contain a hyperbolic tangent function. In some instances, the output... The upper block 884, which can be provided to subsequent building blocks, and the synthesizer 886, which is provided to the current building block.

[0104] In some instances, synthesizer 886 may resemble a GRU cell with two gates: an update gate 848 and a reset gate 850. The update gate 848 determines the amount of input provided by the previous layer to be retained (e.g., included) in the output of synthesizer 886. The reset gate 850 determines the amount of input received from the previous layer to be omitted (e.g., excluded) from the output. In other words, the update gate 848 determines what information to use and / or "remember," and the reset gate 850 determines what information to not use and / or "forget." The output of reset gate 850 may be used by current unit 862 to produce an output combined with the output of update gate 848 to provide the output from synthesizer 886.

[0105] Update gate 848 may contain convolutional blocks 808 and 810. Convolutional block 808 may receive the output of the upper block 884, and convolutional block 810 may receive... Convolutional blocks 808 and 810 can perform convolution operations on their respective inputs and provide corresponding outputs to arithmetic block 820, which can perform summation (e.g., addition) operations on the convolutions. The output of arithmetic block 820 can be provided to activation block 824, which can perform an activation operation on the summed convolutions to provide the output u of update gate 848. l In some instances, activation block 824 may contain a sigmoid function.

[0106] The reset gate 850 may include convolution blocks 812 and 814. Convolution block 812 may receive the output of the upper block 884, and convolution block 814 may receive... Convolutional blocks 812 and 814 can perform convolution operations on their respective inputs and provide corresponding outputs to arithmetic block 822, which can perform summation (e.g., addition) operations on the convolutions. The output of arithmetic block 822 can be provided to activation block 826, which can activate the summed convolutions to provide the output r of reset gate 850. l In some instances, activation block 822 may contain a sigmoid function.

[0107] The current unit 862 may include convolution blocks 816 and 818. Convolution block 816 may receive the output of the upper block 884, and convolution block 818 may receive... Convolution block 816 performs a convolution operation on the output of upper block 884 and provides the output to arithmetic block 828. Arithmetic block 828 can further receive the output r of reset gate 850. l And for the output of convolution block 816 and r l Perform element-wise multiplication. The convolution block 818 can perform element-wise multiplication. A convolution operation is performed, and the output is provided to arithmetic block 830. Arithmetic block 830 sums the outputs of arithmetic block 828 and convolution block 818. The output of arithmetic block 830 can be provided to activation block 838, which can perform an activation operation to generate the output c of current unit 862. l In some instances, activation block 838 may contain a hyperbolic tangent function.

[0108] Update the output u of gate 848 l It can provide outputs from 1 to u. l The unit function block 834. The arithmetic block 836 can output 1 to u. l With the output c of current unit 862 l Element-wise multiplication. The output of update gate 848 can also be provided to arithmetic block 832. Arithmetic block 832 can further receive the output of upper block 884 and multiply the input element-wise. The outputs of arithmetic block 832 and arithmetic block 836 can be provided to arithmetic block 840, which sums the inputs to produce the output.

[0109] The operation of building unit 800 can be summarized by the following equation, where * indicates convolution and The mark is the Hadamard.

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] Where l∈{0,…L}, j∈{0,…N×M}, where N is the number of samples, and M is the number of images acquired from each sample, and W1 is a shared parameter across all building units. W0, W1, W u W r W c V u V r and V c This is the weight matrix applied to the corresponding input. The weights can be determined during training of a machine learning model containing building blocks 800 and / or 802.

[0118] Figure 9 This is a functional block diagram of a noise concern block according to an example of this disclosure. Noise concern block 900 can be used to implement one or more of noise concern blocks 512, 534, 612, 712, and / or 1732. In some instances, noise concern block 900 can be implemented by a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium encoded with instructions that, when executed by a processor of computing system 400, implement noise concern block 900. In some instances, noise concern block 900 can be implemented wholly or partially by hardware (e.g., circuit systems such as ASICs, programmable FPGAs, etc.). Noise concern block 900 can provide an image X in a sequence. i A portion of the noise (e.g., the noise component) is output.

[0119] The noise concern block 900 may receive feature vector 901 as input. In the example shown, feature vector 901 contains 64 channels, but in other examples it may contain more or fewer channels. In some examples, feature vector 901 may be the result of a convolution operation, such as the result of the convolution operation indicated by equation 903. In some examples, the convolution operation may have been performed by a convolution block of a unit, such as convolution block 658 of unit 662. In some examples, feature vector 901 may be the output of synthesizer 905, such as synthesizer 786 and / or 886.

[0120] Noise interest block 900 may contain a max-pooling block 968 of the channel with a maximum set feature vector 901. In some instances, the output of max-pooling block 968 may be provided to activation block 972 to generate an interest map. In some instances, activation block 972 may contain a sigmoid function. Feature vector 901 may also be provided to extraction block 970, which extracts the channel from feature vector 901. In some instances, for example in... Figure 9In the example shown, extraction block 970 can extract a first channel from feature vector 901 (e.g., channel 0 in some instances). The extracted channel and interest map can be provided to arithmetic block 974. In some instances, arithmetic block 974 can element-wise multiply the extracted channel with the interest map to provide a noise component (e.g., a portion of noise) as output. The output can be provided to adder blocks, such as adder blocks 514, 614, and / or 714. The adder blocks can extract the first channel from image X. i Remove the portion of noise provided by the noise concern block 900 and provide a less noisy (e.g., denoised) image. In some instances, the adder block may contain an adder.

[0121] Figure 10 This is a functional block diagram of a feedback block according to an example of the present disclosure. In some instances, feedback block 1000 may be used to implement one or more of feedback blocks 516, 530, 716, and 730. In some instances, feedback block 1000 may be implemented by a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium encoded with instructions that, when executed by a processor of the computing system, implement feedback block 1000. In some instances, feedback block 1000 may be implemented wholly or partially by hardware (e.g., circuit systems such as ASICs, programmable FPGAs, etc.). In some instances, feedback block 1000 may receive the output M of a building unit, such as building unit 510 and / or building unit 600. t As input. In some instances, feedback block 1000 may receive the output of a synthesizer, such as synthesizer 786 and / or 886. In some instances, feedback block 1000 may receive the updated image X from adder blocks, such as adder blocks 514, 614 and / or 714. iupdated 、X' l j .

[0122] Feedback block 1000 may include activation block 1076 that receives the updated image as input. In some instances, activation block 1076 may contain a hyperbolic tangent function. Feedback block 1000 may include receiving the output of activation block 1076 and the output M from the building unit. t And / or a copy block 1078 from the output of the synthesizer. In some instances, the output provided to the activation block 1076 may contain multiple channels. The copy block 1078 may copy the output of the activation block 1076 through channels provided to the output of the feedback block 1000. In some instances, the copy block 1078 may copy the output of the activation block 1076 through output M. t The output of the synthesizer can be copied to the first channel (e.g., channel 0 in some instances). The output of copy block 1078 can provide feedback FM as the output from feedback block 1000. t and / or feedback In some instances, for example Figure 10 The examples shown in the document provide feedback FM. t , It can contain feature vectors with 64 channels. In other instances, feedback FM... t , It can have more or fewer channels. In some instances, feedback FM... t , It can be provided to building blocks and / or another layer, such as ReLUConv layer 518 and / or ConvReLU layer 718.

[0123] Any of the machine learning models disclosed herein, comprising machine learning models 200, 500, and / or 700, and portions thereof, such as layers, building blocks (e.g., building blocks 510, 710, 600, 800, and 802), arithmetic blocks, convolutional blocks, noise concern blocks, feedback blocks, convolutionals, synthesizers, and / or other blocks or units, may be implemented in hardware, software, or a combination thereof. For example, any of the machine learning models disclosed herein may be implemented by a computing system, such as computing system 400. For example, the computing system may include non-transitory computer-readable medium encoded with instructions that, when executed by one or more processors of the computing system, implement the machine learning model or one or more portions thereof. In some instances, any and all of the machine learning models disclosed herein, or portions thereof, may be implemented wholly or partially by a circuit system, such as an ASIC, a programmable FPGA, logic circuitry (e.g., logic), or other hardware.

[0124] In some instances, instructions encoded in a non-transitory computer-readable medium may correspond to a software program comprising one or more modules, which contain one or more instructions corresponding to some or all of an implementation of a machine learning model. For example, a module may contain instructions for implementing a noise concern block (e.g., noise concern block 900). In some instances, a module may contain instructions containing a sub-section implementing a portion of a machine learning model. For example, a module for building blocks of a machine learning model may comprise: a sub-module containing instructions for implementing a convolutional unit (e.g., convolutional unit 784, 884) and another sub-module containing instructions for implementing a synthesizer (e.g., synthesizer 786, 886). In other instances, other techniques may be used to organize / construct software, when executed by one or more processors, to provide instructions for implementing a machine learning model (or a portion thereof).

[0125] Despite the reference sequence image description Figures 2 to 10However, the described devices, models, systems, and techniques can also be applied to portions of the image. For example, in biomedical imaging, image files can be hundreds of megabytes or larger (e.g., gigabytes). Stably available computing systems and / or hardware components may not be able to implement machine learning models to process the entire image file at once. In these cases, sequences (e.g., X sequences) i X i+1 X i-1 Images can be subdivided into sub-images before being fed to a machine learning model. The machine learning model can then analyze the sub-images individually to provide outputs for each sub-image. The outputs from the sub-images can then be reconstructed into a complete sequence of images.

[0126] In some applications, the ability to subdivide an image can offer advantages over other techniques. For example, some image processing techniques derive information from the entire image to identify real signals and / or noise signals within it. However, because the techniques disclosed herein use different images from an image sequence, similar or identical results will be obtained when corresponding sub-images from two images are provided, compared to processing the entire image at once.

[0127] Figure 11A This is a flowchart of a method according to an example of this disclosure. Method 1100A may be used for processing an image (e.g., image X). i Image X j Denoising methods, such as FIB-SEM, CT, and / or MRI images. Images may be images from sequences such as temporal and / or spatial sequences. In some instances, the method may be performed at least partially by machine learning models, such as machine learning model 202, machine learning model 500, and / or machine learning model 700. In some instances, method 900A may be performed at least partially by one or more building blocks, such as building blocks 510, 600, 710, 800, and / or 802. In some instances, the machine learning model and / or building blocks may be implemented by a computing system, such as computing system 400. In some instances, method 1100A may be implemented by a computing system, such as computing system 400. For example, the computing system may contain non-transitory computer-readable media encoded with instructions that, when executed by the processor of the computing system, perform method 1100A.

[0128] At box 1102A, the process of "identifying noise in images of an image sequence, at least in part, based on images in the image sequence and at least one other image" can be performed. In some instances, noise identification may involve estimating and / or predicting noise in the images. In some instances, the images and at least one other image may be spatially different, temporally different, or a combination thereof. At box 1104A, the process of "removing at least a portion of the noise from the image to provide an output image" can be performed. The output image may correspond to an image in a sequence in which the portion of the noise has been removed (e.g., a denoised image). In some instances, the identification and removal are at least in part performed by a machine learning model. In some instances, the machine learning model is implemented by at least one processor, such as processor 402. In some instances, the machine learning model may include RNNs, such as machine learning model 500 and machine learning model 700.

[0129] Optionally, in some instances, at box 1106A, "acquiring image sequences using a medical imaging system" can be performed. For example, medical imaging system 422 can be used to acquire image sequences. In some instances, the medical imaging system may include FIB-SEM, MRI, CT, and / or ultrasound imaging systems. In some instances, the image sequence may include acquiring multiple image planes from the volume of a biological sample.

[0130] Optionally, in some instances, at box 1108A, "detection of cancer cells based at least in part on the output image" can be performed. Optionally, in some instances, at box 1110A, "detection of intercellular interactions based at least in part on the output image" can be performed. In some instances, detection can be performed by another machine learning model trained to detect when cancer cells and / or intercellular interactions are present in the output image (e.g., a denoised image) provided by the machine learning model implementing boxes 1102A and / or 1104A. In some instances, detection can be performed by one or more image processing techniques (e.g., segmentation, histogram analysis) applied to the denoised image.

[0131] Optionally, when the hardware and / or the computing system, such as computing system 400, cannot process the entire image at once, blocks 1112 to 1114A can be performed. At block 1112A, "subdividing the image into a first plurality of sub-images" can be performed. At block 1114A, "subdividing at least one other image into a second plurality of sub-images" can be performed. In these instances, blocks 1102A and 1104A can be performed on the first plurality of sub-images and the second plurality of sub-images to produce a plurality of output sub-images and reconstruct the plurality of output sub-images to provide an output image.

[0132] Figure 11B For examples based on this disclosure Figure 11AThe flowchart shown illustrates the implementation of identification and removal. In some instances, method 1100B can be used for implementation. Figure 11A The recognition and removal boxes 1102A and 1104A are shown in the figure. At box 1102B, "convolution and linear correction of the image sequence and at least one other image can be performed to produce a first output". In some instances, this can be done by, for example, a ConvReLU layer of ConvReLU layer 508.

[0133] At box 1104B, “a memory output of the preceding layer of a plurality of layers is received at a plurality of layers and a portion of noise of the image is provided, wherein the first layer of the plurality of layers is configured to receive a first output.” In some instances, the plurality of layers may contain one or more building units, such as building unit 510 and / or building unit 600. In some instances, the noise portion may be provided by noise concern blocks, such as noise concern blocks 512, 532 and / or 612 and / or 900.

[0134] At box 1106B, "removal of noise portions received from multiple layers from an image to provide an output image" can be performed. In some instances, the removal can be performed by adder blocks, such as adder blocks 514, 534, and / or 614.

[0135] In some instances, method 1100B includes block 1108B, where "providing feedback output to the current layer of multiple layers" can be performed. In some instances, the feedback may be based at least in part on the memory output of the previous layer among the multiple layers and the image with noise removed from the portion provided by the previous layer among the multiple layers. In some instances, the feedback may be provided by feedback blocks, such as feedback blocks 516, 530, and / or 1000. Although block 1108B is shown between blocks 1104B and 1106B, in some instances, all or part of block 1108B may be performed concurrently with blocks 1104B and / or 1106B.

[0136] Figure 11C For examples based on this disclosure Figure 11A The flowchart shown illustrates the implementation of identification and removal. In some instances, method 1100C can be used for implementation. Figure 11A The recognition and removal boxes 1102A and 1104A are shown in the figure. At box 1102C, "convolution and linear correction of the image in the image sequence and at least one other image can be performed to produce a first output based on the image and a second output based on at least one other image". In some instances, box 1102C can be performed by a ConvReLU layer, such as ConvReLU layer 708.

[0137] At box 1104C, it is possible to "receive the output of the preceding layer of a plurality of layers and provide a portion of the noise of the image, wherein the first layer of the plurality of layers is configured to receive a first output and a second output." In some instances, the plurality of layers may contain one or more building units, such as building unit 710, building unit 800, and / or building unit 802.

[0138] In some instances, the noise portion can be provided by noise concern blocks, such as noise concern blocks 712, 732, and / or 900.

[0139] At box 1106C, "removal of noise portions received from multiple layers from the image to provide an output image" can be performed. In some instances, the removal can be performed by adder blocks, such as adder blocks 714 and / or 734.

[0140] In some instances, method 1100C includes box 1108C, where "providing feedback output to the current layer of multiple layers" can be performed. In some instances, the feedback can be based at least in part on a first and / or second output of the preceding layer of multiple layers and a portion of the image from which noise has been removed from the preceding layer. In some instances, the feedback can be provided by feedback blocks, such as feedback blocks 716, 730, and / or 1000. Although box 1108C is shown between boxes 1104C and 1106C, in some instances, all or part of box 1108C can be performed concurrently with boxes 1104C and / or 1106C.

[0141] Figure 11D For examples based on this disclosure Figure 11A The flowchart illustrates the implementation of the identification and removal process. In some instances, at least one other image in the image sequence may include a first image and a second image (e.g., X). i-1 and X i+1 In some instances, method 1100D can be used for implementation. Figure 11A The recognition and removal boxes 1102A and 1104A are shown in the figure.

[0142] At box 1102D, noise can be removed from the image at least partially based on the first image to provide a first intermediate image. In some instances, this removal can be performed by a branch of the machine learning model, such as branch 502 or 504, or branch 702 or 704. At box 1104D, noise can be removed from the image at least partially based on the second image to provide a second intermediate image. In some instances, this removal can be performed by another branch of the machine learning model, such as another of branch 502 or 504, or branch 702 or 704. At box 1106D, the first intermediate image and the second intermediate image can be combined to provide an output image. In some instances, the combination may involve taking the average of the two intermediate images.

[0143] For reference Figure 1A , 1B As described in section 2, the machine learning model is trained to perform the desired task before being deployed on “real data.” During training, acceptable parameters (e.g., network coefficients) of the machine learning model are determined based on the accuracy of predictions / inferences (e.g., noise identification) made by the parameterized machine learning model. Parameters may include values ​​of weights in matrices and / or vectors. The accuracy of the predictions may be represented by a loss function. The value of the loss function may be high when the machine learning model makes poor predictions (e.g., inaccurately identifying noise in an image) and low when the machine learning model makes good predictions (e.g., accurately identifying noise in an image). The machine learning model can be considered “trained” when the loss function reaches its minimum (e.g., the loss function is minimized). The parameters of the machine learning model that provide the minimum value of the loss function can be used with the machine learning model when it is deployed on non-training data (e.g., new data). In some instances, training may be performed at least partially by an optimizer. An example of a suitable optimizer is the ADAM optimizer. However, other optimizers may be used.

[0144] Figure 12 This diagram illustrates the process of training a machine learning model for image denoising according to an example of this disclosure. In some instances, process 1200 may be performed by a computing system, such as computing system 400. In some instances, an optimizer may perform some or all of process 1200. In some instances, the optimizer may be implemented by the computing system. The machine learning model and / or optimizer may be implemented by the computing system by executing (e.g., by processor 402) instructions (e.g., executable instructions 408) stored on a computer-readable medium and / or by hardware (e.g., logic circuits, ASICs, and / or FPGAs). Sets of training images 1202, 1204 may be provided as input to machine learning model 1206. For example, by providing images of 1202, 1204 to computer-readable medium 404 and / or memory 412. In some instances, machine learning model 1206 may include machine learning models 202, 500, and / or 700. In some instances, the training image sets 1202 and 1204 are image pairs from an image sequence. In some instances, it is common for sets 1202 and 1204 to contain one or more images from a sequence. Figure 12 In the example shown, set 1202 contains image X. i and image X i+1 And the training image set 1204 contains image X i and image X i-1In some instances, set 1202 can be provided to one branch of the machine learning model (e.g., branches 502, 702), and set 1204 can be provided to another branch of the machine learning model (e.g., branches 504, 704).

[0145] Initially, for training, the feature matrix of the machine learning model 1206 and / or the weights of other network coefficients (e.g., W) can be used. u V u W r V r W c and / or V c The weights are set as initial values. In some instances, these values ​​may be stored in computer-readable media 404 and / or memory 412. Based at least in part on sets 1202, 1204 and the initial values, machine learning model 1206 can provide two output images 1208 and 1210. In some instances, output image 1208 (e.g., image X) is... i '520, Image X' j '720) may be at least partially based on set 1202, and output image 1210 (e.g., image X) i "522, Image X" j "722) may be based at least in part on set 1204. The output image may be analyzed by loss function 1212. One or more of the values ​​of machine learning model 1206 may be adjusted at least in part based on the value of the loss function. Process 1200 may be repeated (e.g., process 1200 is iterative) until the minimum value of loss function 1212 is reached."

[0146] Extending from process 1200, in some instances, the machine learning model can be implemented using three consecutive images x in an image sequence of k = 1…N. k,i-1 ,x k,i ,x k,i+1 During training, the three consecutive images are grouped into two pairs (x... k,i ,x k,i-1 ) and (x k,i ,x k,i+1 Two pairs can be fed forward through a machine learning model, such as sets 1202, 1204 and two denoised images out for k = 1…N. k,i-1 and out k,i+1 (As illustrated in images 1208 and 1210), where N equals the number of image sequences. The loss function L can contain two main terms as shown below: noise-to-noise term and regularizer:

[0147] L = L n2n +ω R L REquation 13

[0148] Where L n2n Provided by:

[0149]

[0150] in

[0151]

[0152] In instances where sequences are obtained from individual samples, equations 14 and 15 can be written as:

[0153]

[0154] Where N is the number of samples, and M is the number of images obtained from each sample.

[0155] In some instances, the regularization sub-term ω R L R This can be the L2 norm weight decay term on the network coefficients. However, in other instances, a regularization sub-term is not necessary in the loss function L:

[0156] L = L n2n Equation 17

[0157] Where L n2n As provided in equations 14 to 16.

[0158] The loss function provided in Equations 13 to 17 can be used, at least in part, to attribute the spatial and / or temporal independence of noise in each image of the image sequence. In some instances, each image in the sequence can be viewed as a discrete version of a signal along another dimension (e.g., if each image extends in the xy-plane of the volume, then the other dimension may be along the z-axis of the volume). Therefore, for each image in the sequence, it can be viewed as:

[0159] x i±1 =s i +n+e0 Equation 18

[0160] Where s i For image x i The "real" signal, where n is the image x i The noise in the image is represented by e0, where e0 is the error. The error e0 can be at least partially attributed to the image x. i With image x i-1 and x i+1 Spatial and / or temporal differences between them. However, the error can be given as:

[0161] e0 = O(s) i±1 -s i Equation 19

[0162] Therefore, the error is small enough that the Taylor expansion along another dimension (e.g., the z-dimensional) can be used for the loss function as shown in Equations 15 to 16.

[0163] Figure 13 This is a flowchart illustrating a method for training a machine learning model for image denoising, according to an example of this disclosure. In some instances, method 1300 may be used to train machine learning models 202, 500, 700, and / or 1206. In some instances, method 1300 may be performed wholly or partially by a computing system, such as computing system 400. For example, the computing system may contain non-transitory computer-readable media encoded with instructions that, when executed by the processor of the computing system, perform method 1300. In some instances, the computing system may implement an ADAM optimizer to perform some or all of method 1300.

[0164] At box 1302, the function "sets the weights of the features in the feature matrix of the machine learning model to initial values" can be performed. For example, one or more values ​​of the weights of convolutional blocks 640 and / or 658 and / or convolutional layers (e.g., ConvReLU layers 508, 708, 718 and / or ReLUConv layer 518) can be set to initial values. These values ​​can be stored in non-transitory computer-readable media (e.g., computer-readable media 404) and / or memory (e.g., memory 412).

[0165] At box 1304, the action of "providing the first and second images of the image sequence as input to the first branch of the machine learning model" can be performed. At box 1306, the action of "providing the first and third images of the image sequence as input to the second branch of the machine learning model" can be performed. In some instances, the first, second, and third images contain noise. In some instances, the noise may be random and / or independent. In some instances, the noise may originate from multiple sources (e.g., heat, beam irregularities, etc.). In some instances, the first and second images are consecutive images in the image sequence, and the first and third images are consecutive images in the image sequence. In some instances, the images may be biological cells or other biological material. In some instances, the images may be FIB-SEM images. In some instances, the images may have been acquired by a medical imaging system, such as imaging system 422. An image can be provided to a machine learning model by providing the image directly to one or more processors implementing the machine learning model (e.g., processor 402) or by providing the image to a computer-readable medium (e.g., computer-readable medium 404) and / or a memory (e.g., memory 412).

[0166] At box 1308, the process of "receiving a first output from a first branch and a second output from a second branch" can be performed. At box 1310, the value of the loss function is calculated "based at least in part on the first and second outputs". In some instances, the calculation can be performed by a processor of a computing system. In some instances, the loss function may include a noise-to-noise term. In some instances, the loss function may include a regularization sub-term. In some instances, the noise-to-noise term may be based at least in part on a term containing the difference between the second image and the second output, a term containing the difference between the third image and the first output, and a term containing the difference between the first and second outputs. In some instances, the noise-to-noise term includes a function containing a weighted sum of said terms. In some instances, at least two of said terms are equally weighted. In some instances, at least one of the weights is negative. In some instances, the noise-to-noise term includes the average of the function. In some instances, the regularization sub-term includes an L2 norm weight decay term. In some instances, the loss function provided in Equations 13 to 17 can be used.

[0167] At box 1312, "weights may be adjusted at least in part based on the value of the loss function." In some instances, boxes 1302 through 1312 may be repeated until the value of the loss function reaches its minimum. Adjusting the weights may involve writing or rewriting the values ​​of the weights to computer-readable media and / or memory.

[0168] Once a machine learning model has been trained, it can be used to make inferences, estimates, and / or predictions, such as identifying noise in a sequence of images. For example, Figure 2The machine learning models 202, 500, and / or 700 shown can be trained models. The identified noise can then be removed from the images to provide cleaner (e.g., less noisy) images. Ideally, the machine learning model is trained whenever a new set of images to be analyzed is provided. A given number of images required to train the machine learning model may be infeasible. However, good results can be achieved by using machine learning models trained on different datasets, specifically when the different datasets have similar characteristics to the new dataset, such as being acquired through the same imaging modality, the same imaging system, imaging parameters, and / or when the different datasets contain images containing the same target (e.g., breast cells) as the new dataset. For example, the machine learning model can be trained and used to denoise images from a single imaging system (e.g., a specific FIB-SEM system). In another instance, the machine learning model can be trained and used to denoise images from different imaging systems of the same modality (e.g., multiple MRI machines). Alternatively, when a new dataset has similar characteristics to different datasets, a trained machine learning model can be used as a starting point for training (e.g., using the network coefficients of the trained model as initial values). By using a trained model as a starting point, the machine learning model may require less training than typically is needed.

[0169] The following describes an example implementation and results of a machine learning model. The examples are provided for illustrative purposes only, and this disclosure is not limited to the examples provided. In the following examples, the machine learning model comprises a two-branch architecture, each with an RNN, such as... Figure 5 and Figure 7 As shown in the diagram, the machine learning model is implemented using three to five building blocks for each branch. The machine learning model based on the instance is called the Noise Reconstruction and Removal Network (NRRN).

[0170] NRRN is trained to denoise images obtained using FIB-SEM from images acquired through Oregon Health & Science University (OHSU). The OHSU dataset contains images of samples taken from cancerous tissue. The dataset contains five volume data files converted into sequences of ten images per volume. The images are 4K x 6K pixels in size. The OHSU dataset includes both noisy images and ground truth (longer exposure) images.

[0171] To analyze the quality of the denoising technique, two classic metrics were used: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Both metrics compare the denoised image with the ground truth image. However, for FIB-SEM, the ground truth image is not available. In the case of the OHSU dataset, an average of ten sample scans were obtained before slicing the samples with FIB-SEM to approximate the ground truth image.

[0172] In the case of FIB-SEM, the flow from tissue collection to final image acquisition takes approximately two weeks. During this process, clinical samples undergo several resin infiltrations, where the spaces between cellular components are filled with resin. To analyze NRRN, the presence of noise in the resin is assessed. A good quality denoising technique should remove most of the noise in the resin. A flat signal should be observed at the resin region. To quantify this property, the interquartile range (IQR) of the signal across a straight line in the resin region is obtained. A smaller IQR indicates less variability in the signal across the resin. Simultaneously, the edges of the cellular components should be preserved. In other words, when denoising is successful, high PSNR, a SSIM close to 1, a smaller IQR in the resin, and sharp edges on the cellular components are observed. It should be noted that this type of analysis is only applicable to images of resin-embedded samples. Other analytical methods can be used for different image types and / or imaging modalities.

[0173] Using the OHSU FIB-SEM dataset, for reference... Figures 5 to 10 The described architecture and components' NRRNs are trained based on references. Figures 12 to 13 The described process involves image denoising. As previously mentioned, FIB-SEM images have a significant size of 4Kx6K. The initial large images are stitched together into a 256x256 pixel image, resulting in a training set of 8190 images, a test set of 2730 images, and an additional 4095 images for validation. Furthermore, the images are converted to YCbCr, using only the Y channel. The architecture is implemented in PyTorch. Training begins at an initial learning rate of 10. -4 The ADAM optimizer was used, with β1 = 0.9, β2 = 0.999, and ∈ = 10. -8 When using Figure 5 and 6 In the machine learning model example shown, an NRRN with PSNR of 31.0110 ± 0.1718 dB and SSIM of 0.9703 ± 0.0006 was achieved on the validation dataset. When using... Figure 7 and 8 In the example machine learning model shown, NRRN achieves a PSNR of 31.0197 ± 0.1905 dB and an SSIM of 0.9705 ± 0.0006.

[0174] Figure 14 Example images from a verification volume are shown according to an embodiment of this disclosure. Image 1400 is a FIB-SEM of cancerous tissue. Image 1404 is a magnified view of a portion of image 1400 indicated by box 1402. Image 1406 is a magnified view of a portion of image 1400 indicated by box 1402. However, compared to image 1404, the application according to... Figures 5 to 6The machine learning model NRRN is shown in Figure 1400 for denoising image 1400. Figure 1408 illustrates the variability of the image signal across the resin within the sample. Figure 1408 shows that NRRN achieves a PSNR of 31.09 dB, indicating a significant reduction in noise on the resin and an IQR reduction to 0.96. Figure 1410 illustrates the image signal across mitochondria. Figure 1410 shows that, in addition to noise removal, NRRN preserves sharp edges at cell organelle boundaries.

[0175] NRRN is trained on three image pairs, and inference (e.g., noise recognition) is based on three images providing sufficient denoising. As previously mentioned, inference can be performed using two or even one image. While more images generally provide higher PSNR and lower noise levels across the resin, inference from two input images also provides very satisfactory quality (e.g., less than 2% worse than three-image input).

[0176] NRRN was compared with other denoising techniques that incorporate Nonlocal Means (NLM), an untrained technique currently considered the "gold standard" for denoising EM images. NRRN was also compared with other machine learning models, including U-Net and DenoiseNet, specifically DenoiseNet as described by Tal Remez, Or Litany, Raja Giryes, and Alex M. Bronstein. Class-aware fully convolutional Gaussian and Poisson denoising. IEEE Transactions on Image Processing, 27(11):27(11):5707-5722, November 2018.

[0177] Images from FIB-EM are large and do not need to be processed directly on the GPU. Instead, the image is divided into smaller patches, denoised from the smaller images (e.g., sub-images), and then put back together to recreate the original whole image. For denoising images from FIB-EM, the 4Kx6K image is divided (e.g., subdivided) into 346 smaller overlapping patches of 256x256 pixels each. The overlap is set to 20 pixels in each direction to avoid boundary artifacts.

[0178] Networks like U-Net, which attempt to reconstruct images, cannot utilize their strength because these networks do not "see" the entire image when it is subdivided. On the other hand, NRRN and DenoiseNet target noise and utilize their random and / or non-dependent properties. Therefore, these techniques are less disadvantaged by image patching. Reconstructing and removing noise from an image, rather than reconstructing a less noisy version of the image, produces a more uniform result across the entire image. NRRN achieves consistent PSNR across all patches, in contrast to U-Net.

[0179] Figure 15 This section showcases example FIB-SEM images denoised using various techniques and corresponding noise curves. Image 1500 is a noisy FIB-SEM image of cells provided as input to various denoising techniques. Image 1502 is image 1500 denoised using NLM. Image 1504 is image 1500 denoised using U-Net. Image 1506 is an image denoised using DenoiseNet. Image 1508 is an image denoised using... Figures 5 to 6 The image 1500 shown is an example of the NRRN denoising method disclosed in this disclosure. Figure 15 The curves 1510 to 1518 below each image are curves of noise in the image indicated in image 1500 along line 1501.

[0180] NLM, DenoiseNet, and U-Net all achieve higher PSNR and SSIM than NRRN. However, NRRN achieves better results in removing noise across the resin. Furthermore, signal analysis across image structures shows blurred images from NLM and U-Net (1500). In some cases, the DenoiseNet image contains speckle, indicating that DenoiseNet struggles with significantly damaged / noisy images. Additionally, as can be seen from the comparison of graphs 1510 to 1518, all three methods (NLM, U-Net, and DenoiseNet) exhibit more noise than NRRN.

[0181] Figure 16 This section showcases example FIB-SEM images denoised using various techniques and corresponding noise curves. Image 1600 is a noisy FIB-SEM image of cells provided as input to various denoising techniques. Image 1602 is image 1600 denoised using NLM. Image 1604 is image 1600 denoised using U-Net. Image 1606 is an image denoised using DenoiseNet. Image 1608 is an image denoised using... Figures 7 to 8 The image 1600 shown is an example of the NRRN denoising method of this disclosure. Figure 16 The curves below each image, 1610 to 1618, are curves of noise in the image along the line in image 1600.

[0182] like Figure 15 In the comparison, NLM, DenoiseNet, and U-Net all achieve higher PSNR and SSIM than NRRN. However, NRRN achieves better results in removing noise across the resin. Furthermore, the signal across the image structure shows blurred images from NLM and U-Net (1600). In some cases, the DenoiseNet image contains speckle, indicating that DenoiseNet struggles with significantly damaged / noisy images. Additionally, as can be seen from the comparison of graphs 1610 to 1618, all three methods (NLM, U-Net, and DenoiseNet) exhibit more noise than NRRN.

[0183] Figure 17 The diagram illustrates a scatter plot of the input signal across resin layers and the machine learning model, according to an example of this disclosure. The modular architecture (e.g., building blocks) of the NRRN allows for intermediate noise estimation at the hidden layers. The scatter plot 1700 and the SSIM values ​​of the layers indicate that most of the denoising occurs in the first to third layers, after which more progressive improvements in noise are observed. In some applications, the last two layers can be more significant in driving SSIM for cases with more image corruption (e.g., higher noise) (e.g., Poisson noise with PEAK = 1 and Gaussian noise σ = 75).

[0184] The ability to discover intermediate denoising results and modular architectures can facilitate fine-tuning of machine learning models. More significant noise may require more building blocks, while less noise may require only a few. Therefore, it is easier to appropriately balance the quality and efficiency of machine learning models (e.g., NRRNs) compared to other denoising techniques. For example, users and / or system designers can analyze data indicating the amount of noise removed by each layer from different layers of a machine learning model (e.g., the data provided in scatter plot 1700), which is generated by training the machine learning model on a full or partial training dataset. Based on this data, users and / or system designers can select the number of building blocks to provide the desired balance between noise removal and computational resources.

[0185] The devices, systems, apparatuses, and methods disclosed herein provide for image denoising. In some instances, machine learning models can be trained to denoise images, even when no ground truth image is available. In some instances, the machine learning models can be modular, providing easier scaling of the architecture and / or comparison of performance across layers of the machine learning model. In some applications, the denoising techniques disclosed herein are more robust than other techniques when images are segmented into sub-images (e.g., due to processor constraints). In some applications, the denoising techniques disclosed herein provide reduced noise and / or reduced blur compared to other denoising techniques.

[0186] Figure 18 This illustrates the application of examples based on this disclosure. Figure 18 In the examples described herein, the systems and / or methods arranged according to the examples described herein can be used to analyze images for the presence of tumors. According to the systems and / or methods described herein, other features can be detected in imaging denoising. Figure 18 In this example, patient 1800 may have tumor 1802. Tumor 1802 can be biopsied and provided to imaging system 1804, for example... Figure 18 The FIB-SEM imaging system is shown. Of course, other imaging systems can be used, such as MRI, CT, US, SEM, and / or photolithography. In other instances, biopsies can be obtained from cadavers, animal models, or cell cultures. In some instances, multiple tumor biopsies can be obtained from multiple patients and / or sources and presented to imaging system 1804. Although... Figure 18 The description includes solid tumor 1802, but its application is not limited to this. For example, instead of tumor 1802, blood and / or bone marrow can be provided to imaging system 1804 (e.g., for detecting and treating blood cancers or malignant cells circulating from the primary tumor).

[0187] Imaging system 1804 may generate one or more images 1806, such as image sequences of tumor 1802 (e.g., sequences 206, 300, 302, 506, and / or 706) or portions thereof (e.g., cells of tumor 1802 or portions of one or more cells). Data representing image sequence 1806 (e.g., pixel or volumetric data) may be provided to computing system 1808. In some instances, computing system 1808 may be located remotely from imaging system 1804. In some instances, data may be provided from imaging system 1804 via a wired or wireless connection. In some instances, computing system 1808 may be integrated with imaging system 1804. In some instances, computing system 1808 may include computing system 400. Computing system 1808 may analyze image sequence 1806 according to machine learning model 1809. In some instances, computing system 1808 may include non-transitory computer-readable medium encoded with instructions that, when executed by one or more processors of computing system 1808, implement machine learning model 1809. In some instances, machine learning model 1809 may include machine learning model 202, machine learning model 500, machine learning model 700, and / or machine learning model 1206. Computing system 1808 may output data representing one or more denoised images 1810 (e.g., image sequences) corresponding to a sequence of images 1806.

[0188] Data representing the sequence of denoised images 1810 can be provided to computing system 1812. In some instances, the sequence of denoised images 1810 can be viewed on a display of computing system 1812. In some instances, computing system 1812 may be the same as computing system 1808, but in others it may be a different computing system, and data representing denoised images 1810 can be provided to computing system 1812 via a wired or wireless connection. In some instances, computing system 1808 and / or computing system 1812 may be integrated with imaging system 1804. In some instances, computing system 1812 analyzes the sequence of denoised images 1810 using one or more techniques. For example, image segmentation algorithms and / or other machine learning models may receive the sequence of denoised images 1810 as input. In other instances, the sequence of denoised images 1810 may be analyzed manually by researcher 1814, for example.

[0189] The sequence of the denoised image 1810 can be analyzed for a variety of purposes. In some instances, the sequence of the denoised image 1810 can be analyzed for diagnosis (e.g., carcinoma versus benign cancer type) and / or to determine tumor progression (e.g., developmental stage, degree of angiogenesis, presence of necrosis). In some instances, the sequence of the denoised image 1810 can be analyzed to identify one or more targets for therapy (e.g., receptors on the surface of tumor cells to which drug delivery mechanisms can bind, interactions between tumor cells and / or between tumor cells and normal cells that can be inhibited by therapy, etc.). In some instances, the sequence of the denoised image 1810 can be analyzed to determine whether a treatment delivered to patient 1800 is effective and / or to provide a prediction of effective treatment for patient 1800. In some instances, treatment recommendations can be made based on the prediction of treatment efficacy. In some instances, the analysis, identification, and / or prediction based on the sequence of the denoised image 1810 can be more accurate compared to the case where the original image 1806 has been analyzed. In some instances, such as in the case of image 1806, analysis, determination, and / or prediction based on the sequence of denoised image 1810 may be impossible.

[0190] In some instances, based on the analysis of the sequence of the denoised image 1810, novel therapies and / or novel therapy delivery systems (e.g., nanoparticles loaded with a known chemotherapeutic agent and coated with a receptor designed based on the sequence of the denoised image 1810) can be designed. In some instances, one or more known therapies can be identified as potentially effective against tumor 1802. In some instances, the designed and / or identified therapy 1816 can be delivered to patient 1800 to treat tumor 1802.

[0191] Figure 18 The applications provided are merely exemplary, and the embodiments disclosed herein are not limited to those described herein. Figure 18 The applications provided in [the document / platform].

[0192] The foregoing description of certain embodiments is merely exemplary in nature and is in no way intended to limit the scope of this disclosure or its application or use. In this detailed description of embodiments of the devices, systems, and methods of the present invention, reference is made to the accompanying drawings, which form part of the invention, and illustratively illustrate specific embodiments in which the described devices, systems, and methods may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the devices, systems, and methods of the present invention, and it should be understood that other embodiments may be utilized, and structural and logical changes may be made without departing from the spirit and scope of this disclosure. Furthermore, for clarity, certain features will not be discussed in detail when they are obvious to those skilled in the art, so as not to obscure the description of embodiments of this disclosure. Therefore, the discussions herein should not be construed in a limiting sense, and the scope of this disclosure is defined by the appended claims. As used herein, the term "device" may refer to a circuit, apparatus, system, component, or combination thereof. For example, a device may be a computing device, processor, memory, memory device, mobile device, edge device, server, and / or cloud computing system.

[0193] Of course, it should be understood that any of the examples, embodiments, or processes described herein may be combined with or separated from one or more other examples, embodiments, and / or processes and / or carried out in a separate device or part of a device according to the invention.

[0194] Finally, the foregoing discussion is intended to be illustrative only and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Therefore, although various embodiments of this disclosure have been described in particular detail, it should be understood that numerous modifications and alternative embodiments can be devised by those skilled in the art without departing from the broader and contemporaneous spirit and scope of this disclosure as set forth in the appended claims. Thus, the specification and drawings should be viewed in an illustrative manner and are not intended to limit the scope of the appended claims.

Claims

1. An apparatus for image processing, comprising: A logic circuit configured to receive a first input and a second input, and further configured to provide a first output, a second output, and a noise portion of an image, wherein the first input and the second input contain information about the image; as well as Adder; The logic circuitry described includes a convolutional unit, a synthesizer, and a noise concern block. The convolutional unit is configured to convolve the first input to produce the first output; The synthesizer is configured to receive the first output and the second input to produce the second output, wherein the synthesizer includes: an update gate configured to determine the amount of the first input and the amount of the second input to be included in the second output; and a reset gate configured to determine the amount of the first input and the amount of the second input to be excluded from the second output; The noise concern block is configured to provide the noise portion at least in part based on the second output; and The adder is configured to combine the noise portion with the image; and The update gate and the reset gate each include: The first convolutional block is configured to convolve the first output; A second convolutional block, configured to convolution the second input; and An arithmetic block configured to combine the outputs of the first convolutional block and the second convolutional block.

2. The apparatus of claim 1, wherein the update gate further comprises an activation block configured to apply an activation function to the output of the arithmetic block.

3. The device of claim 2, wherein the activation function includes logic configured to implement an S-shaped function.

4. The device of claim 1, wherein the arithmetic block includes logic configured to implement a summation function.

5. The apparatus of claim 1, wherein the synthesizer further comprises a current unit configured to receive the first input, the second input, the output of the update gate, and the output of the reset gate. The second output is at least partially based on the first output, the output of the update gate, and the output of the current unit.

6. The device according to claim 5, wherein the current unit comprises: The first convolutional block is configured to convolve the first output; The second convolutional block is configured to convolve the second input; A first arithmetic block, configured to combine the output of the first convolution block with the output of the reset gate; as well as A second arithmetic block is configured to combine the output of the second convolution block with the output of the first arithmetic block.

7. The device of claim 6, wherein the current unit further comprises an activation block configured to apply an activation function to the output of the second arithmetic block.

8. The device of claim 7, wherein the activation function includes logic configured to implement a hyperbolic tangent function.

9. The apparatus of claim 6, wherein the first arithmetic block comprises a function configured to implement an element-wise multiplication function.

10. The device of claim 6, wherein the second arithmetic block includes logic configured to implement a summation function.

11. The apparatus of claim 5, wherein the synthesizer further comprises: A first arithmetic block, configured to combine the first output with the output of the update gate; A unit function block configured to apply a unit function to the output of the update gate; A second arithmetic block is configured to combine the output of the current unit with the output of the unit function block; as well as A third arithmetic block is configured to combine the outputs of the first arithmetic block and the second arithmetic block to provide the second output.

12. The apparatus of claim 11, wherein the first arithmetic block includes logic configured to implement an element-wise multiplication function.

13. The apparatus of claim 11, wherein the second arithmetic block includes logic configured to implement an element-wise multiplication function.

14. The device of claim 11, wherein the third arithmetic block includes logic configured to implement a summation function.

15. The apparatus of claim 1, wherein the convolutional unit comprises: A convolutional block configured to convolve the first input; as well as An activation block, configured to activate the output of the convolution block.

16. The device of claim 15, wherein the activation block includes logic configured to implement a hyperbolic tangent function.

17. The device of claim 1, wherein the noise concern block comprises: The maximum pool block, configured to maximize the set of multiple channels of the second output; Activate the block, which is configured to generate an attention map; An extraction block configured to extract a first channel from the plurality of channels; as well as An arithmetic block configured to multiply the first channel element-wise with the interest map to provide the noise portion.

18. The device of claim 1, further comprising a feedback block configured to combine the second output with the output of the adder block.

19. The device of claim 18, wherein the feedback block comprises: An activation block configured to apply an activation function to the output of the adder block to produce an activated output; as well as A copy block configured to copy via a first channel having the second output having the activated output.

20. A system for image processing, comprising: At least one processor; as well as The processor can access at least one non-transitory medium, which is encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model includes multiple building blocks and multiple adder blocks: Individually, one of the plurality of building units is configured to receive a first input based on a second image and at least partially based on the image to receive a second input, and is further configured to provide a first output, a second output, and a noise portion, wherein the first input and the second input contain information about the image, wherein the second output is at least partially based on the first input, the second input, and the first output, and the noise portion is at least partially based on the second output. Individual of the plurality of building units includes a synthesizer, the synthesizer including an update gate and a reset gate, the update gate being configured to determine the amount of the first input and the amount of the second input to be included in the second output, and the reset gate being configured to determine the amount of the first input and the amount of the second input to be excluded from the second output; The plurality of adder blocks are configured to combine the noise portion provided by the plurality of building units into the image to provide an output image; as well as The update gate and the reset gate each include: The first convolutional block is configured to convolve the first output; A second convolutional block, configured to convolution the second input; and An arithmetic block configured to combine the outputs of the first convolutional block and the second convolutional block.

21. The system of claim 20, wherein an individual of the plurality of building blocks includes a noise concern block configured to provide the noise portion at least in part based on the second output.

22. The system of claim 20, wherein the machine learning model further comprises a plurality of feedback blocks configured to combine the output image of the second output provided by the plurality of building units with the corresponding output images of the plurality of adder blocks.

23. The system of claim 22, wherein the outputs of at least some of the plurality of feedback blocks include the second inputs of at least some of the plurality of building blocks.

24. The system of claim 20, wherein the first output is at least partially based on the convolution of the first input.

25. The system of claim 20, wherein the number of the plurality of building blocks is at least partially based on the amount of noise contained in the image.

26. The system of claim 20, further comprising: The second plurality of building units, wherein an individual of the building units is configured to receive a third input based on a third image and at least partially based on the image, and is further configured to provide a third output, a fourth output, and a second noise portion; as well as A second plurality of adder blocks are configured to combine the second noise portion provided by the second plurality of building units into the image to provide a second output image.

27. The system of claim 26, wherein the machine learning model is configured to combine a first output image of the last of the plurality of adder blocks with a second output image of the last of the second plurality of adder blocks to provide a final output image.

28. The system of claim 27, wherein the final output image comprises the average of the first output image and the second output image.

29. The system of claim 20, further comprising an ion beam scanning electron microscope configured to acquire the image and the second image.

30. The system of claim 20, wherein the image and the second image are contained in an image sequence.

31. The system of claim 30, wherein the image sequence comprises a plurality of image planes obtained from a volume.

32. The system of claim 20, wherein each of the building units comprises: An update gate is configured to determine the amount of the first input and the amount of the second input to be included in the second output; A reset gate configured to determine the amount of the first input excluded from the second output and the amount of the second input; as well as A current unit configured to receive the first input, the second input, the output of the update gate, and the output of the reset gate. The second output is at least partially based on the first output, the output of the update gate, and the output of the current unit.

33. The system of claim 20, wherein the instructions, when executed, further cause the system to determine when cancer cells are present in one or more of the output images.

34. An apparatus for image processing, comprising: A logic circuit configured to receive memory inputs and new inputs and further configured to provide a memory output and a noise portion of an image, wherein the memory inputs and the new inputs contain information about the image; The logic circuit mentioned above includes: An update gate is configured to determine the amount of memory input to be included in the memory output and the amount of new input; A reset gate configured to determine the amount of the memory input excluded from the memory output and the amount of the new input; and Nonlinear blocks are used to extract features from the memory input; A first arithmetic block is configured to combine the features with the new input and provide a combined output; A noise concern block configured to provide the noise portion at least in part based on the new input, the memory input, and the output of the reset gate; and An adder configured to combine the noise portion with the image; The update gates mentioned therein include: The first convolutional block, configured to convolve the combined output; and A first activation block, configured to apply a first activation function to the output of the first convolutional block and provide the output of the update gate; and The reset gate includes: A second convolutional block, configured to convolve the combined output; and The second activation block is configured to apply the second activation function to the output of the second convolutional block and provide the output of the reset gate.

35. The device of claim 34, wherein the first arithmetic block includes logic configured to implement a summation function.

36. The device of claim 34, wherein the first activation block comprises an S-shaped function.

37. The device of claim 34, wherein the second activation block includes logic configured to implement an S-shaped function.

38. The device of claim 34, further comprising a second arithmetic block configured to combine the feature with the output of the reset gate and provide a first combined output.

39. The apparatus of claim 38, wherein the second arithmetic block includes logic configured to implement an element-wise multiplication function.

40. The device of claim 38, further comprising a third arithmetic block configured to combine the first combined output with the new input and provide a second combined output.

41. The device of claim 40, wherein the third arithmetic block includes logic configured to implement a summation function.

42. The device according to claim 40, further comprising: The third convolutional block is configured to convolve the second combined output; as well as The third activation block is configured to apply a third activation function to the output of the third convolution block and provide a current output.

43. The device of claim 42, wherein the third activation function includes logic configured to implement a hyperbolic tangent function.

44. The device of claim 42, wherein the logic is configured to combine the current output with the output of the update gate to provide the memory output.

45. The device of claim 44, wherein the logic is further configured to combine the feature with the output of the update gate to provide the memory output.

46. ​​The device of claim 42, wherein the output of the third convolutional block is provided as an input to the noise concern block.

47. The apparatus of claim 46, wherein the output of the third convolutional block comprises a feature vector.

48. The apparatus of claim 34, further comprising an arithmetic block configured to combine the feature with the output of the update gate to produce a first combined output.

49. The apparatus of claim 48, wherein the arithmetic block includes logic configured to implement an element-wise multiplication function.

50. The device according to claim 48, further comprising: A second arithmetic block is configured to combine the output of the update gate with the current output and provide a second combined output, wherein the current output is based at least in part on the output of the reset gate, the feature, and the new input; as well as A third arithmetic block is configured to combine the first combined output with the second combined output to provide the memory output.

51. The apparatus of claim 50, wherein the second arithmetic block includes logic configured to implement an element-wise multiplication function, and the third arithmetic block includes logic configured to implement a summation function.

52. The device of claim 34, wherein the logic is configured to provide a feature vector comprising a plurality of channels to the noise concern block, wherein the feature vector is based at least in part on the new input, the memory input, and the output of the reset gate; as well as The noise concern block includes: The largest pool block, configured to maximize the set of the multiple channels; Activate the block, which is configured to generate an attention map; Extraction block, configured to extract a first channel from the plurality of channels; and An arithmetic block configured to multiply the first channel element-wise with the interest map to provide the noise portion.

53. A system for image processing, comprising: At least one processor; as well as The processor can access at least one non-transitory medium, the at least one non-transitory medium being encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model includes: A plurality of building units, wherein an individual of the building units is configured to receive memory input and a new input and is further configured to provide a memory output and a noise portion, wherein the plurality of building units are configured as a recurrent neural network, wherein an individual of the plurality of building units includes a noise concern block, the noise concern block being configured to provide the noise portion at least in part based on the new input and the memory input, wherein an individual of the plurality of building units further includes: Nonlinear blocks are used to extract features from the memory input; A first arithmetic block is configured to combine the features with the new input and provide a combined output; An update gate, configured to determine the amount of memory input and the new input to be included in the memory output; and A reset gate configured to determine the amount of memory input excluded from the memory output and the amount of the new input, wherein the noise portion is further based on the output of the reset gate; A convolutional layer configured to convolve an image and a second image to produce the memory input provided to the first of the plurality of building units, wherein the others of the plurality of building units receive the memory output as the memory input from the preceding of the plurality of building units; and Multiple adders configured to combine the noise portions provided by the multiple building units into the image to provide an output image; The update gates mentioned therein include: The first convolutional block, configured to convolve the combined output; and A first activation block, configured to apply a first activation function to the output of the first convolutional block and provide the output of the update gate; and The reset gate includes: A second convolutional block, configured to convolve the combined output; and The second activation block is configured to apply the second activation function to the output of the second convolutional block and provide the output of the reset gate.

54. The system of claim 53, wherein the machine learning model further comprises a plurality of feedback blocks configured to combine the memory outputs provided by the plurality of building units with the output images of the counterparts of the plurality of adders to provide feedback, wherein the feedback of an individual of the plurality of feedback blocks is provided as the new input to the counterparts of the other entities of the plurality of building units.

55. The system of claim 54, wherein each of the plurality of feedback blocks comprises: An activation block, configured to apply an activation function to the output image to produce an activated output; as well as A copy block configured to copy the output image through a first channel having the activated output to generate the feedback.

56. The system of claim 53, wherein the noise concern block comprises: A maximum pooling block, configured as multiple channels with a maximum set of feature vectors and generating an interest map, wherein the feature vectors are at least partially based on the new input and the memory input; An extraction block configured to extract a first channel from the plurality of channels; as well as An arithmetic block configured to multiply the first channel element-wise with the interest map to provide the noise portion.

57. The system of claim 53, wherein the image and the second image are contained in an image sequence.

58. The system of claim 57, wherein the image and the second image are consecutive images in the image sequence.

59. The system of claim 57, wherein the image sequence comprises a time series of images.

60. The system of claim 57, wherein the image sequence comprises a plurality of image planes obtained from a volume.

61. The system of claim 53, wherein the number of the plurality of building blocks is at least partially based on the amount of noise contained in the image.

62. The system of claim 53, wherein the image and the second image are focused ion beam scanning electron microscopy images.

63. The system of claim 53, wherein the instructions, when executed, further cause the system to determine when cancer cells are present in one or more of the output images.

Images