Segmenting irregular shapes in images using deep region growth

Abstract

A method for identifying an object of interest in a medical image, the method comprising: Creating an image pyramid for the medical image, the image pyramid including a plurality of layers, each layer including a plurality of values, each successive layer of the plurality of layers included in the image pyramid representing the medical image at a lower resolution than an image represented in an immediately preceding layer of the image pyramid; for each layer of the image pyramid Initializing internal states of nodes of a spatial grid, each node in the spatial grid representing a block of one or more pixels in the medical image and connected to at least one node representing a neighboring block of one or more pixels in the medical image; Iteratively performing a first convolution comprising a first concatenation of previous internal states of the nodes representing the values ​​contained in a layer of the image pyramid and the values ​​contained in the layer of the image pyramid; Performing, at each iteration, a second convolution comprising a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid immediately above the current layer of the image pyramid, and a layer of the image pyramid immediately below the current layer of the image pyramid; and Updating, at each iteration, the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network, wherein each node updates its internal state at each iteration based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node, using a squash function and results of performing the second convolution; and Identifying the object of interest in the medical image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid.

Description

AREA

[0001] Embodiments described herein relate to segmenting images, such as biomedical images, and more particularly to segmenting images using a neural network that temporally and spatially gates data propagation.

[0002] Regarding the state of the art, please refer to the following documents: STOLLENGA, Marijn F., et al. Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation, Advances in neural information processing systems, 2015, pp. 1-9; KONG, Bin, et al. Cancer metastasis detection via spatially structured deep network, in: International Conference on Information Processing in Medical Imaging. Springer, Cham, 2017, pp. 1-12; as well as LIANG, Xiaodan, et al. Interpretable structure-evolving Istm. in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, S. 2175-2184. KURZDARSTELLUNG

[0003] Embodiments described herein relate to a new type of neural network unit that combines principles used in recurrent neural networks (RNNs) and convolutional neural networks (CNNs). An RNN receives an input sequence and reads and processes one element of the sequence at a time. As the RNN processes each element in the sequence, the RNN changes its knowledge of the sequence, which is stored in the RNN's internal state. An RNN uses all or part of the internal state to either output a second sequence or make a single prediction after reading the entire input sequence. An example of an RNN is a long short-term memory (LSTM) neural network that contains one or more LSTM cells. Each LSTM cell stores previous states for the cell, which can be provided to other components of the LSTM neural network.Each LSTM cell contains an input gate, a forgetting gate, and an output gate. The LSTM was introduced to solve a problem in RNN training involving vanishing gradients.

[0004] A CNN applies filters (kernels) to an input (for example, an image) to make a prediction about the input. In one example, the prediction is which set of categories the image belongs to. Filters correspond to features that can be found in the input image. For example, when an image is input to a CNN, the filters are applied to blocks of neighboring pixels in the input image to generate an intermediate image that indicates how strongly each feature is represented at each position in the image. The content of a feature is indicated by the weights of the filter associated with the feature. The weights multiply the pixels contained in each block of neighboring pixels. For example, if the input to the CNN is a handwritten digit, the CNN classifies the handwritten digit as belonging to one of several categories (in this case, the categories are the numbers 1 to 9).The CNN's classification of the handwritten digit is based on the features of the image that the CNN has identified as belonging to the digit and on how strongly these features indicate that the handwritten digit is one of the numbers 1 to 9.

[0005] Embodiments described herein relate to segmenting biomedical images. In segmenting biomedical images, boundaries of objects in images, particularly medical images, are identified. Previously, region growing was used to identify objects in images. In region growing, a seed pixel is placed anywhere in an object of interest. Once placed in the image, the seed pixel is repeatedly spread to neighboring pixels with similar intensity and brightness. The spreading of the pixel stops when a boundary of the object is reached. In region growing, a boundary can be defined by a drop below a threshold for intensity or brightness.

[0006] One problem with region growth is that even a minimal connection with a neighboring bright pixel in a medical image can cause the region to expand outside the object of interest. For example, if Fig. 1 shows two bright tissue regions (one tissue region located inside the lung and one tissue region located outside the lung) connected by a small bright tissue fragment, region growing incorrectly represents the two bright tissue regions as belonging to the same mass or object. Region growing is therefore often omitted in favor of more sophisticated techniques such as plane fitting, conditional random fields (CRFs), active contours, and graph cuts.

[0007] CNNs rule out the precedence of pixel proximity. Instead, CNNs identify objects that exhibit a regularity. Given an object exhibiting a regularity, the CNN can be trained to classify the object as an object type. However, CNNs may not be able to accurately detect and segment shapes that are not regular, such as tumor masses, lesions, and the like. Therefore, CNNs often cannot precisely determine the boundaries of irregular shapes in medical images, such as shapes that differ in geometry, intensity, and the like.

[0008] Embodiments described herein therefore provide a technical solution to the problems described above in response to previous solutions for identifying the boundaries of irregularly shaped objects of interest. Specifically, embodiments described herein include spatially coupling a CNN with temporal gating, as used in RNN, to provide a more intelligent method for segmenting irregular structures in images. Specifically, embodiments described herein provide a new type of unit for classifying pixels in an image based on the previous internal states and current values ​​of the nodes representing pixels adjacent to the classified pixel.This new type of unit is referred to here as a gated spatiotemporal unit, which is a gated recurrent unit with spatial awareness, typically associated with a CNN. At each time step, for example, each node decides whether to update its internal state with the value of its previous internal state or the internal state of one of its neighboring nodes.

[0009] The methods and systems described herein thus provide a neural network that propagates information temporally and spatially. Compared to simply gating the flow of information temporally, temporal and spatial gating allows a recurrent unit to make decisions about a pixel's internal state based on the internal states and values ​​of surrounding pixels in the image. Furthermore, in some embodiments, the neural network can propagate information between image resolutions both temporally and spatially.

[0010] As described in more detail below, embodiments described herein use machine learning to learn an algorithm. Specifically, the network performs updates until the values ​​associated with internal states converge. A single-pass network, on the other hand, learns a function. As described above, embodiments herein provide a gated spatiotemporal unit that controls how much information propagates from one pixel to another. As described in more detail below, in some embodiments, an image is input to the system, and the system generates an image pyramid with a plurality of layers. Each layer of the image pyramid contains a different number of variables representing the input image. The base of the pyramid contains a large number of values ​​representing the image (in other words, the base layer represents the high-resolution image).At each successive level of the pyramid, fewer and fewer values ​​are used to represent the image (in other words, each successive layer represents the image at a lower resolution than the immediately preceding layer). The image pyramid allows information to be propagated from one part of the image to a lower resolution and then back to a higher resolution in another part of the image, with fewer iterations than if the system did not use an image pyramid. This is advantageous when, for example, an image with thousands of pixels is input to the system. Such an input might require the system to perform thousands of iterations before producing a prediction. The system performs convolutions using an internal state of the system from a previous time step and the representations of the image in the image pyramid.The results of the convolutional layers are used by the gated spatiotemporal unit to determine values ​​that are incorporated into a current internal state of a node in the network. Iterations are performed in the gated spatiotemporal unit until the internal states of the nodes in the network converge. When the internal states of the nodes in the system converge, a probability that each pixel belongs to an object of interest is calculated. In particular, embodiments described herein provide a network for segmenting irregular structures in medical images that is intelligent with respect to the data flow across the grid and learns other factors such as homogeneity to determine how the pixel should be spread.However, these embodiments may be applied in fields other than segmenting medical imaging, including, for example, weather forecasting, oil and gas modeling, and the like.

[0011] For example, one embodiment provides a method for identifying an object of interest in a medical image. The method includes initializing internal states of nodes in a spatial grid. Each node in the spatial grid corresponds to a pixel of the medical image and is connected to at least one node representing a neighboring pixel of the medical image. The method further includes iteratively updating the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network. At each iteration, each node updates its internal state based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node.The method further includes identifying objects of interest in the medical image based on the values ​​of the nodes upon convergence of the spatial grid.

[0012] Another embodiment further provides a method for identifying an object of interest in a medical image. However, the method provided by this embodiment includes generating an image pyramid for the medical image. The generated image pyramid includes a plurality of layers, each layer containing a plurality of values, each value representing a block of one or more pixels in the medical image. Each successive layer in the image pyramid contains fewer values ​​than the immediately preceding layer. The method further includes, for each layer of the image pyramid, initializing internal states of nodes of a spatial grid. Each node in the spatial grid represents a block of one or more pixels in the medical image and is connected to at least one node representing a neighboring block of one or more pixels in the medical image.The method further comprises, for each layer of the image pyramid, iteratively updating the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network. At each iteration, each node updates its internal state based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node. The method further includes identifying the object of interest in the medical image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid.

[0013] One embodiment provides a system for determining a region of interest in an image. The system comprises a memory and an electronic processor. The electronic processor included in the system is connected to the memory and configured to initialize internal states of nodes of a spatial grid. Each node of the spatial grid corresponds to a pixel of the image and is connected to at least one node representing a neighboring pixel of the image. The electronic processor is further configured to iteratively update the internal states of each node in the spatial grid with spatially gated propagation using a neural network and to identify the region of interest in the image based on the internal states of the nodes upon convergence of the spatial grid.

[0014] Another embodiment further provides a system for determining a region of interest in an image. Similar to the system of the embodiment described above, the system described in this embodiment also includes a memory and an electronic processor connected to the memory. However, the electronic memory of the system provided in this embodiment is configured to generate an image pyramid for the image. The image pyramid includes a plurality of layers. For each layer of the image pyramid, the electronic processor is configured to initialize internal states of nodes of a spatial grid and iteratively update the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network.Each node in the spatial grid represents a block of one or more pixels in the image and is connected to at least one node representing a neighboring block of one or more pixels in the image. The electronic processor is further configured to identify the region of interest in the image based on the internal states of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid.

[0015] One embodiment provides a non-transitory, computer-readable medium containing instructions executable by an electronic processor to perform a set of functions. The set of functions includes initializing internal states of nodes in a spatial grid. Each node represents a pixel of an image and is connected to at least one neighboring pixel of the image. The set of functions further includes iteratively updating the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network. At each iteration, each node updates its internal state based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node.The set of functions further includes identifying an object of interest in the image based on the values ​​of the nodes upon convergence of the spatial grid.

[0016] Another embodiment further provides a non-transitory, computer-readable medium containing instructions executable by an electronic processor to perform a set of functions. In contrast to the set of functions in the embodiment described above, the set of functions performed by the electronic processor of this embodiment includes generating an image pyramid for an image. The generated image pyramid includes a plurality of layers, each layer containing a plurality of values, and each value representing a block of one or more pixels in the image. Each successive layer in the image pyramid contains fewer values ​​than an immediately preceding layer. The set of functions further includes, for each layer of the image pyramid, initializing internal states of nodes of a spatial grid.Each node of the image pyramid represents a block of one or more pixels in the image and is connected to at least one node representing a neighboring block of one or more pixels in the image. The set of functions further comprises, for each layer of the image pyramid, iteratively updating the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network. At each iteration, each node updates its internal state based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node.The set of functions further includes identifying an object of interest in the image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a medical image where region growing was applied to identify an object of interest. Fig. Figure 2 illustrates a system for determining a region of interest in an image. Fig. 3 illustrates a system of Fig. 2 included neural network. Fig. Figure 4 illustrates an example of an input to a node in a gated spatio-temporal unit. Fig. Figure 5 illustrates an example of a medical image that the neural network of Fig. 4 as input. Fig. Figure 6 illustrates an example of an area of ​​interest that the neural network of Fig. 4 in the medical image of Fig. 5 recognizes. DETAILED DESCRIPTION

[0017] One or more embodiments are described and illustrated in the following description and the accompanying drawings. These embodiments are not limited to the specific details provided herein and are subject to modification in many ways. Moreover, there may be other embodiments not described herein. Functionality described herein as being performed by one component may be further performed by multiple components in a distributed manner. Likewise, functionality performed by multiple components may be aggregated and performed by a single component. Similarly, a component described as performing a particular functionality may also perform additional functionality not described herein.Furthermore, a device or structure "configured" in a particular manner is configured at least in that manner, but may also be configured in a manner not listed. Further, some embodiments described herein may include one or more electronic processors configured to perform the described functionality by executing instructions stored on a non-transitory, computer-readable medium. Similarly, embodiments described herein may be implemented as a non-transitory, computer-readable medium storing instructions executable by one or more electronic processors to perform the described functionality.As used in this application, a "non-transitory computer-readable medium" includes any computer-readable media, but does not consist of a volatile, propagating signal. A non-transitory computer-readable storage medium may therefore include, for example, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, ROM (read-only memory), RAM (random access memory), register memory, a processor cache, or a combination thereof.

[0018] Furthermore, the phraseology and terminology used herein is for the purpose of description and should not be considered limiting. The use of "comprise," "include," "include," "have," and variations thereof herein is intended to include the items listed below and their equivalents, as well as additional items. The terms "connected" and "coupled" are used generally and include both direct and indirect connections and couplings. Furthermore, "connected" and "coupled" are not limited to physical or mechanical connections or couplings and can include electrical connections or couplings, whether direct or indirect.Furthermore, electronic data transmissions or communications may be conducted using wired connections, wireless connections, or a combination thereof, and may be transmitted directly or through one or more intermediaries over various types of networks, data transmission channels, and connections. Furthermore, relationship terms such as "first" and "second," "top" and "bottom," and the like may be used herein only to distinguish one entity or action from another, without necessarily requiring or implying any such relationship or order between such entities or actions.

[0019] As described above, biomedical image segmentation aims to identify pixels in an image that represent an object of interest, allowing various calculations and data processing to be performed on the object (e.g., volume calculation and the like). However, many techniques for performing image segmentation rely on identifying consistent shapes and relationships. For example, as described above, CNNs excel at recognizing shapes and objects in images that they have been trained to recognize, but CNNs struggle to recognize irregular shapes in images. Techniques that rely on identifying consistent shapes and relationships may therefore be ineffective at identifying irregular objects such as tumor masses, lesions, and the like.

[0020] Other techniques rely on pixel propagation to determine the boundaries of an object of interest in an image. As described above, region growing does not rely on regularity; instead, a starting pixel is propagated to neighboring pixels until boundaries are identified. The shape of an object of interest therefore does not affect the performance of region growing. However, if an object does not have a well-defined boundary (e.g., if the object is connected to neighboring bright tissue by even a small connection), region growing can be used as in Fig. 1 shows that an object can grow outside its original boundary in an inappropriate manner.

[0021] To address the shortcomings of the techniques described above, embodiments described herein combine the advantages of CNNs and RNNs in a spatio-temporal unit to improve the detection of irregular objects in images. As described in more detail below, embodiments described herein specifically utilize spatially gated propagation. In gating, one part of a network generates a new state for the system (based on its previous state and newly received information), and a separate part of the network gates this new state and decides whether to use the new state and propagate it forward in time. As described herein, the immediately previous states of a pixel and the pixel's nearest neighbors are gated and used to determine the pixel's internal state at a current time step.The systems and methods described herein therefore propagate values ​​both spatially and temporally. Furthermore, the creation of the image pyramid described above enables the propagation of values ​​across different image resolutions.

[0022] Fig. Figure 2 illustrates a system 200 for implementing a neural network. Neural networks are machine learning models that use one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an input layer and an output layer. The output of each hidden layer is used as input to the next layer in the network (the next hidden layer or the output layer). Each layer of the network generates an output from a received input according to current values ​​of a respective set of parameters.

[0023] As in Fig. 2, the system 200 includes a data processing unit 202 that includes an electronic processor 204 and a memory 206. The electronic processor 204 and the memory 206 communicate wirelessly, via wired data transmission channels or buses, or a combination thereof. The data processing unit 202 may include additional components beyond those illustrated in Fig. 2 in various configurations. In some embodiments, the computing unit 202 includes, for example, multiple electronic processors, multiple memory modules, or a combination thereof. In some embodiments, the computing unit 202 includes one or more input / output interfaces that enable the computing unit 202 to communicate with networks, peripherals, and the like.

[0024] It should be understood that the functionality described herein, as performed by computing device 202, may be performed in a distributed manner by a plurality of computing devices located in different geographic locations. For example, the functionality described herein, as performed by computing device 202, may be performed by a plurality of computing devices 202 included in a cloud computing environment. Electronic processor 204 may be a microprocessor, an application-specific integrated circuit (ASIC), and the like. Electronic processor 204 is generally configured to execute software instructions to perform a set of functions, including the functions described herein.Memory 206 includes a non-transitory, computer-readable medium and stores data, including instructions executable by electronic processor 204. As shown, for example, in . Fig. 2, the memory 206 stores a neural network 208 containing a computer program executed by the electronic processor 204.

[0025] Fig. Figure 3 illustrates a visual representation of an example of the neural network 208 that the electronic processor 204 executes to perform the methods described herein. As shown in Fig. 3, the neural network 208, when executed by the electronic processor 204, provides a machine learning system that receives an input and generates an output 305. The input includes, for example, an image (an input image 300), e.g., a biomedical image, or another type of multidimensional data, and the output 305 similarly includes an image or another type of multidimensional data.

[0026] As in Fig. 3, the input image 300 is input to a first layer 310 of the neural network 208. It should be understood that although the first layer 310 is shown as a single layer, this is for illustrative purposes only, and the first layer 310 may include any number of layers. In the first layer 310, the neural network 208 may perform a plurality of convolutions at values ​​representing the brightness of each pixel. In other embodiments, the neural network 208 may perform a plurality of convolutions in the first layer 310 to generate an image pyramid 315 from the input image 300 (I0), as described below.

[0027] The image pyramid 315 is a sequence of tensors (I1 - I l) convolved from the input image 300. The tensor generated for l = 1 has the same spatial dimensions as the input image 300 (I0), but the tensors halve with each subsequent convolution / reduction. Tensors for each value of l therefore have a different resolution, with tensor I1 having the highest resolution and tensor I l the lowest resolution. The following equations illustrate the process for generating the image pyramid 315, which is performed in the first layer 310. I1=K1l∗I0 I2=K2I∗D1I∗I1⋮ Il=KlI∗Dl−1I∗Il−1

[0028] The operator * represents a convolution operation. For example, the equation A * B represents a convolution between an input B and a kernel A.

[0029] I0 is a variable representing the original input image 300. I0 has dimensions N0 × N0 × 1. In other words, the input image 300 has N0 rows, N0 columns, and (since the input image 300 is a grayscale image in this exemplary embodiment) one channel.

[0030] I l is a variable representing an intermediate form of image data (a tensor) that is generated after one or more reductions have been applied to the input image 300 (I0). If l > 1, then I l as described above, a lower resolution than the input image 300. I0). I l has the dimensions N l × N l × C, where N l = 2 -(l-1) N0 and C is the number of channels. KlI is a variable representing a convolution operator (a kernel) that preserves the dimensions of the input image data. The input image data has dimensions N l × N l× C I while the output image data has dimensions N l × N l × C0. K can represent the combination of several sequential convolution operations arranged, for example, as in AlexNet, DenseNet, or a number of other architectures, as well as the learnable parameters of the convolution operators. D1I is a variable representing a convolution operator that reduces the dimensions of the input image data by half. The input image data has dimensions N l-1 × N l-1 × C I while the output image data has dimensions N l × N l× C0. Similar to K, D can represent multiple sequential convolution operations, arranged, for example, as in AlexNet, DenseNet, or a number of other architectures, as well as the learnable parameters of the convolution operators. However, the convolution operator D also represents a max-pooling convolution layer or a strided convolution layer, which reduces the dimensions of the input image data by half.

[0031] The tensor calculated for each level of the image pyramid 315 is fed to a second layer 320. The equation illustrating the operations performed in the second layer 320 is Xlt=KlIH∗[Il,Hlt]

[0032] Here too, the operator * represents a convolution operation, and I lis a variable representing an intermediate form of image data (a tensor) that is generated after one or more reductions have been performed on the input image 300 (I0).

[0033] [A, B] is a join operation between tensors, e.g., tensors A and B. A join operation performed on two tensors connects the channels contained in each of the tensors. For example, if tensor A has dimensions M × M × C1 and tensor B has dimensions M × M × C2, the output of [A, B] has dimensions M × M × (C1 + C2). Holds is a tensor 322 that stores an internal state for each node in the spatial grid at a resolution l and at a time step t. As described above, the internal state of each node in the spatial grid is updated at each time step. The tensor 322 has dimensions N l × N l × C H . Therefore there is CH Variables that describe each block of one or more pixels of the image with a resolution l. KlIH is a variable representing a convolution operator that preserves the dimensions of the input image data. The input image data has dimensions N l × N l × C I , while the output image data has dimensions N l × N l × C0. K can represent the combination of several sequential convolution operations arranged, for example, as in AlexNet, DenseNet, or a number of other architectures, as well as the learnable parameters of the convolution operators. XLt is a variable that represents the results 323 of performing equation (4). XLt has the dimensions N l × N l × C X and is input to the third layer 325 of the neural network 208.

[0034] In summary, equation (4) links the tensor (I l ) with the tensor 322 (Hlt) (performs a first operation), applies a convolution operator KlIH on the link (performs a first convolution for a current layer of the image pyramid) and stores the results 323 in the tensor Xlt.

[0035] The equation illustrating the operations performed in the third layer 325 is Zlt=KlL3∗[DlX∗Xl−1t,Xlt,UlX∗Xl+1t]

[0036] As described above, the operator * also represents a convolution operation, and [A, B] is a concatenation operation between tensors, e.g., tensors A and B. KlL3 is similarly a variable representing a convolution operator (a kernel) that preserves the dimensions of the input image data, DlX is a variable representing a convolution operator (a kernel) that reduces the dimensions of the input image data by half, and XLt is a variable that represents the result of equation (4), which is calculated using the tensor I l , the internal state Holds and the kernel KlIH was calculated. XL+1t is a variable that represents the result of equation (5), which is calculated using the tensor I l+1 , the internal state Hl+1t and the kernel Kl+1IH was calculated, and Xl−1t is a variable that represents the result of equation (5), which is calculated using the tensor I l-1 , the internal state Hl−1t and the kernel Kl−1IH was calculated. UlX is a variable representing a convolution operator (a kernel) that upsamples the dimensions of the input image data by doubling the dimensions. For example, if the input image data has dimensions N l+1 × N l+1 × C I the output image data has dimensions N l × N l × C0. Similar to the convolution operator K, the convolution operator U can represent the combination of multiple sequential convolution operations arranged as in AlexNet, DenseNet, or a number of other architectures, along with the learnable parameters of the convolution operations. However, the convolution operator U can also represent a transposed convolution layer to double the image dimensions. Zlt is a tensor containing the result of performing equation (5). Zlt contains information that is passed to the gate-controlled spatio-temporal unit. In summary, equation (5) involves reducing the results (Xl−1t) calculating equation (4) with a tensor that represents the input image 300 with a higher resolution (I l-1 ) (a layer of the image pyramid immediately below the current layer of the image pyramid), and upsampling the results (XL+1t) calculating equation (4) with a tensor that represents the input image 300 with a lower resolution (I l+1 ) (a layer of the image pyramid immediately above the current layer of the image pyramid). Equation (5) also includes linking the results of upsampling with the results of downsampling (sampling rate reduction) and with the results of calculating equation (4) from a tensor representing the input image 300 with a resolution I l(performs a second operation). The equation convolves the results of the operation with the kernel KlL3 (performs a second convolution) and stores the results in the variable Zlt.

[0037] In the first iteration of the neural network 208, the internal state for each node contained in a spatial grid of a gated spatiotemporal unit 330 is initialized only with the output from the third layer 325. Each node contains a vector of values ​​representing the internal state of the node and values ​​derived in the image pyramid from the brightness of a block of one or more pixels centered at that node. In each successive iteration, the internal state of each node is inherited from the previous iteration via the tensor 322. (Hlt) input to the second layer 320 of the neural network 208. The process described above is then repeated starting with the second layer 320.

[0038] As described above, the neural network 208 includes the gated spatiotemporal unit 330 having a plurality of nodes arranged in a spatial grid. Each node in this grid corresponds to a pixel in the input image 300. The gated spatiotemporal unit 330 performs data processing at each of a plurality of time steps. At each time step, the gated spatiotemporal unit 330 receives a plurality of values. Based on the received values ​​and the values ​​representing the internal state of each node gated in the previous time step, the gated spatiotemporal unit 330 determines how to update the internal state of each node in the current time step.As described in more detail below, the gated spatio-temporal unit 330 determines how to update the internal state of each node by deciding for each node in the lattice whether to retain the node's internal state from a previous time step, set the node's internal state to a value that represents the internal state of the neighboring node from a previous time step, or create a new internal state for the node.

[0039] The following equation is an example of a calculation used to determine the internal state Hold+1 327 of a node in the grid of the gate-controlled spatial-temporal unit 330 in the current time step and is distributed over seven rows (labeled I to VII) for easier understanding. Hlt+1=tanh[ σ(Zl,1t)⊙S0,0∗Hlt +σ(Zl,2t)⊙S0,−1∗Hlt +σ(Zl,3t)⊙S−1,0∗Hlt +σ(Zl,4t)⊙S1,0∗Hlt +σ(Zl,5t)⊙S0,1∗Hlt +σ(Zl,6t)⊙tanh(Zl,7t)] σ(A) represents the application of the sigmoid function 1 / (1 + e -a ) elementwise to each element a of the tensor A. The sigmoid function can also be called a "squashing function." The sigmoid function takes any input value from +∞ to -∞ and squeezes the input value into an output value from 0 to 1.

[0040] Tanh is also a squeezing function. It also takes any input value from +∞ to -∞, but the tanh function squeezes the input value to an output value from -1 to 1.

[0041] The operator ⊙ represents a Hadamard product operation. For example, given the equation ⊙, a Hadamard product operation is performed between input B and input A. The Hadamard product is an element-wise multiplication of each pair of elements of two inputs of equal size. Zlt is a tensor containing the results 326 of the calculation performed in the third layer 325 with a resolution l and in time step t. Zlt has the dimensions N l × N l × 7 × C H Each of the seven elements of the third dimension plays a special role in the spatio-temporal gate control process. The variables Zl,1t until Zl,7t in the equation refer to the tensor that results when one of the seven elements is selected. Each tensor associated with each of the seven elements has dimensions N l × Nl × C H . Holds is a tensor 322 that stores an internal state for each node in the spatial grid with a resolution l and at a time step t. As described above, the internal state is dynamically updated at each time step. The tensor has dimensions N l × N l × C H . Therefore there is C H Variables that describe each node with a resolution l.

[0042] S Δx,Δy is a convolution operator with spatial shift. It has no learnable parameters. S Δx,Δy allows information from the internal states of the nearest neighbor nodes to be taken into account when determining the current internal state of a node.

[0043] To return to the above equation, the results 326 contained in the tensor Zlt are divided into seven parts. Each part represents an element in the third dimension of the tensor, as described above. Zlt Line (I) of equation (6) applies a tanh squeezing function to the sum of lines II to VII of equation (6) to determine the internal state of a node at the current iteration. Line (II) of equation (6) corresponds to the ability to copy the internal state of the node from the previous timestamp to the current timestamp depending on the gated value. The next four lines (III to VI) (+σ(Zl,2t)⊙S0,−1∗Hlt,+σ(Zl,3t)⊙S−1,0∗Hlt,+σ(Zl,4t)⊙S1,0∗Hlt,+σ(Zl,5t)⊙S0,1∗Hlt) Each corresponds to the possibility of copying the internal states of one of the nearest neighbors from the previous iteration into the internal state of the node in the current iteration. The last line (VII) corresponds to creating a completely new value and possibly setting the internal state of the node to the new value in the current iteration.

[0044] Fig. Figure 4 illustrates the connections between a node 400 whose internal state is being determined in a current iteration and nodes with internal states determined in a previous iteration. Each node is connected to nodes whose internal states were calculated in an immediately preceding iteration of the gated spatiotemporal unit 330. In particular, each node is connected to a node representing its own internal state in the immediately preceding iteration of the gated spatiotemporal unit 330, as well as to nodes representing internal states of its neighboring nodes in the immediately preceding iteration of the gated spatiotemporal unit 330. In Fig. 4, nodes in group 405 are nodes that each correspond to internal states determined in an immediately preceding iteration of neural network 208. Nodes in group 410 are nodes that correspond to internal states determined in the current iteration of neural network 208. As described above, each node corresponds to a pixel (or a block of one or more pixels) in input image 300. Each neighbor node of a node represents a pixel (or a block of one or more pixels) that is adjacent to the pixel (or block of one or more pixels) represented by the node.For example, if a node 400 represents a pixel in the image at coordinates (i,j), the node 400 representing the pixel at (i,j) is connected to a node 415 representing a pixel at coordinates (i - 1,j) (immediately to the left of the pixel at (i,j)), to a node 420 representing a pixel at coordinates (i + 1,j) (immediately to the right of the pixel at (i,j)), to a node 425 representing a pixel at coordinates (i,j + 1) (immediately above the pixel at (i,j)), and to a node 430 representing a pixel at coordinates (i,j - 1) (immediately below the pixel at (i,j)). Each of the above nodes described as being connected to node 400 representing pixel (i,j) is the neighbor node of node 400. The gated spatio-temporal unit 330 therefore determines whether to set the internal state of node 400 to the internal state of one of the nodes in group 405.

[0045] When the internal states of the nodes in the spatial grid of the gated spatiotemporal unit 330 converge (change less than a predetermined amount), the internal states of the nodes representing the input image 300 with the highest resolution are output to a final layer 335 of the neural network 208. The final layer 335 uses a value contained in the internal state of each node to calculate the probability (for example, a value between zero and one) that the pixel represented by that node belongs to the object of interest in the input image 300. The following equation describes the operation performed in the final layer 335 to determine the probability that each pixel is part of an object of interest: Yt=σ(K1Y∗H1t) Y tis a variable representing the output 305 of the neural network 208 at a time step t and has dimensions N0 × N0 × 1 (the same dimensions as the input). H1t is a tensor that stores an internal state for each node in the spatial grid with a resolution of 1 and at time step t. K1Y is a variable representing a convolution operator that preserves the dimensions of the input image data. The input image data has dimensions N0 × N0 × C I , while the output image data has dimensions N0 × N0 × 1. K. K can represent the combination of several sequential convolution operations arranged, for example, as in AlexNet, DenseNet, or in a number of other architectures, as well as the learnable parameters of the convolution operators.

[0046] In summary, equation (7) applies a final convolution with K1Y on the internal state H1t with the highest resolution, increasing the number of input channels (C H ) is reduced to 1 output channel. The equation applies the sigmoid function to the results of applying a final convolution with K1Y on the internal state H1t with the highest resolution, allowing everyone in Y t contained value is squeezed to a value between 0 and 1. Each value in Y t between 0 and 1 corresponds to the probability that an image pixel is located in an object of interest in the input image 300. For example, if the value generated by the sigmoid function for a single pixel is 5, there is a fifty percent (50%) probability that the pixel is located in the object of interest.

[0047] The neural network 208 may store the generated output 305 (the calculated probabilities for each node) in an output data repository (e.g., memory 206) or make the generated output 305 available for use or consumption, for example, by displaying the output 305 to a user on a display device. Regardless, for each pixel, the electronic processor 204 compares the probability that the pixel is contained within an object of interest to a predetermined threshold. If the probability that a pixel is part of an object of interest is above a predetermined threshold, the electronic processor 204 determines that the pixel is part of the object of interest.

[0048] It is understood that in some embodiments, equations (4 to 6) are implemented in the neural network 208 described above for each representation level of the input image 300 (I1 - I l ) contained in the image pyramid 315. It is further understood that while the neural network 208 is described above as passing values ​​temporally, spatially, and resolution-wise, the neural network 208 can be modified to only pass values ​​temporally and spatially.

[0049] It should also be understood that the values ​​of the gates used to determine the internal state of each node at each iteration must be either zero or one, but can have any value between 0 and 1 (see equation (6) above). Therefore, in some embodiments, the updated internal state of a node may be a mixture (or, more mathematically, a linear combination) of two or more of the options described above (a value of the node from a previous iteration, values ​​of one or more neighboring nodes from a previous iteration, and a new value of the node).

[0050] Fig. 5 and Fig. 6 provide an example of a practical application of the neural network. Fig. Figure 5 illustrates an example of a medical image 500 that the neural network 208 may receive as input. The object of interest in the image 500 is a tumor 505 in a left lung 510. Fig. Figure 6 illustrates the region of the medical image 500 that the neural network 208 identifies as an object of interest (the tumor 505). Unlike using region growth (see Fig. 1) the boundaries of the object of interest do not extend beyond the left lung 510.

[0051] Embodiments described herein thus provide a neutral network that includes a spatiotemporal unit. The spatiotemporal unit is a spatially extended grid of nodes. Each node corresponds, for example, to a pixel in an image. The neural network determines an initial internal state for each node and iteratively updates the internal state for each node to continually generate a new internal state by propagating values ​​temporally, spatially, or both and calculating new values ​​to represent the internal state for each node.Therefore, compared to other types of RNNs such as long short-term memory (LSTM) networks and gated recurrent unit (GRU) networks that iterate on one-dimensional sequences of letters or words, embodiments described herein consider the decisions of neighboring nodes when updating each node's internal state. Specifically, embodiments described herein apply to both spatial and temporal dimensions. Therefore, although the temporal dimension only iterates forward, spatial gating allows spatial information to propagate back and forth across the spatial grid for as long as necessary, as new conclusions are determined in one part of the image and propagated to other parts of the image to influence decision-making in those parts.In some embodiments described herein, values ​​in the neural network 208 may additionally be propagated between different resolutions of the image.

[0052] The embodiments described herein are closed. In particular, the neural network 208 described herein receives all information about the outside world as its first input (the image to be processed), and from that point on, the neural network 208 evolves over time only according to its own internal state and rules, without incorporating any further information from the outside. Iteration therefore continues until convergence, when the internal state no longer changes. This makes the neural network 208 more of an algorithm than a function. In contrast, RNNs receive a new part of the problem (e.g., a word) at each time step, so iteration continues only as long as new information is available.

[0053] The following claims set forth various features and advantages of some embodiments.

Claims

[1] A method for identifying an object of interest in a medical image, the method comprising: Creating an image pyramid for the medical image, the image pyramid including a plurality of layers, each layer including a plurality of values, each successive layer of the plurality of layers included in the image pyramid representing the medical image at a lower resolution than an image represented in an immediately preceding layer of the image pyramid; for each layer of the image pyramid Initializing internal states of nodes of a spatial grid, each node in the spatial grid representing a block of one or more pixels in the medical image and connected to at least one node representing a neighboring block of one or more pixels in the medical image; Iteratively performing a first convolution comprising a first concatenation of previous internal states of the nodes representing the values ​​contained in a layer of the image pyramid and the values ​​contained in the layer of the image pyramid; Performing, at each iteration, a second convolution comprising a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid immediately above the current layer of the image pyramid, and a layer of the image pyramid immediately below the current layer of the image pyramid; and Updating, at each iteration, the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network, wherein each node updates its internal state at each iteration based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node, using a squash function and results of performing the second convolution; and Identifying the object of interest in the medical image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid. [2] The method of claim 1, wherein iteratively updating the internal states of the nodes using a neural network includes updating a value in a vector of values ​​associated with the internal states of the nodes. [3] The method of claim 1, wherein the method further comprises: Saving results of performing the first convolution. [4] The method of claim 1, wherein generating the image pyramid includes performing convolutions on each value representing a brightness of each block of one or more pixels in the medical image, each convolution comprising a reduction of dimensions of input data of a medical image producing values ​​used to represent the medical image in a next layer of the image pyramid. [5] The method of claim 1, wherein each value representing the medical image in the first layer of the image pyramid corresponds to a pixel in the medical image. [6] The method of claim 5, wherein identifying the object of interest in the medical image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid comprises using a last layer of the neural network to calculate a probability that each pixel in the medical image is included in the object of interest based on a value contained in each vector of values ​​associated with a node representing the values ​​contained in a first layer of the image pyramid; and Determine for each pixel whether the calculated probability is above a previously specified threshold. [7] The method of claim 1, wherein each node updates its internal state based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighbor node from the previous iteration, and a new value of the node, comprising using a squash function and the results of performing the second convolution. [8] The method of claim 1, wherein the neighbor node is selected from a group consisting of a node representing a block of one or more pixels located immediately above, immediately below, to the right, and to the left of a block of one or more pixels represented by the node. [9] The method of claim 1, wherein each value represents a block of one or more pixels in the medical image, and wherein each successive layer contains fewer values ​​than an immediately preceding layer. [10] A system for determining a region of interest in an image, the system comprising: a memory; and an electronic processor connected to the memory and configured to generate an image pyramid for the image, the image pyramid including a plurality of layers, each successive layer of the plurality of layers included in the image pyramid representing the image at a lower resolution than an image represented in an immediately preceding layer of the image pyramid, for each layer of the image pyramid, internal states of nodes of a spatial grid are initialized, each node representing a block of one or more pixels in the image and being connected to at least one node representing a neighboring block of one or more pixels in the image, iteratively performs a first convolution, which comprises a first linking of previous internal states of the nodes representing the values ​​contained in the layer of the image pyramid and the values ​​contained in the layer of the image pyramid, performs a second convolution at each iteration comprising a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid immediately above the current layer of the image pyramid, and a layer of the image pyramid immediately below the current layer of the image pyramid; and the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network are updated at each iteration based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node, by the processor using a squash function and results of performing the second convolution; and identifies the region of interest in the image based on the internal states of the nodes upon convergence of the spatial grid with nodes representing values ​​contained in a first layer of the image pyramid. [11] The system of claim 10, wherein the electronic processor is configured to display the image at a lower resolution by displaying the image with fewer values. [12] The system of claim 10, wherein the electronic processor is configured to update the internal states of the nodes by deciding at each iteration for each node whether to retain a value of the node from a previous iteration, set a value of the node to a value of a neighboring node from a previous iteration, or set a new value of the node. [13] The system of claim 10, wherein the electronic processor is configured to iteratively update the internal states of the nodes using a neural network by updating a value in a vector of values ​​associated with the internal states of the nodes. [14] The system of claim 10, wherein the electronic processor is configured to store the results of performing the first convolution at each iteration for each layer of the image pyramid. [15] The system of claim 10, wherein the electronic processor is further configured to perform, in the first iteration, convolutions on each value representing a brightness of each block of one or more pixels in the image, wherein each convolution comprising a reduction of dimensions of the input image data produces values ​​used to represent the image in a next layer of the image pyramid. [16] The system of claim 10, wherein the electronic processor is configured to identify an object of interest in the image based on the values ​​of the nodes at a convergence of the spatial grid with nodes representing values ​​contained in a first layer of the image pyramid by using a final layer of the neural network to calculate a probability that each pixel in the image is contained in the object of interest based on each vector associated with a node representing the values ​​contained in a first layer of the image pyramid, and Determine for each pixel whether the calculated probability is above a previously specified threshold. [17] The system of claim 12, wherein the neighbor node is selected from a group consisting of a node representing a block of one or more pixels in the image that is immediately above, immediately below, to the right, and to the left of the block of one or more pixels in the image represented by the node. [18] A non-transitory, computer-readable medium storing instructions that, when executed by an electronic processor, perform a set of functions, the set of functions comprising: Creating an image pyramid for an image, the image pyramid including a plurality of layers, each successive layer of the plurality of layers included in the image pyramid representing the image at a lower resolution than an image represented in an immediately preceding layer of the image pyramid; for each layer of the image pyramid, initializing the internal states of nodes in a spatial grid, each node representing a block of one or more pixels in the image and connected to at least one node representing a neighboring block of one or more pixels in the image; Iteratively performing a first convolution comprising a first concatenation of previous internal states of the nodes representing the values ​​contained in a layer of the image pyramid and the values ​​contained in the layer of the image pyramid; Performing, at each iteration, a second convolution comprising a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid immediately above the current layer of the image pyramid, and a layer of the image pyramid immediately below the current layer of the image pyramid; and Updating, at each iteration, the internal states of the nodes in the spatial grid with spatially gated propagation using a neural network, wherein each node updates its internal state at each iteration based on at least one selected value from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node, using a squashing function and results of performing the second convolution; and Identifying an object of interest in the image based on the values ​​of the nodes upon convergence of the spatial grid having nodes representing the values ​​contained in a first layer of the image pyramid. [19] The non-transitory computer-readable medium of claim 18, wherein iteratively updating the internal states of the nodes using a neural network includes updating a value in a vector of values ​​associated with the internal states of the nodes. [20] The non-transitory computer-readable medium of claim 18, wherein identifying an object of interest in the image based on the values ​​of the nodes upon convergence of the spatial grid with nodes representing the values ​​contained in a first layer of the image pyramid comprises using a final layer in the neural network to calculate a probability that each pixel in the image is included in the object of interest based on the vector associated with a node representing the values ​​contained in a first layer of the image pyramid; and Determine for each pixel whether the calculated probability is above a previously specified threshold.

Images