Segmenting irregular shapes in images using deep region growing

Abstract

A system for determining a region of interest in an image. The system includes a memory and an electronic processor. An electronic processor included in the system is connected to the memory and configured to initialize internal states of nodes of a spatial lattice. Each node of the spatial lattice corresponds to a pixel of the image and is connected to at least one node representing an adjacent pixel of the image. The electronic processor is also configured to use a neural network to iteratively update the internal state of each node in the spatial lattice using spatial gated propagation, and identify a region of interest in the image based on the internal states of the nodes at which the spatial lattice converges. In one embodiment, the electronic processor is configured to create an image pyramid for the image.

Description

Technical Field

[0001] Embodiments described herein relate to segmenting images, such as biomedical images, and in particular, segmenting images using neural networks that gate the propagation of data in both time and space. Summary of the invention

[0002] Embodiments described herein relate to a novel neural network unit that combines the principles used in recurrent neural networks (RNNs) and convolutional neural networks (CNNs). The RNN receives an input sequence and reads and processes one element of the sequence at a time. As the RNN processes each element in the sequence, the RNN modifies its knowledge about the sequence, which is stored in the internal state of the RNN. The RNN uses some or all of the internal state to output a second sequence or make a single prediction after it has read the entire input sequence. An example of an RNN is a long short-term memory (LSTM) neural network including one or more LSTM cells. Each LSTM unit stores the previous state of the unit, which can be provided to other components of the LSTM neural network. Each LSTM unit includes an input gate, a forget gate, and an output gate. LSTM is introduced to solve the problem of RNN training associated with vanishing gradients.

[0003] CNN applies a filter (kernel) to an input (e.g., an image) to make predictions about the input. In one example, the prediction is which category of a set of categories the image belongs to. The filter corresponds to features that can be found in the input image. For example, when an image is input to a CNN, a filter is applied to adjacent blocks of pixels in the input image to produce an intermediate image that indicates how strongly each feature is represented at each location in the image. The content of the feature is indicated by the weight of the filter associated with the feature. The weight is multiplied by the pixels included in each adjacent block of pixels. For example, if the input to the CNN is a handwritten digit, the CNN classifies the handwritten digit as belonging to one of multiple categories (in this case, the categories are digits 1-9). The classification of the handwritten digit by the CNN is based on the features of the image that the CNN finds to be associated with the digit and the strength with which these features indicate that the handwritten digit is one of the digits 1-9.

[0004] Embodiments described herein relate to biomedical image segmentation. Biomedical image segmentation involves identifying the boundaries of objects in an image, particularly in a medical image. Region growing has previously been used to identify objects in an image. As the region grows, a seed pixel is placed somewhere within the object of interest. Once placed in the image, the seed pixel is repeatedly expanded to adjacent pixels with similar intensity or brightness. When the boundary of the object is reached, the expansion of the pixel stops. In region growing, the boundary can be defined by a drop below a threshold intensity or brightness.

[0005] One problem with region growing is that even tiny connections to neighboring bright pixels in medical images can cause the region to expand outside the object of interest. Figure 1 As shown in the figure, two bright tissue regions (one tissue region is located inside the lung and one tissue region is located outside the lung) are connected by a small bright tissue fragment. Region growing will incorrectly show that the two bright tissue regions belong to the same mass or object. Therefore, region growing is usually discarded in favor of more complex methods such as level sets, conditional random fields (CRFs), active contours, and graph cuts.

[0006] CNN discards the primary factor of pixel adjacency. Instead, CNN recognizes objects with regularity. Objects with regularity allow CNN to be trained to classify objects into object types. However, CNN may not be able to accurately distinguish and segment irregular shapes, such as tumor masses, lesions, etc. Therefore, CNN is generally unable to accurately determine the boundaries of irregular shapes in medical images, such as shapes that vary in geometry, intensity, etc.

[0007] Therefore, the embodiments described herein provide a technical solution to the above-mentioned problems in response to previous solutions for identifying the boundaries of irregularly shaped objects of interest. Specifically, the embodiments described herein combine the spatial connectivity of CNNs with the temporal gating used in RNNs to provide a smarter method for segmenting irregular structures in images. In particular, the embodiments described herein provide a new type of unit for classifying pixels in an image based on the previous internal state and current value of nodes representing pixels adjacent to the pixel being classified. This new type of unit is referred to herein as a gated spatiotemporal unit, which is a gated recurrent unit with spatial perception typically associated with CNNs. For example, at each time step, each node decides whether to update its internal state with the value of its previous internal state or the internal state of one of its neighboring nodes.

[0008] Thus, the methods and systems described herein provide a neural network that propagates information in both time and space. Gating in time and space allows the recurrent unit to decide the internal state of a pixel based on the internal states and values ​​of surrounding pixels in the image, compared to simply gating the flow of information in time. Furthermore, in some embodiments, the neural network can propagate information between image resolutions in time and space.

[0009] As described in more detail below, the embodiments described herein use machine learning to learn algorithms. In particular, the network updates until the value associated with the internal state converges. In contrast, a single-pass network learns a function. As described above, the embodiments herein provide a gated spatiotemporal unit that controls how much information is propagated from one pixel to another. As described in more detail below, in some embodiments, an image is input to a system, and the system creates an image pyramid comprising multiple layers. Each layer of the image pyramid includes a different number of variables representing the input image. The bottom of the pyramid includes a large number of values ​​representing the image (in other words, the base layer represents a high-resolution image). At each successive level of the pyramid, fewer and fewer values ​​are used to represent the image (in other words, each successive layer represents the image at a lower resolution than the frontmost layer). The image pyramid allows information from one part of the image to propagate to a lower resolution in fewer iterations than if the system did not use the image pyramid, and then return to a higher resolution in a different part of the image. For example, this is beneficial if an image with thousands of pixels is input to the system. Such an input may require the system to perform thousands of iterations before generating a prediction. The system performs convolution using the internal state of the system from the previous time step and the representation of the image in the image pyramid. The results of the convolutional layers are used by the gated spatiotemporal unit to determine the values ​​to be included in the current internal state of the nodes in the network. Iterations are performed on the gated spatiotemporal unit until the internal states of the nodes in the network converge. When the internal states of the nodes in the system converge, the probability of each pixel belonging to the object of interest is calculated. In particular, the embodiments described herein provide a network for segmenting irregular structures in medical images that is intelligent about how data flows on the lattice and learns other factors such as uniformity to determine how to expand pixels. However, these embodiments can be applied to fields other than medical imaging segmentation, including, for example, weather prediction, oil and gas modeling, etc.

[0010] For example, one embodiment provides a method for identifying an object of interest in a medical image. The method includes initializing internal states of nodes of a spatial lattice. Each node in the spatial lattice corresponds to a pixel of the medical image and is connected to at least one node representing an adjacent pixel of the medical image. The method also includes iteratively updating the internal states of the nodes in the spatial lattice using a neural network using spatial gated propagation. In each iteration, each node updates its internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of an adjacent node from a previous iteration, and a new value of the node. The method also includes identifying an object of interest within the medical image based on the value of the node at a convergence of the spatial lattice.

[0011] Another embodiment also provides a method for identifying an object of interest in a medical image. However, the method provided by this embodiment includes creating an image pyramid of the medical image. The created image pyramid includes a plurality of layers, each layer includes a plurality of values, and each value represents a block of one or more pixels in the medical image. Each successive layer in the image pyramid includes fewer values ​​than the previous layer. The method also includes initializing the internal state of a node of a spatial lattice for each layer of the image pyramid. Each node in the spatial lattice represents a block of one or more pixels in the medical image and is connected to at least one node representing an adjacent block of one or more pixels in the medical image. The method also includes, for each layer of the image pyramid, iteratively updating the internal state of the node in the spatial lattice using a neural network using spatial gated propagation. In each iteration, each node updates its internal state based on at least one selected from the group consisting of a value of a node from a previous iteration, a value of an adjacent node from a previous iteration, and a new value of the node. The method also includes identifying an object of interest within the medical image based on the value of a node at a convergence of the spatial lattice with nodes representing values ​​included in the first layer of the image pyramid.

[0012] One embodiment provides a system for determining a region of interest in an image. The system includes a memory and an electronic processor. An electronic processor included in the system is connected to the memory and configured to initialize internal states of nodes of a spatial lattice. Each node of the spatial lattice corresponds to a pixel of the image and is connected to at least one node representing an adjacent pixel of the image. The electronic processor is also configured to iteratively update the internal state of each node in the spatial lattice using spatial gated propagation using a neural network, and identify a region of interest within the image based on the internal state of the nodes at the convergence of the spatial lattice.

[0013] Another embodiment also provides a system for determining a region of interest in an image. Similar to the system of the above embodiment, the system described in this embodiment also includes a memory and an electronic processor connected to the memory. However, the electronic processor of the system provided by this embodiment is configured to create an image pyramid of the image. The image pyramid includes multiple layers. For each layer of the image pyramid, the electronic processor is configured to initialize the internal state of the nodes of the spatial lattice, and iteratively update the internal state of the nodes in the spatial lattice using a neural network using spatial gated propagation. Each node in the spatial lattice represents a block of one or more pixels in the image and is connected to at least one node representing an adjacent block of one or more pixels in the image. The electronic processor is also configured to identify the region of interest within the image based on the internal state of the nodes at the convergence of the spatial lattice, the spatial lattice having nodes representing values ​​included in the first layer of the image pyramid.

[0014] One embodiment provides a non-transitory computer-readable medium comprising instructions executable by an electronic processor to perform a group of functions. The group of functions includes initializing the internal states of nodes of a spatial lattice. Each node represents a pixel of an image and is connected to at least one adjacent pixel of the image. The group of functions also includes iteratively updating the internal states of nodes in the spatial lattice using spatial gated propagation using a neural network. In each iteration, each node updates its internal state based on at least one selected from the group consisting of the value of the node from the previous iteration, the value of the adjacent node from the previous iteration, or the new value of the node. The group of functions also includes identifying objects of interest within the image based on the values ​​of the nodes at the convergence of the spatial lattice.

[0015] Another embodiment also provides a non-transitory computer-readable medium including instructions executable by an electronic processor to perform a group of functions. However, unlike the group of functions in the above-described embodiments, the group of functions performed by the electronic processor of this embodiment includes creating an image pyramid of an image. The created image pyramid includes a plurality of layers, each layer includes a plurality of values, and each value represents a block of one or more pixels in the image. Each successive layer in the image pyramid includes fewer values ​​than the previous layer. The group of functions also includes initializing the internal state of a node of a spatial lattice for each layer of the image pyramid. Each node of the image pyramid represents a block of one or more pixels in the image and is connected to at least one node representing an adjacent block of one or more pixels in the image. The group of functions also includes iteratively updating the internal state of a node in the spatial lattice using a neural network using spatial gated propagation for each layer of the image pyramid. In each iteration, each node updates its internal state based on at least one selected from the group consisting of a value of a node from a previous iteration, a value of an adjacent node from a previous iteration, or a new value of a node. The group of functions also includes identifying an object of interest within the image based on the value of a node at a convergence of a spatial lattice having nodes representing values ​​included in a first layer of the image pyramid. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A medical image to which region growing has been applied to identify an object of interest is shown.

[0017] Figure 2 A system for determining a region of interest in an image is shown.

[0018] Figure 3 Shown included in Figure 2 Neural networks in the system.

[0019] Figure 4 An example of input to a node in a gated spatiotemporal cell is shown.

[0020] Figure 5 Shows Figure 4 An example of a neural network receiving as input a medical image.

[0021] Figure 6 Shows Figure 4 The neural network in Figure 5 Example of regions of interest detected in a medical image. DETAILED DESCRIPTION

[0022] One or more embodiments are described and illustrated in the following description and accompanying drawings. These embodiments are not limited to the specific details provided herein and may be modified in various ways. In addition, there may be other embodiments not described herein. In addition, the functions described herein as being performed by one component may be performed by multiple components in a distributed manner. Similarly, the functions performed by multiple components may be merged and performed by a single component. Similarly, the components described as performing specific functions may also perform additional functions not described herein. For example, a device or structure "configured" in a certain way is configured at least in that way, but may also be configured in an unlisted manner. In addition, some embodiments described herein may include one or more electronic processors configured to perform the described functions by executing instructions stored in a non-transitory computer-readable medium. Similarly, the embodiments described herein may be implemented as a non-transitory computer-readable medium storing instructions, which may be executed by one or more electronic processors to perform the described functions. As used in this application, "non-transitory computer-readable medium" includes all computer-readable media, but does not include temporary propagation signals. Thus, a non-transitory computer readable medium may include, for example, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (read only memory), a RAM (random access memory), a register memory, a processor cache, or any combination thereof.

[0023] In addition, the words and terms used herein are for descriptive purposes and should not be considered as limiting. For example, "including", "containing", "comprising", "having" and variations thereof used herein are meant to cover the items listed thereafter and their equivalents and additional items. The terms "connect" and "couple" are widely used and include direct and indirect connections and couplings. In addition, "connect" and "couple" are not limited to physical or mechanical connections or couplings, and may include electrical connections or couplings, whether direct or indirect. In addition, electronic communications and notifications may be performed using wired connections, wireless connections, or a combination thereof, and may be transmitted directly or through one or more intermediate devices over various types of networks, communication channels, and connections. In addition, relational terms such as first and second, top and bottom, etc. are used herein only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions.

[0024] As described above, biomedical image segmentation attempts to identify pixels within an image that represent an object of interest, which allows various calculations and data processing (e.g., volume calculations, etc.) to be performed on the object. However, many techniques for performing image segmentation rely on recognizing consistent shapes and context. For example, as described above, CNNs are good at recognizing shapes and objects in images, and CNNs have been trained to recognize these shapes and objects, but CNNs struggle to recognize irregular shapes in images. Therefore, techniques that rely on recognizing consistent shapes and backgrounds may be ineffective in recognizing irregular objects, such as tumor masses, lesions, etc.

[0025] Other techniques rely on pixel expansion to determine the boundaries of objects of interest in an image. As mentioned above, region growing does not rely on regularity, but instead expands the seed pixel to neighboring pixels until a boundary is identified. Therefore, the shape of the object of interest does not affect the performance of region growing. However, as Figure 1 As shown, when an object does not have a clear boundary (eg, when the object is connected to adjacent bright tissue by even small connections), region growing may inappropriately grow the object beyond its true boundary.

[0026] To address the deficiencies of the above-mentioned techniques, the embodiments described herein combine the advantages of CNNs and RNNs in a spatiotemporal unit to improve the recognition of irregular objects in images. In particular, as described in more detail below, the embodiments described herein employ spatial gated propagation. Gating involves a network generating a new state for the system (based on its previous state and newly received information), and a separate network gating the new state and deciding whether to use the new state and propagate forward in time. As described herein, the most previous internal state of a pixel and its nearest neighboring pixels is gated and used to determine the internal state of the pixel at the current time step. Therefore, the systems and methods described herein propagate values ​​in both space and time. In addition, the creation of the above-mentioned image pyramid allows values ​​to be propagated at different image resolutions.

[0027] Figure 2 A system 200 for implementing a neural network is shown. A neural network is a machine learning model that employs one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to an input layer and an output layer. The output of each hidden layer is used as input to the next layer in the network (either the next hidden layer or the output layer). Each layer of the network generates an output from the received input according to the current values ​​of the corresponding parameter set.

[0028] like Figure 2 As shown, system 200 includes a computing device 202 that includes an electronic processor 204 and a memory 206. Electronic processor 204 and memory 206 communicate wirelessly, via a wired communication channel or bus, or a combination thereof. Computing device 202 may include various configurations except Figure 2 . For example, in some embodiments, computing device 202 includes multiple electronic processors, multiple memory modules, or a combination thereof. Furthermore, in some embodiments, computing device 202 includes one or more input-output interfaces that allow computing device 202 to communicate with a network, peripheral devices, etc.

[0029] It should be understood that the functions described herein as being performed by computing device 202 may be performed in a distributed manner by multiple computing devices located in various geographic locations. For example, the functions described herein as being performed by computing device 202 may be performed by multiple computing devices 202 included in a cloud computing environment. Electronic processor 204 may be a microprocessor, an application specific integrated circuit (ASIC), etc. Electronic processor 204 is generally configured to execute software instructions to perform a group of functions, including the functions described herein. Memory 206 includes non-transitory computer-readable media and stores data, including instructions executable by electronic processor 204. For example, Figure 2 As shown, memory 206 stores a neural network 208 , which comprises a computer program executed by electronic processor 204 .

[0030] Figure 3 A visual representation of an example of a neural network 208 executed by an electronic processor 204 to perform the methods described herein is shown. Figure 3 As shown, when executed by electronic processor 204, neural network 208 provides a machine learning system that receives input and generates output 305. As an example, the input includes an image (input image 300), such as a biomedical image, or another type of multi-dimensional data, and output 305 similarly includes an image or another type of multi-dimensional data.

[0031] like Figure 3 As shown, the input image 300 is input to the first layer 310 of the neural network 208. It should be understood that although the first layer 310 is illustrated as a single layer, this is purely for illustrative purposes and the first layer 310 may include any number of layers. In the first layer 310, the neural network 208 may perform multiple convolutions on the values ​​representing the brightness of each pixel. In other embodiments, as described below, the neural network 208 may perform multiple convolutions in the first layer 310 to create an image pyramid 315 based on the input image 300 (I0).

[0032] The image pyramid 315 is a sequence of tensors (I1-I l ). The tensor produced for l=1 has the same spatial dimensions as the input image 300 (I0), but the tensor is halved in size for each subsequent convolution / reduction. Therefore, the tensor for each value of l has a different resolution, with tensor I1 having the highest resolution and tensor I l The following equation illustrates the process of generating the image pyramid 315 performed in the first layer 310 .

[0033]

[0034] The operator * represents the convolution operation. For example, the formula A*B represents the convolution between input B and kernel A.

[0035] I0 is a variable representing the original input image 300. I0 has dimensions N0 x N0 x 1. In other words, the input image 300 has N0 rows, N0 columns, and (because in this example embodiment, the input image 300 is a grayscale image) one channel.

[0036] I l is a variable representing the intermediate form of image data (tensor) generated after performing one or more reductions on the input image 300 (I0). As described above, when l>1, I l has a lower resolution than the input image 300 (I0). l With dimension Nl ×N l ×C, where N l =2 -(l-1) N0, and C is the number of channels.

[0037] is a variable representing the convolution operator (kernel) that preserves the dimensionality of the input image data. The input image data has size N l ×N l ×C I , and the output image data has size N l ×N l × C0. K can represent a combination of several sequential convolution operations arranged into, for example, AlexNet, DensNet, or a range of other architectures, along with learnable parameters of the convolution operators.

[0038] is a variable representing a convolution operator that halves the dimension of the input image data. The input image data has a dimension N l-1 ×N l-1 ×C I , and the output image data has size N l ×N l ×C0. Like K, D can represent several sequential convolution operations as arranged in AlexNet, DensNet, or a range of other architectures, as well as the learnable parameters of the convolution operator. However, the convolution operator D also represents a maximum pooling or strided convolution layer that halves the dimensionality of the input image data.

[0039] The tensors computed for each level of the image pyramid 315 are fed into the second layer 320. The equation describing the operations performed in the second layer 320 is

[0040]

[0041] In addition, the operator * represents the convolution operation, I l A variable representing an intermediate form of image data (tensor) obtained by reducing the input image 300 (I0) one or more times.

[0042] [A,B] is a concatenation operation between tensors, such as tensors A and B. The concatenation operation performed on two tensors combines the channels included in each tensor. For example, if tensor A has dimensions M×M×C1 and tensor B has dimensions M×M×C2, then the output of [A,B] has dimensions M×M×(C1+C2).

[0043] is a tensor 322 that holds the internal state of each node in the spatial lattice at resolution l and time step t. As described above, the internal state of each node in the spatial lattice is updated at each time step. Tensor 322 has size N l ×N l ×C H Thus, there is a C for each block of one or more pixels describing the image at resolution l. H variable.

[0044] is a variable representing the convolution operator that preserves the dimensionality of the input image data. The input image data has size N l ×N l ×C I , and the output image data has size N l ×N l × C0. K can represent a combination of several sequential convolution operations arranged into, for example, AlexNet, DensNet, or a range of other architectures, along with learnable parameters of the convolution operators.

[0045] is a variable representing the result 323 of executing equation (4). With dimension N l ×N l ×C X , and is input to the third layer 325 of the neural network 208.

[0046] In summary, equation (4) transforms the tensor (I l ) and tensor 322 Cascade (perform the first cascade), convolution operator Applied to this cascade (performs the first convolution on the current level of the image pyramid) and saves the result 323 in a tensor middle.

[0047] The equation describing the operations performed in the third layer 325 is

[0048]

[0049] Likewise, as mentioned above, the operator * represents a convolution operation, and [A,B] is a concatenation operation between tensors, such as tensor A and tensor B. Similarly, is a variable representing the convolution operator (kernel) that preserves the dimensions of the input image data, is a variable representing a convolution operator (kernel) that reduces the dimensionality of the input image data by half, and It means that from the tensor I l , internal status and kernel Calculate the variable as a result of equation (4).

[0050] It means that according to the tensor I l+1 , internal status and nuclear The variables of the result of calculating equation (5) and It means that according to the tensor I l-1 , internal status and nuclear Calculate the variable of the result of equation (5).

[0051] is a variable representing a convolution operator (kernel) that upsamples the dimensions of the input image data by doubling the dimensions. For example, if the input image data has size N l+1 ×N l+1 ×C I , then the output image data has size N l ×N l ×C0. Similar to the convolution operator K, the convolution operator U can represent a combination of several sequential convolution operations as arranged in AlexNet, DensNet, or a range of other architectures, as well as learnable parameters for the convolution operations. However, the convolution operator U can also represent a transposed convolution layer to double the image dimension.

[0052] is a tensor containing the result of executing equation (5). contains the information to be passed to the gated spatiotemporal unit. In summary, equation (5) includes the use of the higher resolution (I l-1 ) (the layer of the image pyramid directly below the current layer of the image pyramid) to reduce the result of calculating equation (4) And according to the lower resolution (I l+1 ) (the layer of the image pyramid directly above the current layer of the image pyramid) to upsample the result of equation (4) Equation (5) also includes concatenating the up-sampled result with the down-sampled result and combining the result with the result according to the representation resolution I l The tensor of the input image 300 is used to calculate the result of equation (4) cascade (perform the second cascade). This equation combines the result of the cascade with the kernel Convolution (perform the second convolution) and save the result in the variable middle.

[0053] In the first iteration of the neural network 208, only the output from the third layer 325 is used to initialize the internal state of each node included in the spatial lattice of the gated spatiotemporal unit 330. Each node includes a vector of values ​​representing the internal state of the node and a value derived from the brightness of one or more pixel blocks centered at the node in the image pyramid. In each successive iteration, the internal state of each node from the previous iteration is initialized via the tensor 322 is input into the second layer 320 of the neural network 208. Then, the above process is repeated starting from the second layer 320.

[0054] As described above, the neural network 208 includes a gated spatiotemporal unit 330 having a plurality of nodes arranged in a spatial lattice. Each node in the lattice corresponds to a pixel in the input image 300. The gated spatiotemporal unit 330 performs data processing at each of a plurality of time steps. At each time step, the gated spatiotemporal unit 330 receives a plurality of values. Based on the received values ​​and the value representing the internal state of each node gated in the previous time step, the gated spatiotemporal unit 330 determines how to update the internal state of each node at the current time step. As will be described in more detail below, the gated spatiotemporal unit 330 determines how to update the internal state of each node by deciding for each node in the lattice whether to keep the internal state of the node at the previous time step, set the internal state of the node to the value representing the internal state of the neighboring node from the previous time step, or generate a new internal state for the node.

[0055] The following equation is used to determine the internal state of the nodes included in the lattice of the gated spatiotemporal unit 330 at the current time step: 327, and for ease of explanation the equation is spread over seven lines (labeled I-VII).

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] σ(A) represents the S-shaped function 1 / (1+e -a) is applied element-wise to each element a of the tensor A. The sigmoid function can be called a "squeeze" function. The sigmoid function takes an input value anywhere from +∞ to -∞ and squares the input value into an output value from 0 to 1.

[0064] Tanh is also a squeezing function. It also takes input values ​​anywhere from +∞ to -∞, but the Tanh function squares the input values ​​to output values ​​from -1 to 1.

[0065] The operator ⊙ represents a Hadamard product operation. For example, if given the equation A⊙B, a Hadamard product operation is performed between input B and input A. The Hadamard product is an element-by-element multiplication of every pair of elements from two inputs of the same size.

[0066] A tensor containing the results 326 of the computations performed in the third layer 325 at resolution l and time step t. With dimension N l ×N l ×7×C H Each of the seven elements of the third dimension has a specific role in the space-time gating process. The variables in Eq. arrive is the tensor produced when one of the seven elements is selected. Each tensor associated with each of the seven elements has dimension N l ×N l ×C H .

[0067] is a tensor 322 that holds the internal state of each node in the spatial lattice at resolution l and time step t. As described above, the internal state is dynamically updated at each time step. The tensor has dimension N l ×N l ×C H Therefore, there exists a C that describes each node at resolution l H variable.

[0068] S Δx，Δy is the spatial shift convolution operator. It has no learnable parameters. Δx,Δy Allows information from the internal states of nearest neighboring nodes to be taken into account when determining the current internal state of a node.

[0069] Going back to the above equation, stored in the tensor The result 326 in is divided into seven parts. As mentioned above, each part represents a tensor The elements in the third dimension of . Line (I) of equation (6) applies the tanh squeeze function to the sum of lines II-VII of equation (6) to determine the internal state of the node at the current iteration. Line (II) of equation (6) corresponds to the probability of copying the internal state of the node from the previous timestamp to the current timestamp according to the gate value. The next four lines (III-VI) Each corresponds to the possibility of copying the internal state of one of the nearest neighbors from the previous iteration to the internal state of the node in the current iteration. The last row (VII) corresponds to generating a completely new value and possibly setting the internal state of the node at the current iteration to the new value.

[0070] Figure 4 The connections between nodes 400 whose internal states are determined in the current iteration and nodes determined to have internal states in previous iterations are shown. Each node is connected to the node whose internal state was calculated at the most previous iteration of the gated spatiotemporal unit 330. Specifically, each node is connected to the node representing its own internal state in the most previous iteration of the gated spatiotemporal unit 330 and the nodes representing the internal states of its neighboring nodes in the most previous iteration of the gated spatiotemporal unit 330. Figure 4 4, the nodes in group 405 are nodes each associated with an internal state determined in the most previous iteration of neural network 208. The nodes in group 410 are nodes associated with an internal state determined in the current iteration of neural network 208. As described above, each node corresponds to a pixel (or a block of one or more pixels) in input image 300. Each neighboring node of a node represents a pixel (or a block of one or more pixels) that is adjacent to the pixel (or block of one or more pixels) represented by the node. For example, if node 400 represents a pixel at coordinate (i, j) in an image, then node 400 representing the pixel at (i, j) is connected to node 415 representing the pixel at coordinate (i-1, j) (directly to the left of the pixel at (i, j)), node 420 representing the pixel at coordinate (i+1, j) (directly to the right of the pixel at (i, j)), node 425 representing the pixel at coordinate (i, j+1) (directly above the pixel at (i, j)), and node 430 representing the pixel at coordinate (i, j-1) (directly below the pixel at (i, j)). Each of the above nodes described as being connected to node 400 representing pixel (i, j) is a neighboring node of node 400. Thus, gated spatiotemporal unit 330 determines whether to set the internal state of node 400 to the internal state of one of the nodes in group 405.

[0071] When the internal states of the nodes in the spatial lattice of the gated spatiotemporal cells 330 converge (change by less than a predetermined amount), the internal states of the nodes representing the input image 300 at the highest resolution are output to the final layer 335 of the neural network 208. The final layer 335 uses a value included in the internal state of each node to calculate the probability (e.g., a value between zero and one) that the pixel represented by the node belongs to the object of interest in the input image 300. The following equation describes the operations performed in the final layer 335 to determine the probability that each pixel is part of the object of interest:

[0072]

[0073] Y t is a variable representing the output 305 of the neural network 208 at time step t, and has dimensions N0×N0×1 (the same dimensions as the input).

[0074] is a tensor that holds the internal state of each node in the spatial lattice at resolution 1 and time step t.

[0075] is a variable representing the convolution operator that preserves the dimensions of the input image data. The input image data has dimensions N0×N0×C I , while the output image data has size N0×N0×1. K may represent a combination of several sequential convolution operations arranged as in, for example, AlexNet, DenseNet, or a range of other architectures and learnable parameters of convolution operators.

[0076] In summary, equation (7) uses Apply the final convolution to the highest resolution internal state Thus, the number of input channels (C H ) is reduced to 1 output channel. This equation applies the sigmoid function to the Apply the final convolution to the highest resolution internal state As a result, Y t Each value included in is squeezed to a value between 0 and 1. t Each value in corresponds to the probability that the image pixel is located inside the object of interest in the input image 300. For example, if the value produced by the sigmoid function for a single pixel is .5, then there is a fifty percent (50%) probability that the pixel is inside the object of interest.

[0077] The neural network 208 may store the generated output 305 (the calculated probability for each node) in an output data repository (e.g., memory 206), or provide the generated output 305 for use or consumption, such as by displaying the output 305 to a user on a display device. In any case, the electronic processor 204 compares the probability of each pixel being included in the object of interest to a predetermined threshold. If the probability that the pixel is part of the object of interest is above the predetermined threshold, the electronic processor 204 determines that the pixel is part of the object of interest.

[0078] It should be understood that in some embodiments, in the above neural network 208, for the input image 300 (I1-I2) included in the image pyramid 315, l ) to perform equations (4-6) for each layer of the representation of ). It should also be understood that although the neural network 208 is described above as propagating values ​​over time, space, and resolution, the neural network 208 can be modified to only propagate values ​​over time and space.

[0079] It should also be understood that the value of the gate used to determine the internal state of each node at each iteration need not be zero or one, but can be any value between 0 and 1 (see equation (6) above). Thus, in some embodiments, the updated internal state of a node may be a mixture (or, more mathematically, a linear combination) of two or more of the options described above (the node's value from the previous iteration, the values ​​of one or more neighboring nodes from the previous iteration, and the node's new value).

[0080] Figure 5 and Figure 6 Examples of practical applications of neural network 208 are provided. Figure 5 An example of a medical image 500 that the neural network 208 may receive as input is shown. The object of interest in the image 500 is a tumor 505 in the left lung 510. Figure 6 The region of the medical image 500 that the neural network 208 identifies as an object of interest (tumor 505) is shown. Unlike when using region growing techniques (see Figure 1 ), the boundary of the object of interest does not extend beyond the left lung 510.

[0081] Thus, the embodiments described herein provide a neutral network including spatiotemporal units. A spatiotemporal unit is a node lattice that is spatially extended. Each node corresponds to a pixel in an image, for example. The neural network determines the initial internal state of each node and iteratively updates the internal state of each node to repeatedly generate new internal states again by propagating values ​​in time, in space, or both and by calculating new values ​​to represent the internal state of each node. Thus, compared to other types of RNNs (such as long short-term memory (LSTM) networks and gated recurrent unit (GRU) networks that iterate on a one-dimensional sequence of letters or words), the embodiments described herein consider the decisions of neighboring nodes when updating the internal state of each node. Specifically, the embodiments described herein apply both spatial and temporal dimensions. Thus, although the temporal dimension is only iterated forward, spatial gating allows spatial information to resonate back and forth on the spatial lattice for as long as needed, because new conclusions are reached in a portion of the image and the new conclusions are propagated to other portions of the image to inform decisions made at those portions. In addition, in some embodiments described herein, values ​​in the neural network 208 can be propagated between different resolutions of the image.

[0082] The embodiments described herein are closed. In particular, the neural network 208 described herein is given all the information about the outside world as an initial input (the image that needs to be processed), and from that point forward, the neural network 208 evolves in time only according to its own internal state and rules, without taking any further information from the outside. As such, iterations continue until convergence when there are no further changes to the internal state. This makes the neural network 208 like an algorithm rather than a function. In contrast, the RNN is given a new piece of the problem (e.g., a word) at each time step, so iterations continue only when there is new information available.

[0083] Various features and advantages of some embodiments are set out in the following claims.

Claims

1. A method for identifying an object of interest in a medical image, the method comprising: Initializing internal states of nodes of a spatial lattice, wherein each node corresponds to a pixel of the medical image and is connected to at least one node representing an adjacent pixel of the medical image; Iteratively updating the internal states of the nodes in the spatial lattice using a neural network and employing spatial gated propagation, wherein in each iteration, each node updates its internal state based on at least one item selected from the group consisting of: a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node; as well as The object of interest within the medical image is identified based on the values ​​of the nodes at the convergence of the spatial lattice.

2. The method of claim 1 , wherein iteratively updating the internal state of the node using a neural network comprises updating a value in a vector of values ​​associated with the internal state of the node. 3 . The method of claim 2 , wherein the values ​​in the vector of values ​​include a value representing the brightness of a pixel corresponding to the node and a value representing the internal state of the node. The method of claim 1 , wherein a convolution involving a previous internal state of the node is performed for each iteration.

5. The method according to claim 1, wherein the method further comprises: In the first iteration, the convolution is performed on each value representing the brightness of each pixel.

6. The method of claim 1 , wherein identifying an object of interest within the medical image based on the values ​​of the nodes at the convergence of the spatial lattice comprises: calculating, using a final layer of the neural network, a probability for each pixel to be included in the object of interest based on values ​​included in the vector of values ​​associated with each pixel; as well as For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

7. The method of claim 1, wherein each node updates its internal state using a squeeze function based on at least one item selected from the group consisting of: a value of the node from a previous iteration, values ​​of neighboring nodes from the previous iteration, and a new value of the node.

8. The method of claim 1, wherein the neighboring node is one selected from the group consisting of nodes representing pixels directly above, directly below, to the right, and to the left of a pixel represented by the node.

9. The method according to claim 1, further comprising: An image pyramid having a plurality of levels is generated, wherein each successive level represents the medical image with fewer values.

10. The method of claim 9, further comprising concatenating values ​​from multiple levels of the image pyramid in each iteration.

11. A system for determining a region of interest in an image, the system comprising: Memory; as well as An electronic processor coupled to the memory and configured to: initializing the internal states of nodes of the spatial lattice, wherein each node corresponds to a pixel of the image and is connected to at least one node representing a neighboring pixel of the image, Using a neural network and adopting spatial gated propagation to iteratively update the internal state of each node in the spatial lattice; as well as identifying the region of interest within the image based on the internal states of the nodes at the convergence of the spatial lattice; wherein the electronic processor is configured to update the internal state of the node by updating the internal state at each iteration based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node.

12. The system of claim 11, wherein the electronic processor is configured to iteratively update the internal state of the node using a neural network by updating a value in a vector of values ​​associated with the internal state of the node.

13. The system of claim 12, wherein the values ​​in the vector of values ​​include a value representing a brightness of a pixel corresponding to the node and a value representing the internal state of the node.

14. The system of claim 11, wherein the electronic processor is further configured to perform a convolution involving a previous internal state of the node in each iteration.

15. The system of claim 11, wherein the electronic processor is further configured to perform a convolution on each value representing the brightness of each pixel in a first iteration.

16. The system of claim 11, wherein the electronic processor is configured to: identify an object of interest within the image based on the values ​​of the nodes at the convergence of the spatial lattice by calculating the probability of each pixel being included in the object of interest based on the vector associated with each pixel using a last layer of the neural network, and For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

17. The system of claim 11, wherein the electronic processor is configured to update its internal state based on at least one selected from the group consisting of: a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node by using a squeeze function.

18. The system of claim 11, wherein the neighboring node is one selected from the group consisting of nodes representing pixels directly above, directly below, to the right, and to the left of a pixel represented by the node.

19. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a group of functions comprising: Initializing internal states of nodes of a spatial lattice, wherein each node represents a pixel of an image and is connected to at least one neighboring pixel of the image; Iteratively updating the internal states of the nodes in the spatial lattice using a neural network and employing spatial gated propagation, wherein in each iteration, each node updates its internal state based on at least one selected from the group consisting of: a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node; as well as Objects of interest within the image are identified based on the values ​​of the nodes at the convergence of the spatial lattice.

20. The non-transitory computer-readable medium of claim 19, wherein iteratively updating the internal state of the node using a neural network comprises updating a value in a vector of values ​​associated with the internal state of the node.

21. The non-transitory computer readable medium of claim 19, wherein identifying an object of interest within the image based on the values ​​of the nodes at the convergence of the spatial lattice comprises: calculating, using a final layer in the neural network, a probability for each pixel to be included in the object of interest based on the vector associated with each pixel; as well as For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

22. A method for identifying an object of interest in a medical image, the method comprising: creating an image pyramid for the medical image, wherein the image pyramid comprises a plurality of layers, each layer comprises a plurality of values, each value represents a block of one or more pixels in the medical image, and each successive layer comprises fewer values ​​than a previous layer; For each level of the image pyramid; Initializing internal states of nodes of a spatial lattice, wherein each node in the spatial lattice represents a block of one or more pixels in the medical image and is connected to at least one node representing an adjacent block of one or more pixels in the medical image; as well as Iteratively updating the internal states of the nodes in the spatial lattice using a neural network and employing spatial gated propagation, wherein in each iteration, each node updates its internal state based on at least one item selected from the group consisting of: a value of the node from a previous iteration, a value of a neighboring node from a previous iteration, and a new value of the node; as well as The object of interest within the medical image is identified based on the values ​​of the nodes at the convergence of the spatial lattice, the spatial lattice having nodes representing the values ​​included in a first level of the image pyramid.

23. The method of claim 22, wherein iteratively updating the internal state of the node using a neural network comprises updating a value in a vector of values ​​associated with the internal state of the node.

24. The method according to claim 22, further comprising: performing, at each iteration of each layer of the image pyramid, a first convolution involving a first concatenation of previous internal states of the nodes representing the values ​​comprised in the layer of the image pyramid and the values ​​comprised in the layer of the image pyramid, and The result of performing the first convolution is stored.

25. The method according to claim 24, further comprising: A second convolution is performed in each iteration of each layer of the image pyramid, the second convolution involving performing a second cascade of results of the first convolution on a current layer of the image pyramid, a layer of the image pyramid directly above the current layer of the image pyramid, and a layer of the image pyramid directly below the current layer of the image pyramid.

26. The method of claim 22, wherein creating the image pyramid comprises performing a convolution on each value representing the brightness of each block of one or more pixels in the medical image, wherein each convolution involving reducing the dimensionality of the input medical image data produces a value for representing the medical image in a next level of the image pyramid.

27. The method of claim 22, wherein each value representing the medical image in the first level of the image pyramid corresponds to a pixel in the medical image.

28. The method of claim 27, wherein identifying the object of interest within the medical image based on the values ​​of the nodes at the convergence of the spatial lattice of nodes having values ​​included in the first level of the image pyramid comprises: using a last layer of the neural network to calculate a probability that each pixel in the medical image is included in the object of interest based on values ​​included in each vector of values ​​associated with nodes representing the values ​​included in the first layer of the image pyramid; as well as For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

29. The method of claim 25, wherein each node updates its internal state based on at least one item selected from the group consisting of: a value of the node from a previous iteration, values ​​of neighboring nodes from the previous iteration, and a new value of the node comprising a result of using a squeeze function and performing the second convolution.

30. The method of claim 22, wherein the neighboring node is one selected from the group consisting of nodes representing blocks of one or more pixels directly above, directly below, to the right, and to the left of the block of one or more pixels represented by the node.

31. The method of claim 22, wherein representing the medical image with fewer values ​​creates a medical image with a lower resolution.

32. A system for determining a region of interest in an image, the system comprising: Memory; as well as an electronic processor connected to the memory and configured to: creating an image pyramid for the image, the image pyramid comprising a plurality of layers, For each level of the image pyramid, Initializing internal states of nodes of a spatial lattice, wherein each node represents a block of one or more pixels in the image and is connected to at least one node representing an adjacent block of one or more pixels in the image, and Using a neural network and adopting spatial gated propagation to iteratively update the internal states of the nodes in the spatial lattice; as well as The region of interest within the image is identified based on internal states of nodes at convergence of the spatial lattice, the spatial lattice having nodes representing values ​​included in a first level of the image pyramid.

33. The system of claim 32, wherein each successive layer of the plurality of layers included in the image pyramid represents the image at a lower resolution than an image represented in a preceding layer of the image pyramid.

34. The system of claim 33, wherein the electronic processor is configured to represent the image at a lower resolution by representing the image with fewer values.

35. A system according to claim 32, wherein the electronic processor is configured to update the internal state of the nodes by deciding for each node at each iteration whether to keep the value of the node from the previous iteration, set the value of the node to the value of a neighboring node from the previous iteration, or set a new value for the node.

36. The system of claim 32, wherein the electronic processor is configured to iteratively update the internal state of the node using a neural network by updating a value in a vector of values ​​associated with the internal state of the node.

37. The system of claim 34, wherein the electronic processor is configured to perform a first convolution at each iteration for each layer of the image pyramid and store a result of performing the first convolution, the first convolution involving a first cascade of previous internal states of nodes representing values ​​included in the layer of the image pyramid and values ​​included in the layer of the image pyramid.

38. The system of claim 37, wherein the electronic processor is configured to perform a second convolution at each iteration for each layer of the image pyramid, the second convolution involving performing a second cascade of results of the first convolution for a current layer of the image pyramid, a layer of the image pyramid directly above the current layer of the image pyramid, and a layer of the image pyramid directly below the current layer of the image pyramid.

39. The system of claim 33, wherein the electronic processor is further configured to, in a first iteration, perform a convolution on each value representing the brightness of each block of one or more pixels in the image, wherein each convolution involving a reduction in the dimensionality of the input image data produces a value for representing the image in a next level of the image pyramid.

40. The system of claim 32, wherein the electronic processor is configured to identify an object of interest within the image based on values ​​of nodes at which the spatial lattice of nodes having values ​​included in the first layer of the image pyramid converge by calculating, using a last layer of the neural network, a probability that each pixel in the image is included in the object of interest based on each vector associated with a node representing a value included in the first layer of the image pyramid, and For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

41. The system of claim 38, wherein the electronic processor is configured to update the internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node by using a squeeze function and performing the result of the second convolution.

42. A system according to claim 35, wherein the adjacent node is a node selected from the group consisting of nodes representing blocks of one or more pixels in the image, wherein the blocks of one or more pixels are directly above, directly below, to the right, and to the left of the blocks of one or more pixels in the image represented by the node.

43. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a group of functions comprising: creating an image pyramid for the image, wherein the image pyramid comprises a plurality of levels, each level comprises a plurality of values, each value represents a block of one or more pixels in the image, and each successive level comprises fewer values ​​than a most previous level; For each level of the image pyramid; Initializing internal states of nodes of a spatial lattice, wherein each node represents a block of one or more pixels in the image and is connected to at least one node representing an adjacent block of one or more pixels in the image; as well as iteratively updating the internal states of the nodes in the spatial lattice using a neural network and employing spatial gated propagation, wherein at each iteration, each node updates its internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node; as well as Objects of interest within the image are identified based on the values ​​of the nodes at the convergence of the spatial lattice, the spatial lattice having nodes representing the values ​​included in the first level of the image pyramid.

44. The non-transitory computer-readable medium of claim 43, wherein iteratively updating the internal state of the node using a neural network comprises updating a value in a vector of values ​​associated with the internal state of the node.

45. The non-transitory computer-readable medium of claim 43, wherein identifying the object of interest within the image based on the values ​​of the nodes at the convergence of the spatial lattice having nodes representing values ​​included in the first level of the image pyramid comprises using a last layer in the neural network to calculate a probability of each pixel in the image being included in the object of interest based on vectors associated with the nodes representing values ​​included in the first level of the image pyramid; For each pixel, it is determined whether the calculated probability is above a predetermined threshold.

Images