System and method for marker detection

Abstract

The present application relates to systems and methods for landmark detection. Systems, methods, and apparatus associated with detection are described herein. Detection can be achieved by determining a graph representation of a plurality of hypothetical landmarks detected in one or more medical images. The graph representation can include nodes representing the hypothetical landmarks and edges representing relationships between paired hypothetical landmarks. The graph representation can be processed using a graph neural network, such as a message passing graph neural network, by which the landmark detection problem can be transformed and solved as a graph node labeling problem.

Description

Technical Field

[0001] The present application relates to the field of medical imaging, and in particular to the detection of landmarks in medical images. Background Art

[0002] Landmark detection plays an important role in the field of medical imaging. The ability to determine and / or monitor the positions of certain landmarks (such as key anatomical structures, implanted medical devices, surgical devices, etc.) based on scanned images of a patient's body may be crucial to the success of many medical applications or processes, including, for example, motion estimation, surgical navigation, device alignment, etc. Conventional landmark detection techniques analyze these scanned images independently of each other and lack the ability to determine and utilize the relationships that may exist between landmarks (in the same image frame or in different image frames). Therefore, the results produced by these conventional techniques are often inconsistent across different image frames and may include a large number of false positives. Model complexity and computational requirements may also be very high. Therefore, systems, methods, and devices for improving the quality and efficiency of landmark detection are desired. Summary of the Invention

[0003] Systems, methods, and apparatus associated with marker detection are described herein. A device configured to perform a marker detection task may include one or more processors configured to determine a plurality of hypothetical markers in one or more medical images of an anatomical structure (e.g., a pair of balloon markers associated with a stent placed inside the anatomical structure, a guidewire placed inside the anatomical structure, etc.), and determine a graph representation of the plurality of hypothetical markers. The graph representation may include a plurality of nodes and a plurality of edges, wherein each node may represent a hypothetical marker among the plurality of hypothetical markers, and each edge may represent a relationship between a pair of hypothetical markers among the plurality of hypothetical markers. The one or more processors of the device may also be configured to identify one or more hypothetical markers among the plurality of hypothetical markers as true markers using a graph neural network (GNN), and to identify one or more hypothetical markers among the plurality of hypothetical markers as false markers. The GNN may be a message passing graph neural network, and may include an encoding network, a core network, and a decoding network. The encoding network can be trained to extract corresponding features from the nodes of the graph representation and the edges of the graph representation, the core network can be trained to estimate the corresponding states of the nodes and edges of the graph representation based on the features extracted by the encoding network, and the decoding network can be trained to indicate true labeled points and false labeled points based on the corresponding states of the nodes and edges estimated by the core network.

[0004] In an example, the core network described herein may include: a first core sub-network trained to estate corresponding states of nodes represented by a graph; and a second core sub-network trained to estate corresponding states of edges represented by the graph. In an example, the encoding network described herein may include: a node encoder trained to extract corresponding features of nodes represented by the graph; and an edge encoder trained to extract corresponding features of edges represented by the graph. In an example, one or more processors of a device configured to determine a plurality of hypothesized landmarks in one or more medical images of an anatomical structure may include: one or more processors configured to extract features from the one or more medical images of the anatomical structure using a landmark detection neural network, and determine corresponding positions of the plurality of hypothesized landmarks in the one or more medical images based on the features extracted by the landmark detection neural network.

[0005] In an example, the plurality of hypothesized marker points described herein may include a first set of hypothesized marker points detected in a first medical image of an anatomical structure and a second set of hypothesized marker points detected in a second medical image of the anatomical structure. In these examples, the one or more processors of the device being configured to determine a graph representation of the plurality of hypothesized marker points may include: the one or more processors being configured to represent the first set of hypothesized marker points and the second set of hypothesized marker points using nodes of the graph representation, pairing the first set of hypothesized marker points with the second set of hypothesized marker points, and representing corresponding relationships between the paired hypothesized marker points using edges of the graph representation. In these examples, features extracted by the encoding network for each edge of the graph representation may indicate similarities or dissimilarities between the paired hypothesized marker points represented by the edges.

[0006] In an example, the plurality of hypothesized markers described herein may include hypothesized markers detected in a single medical image of an anatomical structure, wherein the one or more processors of the device are configured to determine a graph representation of the plurality of hypothesized markers, including: the one or more processors are configured to represent the hypothesized markers detected in the single medical image using nodes of the graph representation, pair a first subset of the hypothesized markers detected in the single medical image with a second subset of the hypothesized markers detected in the single medical image, and represent corresponding relationships between the paired hypothesized markers in the single medical image using edges of the graph representation. In these examples, features extracted by the encoding network for respective edges of the graph representation may indicate geometric constraints between the paired hypothesized markers represented by the edges.

[0007] In an example, the core network described herein can be trained to estimate the respective states of nodes and edges of a graph representation through an iterative process, which iterative process can include: determining the current state of a node of the graph representation based on a previous state of the node and an average state of one or more edges of the graph representation associated with the node, and determining the current state of an edge of the graph representation based on the previous state of the edge and the respective current state of one or more nodes of the graph representation associated with the edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Examples disclosed herein can be understood in more detail from the following description given by way of example in conjunction with the accompanying drawings.

[0009] Figure 1A 、 Figure 1B and Figure 1C is a diagram illustrating example landmarks that may be detected using techniques provided in accordance with one or more embodiments described herein.

[0010] Figure 2 is a simplified diagram illustrating an example landmark detection device according to one or more embodiments disclosed herein.

[0011] Figure 3 is a simplified diagram illustrating example operations that may be performed to determine a graph representation of a plurality of hypothetical markers in accordance with one or more embodiments described herein.

[0012] Figure 4A and Figure 4B is a simplified diagram illustrating example neural network structures and techniques for determining characteristics of graph edges in accordance with one or more embodiments described herein.

[0013] Figure 5A 、 Figure 5B and Figure 5C is a simplified diagram illustrating example techniques for encoding features of graph nodes and edges and for determining corresponding true / false states of hypothetical labeled points in accordance with one or more embodiments described herein.

[0014] Figure 6 is a simplified flowchart illustrating example operations that may be performed to train a neural network in accordance with one or more embodiments described herein.

[0015] Figure 7 is a simplified block diagram illustrating example components of a device that may be configured to perform landmark detection tasks according to one or more embodiments described herein. DETAILED DESCRIPTION

[0016] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0017] Figure 1A 、 Figure 1B and Figure 1C Example landmarks that can be detected using the techniques described herein are illustrated. Figure 1A The example left ventricular myocardium and left atrium intersection and apex are shown, Figure 1B shows the B-spline control points representing the guide wire, Figure 1C An example pair of balloon markers 102 associated with stent placement is shown. These marker points can maintain the same spatial and / or geometric features from one scan image to the next and / or maintain certain geometric relationships (e.g., constraints) within the same scan image. For example, the relative position of the apex point and two intersection points, the relative position of points along the guidewire, and / or the distance between the two balloon markers can remain the same across multiple scan images. It can be seen that these spatial and / or geometric features can be explored to improve the detection of marker points, such as by identifying marker points that maintain the features as true marker points and identifying marker points that do not maintain the features as false marker points.

[0018] Figure 2 2 is a simplified diagram illustrating an example landmark detection apparatus (LDA) 200 according to one or more embodiments disclosed herein. As shown, the LDA 200 can be configured to detect a plurality of landmarks in one or more medical images based on a graph representation 202 of the landmarks. The one or more medical images can be scanned images (e.g., MRI, X-ray, etc.) of an anatomical structure of the human body (e.g., a heart), and the landmarks can include various anatomical and / or artificial structures or components, such as Figure 1A The intersection of the left ventricular myocardium and left atrium and / or the apex of the heart are shown. Figure 1B The guide wire shown, Figure 1CA pair of balloon markers for the stent shown, etc. The marker points may belong to the same scan image (e.g., a single image frame) or different scan images (e.g., multiple scan images of a cardiac movie). The marker points may be initially identified by LDA200 on a hypothesis basis. For example, LDA200 may be configured to extract features from one or more medical images (e.g., using the marker detection neural network described herein) and identify multiple hypothetical marker points in one or more medical images (e.g., the corresponding positions of the hypothetical marker points) based on the extracted features. Due to the similarity between the marker points and their surrounding areas in the medical images, the hypothetical marker points identified by LDA200 may include true positives (e.g., true marker points) and false positives (e.g., background areas that are mistakenly identified as marker points). LDA200 may be configured to represent the hypothetical marker points and / or their mutual relationships with the graph representation 220, and process the graph representation through a graph neural network (GNN) 204 (e.g., a message passing graph neural network) to determine true marker points and / or false marker points among the multiple hypothetical marker points. After making the determination, the LDA 200 may indicate (eg, label) the true / false status of the landmark, for example, via the landmark label 206 (eg, with a true / false label).

[0019] The graph representation 202 may include a plurality of nodes and a plurality of edges. Each node may represent a hypothetical marker point among a plurality of hypothetical marker points determined by the LDA 200, and each edge may represent a relationship between a pair of hypothetical marker points among the plurality of hypothetical marker points determined by the LDA 200. An edge may represent, for example, a relationship between a hypothetical marker point in a first scan image (e.g., a block hypothesized to include a pair of corresponding balloon markers) and a hypothetical marker point in a second scan image (e.g., a block hypothesized to include the same pair of corresponding balloon markers). An edge may also represent, for example, a relationship between two hypothetical marker points in the same scan image (e.g., a relationship between a first block hypothesized to include a pair of corresponding balloon markers and a second block hypothesized to include a guidewire). As will be described in more detail below, by representing the geometric and / or spatial features of the marker points within the same scan image or across different scan images using the nodes and edges of the graph representation 202, the problem of marker detection may be converted into a graph node labeling problem and solved using the GNN 204.

[0020] like Figure 2As shown, the GNN 204 may include an encoding network 204a, a core network 204b, and / or a decoding network 204c. The encoding network 204a may be trained to receive the graph representation 202 and extract corresponding features from the nodes of the graph representation and the edges of the graph representation. The core network 204b may be trained to estimate the corresponding states (e.g., current features) of the nodes and edges of the graph representation 202 based on the features extracted by the encoding network 204a, and the decoding network 204c may be trained to indicate (e.g., using the marker labels 206) the true markers and / or false markers among the plurality of hypothetical markers initially determined by the LDA 200 based on the corresponding states of the nodes and edges of the graph representation 202 estimated by the core network 204b.

[0021] Figure 3 exemplifies a graph representation of a plurality of hypothetical marker points in one or more medical images that can be used to determine anatomical structures (e.g., Figure 2 The example will be described in the context of balloon marker detection, but those skilled in the art will appreciate that the same or similar techniques can also be used to detect other marker points. Figure 3 As shown, the plurality of hypothetical marker points may be associated with one or more medical scan images (e.g., 302a, 302b, etc.) and may include corresponding objects of interest detected in the one or more medical scan images. For example, the plurality of hypothetical marker points may include a first balloon marker pair 304a (e.g., which may be a true balloon marker pair), a second balloon marker pair 304b (e.g., which may be a false balloon marker pair that is erroneously detected due to its similarity to the balloon marker pair 304a), a third balloon marker pair 304c (e.g., which may be another true balloon marker pair), and a fourth balloon marker pair 304d (e.g., which may be another false balloon marker pair that is erroneously detected due to its similarity to the balloon marker pair 304c).

[0022] Multiple hypothetical landmarks can be detected using a landmark detection neural network (LDNN) (such as LDNN 306), which can be trained to extract features from the medical scan images (e.g., 302a, 302b, etc.) described herein and predict (e.g., hypothesized) corresponding locations of the landmarks in the medical scan images. LDNN 306 can include, for example, a fully convolutional neural network (CNN) having an encoder-decoder architecture (e.g., a U-Net architecture). The CNN can include multiple convolutional layers and / or one or more pooling layers. The convolutional layers can be followed by batch normalization and / or linear or nonlinear activations (e.g., rectified linear units or ReLU activations). Each convolutional layer can include multiple convolution kernels or filters with corresponding weights, the values ​​of which can be learned through a training process to extract features from the medical scan images 302a, 302b, etc. The features extracted by the convolutional layers can be downsampled by one or more pooling layers to obtain a representation of the features, for example, in the form of one or more feature maps. The CNN (e.g., LDNN 306) can also include one or more non-pooling layers and one or more transposed convolutional layers. Through the non-pooling layer, the CNN can upsample the features extracted from the input medical scan image, and further process the upsampled features through one or more transposed convolution layers (e.g., via multiple deconvolution operations) to derive one or more amplified or dense feature maps. The dense feature maps can then be used to predict (e.g., hypothesize) the positions of the marker points 304a-304d in, for example, corresponding heat maps or Gaussian response maps (e.g., 308a, 308b, etc.) corresponding to the input medical scan image (302a, 302b, etc.), where the center of the Gaussian (e.g., with a predefined σ) can indicate the corresponding predicted (e.g., hypothesized) position of the marker point.

[0023] LDNN 306 can be trained to perform the above functions using annotated marker data. The training process will be described below, for example, Figure 6 Once the hypothetical marker points (e.g., 304a-304d) are determined, a graph representation 310 (e.g., a data structure representing a graph) including nodes and edges can be created to capture information about the hypothetical marker points and their relationships to each other. Each node of the graph representation 310 can correspond to a data structure configured to store information about one hypothetical marker point, and each edge of the graph representation 310 can correspond to a data structure configured to store information about a relationship between two hypothetical marker points. For example, in Figure 3In the example shown, the nodes of the graph representation 310 can correspond to data structures that respectively store information about the hypothetical marker points 304a, 304b, 304c, and 304d, and the edges of the graph representation 310 can correspond to data structures that respectively store information about the corresponding relationships between the marker points 304a and 304c, between the marker points 304a and 304d, between the marker points 304b and 304c, and between the marker points 304b and 304d.

[0024] The nodes and edges of the graph representation 310 may have their own features. By extracting and tracking these features, the GNN described herein (e.g., Figure 2 The GNN 204 of FIG310 distinguishes true markers such as markers 304a and 304c from false markers such as markers 304b and 304d. The features of each node in the graph representation 310 can be multi-dimensional. For example, the features of a node corresponding to a hypothetical pair of balloon markers (e.g., markers 304a, 304b, 304c, or 304d) can be represented by heat map values ​​around the two marker locations (e.g., in their vicinity). These features can be extracted using a pre-trained neural network (such as a neural network with an encoder-decoder structure, e.g., similar to neural network 306). In an example, such an encoder-decoder neural network can be trained to encode an entire block of image data containing a pair of balloon markers represented by the node, and the features extracted by the encoder of the neural network can be used as the features of the node.

[0025] The characteristics of each edge in the graph representation 310 can be derived based on the geometric constraints associated with the two nodes connected by the edge. Using the edge connecting marker point 304a and marker point 304c as an example, because marker points 304a and 304c are both true marker points and, as described above, the geometric characteristics of these true marker points can be preserved between image frames, the distance between the two hypothetical balloon markers of marker point 304a can be similar to the distance between the two hypothetical balloon markers of marker point 304c. Conversely, because marker point 304d is a false marker point and therefore may have different geometric characteristics than marker point 304a, the distance between the two hypothetical balloon markers of marker point 304a can be different from the distance between the two hypothetical balloon markers of marker point 304d. Therefore, the similarity or dissimilarity (e.g., similarity score or dissimilarity score) between the node pairs connected by each edge (e.g., between the corresponding blocks of the medical scan image containing the two marker nodes) can be used as the characteristics of the edge.

[0026] Figure 4A and Figure 4B Example neural network structures and techniques for determining edge features are illustrated. Figure 4AAn example of a feature of a graph edge connecting two graph nodes using a Siamese network structure and contrastive loss to determine the similarity (e.g., similarity score) between the two graph nodes is illustrated. As shown, similarity can be determined using a Siamese network 402 by comparing the features of two image data blocks 404a and 404b containing corresponding hypothetical markers (e.g., corresponding pairs of balloon markers) represented by graph nodes. The Siamese network 402 may include twin sub-networks 402a and 402b that share the same (e.g., roughly similar) weights and are trained to process blocks 404a and 404b in tandem. It should be noted that although the Siamese network 402 can be shown as having two sub-networks in the figures provided herein, some embodiments can use the same network twice (e.g., before backpropagation) to perform the same function.

[0027] Subnetworks 402a and 402b may include respective CNNs (e.g., fully connected CNNs), each CNN having multiple convolutional layers, one or more pooling layers, and / or one or more fully connected layers. The convolutional layers of the CNN may include multiple convolution kernels or filters that are configured to extract features from blocks 404a and 404b through one or more convolution operations. The convolution operation may be followed by batch normalization and / or nonlinear activation, and the features extracted by the convolutional layers may be downsampled through one or more pooling layers to reduce the redundancy and / or size of the features. As a result of the convolutional operation and / or downsampling operation, corresponding feature representations (e.g., latent space representations) of blocks 404a and 404b may be obtained, for example in the form of twin feature vectors 406a and 406b. The feature vectors may then be compared (e.g., at the block level and / or via the relevant layer) to determine the similarity between blocks 404a and 404b. The similarity may be represented in a similarity score graph (eg, or in other suitable similarity metrics such as similarity vectors, similarity scores, etc.) and may be used as a feature of a graph edge connecting respective graph nodes corresponding to blocks 404a and 404b.

[0028] The Siamese network 402 can be trained, for example, by contrastive learning based on a contrastive loss 408. This contrastive loss can be determined, for example, based on the cosine distance or Euclidean distance of the feature vectors 406a and 406b, and the Siamese network 402 can be trained with the goal of keeping the loss low when the blocks 404a and 404b are similar (e.g., have a closer distance) and keeping the loss high when the blocks 404a and 404b are different (e.g., have a greater distance).

[0029] Figure 4BAn example of determining similarity (e.g., similarity score) between two graph nodes using a Siamese network structure and triplet loss is illustrated. As shown in the figure, similarity can be determined using a Siamese network (e.g., including sub-networks 422a, 422b, and 422c), which can have Figure 4A The structure of the Siamese network 402 is similar to that of (eg, except that the number of identical subnetworks or the number of times the same network is used is three instead of two). Also similar to Figure 4A In the example shown, the similarity between two graph nodes can be determined based on blocks 424a and 424b containing corresponding hypothetical labeled points (e.g., corresponding pairs of balloon labels) represented by the graph nodes. More specifically, the similarity between the two graph nodes can be determined based on at least feature vectors 426a and 426b associated with blocks 424a and 424b, respectively, and the similarity can be represented in a similarity score graph (e.g., or with other suitable similarity metrics) and used as a feature of a graph edge connecting the corresponding graph nodes corresponding to blocks 424a and 424b.

[0030] Figure 4B The Siamese network shown can be trained using a triplet loss 428, for example, by additionally utilizing a negative block 424c along with blocks 424a and 424b, which can represent anchor and positive blocks, respectively. For the negative block 424c and the triplet loss 428, the Siamese network can be trained with the goal of minimizing or reducing the distance (e.g., cosine distance or Euclidean distance) between the anchor block 424a and the positive block 424b (e.g., between feature vectors 426a and 426b), while maximizing or increasing the distance between the anchor block 424a and the negative block 424c (e.g., between feature vectors 426a and 426c).

[0031] Figures 5A to 5C Example techniques are illustrated for encoding features of graph nodes (e.g., representing hypothetical labeled points) and graph edges (e.g., representing relationships between the hypothetical labeled points) that can be established using the techniques described herein, and determining corresponding true / false states of the hypothetical labeled points (e.g., by labeling corresponding graph nodes) using a graph neural network (GNN) (e.g., a message passing GNN). Figure 5A The example GNN may include an encoding network 502 trained to encode the corresponding features of the graph nodes and graph edges, which can be obtained using the above-mentioned techniques. In the example, the GNN may include a node encoder 502a, which is trained to encode the node features and utilizes the node feature vector v i (e.g., corresponding to graph node i) represents the encoded node features. The GNN may also include an edge encoder 502b, which is trained to encode edge features and utilize the edge feature vector e i(e.g., corresponding to graph edge i) represents the encoded edge feature. In an example, the node encoder 502a and the edge encoder 502b may include corresponding multi-layer perceptron (MLP) neural networks (e.g., fully connected neural networks). Each MLP neural network may include an input layer, an output layer, and one or more hidden layers, each of which includes corresponding linear or nonlinear activation nodes (e.g., perceptrons), which can be trained to encode the input features into feature representations (e.g., feature vectors v i or e i ), the GNN can then use this feature representation to determine the state (e.g., condition) of a node or edge based on other nodes or edges.

[0032] Figure 5B The graph neural network (GNN) described herein may include a core network 504 that is trained to generate graph-based representations (e.g., Figure 2 The diagram shows 202 or Figure 3 The GNN can be trained to update the corresponding states of the graph nodes and graph edges of the graph representation 310 (e.g., the current features) based on the other nodes and / or edges of the graph representation 310). The GNN can be trained to update the corresponding states of the graph nodes and graph edges through an iterative process (e.g., through multiple iterations k, k+1, k+2, etc.). For example, the core network 504 can include a first core sub-network (e.g., node network 504a), which is trained to update the previous state v of the current graph node (e.g., in iteration k) based on the previous state v of the current graph node (e.g., in iteration k). i k and the average edge state of the edges associated with the current graph node (e.g., the edges before the node and the edges after the node, if available) at iteration k To determine the state v of the current graph node i (e.g., in iteration k+1) i k+1 The core network 504 may also include a second core sub-network (eg, edge network 504b) that is trained to be based on the previous state e of the current edge (eg, in iteration k). n k and the corresponding states v of graph nodes i and j connected by the current edge (e.g., during iteration k) i k and v j k To determine the state e of the current edge n (e.g., in iteration k+1) n k+1. Each of the core node network 504a and the core edge network 504b can be implemented as a recurrent neural network (e.g., RNN) or an iterative neural network (e.g., a cascaded network with a fixed number of iterations), which can be trained to track the previous states (e.g., hidden states) of the nodes and edges described herein and determine the corresponding final states of the nodes and edges based on the hidden states. The initial states of the graph nodes can be set based on (e.g., the same as) the output of the node feature encoder 502a, and the initial states of the graph edges can be set based on (e.g., the same as) the output of the edge feature encoder 502b.

[0033] Figure 5C The graph neural network (GNN) described herein may include a decoder network 506 that is trained to determine (e.g., label) the true / false state of a graph node based on the corresponding final states of the graph node and the graph edge. A graph node may be labeled true to indicate that the hypothetical labeled point represented by the graph node is a true labeled point (e.g., such as Figure 3 The true balloon marker of the marker point 304a or 304c is represented by the graph node, and the graph node can be marked as false to indicate that the hypothetical marker point represented by the graph node is a false marker point (e.g., such as Figure 3 The corresponding final state v of the graph node is i l may correspond to the respective final features of the graph nodes determined by the node encoder 502a and the core node network 504a, and the respective final states of the graph edges The corresponding final features of the graph edges determined by the edge encoder 502b and the core edge network 504b can correspond to each other. As described herein, the corresponding final features of the graph nodes can indicate features of the image data near the hypothetical marker point represented by the graph node, and the corresponding final features of the graph edges can indicate the similarity of the graph nodes connected by the respective graph edges. It can be seen that the decoding network 506 (e.g., a fully connected neural network including one or more fully connected layers) can be trained to mark the graph node (e.g., the corresponding hypothetical marker point) as true if the graph node has certain features and the similarity between the graph node and its neighboring graph nodes meets certain criteria (e.g., the similarity score is higher than a predetermined threshold). Conversely, the decoding network 506 can be trained to mark the graph node (e.g., the corresponding hypothetical marker point) as false if the graph node does not have certain features and the similarity between the graph node and its neighboring graph nodes does not meet certain criteria (e.g., the similarity score is lower than a predetermined threshold).

[0034] It should be noted that even though certain structures are described for the neural networks described herein, the neural networks are not limited to the described structures. In addition, the proposed techniques can be applied to detect marker points in multiple sequentially scanned images or in a single scanned image, where the scanned images can be two-dimensional (2D) or three-dimensional (3D). For example, the proposed techniques can be applied to detect multiple marker points in a single image frame. For each marker point (e.g., a, b, c, etc.), multiple hypothesized positions (e.g., a0, a1, a2, ..., b0, b1, b2, ..., c0, c1, c2, ..., etc.) can be determined. Each hypothesized marker point can be represented as a graph node in a graph representation, and the relationship between a pair of hypothesized marker points can be represented as an edge in the graph representation. Corresponding features of the hypothesized nodes and edges can be determined and / or encoded, where the features of each hypothesized node can represent features of the image region containing the hypothesized marker point, and the features of each edge can represent geometric constraints associated with the hypothesized marker points connected by the edge. In this way, true / false landmarks can be determined in a single image by converting the landmark detection problem into a graph node labeling problem, which can then be solved using a message passing graph neural network as described in this paper.

[0035] Figure 6Example operations that may be associated with training a neural network described herein (e.g., various encoding networks, core networks, and / or decoding networks) to perform the landmark detection tasks described herein are illustrated. As shown, the training operation may include initializing parameters of the neural network (e.g., weights associated with various filters or kernels of the neural network) at 602. The parameters may be initialized, for example, based on samples collected from one or more probability distributions or parameter values ​​of another neural network with a similar architecture. The training operation may also include providing training data (e.g., a medical scan image including landmarks to be detected) to the neural network at 604, and causing the neural network to predict an outcome at 606. Depending on the function the neural network is designed to perform, the predicted outcome may include, for example, a set of image features associated with a landmark or graph node, a set of features of a graph edge representing a relationship between two graph nodes (e.g., a spatial relationship or geometric constraint), a final state of a graph node or graph edge, a hypothesized true / false label for a landmark, etc. At 608, the predicted outcome may be compared to a gold standard to determine a loss associated with the prediction. The loss may be determined using a suitable loss function, such as mean squared error (MSE), L1 norm, L2 norm, etc. Once the loss is determined, the loss can be evaluated at 610 to determine whether one or more training termination criteria have been met. For example, if the loss is below a predetermined threshold, if the change in loss between two training iterations (e.g., between consecutive training iterations) falls below a predetermined threshold, the training termination criteria can be considered met. If it is determined at 610 that the training termination criteria have been met, training can end. Otherwise, the loss can be backpropagated through the neural network at 612 (e.g., based on gradient descent associated with the loss) before training returns to 606.

[0036] For simplicity of illustration, the training steps are depicted and described herein in a particular order. However, it should be understood that the training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training process are depicted and described herein, and not all illustrated operations need to be performed.

[0037] The systems, methods, and / or devices described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 7is a block diagram illustrating an example device 700 that can be configured to perform the landmark detection tasks described herein. As shown, the device 700 may include a processor (e.g., one or more processors) 702, which may be a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other circuit or processor capable of performing the functions described herein. The device 700 may also include communication circuitry 704, a memory 706, a mass storage device 708, an input device 710, and / or a communication link 712 (e.g., a communication bus), through which one or more components shown in the figure can exchange information.

[0038] The communication circuit 704 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including a local area network (LAN), a wide area network (WAN), the Internet, a wireless data network (e.g., Wi-Fi, 3G, 4G / LTE, or 5G network). The memory 706 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions, which, when executed, causes the processor 702 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc. The mass storage device 708 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROM or DVD-ROM disks, etc., on which instructions and / or data may be stored to facilitate the operation of the processor 702. The input device 710 may include a keyboard, a mouse, a voice control input device, a touch-sensitive input device (eg, a touch screen), etc., for receiving user input of the device 700 .

[0039] It should be noted that the apparatus 700 can operate as a standalone device or can be connected (e.g., networked or clustered) with other computing devices to perform the functions described herein. Figure 7 Only one example of each component is shown in the figure, and those skilled in the art will also understand that the device 700 may include multiple examples of one or more components shown in the figure.

[0040] Although the present disclosure has been described in terms of certain embodiments and generally associated methods, variations and transformations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit the present disclosure. Other changes, substitutions, and variations are also possible without departing from the spirit and scope of the present disclosure. In addition, unless otherwise specifically stated, discussions utilizing terms such as "analyze," "determine," "enable," "identify," "modify," etc. refer to the actions and processes of a computer system or similar electronic computing device, which manipulate and transform data represented as physical (e.g., electronic) quantities within the registers and memories of a computer system into other data represented as physical quantities within the computer system memory or other such information storage, transmission, or display device.

[0041] It should be understood that the above description is intended to be illustrative, rather than restrictive. After reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the full scope of equivalents to which the appended claims and such claims are given.

Claims

1. A device for marking point detection, comprising: One or more processors configured to: determining a plurality of hypothesized landmarks in one or more medical images of anatomical structures; determining a graph representation of the plurality of hypothetical markers, wherein the graph representation comprises a plurality of nodes and a plurality of edges, each of the nodes representing a hypothetical marker from the plurality of hypothetical markers, and each of the edges representing a relationship between a pair of hypothetical markers from the plurality of hypothetical markers; and Using a graph neural network, one or more hypothetical marker points among the plurality of hypothetical marker points are identified as true marker points, and one or more hypothetical marker points among the plurality of hypothetical marker points are identified as false marker points, wherein the graph neural network comprises: an encoding network trained to extract corresponding features from the nodes of the graph representation and the edges of the graph representation; a core network trained to estimate respective states of the nodes and edges of the graph representation based on the features extracted by the encoding network; and A decoding network is trained to indicate the true labeled points and the false labeled points based on the respective states of the nodes and edges estimated by the core network.

2. The device according to claim 1, wherein The core network includes: a first core sub-network, which is trained to represent the corresponding states of the nodes represented by the graph; and a second core sub-network, which is trained to represent the corresponding states of the edges represented by the graph, wherein the encoding network includes: a node encoder, which is trained to extract the corresponding features of the nodes represented by the graph; and an edge encoder, which is trained to extract the corresponding features of the edges represented by the graph.

3. The device according to claim 1, wherein The one or more processors are configured to determine the multiple hypothesized marker points in the one or more medical images of the anatomical structure, including: the one or more processors are configured to extract features from the one or more medical images of the anatomical structure using a landmark detection neural network, and determine the corresponding positions of the multiple hypothesized marker points in the one or more medical images based on the features extracted by the landmark detection neural network.

4. The device according to claim 1, wherein The plurality of hypothesized marker points comprises a first set of hypothesized marker points detected in a first medical image of the anatomical structure and a second set of hypothesized marker points detected in a second medical image of the anatomical structure, and wherein the one or more processors are configured to determine the map representation of the plurality of hypothesized marker points comprises: the one or more processors are configured to: representing the first set of hypothetical marker points and the second set of hypothetical marker points using the nodes represented by the graph; pairing the first set of hypothetical markers with the second set of hypothetical markers; and The edges represented by the graph represent corresponding relationships between the paired hypothetical labeled points.

5. The device according to claim 4, wherein The features extracted by the encoding network for each of the edges represented by the graph indicate similarities or dissimilarities between the pairs of hypothesized labeled points represented by the edges.

6. The apparatus according to claim 1, wherein The plurality of hypothesized marker points comprises hypothesized marker points detected in a single medical image of the anatomical structure, and wherein the one or more processors are configured to determine the map representation of the plurality of hypothesized marker points comprises: the one or more processors are configured to: using the nodes of the graph representation to represent the hypothesized landmarks detected in the single medical image; pairing a first subset of the hypothesized marker points detected in the single medical image with a second subset of the hypothesized marker points detected in the single medical image; and The edges of the graph representation are used to represent the corresponding relationships between the paired hypothesized landmark points in the single medical image.

7. The apparatus according to claim 6, wherein The features extracted by the encoding network for each of the edges represented by the graph indicate geometric constraints between the pairs of hypothesized labeled points represented by the edges.

8. The apparatus according to claim 1, wherein The core network is trained to estimate the respective states of the nodes and edges of the graph representation through an iterative process comprising: determining a current state of a node of the graph representation based on a previous state of the node and an average state of one or more edges of the graph representation associated with the node; and A current state of the edge of the graph representation is determined based on a previous state of the edge and corresponding current states of one or more nodes of the graph representation associated with the edge.

9. The apparatus according to claim 1, wherein The graph neural network includes a message passing graph neural network, and the plurality of hypothesized landmarks include a pair of balloon markers placed inside the anatomical structure or a guidewire placed inside the anatomical structure.

10. A method for detecting a marker point, comprising: determining a plurality of hypothesized landmarks in one or more medical images of anatomical structures; determining a graph representation of the plurality of hypothetical markers, wherein the graph representation comprises a plurality of nodes and a plurality of edges, each of the nodes representing a hypothetical marker from the plurality of hypothetical markers, and each of the edges representing a relationship between a pair of hypothetical markers from the plurality of hypothetical markers; and Using a graph neural network, one or more hypothetical marker points among the plurality of hypothetical marker points are identified as true marker points, and one or more hypothetical marker points among the plurality of hypothetical marker points are identified as false marker points, wherein the graph neural network comprises: an encoding network trained to extract corresponding features from the nodes of the graph representation and the edges of the graph representation; a core network trained to estimate respective states of the nodes and edges of the graph representation based on the features extracted by the encoding network; and A decoding network is trained to indicate the true labeled points and the false labeled points based on the respective states of the nodes and edges estimated by the core network.

Images