System and method for tracking a group of objects in a medical image

Abstract

Systems, methods, and apparatuses associated with tracking groups of small objects in medical images are described herein. The tracking can be achieved by, for each image in a sequence of medical images, determining a plurality of candidate objects captured in the medical image, grouping the plurality of candidate objects into a plurality of candidate object groups, and dividing the medical image into a plurality of regions each surrounding a corresponding candidate object group. Each of the plurality of regions can be examined to extract respective features associated with the respective corresponding candidate object group. A match between a first group and a second group can be determined based on first features associated with a first candidate object group in a first medical image and second features associated with a second candidate object group in a second medical image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing. BACKGROUND

[0002] Small object detection plays an important role in the field of medical imaging. The ability to determine and / or track the location of certain small objects (e.g., implanted medical devices, surgical devices, etc.) with medical images that are sequentially captured based on the patient’s body can be critical to the success of many medical applications or procedures including, for example, motion estimation, surgical navigation, device alignment, etc. Object detection techniques can analyze these medical images individually and independently, thus lacking the ability to determine and utilize any relationships between regions associated with these objects across sequentially captured medical images. As a result, the results produced by these detection techniques can often be inconsistent and lack accuracy across different image frames. Accordingly, systems, methods, apparatuses for improving the quality and efficiency of known object detection techniques are desirable. SUMMARY

[0003] Systems, methods, and apparatuses associated with tracking groups of objects (e.g., small objects) in medical images are described herein. An apparatus configured to perform an object detection task can include one or more processors configured to receive a plurality of medical images (e.g., a sequence of images of the same anatomy), and determine a plurality of candidate objects (e.g., a group of balloon markers associated with a stent placed inside the anatomy, a guidewire placed inside the anatomy, etc.) in each of the sequence of medical images of the anatomy using one or more artificial neural networks (ANNs), and determine a match between a first group of candidate objects in a first medical image and a second group of candidate objects in a second medical image, where the first medical image and the second image are related (e.g., sequentially captured over a small time delay). The plurality of candidate objects can be grouped into a plurality of groups of candidate objects, and each medical image in the sequence can be divided into a plurality of regions each surrounding a corresponding group of candidate objects. Respective features associated with each group of candidate objects (e.g., from the corresponding regions) can then be extracted. The match between the first group of candidate objects and the second group of candidate objects can then be determined based on first features associated with the first group of candidate objects in the first medical image and second features associated with the second group of candidate objects in the second medical image.

[0004] The one or more processors of the apparatus can also be configured to determine, for each image in the sequence of medical images, a plurality of candidate objects associated with a target object captured in the medical image by computing a heat map for the medical image using a first neural network (e.g., each pixel in the heat map is associated with a value indicative of a probability of a marker being present at the pixel), and then determining the plurality of candidate objects based on the pixel values in the heat map.

[0005] One or more processors of the device may also be configured to group multiple candidate objects into multiple candidate object groups (e.g., for individual medical images in a sequence) by grouping multiple candidate objects into multiple candidate object pairs, and to divide each medical image into multiple regions surrounding a corresponding candidate object group by dividing each medical image into multiple regions surrounding a corresponding candidate object pair.

[0006] One or more processors in the device may also be configured to extract corresponding features associated with each candidate object group (e.g., for each medical image in a sequence) by computing corresponding features associated with each candidate object group using a second neural network. The second neural network may include multiple layers, each performing a downsampling operation.

[0007] One or more processors of the device can also be configured to determine a graph representation of multiple candidate object groups, wherein the graph representation includes multiple nodes and multiple edges. Each node can represent a candidate object group, and each edge can represent a relationship between a pair of groups.

[0008] One or more processors in the device can also be configured to determine a match between a first candidate object group in a first medical image and a second candidate object group in a second medical image using a graph neural network (GNN) based on a first feature associated with a first candidate object group and a second feature associated with a second candidate object group. The GNN can receive a first graph representation associated with the first medical image and a second graph representation associated with the second medical image as input, and compute node labels indicating whether the group corresponding to a node includes a real object. The GNN can then compute edge values ​​of the edges connecting the first node in the first medical image and the second node in the second medical image, where the edge values ​​indicate whether the first node and the second node are associated with the same candidate object group. In the example, the output of the first neural network (e.g., a heatmap of the medical image) can be updated based on the output of the GNN (e.g., node labels and edge values).

[0009] In the example, the first neural network can be a first convolutional neural network (CNN), and the second neural network can be a second CNN, wherein the first CNN, the second CNN, and the GNN can be trained jointly. Training may include: calculating a heatmap regression loss based on the output of the first CNN from a training sequence of medical images; calculating a label group classification loss based on the output of the second CNN; and calculating a node classification loss based on the output of the GNN. At least one parameter associated with at least one of the first CNN, the second CNN, or the GNN can then be adjusted based on at least one of the heatmap regression loss, the label group classification loss, or the node classification loss.

[0010] The one or more processors of the device can also be configured to obtain a subsequent medical image following the sequence of medical images, and compute a location of the group of candidate objects in the subsequent medical image based on the locations of the group of candidate objects in the sequence of medical images. BRIEF DESCRIPTION OF DRAWINGS

[0011] The examples disclosed herein can be understood more readily by reference to the following detailed description, when taken in connection with the accompanying drawings, and wherein:

[0012] Figure 1A and Figure 1B is a diagram illustrating an example group of candidate objects (e.g., markers) that can be detected and tracked across multiple medical images using techniques provided in accordance with one or more embodiments described herein.

[0013] Figure 2 is a simplified diagram illustrating example operations that can be performed in accordance with one or more embodiments described herein to detect candidate objects (e.g., markers) in medical images and divide the images into regions surrounding groups of candidate objects.

[0014] Figure 3 is a simplified diagram illustrating example operations that can be performed in accordance with one or more embodiments described herein on respective regions to extract respective features associated with corresponding groups of candidate objects (e.g., markers).

[0015] Figure 4 is a simplified diagram illustrating example operations that can be performed in accordance with one or more embodiments described herein to match a first group of candidate objects (e.g., markers) in a first medical image to a second group of candidate objects in a second medical image.

[0016] Figure 5 is a simplified flow diagram illustrating example operations that can be performed in accordance with one or more embodiments described herein to track a group of candidate objects (e.g., markers) across multiple medical images.

[0017] Figure 6 is a simplified flow diagram illustrating example operations that can be performed in accordance with one or more embodiments described herein to jointly train a number of neural networks.

[0018] Figure 7 is a simplified block diagram illustrating example components of a device that can be configured to perform tasks for tracking a group of candidate objects (e.g., markers) in accordance with one or more embodiments described herein. DETAILED DESCRIPTION

[0019] The disclosure is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which:

[0020] Figure 1A andFigure 1B is a diagram illustrating example candidate object (e.g., marker) groups including a first group (102a and 104a) and a second group (102b and 104b), respectively, that can be detected and tracked across multiple medical images using techniques provided in accordance with one or more embodiments described herein. Figure 1A An example first medical image in a sequence of medical images is shown, and Figure 1B An example second medical image in a sequence of medical images is shown, where the first medical image and the second medical image can be captured sequentially. Figure 1A Two candidate object groups (e.g., group 102a enclosed in a dark circle and group 104a enclosed in a light circle) that can indicate the location of a larger object within a patient are shown. For example, the location associated with the presence of a stent can be indicated by the respective candidate marker (e.g., balloon marker) groups 102a and 104a. More specifically, the respective candidate marker groups (e.g., pairs) 102a and 104a can indicate the location of two portions of a stent, respectively. Figure 1B Two candidate object groups (e.g., group 102b enclosed in a dark circle and group 104b enclosed in a light circle) that can be the same two groups tracked from a first medical image ( Figure 1A ) to a second medical image ( Figure 1B ) in the sequence are shown. Due to noise associated with the medical images, some regions in the images can resemble the appearance of a stent and can be erroneously identified as a stent by an image processing system or device. This problem can be addressed by tracking groups of small objects (or other indicators), which can be candidate markers used to indicate the location of any stent. These candidate markers can maintain the same spatial and / or geometric features from one medical image to another in the sequence. For example, the relative position of points along a guidewire and / or the distance between two balloon markers can remain the same across multiple medical images in the sequence of medical images. As such, these spatial and / or geometric features can be explored to improve the tracking of the candidate marker groups, whereby detection of a true marker is achieved based on the candidate markers, e.g., by identifying candidate markers that maintain the features as true markers, while identifying candidate markers that do not maintain the features as false markers.

[0021] It should be noted herein that while markers or stents can be used as examples to describe embodiments in this disclosure, those skilled in the art will appreciate that the disclosed techniques can also be used to detect other small objects, such as small medical devices or anatomical structures.

[0022] Figure 2is a simplified diagram illustrating example operations 200 that can be performed to detect candidate markers in medical images 202 and divide the images into regions 208 surrounding groups of candidate markers, in accordance with one or more embodiments described herein. A device 200 configured to perform the example operations can include one or more processors configured to detect candidate markers in one or more medical images 202 (e.g., a sequence of medical images). The one or more medical images 202 can be scan images (e.g., MRI, X-ray, etc.) of an anatomical structure of a human body (e.g., a heart), and the candidate markers can include various anatomical and / or artificial structures or components that can indicate other phenomena, such as an end of a guidewire or a pair of balloon markers that can indicate a location of a stent in the medical images. The candidate markers can be tracked across multiple different medical images (e.g., a sequence of multiple scan images of a cardiac cine). The candidate markers can initially be identified by the detection operations 200 on a hypothesis basis. For example, the detection operations 200 can identify hypothesized candidate markers (e.g., respective locations of hypothesized candidate markers 102a and 104a as shown) in a sequence of medical images. Because there can be some similarity between a candidate marker and its surrounding region in some medical images, the hypothesized candidate markers identified by the detection operations 200 can include true positives (e.g., real markers) and / or false positives (e.g., background regions that are falsely identified as candidate markers). Figure 1A

[0023] In an example, the detection operations 200 can perform the candidate marker detection task using one or more classification neural networks. In an example, the classification neural network can include a deep neural network, such as a first convolutional neural network (CNN) 204 that has been trained to extract image features from individual input medical images 202 and classify regions of the medical images as containing a candidate marker or not containing a candidate marker based on the extracted image features. A digital image (e.g., a medical image 202) can be composed of picture elements, also referred to as pixels, each having an associated value that is represented as a numerical value that serves as its intensity or grayscale level. Thus, a digital image can be processed as the numerical values of these pixels, and to detect particular image features (e.g., for extraction), the processing can identify patterns and / or regularities in the numerical values that are associated with the image features. For example, image feature extraction can be based on identification of numerical patterns that are associated with image features such as shapes and / or edges in the digital image. In this example, a probability map can then be created to locate candidate markers of individual medical images 202 by identifying candidate marker boundaries based on shape and / or edge image features extracted from the individual medical images 202.

[0024] ​The first CNN 204 can include multiple convolutional layers and / or one or more pooling layers. The convolutional layers can be followed by batch normalization and / or linear or non-linear activation (e.g., a rectified linear unit or ReLU activation). The respective convolutional layers can include one or more convolutional kernels or filters with respective weights, the values of which can be learned through a training process in order to extract image features from the respective medical images 202. The image features extracted by the convolutional layers can be down-sampled through the one or more pooling layers to obtain a representation of the image features, e.g., in the form of one or more image feature maps. The first CNN 204 can further include one or more up-pooling layers and one or more transposed convolutional layers. Through the up-pooling layers, the first CNN 204 can up-sample the image features extracted from the input medical images 202, and further process the up-sampled image features through the one or more transposed convolutional layers (e.g., via a de-convolution operation) to derive one or more upscaled or dense image feature maps. The dense image feature maps can then be used to predict (e.g., hypothesize) candidate markers (e.g., Figure 1A in the respective heat maps 206 corresponding to the input medical images 202, where the pixel values of the heat maps 206 can represent probability values of the presence of a candidate marker at the respective predicted (e.g., hypothesized) locations of the candidate marker in the corresponding medical images 202. As explained more fully below with respect to Figure 6 , a heat map regression loss L hm can be computed for the heat maps 206 and used to train the first CNN 204 or jointly with other neural networks.

[0025] In an example, the detection operation 200 can predict grouping of candidate markers into candidate marker groups, and the respective medical images in the sequence can be divided (based on the heat maps 206) into respective regions 208 that each surround a corresponding (predicted) candidate marker group. Information about the predicted candidate marker groups and the corresponding regions 208 can be provided to a second CNN (e.g., Figure 3 , CNN 302) so that respective group features associated with the respective candidate marker groups can then be extracted from the corresponding regions 208, as explained more fully below with respect to Figure 3 In an example, the detection operation 200 can group the candidate markers into candidate marker groups (e.g., for the respective medical images in the sequence) by grouping the candidate markers into pairs of candidate markers (e.g., Figure 1A , groups 102a) and divide the respective medical images into respective regions (e.g., respective regions 208) that each surround a corresponding pair of candidate markers.

[0026] Figure 3is a simplified diagram illustrating example operations 300 that can be performed on respective regions 208 to extract respective group features associated with corresponding groups of candidate markers according to one or more embodiments described herein. The group features can be, for example, features extracted from the regions 208 (e.g., spatial features such as distances between candidate markers) that are shared by candidate markers associated with a group of candidate markers corresponding to the regions 208. For example, Figure 1A The pairs of candidate markers of the group 102a can all share features of having the same shape as each other and within the corresponding regions 208. A device configured to perform the example operations 300 can include one or more processors configured to extract group features from the respective regions 208 in the one or more medical images 202 using a second CNN 302. The CNN 302 can extract the group features from the respective regions 208 using one or more convolutional operations. The CNN 302 can be configured to extract the group features from the respective regions 208 using one or more convolutional kernels or filters configured to extract the group features of the candidate markers in the regions 208 through the one or more convolutional operations. The convolutional operations can be followed by batch normalization and / or non-linear activation. The group features extracted by the convolutional layers (e.g., at 304-310) can be down-sampled through one or more pooling layers (e.g., average pooling at 312) to obtain a representation of the group features, for example, in the form of one or more graph representations. Thus, for example, the group features of the candidate marker groups (e.g., the groups 102a and 104a) can be obtained in the form of a marker group feature graph having a graph representation (e.g., the graphs 404a and 404b) of the group. The marker group feature graph can then be provided to a GNN (e.g., the GNN 306) to determine the respective groupings of the candidate markers in the one or more medical images 202. Figure 2 The medical images 202 and information about the predicted groups of candidate markers and the corresponding regions 208 (e.g., the marker group predictions) can be received by the 206 of the detection operations 200 from the second CNN 302, as described above. The extraction operations 300 can then determine respective groupings of the candidate markers in the one or more medical images 202 based on the group features extracted by the second CNN 302. The respective groupings of the candidate markers can be determined by the extraction operations 300 using the GNN 306, as described below. Figure 1A The groupings of the candidate markers in the one or more medical images 202 can be determined by the extraction operations 300 using the GNN 306, as described below. The respective groupings of the candidate markers in the one or more medical images 202 can be determined by the extraction operations 300 based on the marker group feature graphs (e.g., the graphs 404a and 404b) of the groups 102a and 104a (e.g., at 304-310), and then the marker group classification loss L mark may be computed at 314 and used to train the second CNN 302 or jointly train with other neural networks, as explained more fully below with respect to Figure 6 .

[0027] The second CNN 302 can include a plurality of convolutional layers, one or more pooling layers, and / or one or more fully connected layers. The convolutional layers of the second CNN 302 can include convolutional kernels or filters configured to extract group features of the candidate markers in the regions 208 through one or more convolutional operations. The convolutional operations can be followed by batch normalization and / or non-linear activation, and the group features extracted by the convolutional layers (e.g., at 304-310) can be down-sampled through one or more pooling layers (e.g., average pooling at 312) to obtain a representation of the group features, for example, in the form of one or more graph representations. Thus, for example, the group features of the candidate marker groups (e.g., the groups 102a and 104a) can be obtained in the form of a marker group feature graph having a graph representation (e.g., the graphs 404a and 404b) of the group. The marker group feature graph can then be provided to a GNN (e.g., the GNN 306) to determine the respective groupings of the candidate markers in the one or more medical images 202. Figure 4 The respective groupings of the candidate markers in the one or more medical images 202 can be determined by the extraction operations 300 using the GNN 306, as described below. The respective groupings of the candidate markers in the one or more medical images 202 can be determined by the extraction operations 300 based on the marker group feature graphs (e.g., the graphs 404a and 404b) of the groups 102a and 104a (e.g., at 304-310), and then the marker group classification loss L Figure 1A may be computed at 314 and used to train the second CNN 302 or jointly train with other neural networks, as explained more fully below with respect to Figure 1B . Figure 4The GNN 402 allows feature maps of labeled groups to be compared with each other to determine any similarity that can be used to match groups (e.g., based on features of the first labeled group associated with the first candidate labeled group and features of the second labeled group associated with the second candidate labeled group), as follows regarding Figure 4 A more comprehensive explanation. For example, candidate markers may include a first group of candidate markers detected in the first medical image (e.g., Figure 1A Group 102a) and the second candidate marker group detected in the second medical image (e.g., Figure 1B (Group 102b). To determine the graph representation of the candidate label groups (e.g., a label group feature map), the first and second candidate label groups can be represented by nodes in the graph representation and paired with each other. The corresponding relationships between the paired candidate labels (e.g., based on the extracted group features) can then be represented by the edges connecting them in the graph representation. The group features extracted by the second CNN 302 for each edge of the graph representation can indicate the similarity or dissimilarity between the paired candidate labels connected by the respective edges (e.g., whether they belong to the same group).

[0028] Figure 4 This illustrates how one or more embodiments described herein can be performed to match a first candidate group of markers in a first medical image (e.g., Figure 1A Group 102a) and the second candidate marker group in the second medical image (e.g., Figure 1B A simplified diagram of example operation 400 for group 102b). A device configured to perform example matching operation 400 may include one or more processors configured to use a graph neural network (GNN) 402 to match candidate label groups in individual medical images. The GNN 402 has been trained to determine (e.g., label) which candidate labels (e.g., those derived from...) Figure 2 The first CNN 204 detects which are real labels and which are fake labels. Based on the real candidate labels, GNN 402 can further determine which candidate label pairs belong to the same group based on shared group features extracted from the corresponding regions 208 around the group (e.g., indicators that are the same stents present in the sequence of medical images 202).

[0029] GNN 402 can be obtained from the above-mentioned... Figure 3 The extraction operation 300 at step 312 receives labeled group feature maps (e.g., Figures 404a and 404b) with group representations. As described above, the edges of these maps can represent, for example, a first medical image ( Figure 1A Candidate marker groups (e.g., group 102a) in the second medical image ( Figure 1Brelationships between possibly matching candidate marker groups (e.g., a group 102b that can include the same candidate markers) in the pairs of medical images (e.g., the pairs of medical images 202a and 202b). By representing the geometric and / or spatial features of candidate marker groups across multiple different medical images (e.g., by the nodes and edges of the marker group feature maps (e.g., the maps 404a and 404b) and the group features extracted by the second CNN 302) the problem of tracking candidate marker groups across multiple different medical images can be converted into a problem of node labeling of graphs and solved using the GNN 402. Figure 3

[0030] For example, the nodes of the graph representation 404a can correspond to data structures that store information about the candidate marker groups 102a, 104a, 102b, and 104b, respectively, and the edges of the graph representation 404a can correspond to data structures that store information about the respective relationships between the candidate marker groups 102a and 102b and between the candidate marker groups 104a and 104b. The nodes and edges of the graph representations 404a and 404b can have their own features. By extracting and tracking these graph features, the true markers (e.g., the corresponding nodes labeled with “R” in the graph outputs 406a and 406b) can be distinguished from the false markers by the GNN 402.

[0031] In an example, the graph features of each edge in the graph representation 404a can be derived based on geometric constraints associated with the two nodes connected by the edge. For example, the edge connecting the true marker pair of the group 102a can represent the distance between the candidate marker pair, and if the edge representing the relationship between the candidate marker pair of the group 102b represents a similar distance, then the two groups (e.g., 102a and 102b) can be matched based on the geometric features of these true markers being maintained across multiple medical image frames (e.g., the pairs of medical images 202a and 202b) in the sequence of medical images 202. Figure 1A Figure 1B Figure 1A Figure 1B Figure 1A Figure 1B In an example, a node classification loss L node can be computed for the graph output (e.g., 406a) and used to train the GNN 402 or jointly train with other neural networks, as described more fully below with respect to Figure 6 In an example, the output of the first CNN 204 (e.g., the pixel values of the heat map 206 of the medical images 202) can be updated based on the output of the GNN 402 (e.g., the node labels and edge values of the graph output 406a).

[0032] ​​​​​​The matching operation 400 can also obtain subsequent medical images following the sequence of medical images 202, and based on candidate marker groups (e.g., Figure 1A The position of the candidate label group in subsequent medical images is calculated by determining the position of group 102a) in the sequence of medical images 202. In the example, the GNN 402 described herein can be trained to estimate the corresponding states of nodes and edges in the graph representation of subsequent images in the sequence of medical images 202 through an iterative process, which may include: determining the current state of a node in the graph representation based on the previous state of the node and the average state of one or more edges associated with the node in the graph representation, and determining the current state of an edge in the graph representation based on the previous state of the edge and the corresponding current state of one or more nodes associated with the edge in the graph representation.

[0033] Figure 5 This illustrates how one or more embodiments described herein can be performed across multiple medical images (e.g., Figure 2 A simplified flowchart of example operation 500 (202) for tracking candidate marker groups. In operation 502, a device configured to perform a target object detection task may include one or more processors configured to receive medical images (e.g., a sequence of medical images 202 of the same anatomical structure). In operation 504, using one or more artificial neural networks (e.g., a first CNN 204, a second CNN 302, and a GNN 402), candidate markers associated with the target object (e.g., a set of balloon markers associated with a stent placed inside an anatomical structure) can be determined in the individual images of the sequence of medical images of the anatomical structure. In operation 506, the candidate markers can be grouped into candidate marker groups (e.g., groups 102a, 104a, 102b, and 104b), and the individual medical images in the sequence of medical images (e.g., Figure 1A The region is divided into regions (e.g., 208) around its corresponding candidate label group. In operation 508, the corresponding features associated with each candidate label group (e.g., label group feature map 404a) can then be extracted (e.g., from the corresponding region 208). In operation 510, the first medical image in the sequence can then be determined based on the first feature associated with the first candidate label group and the second feature associated with the second candidate label group (e.g., based on the first and second graph features of the node labels / edge values ​​from the GNN 402 output map 406a). Figure 1A The first candidate marker group (e.g., 102a) in the second medical image ( Figure 1B Matching between the second candidate tag group (e.g., 102b) in ).

[0034] Figure 6is a simplified flowchart illustrating example operations 600 that can be performed for jointly training several neural networks (e.g., first CNN 204, second CNN 302, and GNN 402) in accordance with one or more embodiments described herein. As shown, the training operations can include initializing parameters of the neural networks (e.g., weights associated with individual filters or kernels of the neural networks) at 602. The parameters can be initialized, for example, based on samples collected from one or more probability distributions or parameter values of another neural network having a similar architecture. At operation 604, the training can also include providing training data to the first CNN 204 (e.g., medical scan images including candidate markers to be detected) and computing a heat map regression loss L hm (e.g., causing the first CNN 204 to predict results and comparing the results to a gold standard to determine a loss associated with the predictions) based on outputs of the first CNN 204. At operation 606, the training can also include computing a marker group classification loss L mark based on outputs of the second CNN 302 (e.g., the group feature map outputs of the second CNN 302 are input). At operation 608, the training can also include computing a node classification loss L node based on outputs of the GNN 402 (e.g., the group feature map outputs of the second CNN 302 are input). Once determined, the losses (e.g., L hm , L mark , and L node ) can be evaluated at 610 to determine whether one or more training termination criteria have been satisfied. For example, the training termination criteria can be considered satisfied if one or more of the losses described above are below a predetermined threshold, if a change in one or more of the losses between two training iterations (e.g., between consecutive training iterations) falls below a predetermined threshold, etc. If it is determined at 610 that the training termination criteria have been satisfied, the training can end. Otherwise, at 612, at least one parameter of at least one of the first CNN, the second CNN, or the GNN can be adjusted based on at least one of the heat map regression loss L hm , the marker group classification loss L mark , or the node classification loss L node before the training returns to operation 604.

[0035] For the sake of simplicity in illustration, the training steps are depicted and described herein in a particular order. However, it should be understood that the training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Moreover, it should be noted that not all operations that can be included in a training process are depicted and described herein, and not all of the example operations depicted need be performed.

[0036] Figure 7is a simplified block diagram illustrating example components of a device 700 that can be configured to perform tasks for tracking candidate marker groups, in accordance with one or more embodiments described herein. In embodiments, the neural network system 700 can be connected (e.g., via a network, such as a local area network (LAN), an intranet, an extranet, or the Internet) to other computer systems. The neural network system 700 can operate in the capacity of a server or a client computer in client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The neural network system 700 can be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any device (or combination thereof) capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, the term “computer” shall also include any collection of computers or computerized devices (whether or not in the same physical location) that together execute a set, or multiple sets, of instructions to perform any one or more of the methodologies discussed herein.

[0037] Further, the neural network system 700 can include a processing device 702, a volatile memory 704 (e.g., random access memory (RAM)), a non-volatile memory 706 (e.g., read-only memory (ROM) or Electrically Programmable (ROM) (EEPROM)), and a data storage device 716, which can communicate with one another via a bus 708. The processing device 702 can be provided by one or more processors, such as a general-purpose processor (e.g., a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a microprocessor implementing a combination of instruction sets) or a special-purpose processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0038] The neural network system 700 can also include a network interface device 722, a video display unit 710 (e.g., an LCD), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), a data storage device 716, and / or a signal generation device 720. The data storage device 716 can include a non-transitory computer-readable storage medium 724 on which can be stored instructions 726 encoding any one or more of the image processing methods or functions described herein. The instructions 726 can further reside completely, or partially, within volatile memory 704 and / or processing device 702 during their execution by computer system 700, such that volatile memory 704 and processing device 702 also can be said to be a machine-readable storage medium.

[0039] While the computer-readable storage medium 724 is shown in the illustrative example as a single medium, the term“computer-readable storage medium” shall include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of executable instructions. The term“computer-readable storage medium” shall also include any tangible medium that is capable of storing or encoding a set of instructions for execution by a computer that cause the computer to perform any one or more of the methods taught herein. The term“computer-readable storage medium” shall include, without being limited to, solid-state memories, optical media, and magnetic media.

[0040] The methods, components, and features described herein can be implemented by discrete hardware components or can be integrated in the functionality of other hardware components such as ASICs, FPGAs, DSPs or similar devices. Additionally, the methods, components and features can be implemented by firmware modules or functional circuitry within hardware devices. Further, the methods, components and features can be implemented in any combination of hardware devices and computer program components, or in computer programs.

[0041] While the present disclosure has been described in terms of certain embodiments and illustrative methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above-described above description of example embodiments does not limit the present disclosure. Other changes, modifications, and variations can be possible. Additionally, unless otherwise stated the use of

[0042] It should be understood that the foregoing description is directed to exemplary embodiments and is not intended to limit the scope of the present disclosure. Many other embodiments will be apparent to those skilled in the art after reading and understanding the above description. The scope of the present disclosure should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims and the full range of equivalents that are taught thereby.

Claims

1. A method for tracking a group of candidate objects, comprising: Obtain the first and second medical images; For each medical image in the first medical image and the second medical image: Identify multiple candidate objects in the medical image; The plurality of candidate objects are grouped into a first group comprising two or more candidate objects and a second group comprising two or more candidate objects; Extract a first feature from the first region in the medical image corresponding to the first group including two or more candidate objects; and Extract a second feature from the second region in the medical image that corresponds to the second group including two or more candidate objects; The graph neural network (GNN) is used to determine the matching between the first group in the first medical image containing two or more candidate objects and the first group in the second medical image containing two or more candidate objects, based on the first features extracted from the first medical image and the second medical image, respectively. as well as The GNN is used to determine the matching between the second group of candidates in the first medical image and the second group of candidates in the second medical image based on the second features extracted from the first medical image and the second medical image, respectively; The GNN is configured as follows: Receive a graph representation including a first node, a second node, and an edge between the first node and the second node, wherein the first node represents the first group in the first medical image that includes two or more candidate objects, the second node represents the first group in the second medical image that includes two or more candidate objects, and the edge represents the relationship between the first group in the first medical image that includes two or more candidate objects and the first group in the second medical image that includes two or more candidate objects. The calculation indicates whether the first group represented by the first node includes a first node label of a real object and whether the first group represented by the second node includes a second node label of a real object; and Calculate the edge value of the edge connecting the first node and the second node, wherein the edge value indicates whether the first node and the second node are associated with the same group of candidate objects.

2. The method according to claim 1, wherein, The plurality of candidate objects are multiple candidate markers. For each medical image in the first medical image and the second medical image, determining the plurality of candidate objects in the medical image includes: A heatmap of the medical image is calculated using a first neural network, wherein the value of each pixel in the heatmap indicates the probability that a candidate label exists at that pixel; and The plurality of candidate objects are determined based on the values ​​of each pixel in the heatmap.

3. The method according to claim 1, wherein, The multiple candidate objects are multiple candidate tags, the first group including two or more candidate objects includes two or more candidate tags, and the second group including two or more candidate objects includes two or more candidate tags.

4. The method according to claim 2, wherein, The first feature and the second feature were extracted using a second neural network.

5. The method according to claim 4, wherein, The first neural network, the second neural network, and the GNN are jointly trained, and the training includes: The heatmap regression loss is calculated from the training sequence of medical images based on the output of the first neural network; The label group classification loss is calculated based on the output of the second neural network; Calculate the node classification loss based on the output of the GNN; and At least one parameter of the first neural network, the second neural network, or the GNN is adjusted based on at least one of the heatmap regression loss, the label group classification loss, or the node classification loss.

6. The method according to claim 5, further comprising: For each medical image in the first medical image and the second medical image, the probability value indicated by the value of each pixel in the heatmap is adjusted based on the output of the GNN.

7. The method according to claim 1, further comprising: Obtain subsequent medical images following the medical image sequence; as well as The positions of the first group and the second group in the subsequent medical images are calculated based on their positions in the first and second medical images.

8. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images