Determining location in medical imaging data at which given feature is represented

Abstract

A computer-implemented method and apparatus for determining a location in target medical imaging data at which a given feature is represented is disclosed. The method includes determining an approximate position at which the given feature is represented by inputting an initial descriptor for an initial position in the target medical imaging data into a trained machine learning model. A plurality of candidate positions are determined based on the approximate positions, and a candidate descriptor for each of the plurality of candidate positions is obtained. A candidate position is selected from among the plurality of candidate positions based on a similarity measure between the candidate descriptor and a reference descriptor for a reference position in reference medical imaging data.

Description

[0001] Cross - reference to related applications This application claims the benefit of priority from European Patent Application No. 23215512.7, filed on December 11, 2023, the content of which is incorporated herein by reference. Technical field

[0002] This framework relates to methods and apparatus for determining the location in medical imaging data at which a given feature is represented. Background art

[0003] In medical imaging, it is often important to determine where in a medical image a certain feature, such as a medical landmark (e.g., carina), is represented. Landmarks can be used to register different images into a coordinate system. For example, when comparing two images of a patient taken at different times, the position and zoom of the images may be different. By registering them into a coordinate system, the images can be compared to evaluate the progression of a disease. Landmarks also have other uses, such as planning radiotherapy and determining geometric features (e.g., distances) in the human body.

[0004] One method for finding a landmark involves: comparing a descriptor of the landmark in a reference medical image with a candidate descriptor for a candidate location in a target medical image; and identifying the candidate location whose descriptor most closely resembles the reference descriptor. However, this requires testing candidate locations across the entire target medical image. In addition, even if the landmark is not actually represented in the target medical image, the method may identify a candidate location as representing the landmark. Thus, the method may be vulnerable to "false positive" indications of the location of a particular landmark in the target medical image, which is undesirable.

[0005] There is a need to provide reliable and efficient methods for determining the location of features in medical imaging data. Summary of the invention

[0006] According to one aspect, there is provided a computer-implemented method for determining a location in target medical imaging data at which a given feature is represented, the medical imaging data including an array of elements having respective values and representing respective locations. The method may include: obtaining an initial descriptor for an initial location in the target medical imaging data, the initial descriptor representing values of elements of the target medical imaging data located relative to the initial location according to a first predefined pattern. Determining an approximate location in the target medical imaging data at which the given feature is represented based on an input of data representing the initial descriptor to a trained machine learning model. Further determining a plurality of candidate locations in the target medical imaging data based on the approximate location. Obtaining a candidate descriptor for each of the plurality of candidate locations, each candidate descriptor representing values of elements of the target medical imaging data located relative to the candidate location according to a second predefined pattern. Obtaining a reference descriptor for a reference location in reference medical imaging data, the reference medical imaging data including one or more sets of reference medical imaging data, for each of the one or more sets of reference medical imaging data, the reference location being a location in the set of reference medical imaging data at which the given feature is represented, for each of the one or more sets of reference medical imaging data, the reference descriptor representing values of elements of the set of reference medical imaging data located relative to the reference location according to the second predefined pattern. For each of the plurality of candidate locations, comparing the reference descriptor and the candidate descriptor for the candidate location to obtain a similarity measure. Selecting a candidate location from among the plurality of candidate locations based on the calculated similarity measure. Determining a location in the target medical imaging data at which the given feature is represented based on the selected candidate location. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more complete appreciation of the present disclosure and many of its attendant aspects will be readily obtained, as the same becomes better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0008] Figure 1 is a flowchart schematically illustrating a method for determining a location in target medical imaging data at which a given feature is represented; Figure 2 is a diagram schematically illustrating target medical imaging data according to an example; Figure 3 is a diagram schematically illustrating target medical imaging data according to an example; Figure 4 is a diagram schematically illustrating a trained machine learning model according to an example; Figure 5ais a diagram schematically illustrating reference medical imaging data according to another example; Figure 5b is a diagram schematically illustrating reference medical imaging data according to another example; Figure 6a is a diagram schematically illustrating target medical imaging data according to another example; Figure 6b is a diagram schematically illustrating target medical imaging data according to another example; Figure 7 is a diagram schematically illustrating reference medical imaging data according to another example; Figure 8 is a diagram schematically illustrating target medical imaging data according to another example; Figure 9 is a diagram schematically illustrating target medical imaging data according to another example; Figure 10 is a diagram schematically illustrating target medical imaging data according to another example; Figure 11 is a graph illustrating the performance of several methods for determining the location in target medical imaging data where a given feature is represented; and Figure 12 is a diagram illustrating an apparatus according to an example. Detailed Description

[0009] According to a first aspect, there is provided a computer-implemented method for determining a location in target medical imaging data at which a given feature is represented, the medical imaging data comprising an array of elements having respective values and representing respective locations, the method comprising: obtaining an initial descriptor for an initial location in the target medical imaging data, the initial descriptor representing the values of the elements of the target medical imaging data located relative to the initial location according to a first predefined pattern; determining an approximate location in the target medical imaging data at which the given feature is represented based on an input of data representing the initial descriptor to a trained machine learning model; determining a plurality of candidate locations in the target medical imaging data based on the approximate location; obtaining a candidate descriptor for each of the plurality of candidate locations, each candidate descriptor representing the values of the elements of the target medical imaging data located relative to the candidate location according to a second predefined pattern; obtaining a reference descriptor for a reference location in reference medical imaging data, the reference medical imaging data comprising one or more sets of reference medical imaging data, for each of the one or more sets of reference medical imaging data, the reference location being the location in that set of reference medical imaging data at which the given feature is represented, and for each of the one or more sets of reference medical imaging data, the reference descriptor representing the values of the elements of that set of reference medical imaging data located relative to the reference location according to the second predefined pattern; for each of the plurality of candidate locations, comparing the reference descriptor and the candidate descriptor for the candidate location to obtain a similarity measure; selecting a candidate location from among the plurality of candidate locations based on the calculated similarity measure; and determining a location in the target medical imaging data at which the given feature is represented based on the selected candidate location.

[0010] Optionally, the method comprises: determining a plurality of further candidate locations in the target medical imaging data based on the determined location in the target medical imaging data at which the given feature is represented, wherein the distance between the further candidate locations is less than the distance between the candidate locations; obtaining a further candidate descriptor for each of the plurality of further candidate locations, each further candidate descriptor representing the values of the elements of the target medical imaging data located relative to the further candidate location according to the second predefined pattern; for each of the plurality of further candidate locations, comparing the reference descriptor and the further candidate descriptor for the candidate location to obtain a further similarity measure; selecting a further candidate location from among the plurality of further candidate locations based on the further similarity measure; and determining an improved location in the target medical imaging data at which the given feature is represented based on the selected further candidate location.

[0011] Optionally, the method includes: performing an image registration process using the position in the target medical imaging data at which the given feature is represented.

[0012] Optionally, the method further includes: using the computer-implemented method of the first aspect to determine the position in the target medical imaging data at which a further feature is represented, wherein performing the registration process includes: performing the registration process using the position in the target medical imaging data at which the further feature is represented.

[0013] Optionally, determining the position at which the given feature is represented includes: determining a selected candidate position as the position in the target medical imaging data at which the given feature is represented.

[0014] Optionally, the one or more sets of reference medical imaging data include multiple sets of reference medical imaging data, and the reference descriptor is an average reference descriptor obtained from the multiple sets of reference medical imaging data.

[0015] Optionally, the average reference descriptor is an average of multiple reference descriptors, each of the multiple reference descriptors corresponding to a reference position in a respective set of the multiple sets of reference medical imaging data and representing the value of an element of the respective set of reference medical imaging data positioned relative to the reference position according to the second predefined pattern.

[0016] Optionally, determining the approximate position includes: generating an initial set of coordinates representing an initial body position in a template body based on an input of data representing the initial descriptor to a trained machine learning model, the initial body position in the template body corresponding to a body position in the body at least partially represented by the target medical imaging data and represented at an initial position in the target medical imaging data; and determining the approximate position based on the initial set of coordinates, a set of feature coordinates representing the position of a given feature in the template body, and the initial position.

[0017] Optionally, determining the approximate position includes: calculating a vector between the initial set of coordinates and the set of feature coordinates; and calculating the approximate position based on the initial position and the vector.

[0018] Optionally, calculating the approximate position includes: adding the vector to a vector representation of the initial position.

[0019] Optionally, the trained machine learning model has been trained to generate, based on an input of data representing a given descriptor, a set of coordinates representing a body position in the template body, the given descriptor representing the values of elements of the given medical imaging data located relative to a given position in the given medical imaging data according to a first predefined pattern, the body position in the template body corresponding to a body position in the body at least partially represented by the given medical imaging data and represented at the given position in the given medical imaging data.

[0020] Optionally, the method includes an improvement process for improving the approximate position, the improvement process including: determining a direction based on the initial set of coordinates and the set of feature coordinates; determining a further initial position in the target medical imaging data based on the initial position and the direction; obtaining a further descriptor for the further initial position, the further descriptor representing the values of elements of the target medical imaging data located relative to the further initial position according to the first predefined pattern; generating, based on an input of data representing the further descriptor to the trained machine learning model, a further initial set of coordinates representing a further initial body position in the template body, the further initial body position in the template body corresponding to a body position in the body at least partially represented by the given medical imaging data and represented at the further initial position in the target medical imaging data; and determining the approximate position based on the further initial set of coordinates, the set of feature coordinates representing the positions of given features in the template body, and the further initial position.

[0021] Optionally, determining the direction includes: determining the direction of a vector between the initial set of coordinates and the set of feature coordinates as the direction.

[0022] Optionally, determining the further initial position includes: adding a vector between the initial set of coordinates and the set of feature coordinates to a vector representation of the initial position.

[0023] Optionally, the trained machine learning model has been trained by a training method comprising: providing a machine learning model configured to generate, based on an input of data representing a given descriptor, a set of coordinates representing a body position in the template body, the given descriptor representing values of elements of the given medical imaging data located relative to a given position in the given medical imaging data according to the first predefined pattern; providing training data comprising a plurality of training descriptors, each training descriptor representing values of elements of a given set of medical imaging data located relative to a given position in the given set of medical imaging data according to the first predefined pattern, and for each training descriptor, the training data further comprising a set of ground truth coordinates associated with the corresponding body position in the template body; and training the machine learning model based on the training data to minimize a loss function between the set of coordinates generated by the machine learning model based on the training descriptors and the corresponding set of ground truth coordinates.

[0024] Optionally, providing the training data includes obtaining one of the plurality of training descriptors by: obtaining a template descriptor for a reference position in template medical imaging data, the reference position in the template medical imaging data being the position at which the corresponding body position is represented, the template descriptor representing values of elements of the template medical imaging data located relative to the reference position according to the second predefined pattern; obtaining a candidate descriptor for each of a plurality of candidate positions in the given set of medical imaging data, each candidate descriptor representing values of elements of the given set of medical imaging data located relative to the candidate position according to the second predefined pattern; for each of the plurality of candidate positions in the given set of medical imaging data, comparing the template descriptor and the candidate descriptor for the candidate position to obtain a template similarity metric; and determining the training descriptor based on the candidate descriptor and the computed template similarity metric.

[0025] According to a second aspect, there is provided an apparatus configured to perform the method of the first aspect.

[0026] According to a third aspect, there is provided a computer program which, when executed by a computer, causes the computer to perform the method of the first aspect.

[0027] Reference Figure 1 , illustrates a computer-implemented method for determining the position in target medical imaging data 330 at which a given feature 226 is represented.

[0028] In Figure 2 , 3Representations of example medical imaging data with respect to which the method can be used are illustrated in FIGS. 5-10. The medical imaging data can be data captured from performing medical imaging on a patient (e.g., computed tomography (CT), magnetic resonance imaging (MRI), X-ray, or other imaging techniques).

[0029] Figure 5a 、 5b and 7 both illustrate representations of reference medical imaging data 220. Figure 2 、 3 、6a, 6b, and 8-10 all illustrate representations of target medical imaging data 330. In each case, the medical imaging data includes an array of elements each having a value. For example, the medical imaging data can include a 2D array of pixels, each pixel having at least one value. As another example, the medical imaging data can include a 3D array of voxels, each voxel having at least one value. The at least one value can correspond to or otherwise represent the output signal of the medical imaging technique used to generate the medical imaging data. For example, for X-ray imaging, the value of an element (e.g., a pixel) can correspond to or represent the extent to which X-rays have been detected at a particular portion of the imaging plane corresponding to the element. As another example, for magnetic resonance imaging, the value of an element (e.g., a voxel) can correspond to or represent the rate at which excited nuclei in the region corresponding to the element return to equilibrium. In some examples, each element can have only one value. However, in other examples, each element can have or otherwise be associated with multiple values. For example, multiple values of a given element can represent the values of corresponding multiple signal channels. For example, each signal channel can represent a different medical imaging signal or characteristic of the imaging subject. In some examples, the at least one value can include an element (e.g., pixel or voxel) intensity value. For example, the output signal from medical imaging can be mapped to pixel or voxel intensity values (e.g., values within a defined range of intensity values). For example, for a grayscale image, the intensity value can correspond to a value within the range 0 to 255, where for example 0 represents a "black" pixel and 255 represents a "white" pixel. As another example, for example, as in the case of USHORT medical image data, the intensity value can correspond to a value within the range 0 to 65536. As another example, in a color image (e.g., where different colors represent different characteristics of the imaging subject), each pixel / voxel can have three intensity values, for example one for each of the red, green, and blue channels. It should be appreciated that other values can be used. In any case, the medical imaging data can be rendered into an image, such as schematically illustrated in Figure 2 、 3 and FIGS. 5-10.

[0030] In the illustrated example, the reference medical imaging data 220 and the target medical imaging data 330 are data captured by performing medical imaging on the same region of different patients using the same modality. However, in some examples, the modality of the medical imaging (i.e., the medical imaging method by which the data was captured) and / or the protocol (i.e., the specific parameters of the given method by which the medical imaging was performed) may be different between the reference medical imaging data 220 and the target medical imaging data 330. Additionally, in some examples, the reference medical imaging data 220 includes a single panoramic "map" medical image of the entire patient's body.

[0031] As can be seen, certain features 226, 242 are represented in both the first medical imaging data 220 and the second medical imaging data 330. The medical fiducials (e.g., carina) are largely used as example features to describe this framework. However, it should be appreciated that in an example, a given feature can be any feature represented in the medical imaging data, e.g., any specific part of the imaging subject (e.g., including internal cavities, etc.).

[0032] Referring again to Figure 1 , broadly overviewed, the method includes: - In step 112, obtain an initial descriptor for an initial position 710 in the target medical imaging data 330, the initial descriptor representing the values of elements of the target medical imaging data 330 positioned relative to the initial position 710 according to a first predefined pattern; - In step 114, based on the input of data representing the initial descriptor to a trained machine learning model 715, determine an approximate position in the target medical imaging data 330 at which a given feature 226 is represented; - In step 122, based on the approximate position, determine a plurality of candidate positions 442 in the target medical imaging data 330; - In step 124, obtain a candidate descriptor for each of the plurality of candidate positions 442, each candidate descriptor representing the values of elements of the target medical imaging data 330 positioned relative to the candidate position according to a second predefined pattern 440; - In step 126, obtain a reference descriptor for a reference position in the reference medical imaging data 220, the reference medical imaging data 220 including one or more sets of the reference medical imaging data 220, for each of the one or more sets of the reference medical imaging data 220, the reference position being the position in the set of the reference medical imaging data 220 at which the given feature 226 is represented, for each of the one or more sets of the reference medical imaging data 220, the reference descriptor representing the values of elements of the set of the reference medical imaging data 220 positioned relative to the reference position according to the second predefined pattern 440; - In step 128, for each of the plurality of candidate locations 442, a reference descriptor and a candidate descriptor for the candidate location are compared to obtain a similarity metric; - In step 130, a candidate location is selected from among the plurality of candidate locations 442 based on the calculated similarity metric; and - In step 132, based on the selected candidate location, a location in the target medical imaging data 330 at which a given feature 226 is represented is determined.

[0033] Accordingly, a technique is provided for determining a location in the target medical imaging data 330 at which a given feature 226 is represented. This can, for example, reduce the burden on a physician in finding a location in the target medical imaging data 330 at which a given feature 226 is represented.

[0034] The method consists of a two-stage process. In a first stage, hereinafter referred to as the BodyGPS method 110, a trained machine learning model 715 is used to determine an approximate location at which a given feature 226 is represented. The BodyGPS method 110 includes steps 112 and 114. In a second stage, hereinafter referred to as the point matching method 120, the approximate location is used to determine a location at which a given feature 226 is represented. The point matching method 120 includes steps 122 to 132.

[0035] As described below, compared to using the point matching method 120 alone, using the BodyGPS method 110 to first estimate an approximate location and subsequently using the point matching method 120 to refine the approximate location reduces the risk of false positive findings of the location of a given feature 226 when the given feature 226 is not represented in the target medical imaging data 330. Additionally, the method is computationally more efficient than using the point matching method 120 alone.

[0036] Furthermore, although the BodyGPS method alone can be used to estimate an approximate location, the additional use of the point matching method 120 can improve the accuracy of the location determination. This is because the machine learning model of the BodyGPS method can output body coordinates with limited accuracy due to the limited granularity of the locations on which the machine learning model has been trained. On the other hand, the point matching method involves directly comparing the target medical imaging data 330 with the reference medical imaging data 220, which can be done at any granularity (e.g., down to the individual pixel level).

[0037] As mentioned above, the method includes: in step 112, obtaining an initial descriptor for an initial location 710 in the target medical imaging data 330. The initial descriptor represents the values of the elements of the target medical imaging data 330 located relative to the initial location 710 according to a first predefined pattern.

[0038] The initial position 710 can be arbitrarily selected. In some examples, the center point of the target medical imaging data 330 is selected as the initial position.

[0039] Alternatively, a medical professional viewing the target medical imaging data 330 can select the initial position 710 by the following operations: examining the target medical imaging data 330, identifying a given feature 226, and selecting a position near where the given feature 226 is represented as the initial position 710.

[0040] In some examples, an initial descriptor can be output for the initial position from a descriptor model applied to the target medical imaging data 330. The descriptor model can be configured to calculate a descriptor for a given position based on the values of elements positioned relative to the given position according to a first predefined pattern.

[0041] The descriptor for a given position can be a vector including multiple entries, each entry representing the value (e.g., intensity value) of an element (e.g., pixel or voxel), and the elements are positioned relative to the given position according to a predefined pattern. The first predefined pattern can be, for example, an e-grid pattern, such as Figure 2 the grid pattern 332 shown in. In some examples, the descriptor can be determined using many such element values (e.g., 100 elements), and correspondingly, the descriptor can be a vector with many entries (e.g., 100 entries). For example, briefly referring to Figure 3 , for illustrative purposes, a medical imaging data set 660 is presented. A complex grid containing a large number of points (shown as white dots) has been applied to the medical imaging data set 660 to determine a descriptor for a given position at the center of the grid. As can be seen, as the distance from the given position increases, the density of the positions in the predefined pattern decreases. A second predefined pattern 440 is used in the point matching method 120, and a first predefined pattern is used in the BodyGPS method 110. These predefined patterns can be the same pattern.

[0042] The descriptor can encode the spatial context of a given position where a given feature 226 is represented and thus can provide a compact representation of the surroundings of the given feature 226.

[0043] It should be appreciated that in some examples, descriptors other than the specific examples described above may be used. For example, in some examples, each entry may represent the value of an element within one or more corresponding ones of a plurality of predefined boxes (i.e., rectangular regions) located relative to a given position according to a first predefined pattern. It should be appreciated that in the case where medical imaging data exists in three spatial dimensions, the term "box" as used herein may refer to a cubic region or volume. In some examples, each entry of the descriptor may represent the value of an element within a corresponding one of a plurality of predefined boxes. For example, each entry of the descriptor may be the average value of the elements within a corresponding one of a plurality of predefined boxes. That is, each entry may be the sum of the values of the elements within a particular box divided by the number of elements included in the box.

[0044] Alternatively, in some examples, Haar-like descriptors may be used, i.e., descriptors where each entry represents the difference between the sums of the element values within each of a plurality of boxes defined in the image data. In some examples, the descriptor may be a gradient descriptor, for example where each entry represents one or more image gradients in a corresponding one of a plurality of regions of the medical image data. For example, the image gradient for a given region may be based on the change in the values (e.g., intensity values) between the elements within that given region. In some examples, the descriptor may be such that each entry is the value of a corresponding one of a plurality of elements randomly distributed in the medical imaging data relative to a given position. In some examples, the descriptor for a given position may be such that each entry is the sum of the values of the elements intersected by a corresponding one of a plurality of randomly oriented rays, each ray originating from that given position. In each case, the descriptor for a given position represents the values of the elements of the medical imaging data located relative to the given position according to a first predefined pattern. Other descriptors may be used.

[0045] As mentioned above, the method includes: in step 114, determining an approximate position in the target medical imaging data 330 at which a given feature 226 is represented, based on an input of data representing an initial descriptor to a trained machine learning model 715. The trained machine learning model 715 may have been trained to determine an approximate position in the target medical imaging data 330 at which a given feature 226 is represented, based on an input of data representing the initial descriptor.

[0046] The given feature 226 may include a medical landmark. Example medical landmarks include the carina of the trachea, the center of the right thumbnail, and the apex (highest point) of the upper lobe of the right lung. Figure 2 The given feature 226 illustrated in the figure is entirely illustrative. It is a landmark that appears in the upper lobe of the right lung.

[0047] Reference Figure 4, an example method performed by a trained machine learning model 715 is illustrated. In this example, determining the approximate location includes: generating an initial set of coordinates 700 representing an initial body position in a template body 705 based on an input of data representing an initial descriptor to the trained machine learning model 715. The template body 705 can be, for example, a human body depicted in reference medical imaging data 220. In Figure 7 an example of the reference medical image data is depicted. The reference medical imaging data 220 includes one or more sets of the reference medical imaging data 220. The reference medical imaging data 220 can consist of, for example, one set of the reference medical imaging data 220, and this set of the reference medical imaging data 220 represents a study on one patient. The patient represented can be selected as an "average male" or "average female" patient; that is, a patient whose body metrics are similar to the population average. The reference medical imaging data 220 can be for a part of the patient, such as their lungs. Alternatively, the reference medical imaging data 220 can include multiple sets of medical imaging data depicting corresponding body parts of one patient. An image registration process can be performed on the sets of medical imaging data to form a single panoramic "map" medical image of the patient. The panoramic image can be a volume image.

[0048] The initial set of coordinates 700 can be considered a general set of coordinates indicating a position in the template body 705. For example, the origin of the coordinate system can be the carina of the trachea. The positive z-axis of the coordinate system can be oriented towards the head. The positive x-axis of the coordinate system can be oriented towards the right arm. The positive y-axis of the coordinate system can be oriented towards the sternum. However, this is entirely exemplary, and spherical or cylindrical coordinates can be used instead of Cartesian coordinates, the origin can be located in any body part, and the coordinate system can have any orientation.

[0049] In this example, the trained machine learning model 715 has been trained to generate a set of coordinates representing a body position in the template body 705 based on an input of data representing a given descriptor. Here, the given descriptor represents the value of an element of the given medical imaging data located according to a first predefined pattern relative to a given position in the given medical imaging data. The body position in the template body corresponds to the body position in the body at least partially represented by the given medical imaging data, at the given position in the given medical imaging data.

[0050] In any case, the initial body position corresponds to the position in the imaged body represented by the initial element. For example, if the initial element (representing the initial position 710 for which the initial descriptor was obtained) represents a point in the right lung of the imaged body, then the initial set of coordinates 700 represents the same point in the right lung of the template body 705.

[0051] An example of training a machine learning model is described below.

[0052] This framework uses the following knowledge: Different bodies depicted in different medical images have similar arrangements of organs, landmarks, and other medical features. Thus, a vector connecting two locations (such as, two landmarks in one body) is likely to be similar (in terms of direction and magnitude) to a vector connecting the same two locations in different bodies. This framework utilizes this knowledge to determine an approximate location where a given feature 226 is represented, as will now be described with reference to Figures 5a to 6b as follows.

[0053] Figure 5a A simplified representation of a template body 705 is shown, on which a set of feature coordinates 730 indicating the location of a given feature 226 in the template body 705 and an initial set of coordinates 700 have been indicated. Figure 6a A portion of target medical imaging data 330 is shown in which an initial location 710 and an approximate location 740 are indicated.

[0054] In this example, determining the approximate location 740 where a given feature 226 is represented includes: determining the approximate location 740 based on the initial set of coordinates 700, the set of feature coordinates 730 indicating the location of the given feature 226 in the template body 705, and the initial location 710 (which is the location described by the initial descriptor).

[0055] The set of feature coordinates 730 is, for example, a set of coordinates indicating the location of a given feature 226 in the right lung of the template body 705. In this case, it is desired to determine the location of this feature of the (imaged) right lung in the target medical imaging data 330.

[0056] In this example, the given feature 226 is a feature selected by the user. The set of feature coordinates 730 can be obtained from a database (not shown) that stores the set of feature coordinates in association with the corresponding feature (e.g., a medical landmark) represented in the template body 705. The database can be used to extract the set of feature coordinates based on the selection of the given feature 226.

[0057] Alternatively, in some examples, the method includes: comparing the initial set of coordinates 700 with the set of feature coordinates stored in the database to identify the feature in the template body 705 that is closest to the initial set of coordinates 700. The set of feature coordinates associated with this closest feature can be selected as the set of feature coordinates 730 to be used.

[0058] In some examples, determining the approximate location includes: calculating a vector 720 between an initial coordinate set 700 and a feature coordinate set 730; and then calculating the approximate location based on the initial location 710 and the vector 720. The vector 720 between the initial coordinate sets 700 can be calculated by subtracting the initial coordinate set 700 from the feature coordinate set 730. The approximate location can be calculated by adding the vector 720 to the vector representation of the initial location 710 in the target medical imaging data 330. For example, if the initial coordinate set 700 is (0.5, 0, -0.2) and the coordinate set of a given feature 226 in the template body 705 is (0.4, 0, 0.7), then the vector 720 (-0.1, 0, 0.9) can be added to the vector representation of the initial location 710 in the target medical imaging data 330. The target medical imaging data 330 can include a grid of voxels arranged along three axes, and in this case, the vector representation of the initial location 710 can be, for example, a vector of the position of the voxel corresponding to the initial location 710 relative to these axes.

[0059] In some cases, the determined approximate location can be a location outside the scope of the target medical imaging data 330. For example, in a case where the target medical imaging data 330 depicts only the abdomen and the feature to be found is a landmark on the foot, the determined approximate location may be a location not represented in the target medical imaging data 330. In such a case, the method can include: determining that the approximate location is outside the imaged part of the imaged body; and outputting data indicating that the approximate location is outside the imaged part of the imaged body. For example, the processor 502 executing the method can output an indication to a display device (not shown) that the approximate location is outside the imaged part of the imaged body, such as a message stating "The feature is not included in this image".

[0060] As described herein, the point matching method alone may cause false positive indications of the location of a given feature 226. However, by using the BodyGPS method 110 before using the point matching method as described in the examples herein, this risk of generating false positive indications of the location of the given feature 226 can be eliminated or reduced.

[0061] In some examples, the trained machine learning model 715 directly outputs the vector 720 between the initial coordinate set 700 and the feature coordinate set 730, rather than outputting the initial coordinate set 700.

[0062] Because the arrangement of the set of feature coordinates 730 and the set of initial coordinates 700 in the template body 705 (i.e., the vector 720 separating them) is similar to the arrangement of the position of the given feature 226 and the initial position 710 in the target medical imaging data 330 (i.e., the vector 720 separating them), the approximate position determined by adding the vector 720 to the initial position 710 provides an approximate position 740 at which the given feature 226 is represented. Accordingly, in some examples, this can be regarded as the approximate position in the target medical imaging data 330 at which the given feature 226 is represented.

[0063] However, as can be seen in Figure 5a and 6a , the lungs in the template body 705 are smaller than the lungs in the imaged body. Thus, the magnitude of the vector 720 used is an underestimate, and the determined approximate position is not exactly where the given feature 226 is represented in the imaged body.

[0064] To mitigate this, in some examples, the BodyGPS method 110 includes: performing an improvement process for improving the approximate position, which is described with reference to Figure 5b and 6b . Figure 6b Illustrates Figure 6a a portion of the target medical imaging data 330 shown in Figure 6a and illustrates the first iteration of the improvement process. Figure 5b Illustrates Figure 5a a portion of the reference medical imaging data 220 shown in Figure 5a and illustrates the first iteration of the improvement process.

[0065] The improvement process includes: determining a direction 721 based on the set of initial coordinates 700 and the set of feature coordinates 730. This can include: determining the direction of the vector 720 between the set of initial coordinates 700 and the set of feature coordinates 730 as the direction 721. However, any direction from the set of initial coordinates 700 towards the set of feature coordinates 730 can be used.

[0066] The improvement process includes: determining a further initial position 740 in the target medical imaging data 330 based on the initial position 710 and the direction 721. The further initial position 740 is positioned relative to the initial position 710 in the calculated direction 721. In including Figure 6bIn some examples of the example shown, determining a further initial position 740 includes adding a vector 720 to an initial position 710; that is, the further initial position 740 is the same as a previously calculated approximate position 740. However, in other examples, a vector having a calculated direction 721 but a predetermined step size (e.g., ten voxels) may be used. That is, the further initial position 740 may be located a fixed distance away from the initial position 710 in the direction 721 of the vector 720. This can provide a robust approximate position determination in examples where the scale of the template body 705 does not match the scale of the target medical imaging data 330.

[0067] The refinement process includes: obtaining a further descriptor for the further initial position 740. The further initial descriptor represents the values of the elements of the target medical imaging data 330 located relative to the further initial position 740 according to a first predefined pattern. The further initial descriptor may be calculated in a manner similar to the initial descriptor.

[0068] The refinement process includes: based on an input of data representing the further descriptor to a trained machine learning model 715, generating a further initial coordinate set 750 representing a further initial body position in the template body 705. Similar to the initial body position, the further initial body position corresponds to a position in the imaged body represented at the further initial position 740 in the target medical imaging data 330 (e.g., a position near a given feature 226 as shown in Figure 6a and 6b ).

[0069] The refinement process further includes: determining an approximate position based on the further initial coordinate set 750, a feature coordinate set 730 representing the position of a given feature in the template body 705, and the further initial position 740. In a manner similar to the way described above, the further initial coordinate set 750 and the feature coordinate set 730 can be used to determine a refined approximate position. For example, the refinement process may include: determining a further vector 760, which is a vector 760 between the further initial coordinate set 750 and the feature coordinate set 730. The further vector 760 can be added to the vector representation of the further initial position 740 to determine the refined approximate position.

[0070] Further iterations of the refinement process can be performed. After a fixed number of iterations have been performed, the refinement process can stop. Alternatively, in some examples, the refinement process can stop in response to determining that a criterion has been met. For example, at each iteration, the distance between the approximate position before the iteration and the refined approximate position after the iteration can be compared to the criterion. The criterion can include a threshold, such as a predefined number of pixels or voxels. The method can include: in response to the comparison, determining the refined approximate position as the approximate position to be used as an input to the point matching method 120. For example, if the distance is below the threshold, the method can include: determining the refined approximate position after the iteration as the approximate position to be used as an input to the point matching method 120. Alternatively and particularly in examples where a predefined step size is used, a different criterion can be used. For example, the refined approximate position is used to perform a second iteration of the refinement process to reach a further refined approximate position. The vector between the approximate position before the first iteration and the refined approximate position after the first iteration can be compared to the vector between the refined approximate position and the further refined approximate position. If the angle between the two vectors is greater than a threshold (e.g., 90 degrees), the method can include: determining the further refined approximate position as the approximate position to be used as an input to the point matching method 120. Stopping the refinement process when the criterion is met can improve the computational efficiency of the BodyGPS system while maintaining the accuracy in the output approximate position.

[0071] In any case, the BodyGPS method 110 includes: determining an approximate position in the target medical imaging data 330 at which a given feature 226 is represented. The approximate position is then used as an input to the point matching method 120.

[0072] The BodyGPS method 110 alone can provide an approximate position in the medical image at which a given feature 226 (e.g., a landmark) is represented. Additionally, the BodyGPS method 110 can avoid false positive determinations of the position of the given feature 226, as described above. However, even when performing the refinement process described above, the trained machine learning model also has limited precision in mapping the position to a body position in the template body. To mitigate this, the present framework uses the determined approximate position as an input to the point matching method 120, which involves directly comparing descriptors obtained from the target medical imaging data 330 to the reference medical imaging data 220.

[0073] Now refer Figures 7 to 9 to describe the point matching method 120.

[0074] As mentioned, the method includes: at step 122, determining a plurality of candidate positions in the target medical imaging data 330 based on the approximate position at which a given feature 226 is represented in the target medical imaging data 330.

[0075] Positions that are less than a predefined distance from the approximate position can be selected as the plurality of candidate positions 442. For example, in the case where the scale of the target medical imaging data 330 is known, the plurality of candidate positions 442 can be selected to be at a distance, e.g., less than 2 cm, from the approximate position. Alternatively, the plurality of candidate positions 442 can be selected to be at a distance less than a predefined number of pixels or voxels from the approximate position. The plurality of candidate positions 442 can include a 3x3 square grid of candidate positions centered on the approximate position, as Figure 9 shown.

[0076] As mentioned, at step 124, the method includes: obtaining a candidate descriptor for each of the plurality of candidate positions 442.

[0077] Each candidate descriptor represents the value of an element of the target medical imaging data 330 positioned relative to the candidate position according to a second predefined pattern 440. The same descriptor model that is applied to the target medical imaging data 330 to generate an initial descriptor for an initial position can be applied to the target medical imaging data 330 to generate a candidate descriptor for each of the plurality of candidate positions 442. For example, referring to Figure 7 and 8 , the grids 222, 228 and the positions of the elements used to calculate the candidate descriptors for each of those candidate positions 340 relative to each candidate position 340 are the same as the grid 332 and the positions of the elements used to calculate the initial descriptor for the initial position.

[0078] As mentioned, the method includes: at step 126, obtaining a reference descriptor for a reference position in the reference medical imaging data 220.

[0079] The reference medical imaging data 220 can be the reference medical imaging data 220 that forms the template body 705 described above in the BodyGPS method. However, in some examples, it is different from the medical imaging data used in the BodyGPS method.

[0080] In this example, the reference position is the position in the set of reference medical imaging data 220 that represents a given feature 226. For each of the one or more sets of reference medical imaging data 220, the reference descriptor represents the value of an element of the set of reference medical imaging data 220 that is positioned relative to the reference position according to a second predefined pattern 440. The same descriptor model that is applied to the target medical imaging data 330 to generate an initial descriptor for the initial position can be applied to the reference medical imaging data 220 to generate a reference descriptor for the reference position.

[0081] In some examples, the reference descriptor can be obtained from a database (not shown). For example, the descriptor for the reference position may have been calculated (e.g., by applying a descriptor model) and stored in the database, e.g., associated with the given feature 226 and the reference position. For example, the database can store multiple reference descriptors in association with the corresponding reference positions in the medical imaging data on which the reference descriptors were determined and the corresponding features represented by the reference positions. Accordingly, in some examples, the method can include: selecting a given feature 226 from among the plurality of features; and extracting a reference descriptor associated with the given feature 226. The given feature 226 can be selected by the user and the database for extracting the reference descriptor. Alternatively, in examples where the BodyGPS method includes determining the feature closest to the set of initial coordinates 700, the identified feature can be used to identify the reference position and the reference descriptor.

[0082] In some examples, the one or more sets of reference medical imaging data 220 include multiple sets of reference medical imaging data 220. In these examples, for each of the one or more sets of reference medical imaging data 220, the reference position is the position in the set of reference medical imaging data 220 that represents a given feature 226, and the reference descriptor is an average reference descriptor obtained from the multiple sets of reference medical imaging data 220. The inventors have recognized that using an average reference descriptor obtained from multiple sets of reference medical imaging data 220 can improve the accuracy of the determined position that represents a given feature 226. In Figure 11 The results of this are illustrated. The multiple sets of reference medical imaging data 220 can be sets representing reference medical imaging data 220 of different patients. This can help mitigate the impact of anatomical deviations of a patient in a set of reference medical imaging data 220 from typical body metrics in a single set of medical imaging data on the comparison in step 128. Alternatively, the multiple sets of medical imaging data can be sets of reference medical imaging data 220 related to the same patient. This can help mitigate the impact of noise in the reference medical imaging data 220 on the comparison. In any case, the multiple sets of reference medical imaging data 220 can include a given feature 226. An image registration process can be performed to transform the multiple sets of reference medical imaging data 220 into a single coordinate system. This will reduce the impact of, for example, slightly different orientations of the sets of medical imaging data.

[0083] In one example, the average reference descriptor is the average of multiple reference descriptors. Each of the multiple reference descriptors is a reference descriptor for a reference position in a corresponding one of the multiple sets of reference medical imaging data 220. Each of the multiple reference descriptors represents the value of an element of the corresponding set of reference medical imaging data 220 positioned relative to the reference position according to the second predefined pattern 440.

[0084] In another example, the average reference descriptor is a reference descriptor for the "average image" of the multiple sets of medical imaging data. For example, for each of a plurality of element (e.g., pixel / voxel) positions, the multiple sets of medical imaging data can represent values, such as intensity values. An image registration process can be performed on the multiple sets of medical imaging data such that each set of medical imaging data is represented in the same coordinate system. For each element position of the registered sets of medical imaging data, the values of the different sets can be averaged to obtain an average value. The "average image" is an image including the average value for each element position.

[0085] As mentioned, the method includes: in step 128, for each of the multiple candidate positions 442, comparing the reference descriptor and a candidate descriptor for the candidate position to obtain a similarity metric.

[0086] In some examples, the similarity metric can include a normalized mutual information similarity between the reference descriptor and the candidate descriptor. In some examples, other similarity metrics between a first descriptor and a second descriptor can be used. For example, alternatively or additionally, cosine similarity, Euclidean distance, and / or cross-correlation can be used.

[0087] As mentioned, the method includes: in step 130, selecting a candidate location from among the plurality of candidate locations 442 based on the calculated similarity metric; and in step 132, determining, based on the selected candidate location, the location in the target medical imaging data 330 at which a given feature 226 is represented.

[0088] In some examples, selecting a candidate location can include: selecting, from among the similarity metrics of the plurality of candidate locations 442, 340, the candidate location having the similarity metric indicating the highest degree of similarity. For example, the candidate location having the highest mutual information similarity metric can be selected. In some examples, determining the location at which a given feature 226 is represented includes: determining the selected candidate location as the location in the target medical imaging data 330 at which a given feature 226 is represented. In Figure 9 the example shown, the approximate location for determining the candidate location is the candidate location having the highest similarity metric and is thus selected as the selected candidate location.

[0089] In some examples, the selected candidate location 448 can be regarded as the location in the target medical imaging data 330 at which a given feature 226 is represented. However, in other examples, the determined location can be refined in successively finer-grained patterns (see, for example, Figure 9 the pattern of black circles 444 in ) by defining further candidate locations 444 (based on the selected candidate location 448). Accordingly, an accurate and efficient determination of the location in the target medical imaging data 330 at which a given feature 226 is represented can be provided.

[0090] Specifically, in these examples, the method includes: determining a plurality of further candidate locations 444 in the target medical imaging data 330 based on the determined location in the target medical imaging data 330 at which a given feature 226 is represented. For example, the further candidate locations can be defined as locations in a region of the selected candidate location. For example, the further candidate locations can be defined as locations in a region (i.e., its local region) of the selected candidate location 448. For example, the candidate locations can be arranged in a square 3x3 grid centered on the approximate location, and the further candidate locations can be arranged in a smaller square 3x3 grid based on the previously determined location at which a given feature 226 is represented.

[0091] The method may then include: obtaining a further candidate descriptor for each of the plurality of further candidate locations 444 in the target medical imaging data 330, each further candidate descriptor representing the value of an element of the second medical imaging data 330 located relative to a corresponding further candidate location 444 according to a second predefined pattern 440. For example, the further candidate descriptor may be the same as the reference and candidate descriptors described above, i.e., it may have been calculated in the same manner as the reference and candidate descriptors described above.

[0092] The method may then include: for each of the plurality of further candidate locations 444, comparing the reference descriptor and the further candidate descriptor for the candidate location to obtain a further similarity measure. For example, the further similarity measure may be the same as the similarity measure used between the reference descriptor and the candidate descriptor, i.e., it may be calculated in the same manner as the similarity measure used between the reference descriptor and the candidate descriptor. The method may then include: selecting a further candidate location from among the plurality of further candidate locations 444 based on the further similarity measure. For example, as Figure 9 illustrated, the further candidate location 446 having the highest similarity measure may be selected. The method may then include: determining, based on the selected further candidate location 446, the location in the target medical imaging data 330 at which a given feature 226 is represented.

[0093] In some examples, the selected further candidate location 446 may be regarded as the location in the target medical imaging data 330 at which a given feature 226 is represented. However, in other examples, the determined location may be further refined by: defining yet further candidate locations based on the selected further candidate location 446; and repeating the method described above for these yet further candidate locations.

[0094] In any case, determining the location in the target medical imaging data 330 at which a given feature 226 is represented. In some examples, the method may further include: generating output data indicating the determined location in the target medical imaging data 330 at which a given feature 226 is represented.

[0095] For example, the output data can include coordinates or pixel / voxel indices corresponding to the determined location 446 within the target medical imaging data 330. In some examples, the output data can further include a reference to the target medical imaging data 330 in which the location has been determined. In some examples, the output data can include the target medical imaging data 330 itself (or a portion thereof). In some examples, the output data can include an image (or data for an image) in which an indicator indicating the determined location is superimposed on a rendering of the target medical imaging data 330. For example, as Figure 10 illustrated in, a representation of the target medical imaging data 330 is shown, and superimposed on (or otherwise included in) the representation is an indicator 550 indicating the location in the target medical imaging data 330 at which a given feature 226 is represented. In this example, the indicator is a box centered on the determined location. However, in other examples, other indicators can be used, such as: a marker or dot or other symbol superimposed at (or otherwise included at) the determined location; or for example, an arrow or pointer or other label that points to or connects to or otherwise indicates the determined location. The output data can allow providing an indication to a user (e.g., a physician) of the determined location in the target medical imaging data 330 at which a given feature 226 is represented. In some examples, the output data can be stored in a storage device. For example, the output data can be stored in a database. This can allow the output data to be accessed, for example, by a user or by an automated downstream process (not shown).

[0096] In some examples, the method can include: transmitting the output data to a display device to display a representation of the target medical imaging data 330 and an indicator 550 indicating the determined location at which a given feature 226 is represented on the representation of the target medical image data. For example, the display device (not shown) can be a computer monitor of a computer or other display screen. For example, the displayed representation can be similar to Figure 10 the representation shown in, in which the indicator 550 is superimposed on the representation of the second medical imaging data 330. Any of the example indicators mentioned above can be used. This can allow a user (e.g., a physician) to immediately and easily appreciate the location in the target medical imaging data 330 at which a given feature 226 is represented, which can for example allow making an assessment based on the given feature 226 more quickly and with minimal burden.

[0097] In some examples, the method includes: performing an image registration process using the location in the target medical imaging data 330 at which a given feature 226 is represented. For example, the location at which a given feature 226 is represented can be used as one of the set of transformations of the target medical imaging data 330 to a coordinate system. For example, the target medical imaging data 330 can be registered with a reference medical image (such as the reference medical imaging data 220) showing the given feature 226 using the location in the target medical imaging data 330 at which the given feature 226 is represented and the location in the reference medical image at which the given feature 226 is represented.

[0098] The registration process described above can be made as accurate as desired by the following operation: adjusting the number of iterations used in the hierarchical point matching method 120 to determine the location at which a given feature 226 is represented. If computational efficiency is desired, the number of iterations can be reduced, and if higher image registration accuracy is desired, the number of iterations can be increased.

[0099] In some examples, the method includes: performing steps 112 to 132 to determine the location in the target medical imaging data 330 at which a further feature is represented. For example, a further feature in the database described above that provides a correspondence between features and their locations in the reference medical imaging data 220 can be used as a basis for performing steps 112 to 132 to determine the location in the target medical imaging data 330 at which the further feature is represented. Multiple locations of features in the target medical imaging data 330 can be determined in this way. The registration process can include: using the location in the target medical imaging data 330 at which the further feature is represented. This can improve the accuracy of the image registration process.

[0100] Using the hierarchical method described above can allow the location to be determined accurately.

[0101] It is possible to use the point matching method to determine the location without first identifying an approximate location using BodyGPS. For example, instead of determining candidate locations based on an approximate location, the candidate locations can initially be spread across a uniform coarse grid spanning the entire target medical image. The iteration of the point matching method 120 can be applied by specifically using the selected candidate locations determined at one iteration of the point matching method 120 to determine further candidate locations for the next iteration, so that the area of the grid that can contain the given feature 226 gradually narrows. After a certain number of iterations, the selected candidate location can be identified as the location at which the given feature 226 is represented.

[0102] However, testing the entire target medical imaging data 330 in this undirected manner would require a large number of iterations of the point matching method 120 to accurately determine the location. At each iteration, many comparisons between the candidate descriptor for that iteration and the reference descriptor would be required. This is computationally expensive. Additionally, as previously discussed, since this method would involve determining the location in the image whose descriptor has the highest similarity to the reference descriptor, this method may produce false positive indications of the location where a given feature 226 is represented - a location where the given feature 226 is not actually represented, yet is the location in the target medical imaging data 330 whose descriptor is most similar to the reference descriptor.

[0103] The present framework addresses the above problems by using the BodyGPS method 110 to provide an approximate location on which to determine candidate locations. Since the BodyGPS method 110 can determine whether a given feature is represented in the target medical imaging data 330, the risk of false positive indications of the location of the given feature 226 is reduced compared to using the point matching method 120 alone. Additionally, since the BodyGPS method 110 provides an approximate location of landmarks on which to initiate the point matching method 120, the area or volume of the target medical imaging data on which the point matching method 120 is applied can be significantly reduced. In other words, the number of iterations of the point matching method 120 required to achieve a particular level of accuracy for the location can be significantly reduced by first using the BodyGPS method 110 to determine the approximate location. Thus, the present framework can be more computationally efficient than using the point matching method 120 alone.

[0104] Additionally, although the BodyGPS method 110 can be applied alone to determine an approximate location where a given feature 226 is represented, the accuracy and / or precision of this approximate location may be limited, for example due to the limited granularity of the body positions on which the machine learning model was trained. However, the point matching method 120 involves directly comparing the descriptor of the candidate location determined from the approximate location to the reference descriptor. This can be performed at any granularity (e.g., down to the individual pixel level), which can improve the accuracy and / or precision of the determined location.

[0105] Figure 11 A graph is illustrated showing the accuracy of each of several methods for determining the location in medical imaging data where the carina is represented. The horizontal axis indicates the error in the determined location, measured as the distance in millimeters between the determined location and the true location of the carina. The vertical axis indicates the fraction of the tested set of medical imaging data for which the error is less than the error indicated on the horizontal axis.

[0106] Curve 910 indicates the accuracy of the approximate position generated by BodyGPS method 110, where BodyGPS method 110 uses 24 iterations of the refinement process (in other words, a total of 25 iterations for the BodyGPS method). As can be seen in the figure, for approximately 74% of the tested set of medical imaging data, the approximate position determined by BodyGPS method 110 is accurate to 6 mm. Curve 920 indicates the accuracy of the position obtained by using four iterations of the hierarchical point matching method 120 alone. For approximately 86% of the tested set of medical imaging data, the position determined using this method is accurate to 6 mm.

[0107] Curve 930 indicates the accuracy of the position obtained by using BodyGPS method 110 and 24 iterations of the refinement process (in other words, a total of 25 iterations for the BodyGPS method) to determine the approximate position and then using one iteration of the point matching method 120 to refine the position, where the reference descriptor used for the point matching method 120 is obtained from a single set of medical imaging data. For approximately 91% of the tested set of medical imaging data, the position determined using this method is accurate to 6 mm. This experiment thus demonstrates that the combination of BodyGPS method 110 and point matching method 120 produces a more accurate method than either BodyGPS method 110 alone or one iteration of the point matching method 120.

[0108] Curve 940 indicates the accuracy of the position obtained by using BodyGPS method 110 to determine the approximate position and then using the point matching method 120 to refine the position (as in curve 930), but where the reference descriptor used for the point matching method 120 is obtained by averaging the descriptors from multiple sets of medical imaging data in the manner described above. For approximately 96% of the tested set of medical imaging data, the position determined using this method is accurate to 6 mm. This experiment thus demonstrates that using an average reference descriptor obtained from multiple sets of medical imaging data produces a more accurate method than using a reference descriptor obtained from a single set of medical imaging data.

[0109] An example method for training the trained machine learning model 715 used in BodyGPS method 110 is described below.

[0110] The training method includes: providing a machine learning model configured to generate a set of coordinates representing a body position in the template body 705 based on an input of data representing a given descriptor. The given descriptor represents the value of an element of the given medical imaging data located relative to a given position in the given medical imaging data according to a first predefined pattern.

[0111] The machine learning model may include, for example, a convolutional neural network. The machine learning model may perform a dimensionality reduction operation, such as an average pooling operation, on any medical imaging data input thereto. Multiple layers of the convolutional neural network may be used to process the dimensionally reduced medical imaging data. The convolutional neural network may be configured to output a set of coordinates. The convolutional neural network may be configured to: apply a sigmoid function to the set of coordinates obtained by processing the dimensionally reduced medical imaging data using the multiple layers to output a set of coordinates each having a reduced range, such as 0 to 1.

[0112] The training method includes: providing training data including a plurality of training descriptors. Each training descriptor represents the value of an element of the given set of medical imaging data located according to a first predefined pattern relative to a given position in the given set of medical imaging data. For each training descriptor, the training data further includes a set of ground truth coordinates associated with a corresponding body position in the template body 705.

[0113] The training descriptors may be obtained from corresponding sets of training medical imaging data. The training descriptors may be obtained using the same descriptor model used to obtain the initial descriptor, the reference descriptor, and the candidate descriptor.

[0114] In some examples, the point matching method 120 may be used to determine the training descriptors. In these examples, providing the training data includes: obtaining one of the plurality of training descriptors by first obtaining a template descriptor for a reference position in the template medical imaging data. The template medical imaging data may be the same as the reference medical imaging data used in the point matching method 120. The reference position is the position in the template medical imaging data at which the corresponding body position is represented. Here, the corresponding body position is the body position associated with the set of ground truth coordinates. The template descriptor represents the value of an element of the template medical imaging data located according to a second predefined pattern 440 relative to the reference position. The template descriptor may be obtained using the same descriptor model used to obtain the initial descriptor, the reference descriptor, and the candidate descriptor.

[0115] In these examples, obtaining the training descriptors includes: obtaining a candidate descriptor for each of a plurality of candidate positions 442 in the given set of medical imaging data. Each candidate descriptor represents the value of an element of the given set of medical imaging data located according to a second predefined pattern 440 relative to the candidate position. The candidate descriptors mentioned here may be obtained using the same descriptor model used to obtain the initial descriptor and the reference descriptor and in a manner similar to that described above with reference to step 124. The candidate positions may be located in a predefined pattern throughout the given set of medical imaging data; for example, the candidate positions may be positioned to cover the nodes of a 3x3 square grid of the given set of medical imaging data.

[0116] In these examples, obtaining a training descriptor includes: for each of the plurality of candidate locations 442 in a given set of medical imaging data, comparing a template descriptor with a candidate descriptor for the candidate location to obtain a template similarity metric. This can be performed in a manner similar to that described above with reference to step 128.

[0117] In these examples, obtaining a training descriptor includes: determining a training descriptor based on the candidate descriptor and the computed template similarity metric. For example, the candidate descriptor having the highest similarity metric with the template descriptor can be selected as the training descriptor.

[0118] The candidate location of the selected candidate descriptor can be used to determine further candidate locations near the selected candidate location to find further candidate descriptors having an even higher similarity with the template descriptor, as described above in the hierarchical point matching method 120.

[0119] The training method can include: determining a set of ground truth coordinates based on a reference location in the template medical imaging data. For example, in the case where the template medical imaging data (which can be the same as the reference medical imaging data 220) is a panoramic image, the set of ground truth coordinates can be determined as the reference location in the panoramic image.

[0120] The above method for obtaining a training descriptor can be used to obtain all training descriptors. The corresponding reference location in the template medical imaging data can be used as a basis for performing the point matching method 120 to obtain training descriptors from a given set of medical imaging data. In some examples, for each set of ground truth coordinates, a plurality of training descriptors can be obtained from the corresponding plurality of medical images.

[0121] The training method includes: training a machine learning model based on training data to minimize a loss function between a set of coordinates generated by the machine learning model based on the training descriptor and the corresponding set of ground truth coordinates. The loss function can include, for example, a function that measures the Euclidean distance or cosine distance between the set of coordinates generated by the machine learning model based on the training descriptor and the corresponding set of ground truth coordinates. The loss function can be any increasing function of one of these distances.

[0122] Thus, the machine learning model can be trained to generate, based on an input of data representing a given descriptor, a set of coordinates representing a body location in the template body 705.

[0123] This method of training a machine learning model does not require any supervision. Instead of manually annotating the positions in a set of training medical imaging data where a given feature 226 (e.g., a landmark) is represented and obtaining a training descriptor for that landmark, the point matching method 120 is used to automatically determine the training descriptor for a set of ground truth coordinates. This training method thus improves the efficiency of training a machine learning model for determining the approximate position in medical imaging data where a given feature 226 (e.g., a landmark) is represented.

[0124] In an example where the trained machine learning model 715 directly outputs a vector 720 between the initial coordinate set 700 and the feature coordinate set 730 in step 114, the training process can be modified by providing a ground truth vector between a coordinate set associated with a body position represented by a reference position and a desired body position, instead of the ground truth coordinate set. For example, an arbitrary vector can be subtracted from the ground truth coordinate set determined using the method described above to obtain an offset position, and a descriptor can be obtained for the offset position. The training process will include: training the machine learning model to minimize a loss function between the vector generated by the machine learning model based on the descriptor for the offset position and the corresponding ground truth vector.

[0125] Reference Figure 12 , illustrates an apparatus 990 according to an example. The apparatus 990 includes an input interface 996, an output interface 998, a processor 992, and a non-transitory memory device 994. The processor 992 and the memory device 994 can be configured to execute a method according to any of the examples described above with reference to Figures 1 to 10 described. The memory device can store computer-readable program code or instructions which, when executed by the processor 992, cause the processor 992 to execute a method according to any of the examples described above with reference to Figures 1 to 10 described. The instructions can be stored on any one or more computer-readable media (e.g., any one or more non-transitory computer-readable media).

[0126] For example, the input interface 996 can receive the target medical imaging data 330 and data representing a given feature 226. The processor 992 can implement a method according to any of the examples described above with reference to Figures 1 to 9 described, and the processor 992 can output, via the output interface 998, data indicating the determined position in the second medical imaging data 330 where the given feature 226 is represented, e.g., the output data as described above with reference to Figure 10 described. In some examples, the output data can be transmitted to a storage (not shown) that implements a database, such that the output data is stored in the storage. In some examples, the output data can be transmitted to a display device (not shown) to allow a user to inspect the output data, e.g., as described above with reference toFigure 10 As described. In some examples, alternatively or additionally, the output data may be stored in the memory 994.

[0127] The apparatus 990 may be implemented as a processing system and / or a computer. It should be appreciated that the methods according to any of the examples described above Figures 1 to 10 are computer-implemented methods, and these methods may be implemented by the apparatus 990.

[0128] The above examples should be understood as illustrative examples of the present invention. It should be understood that any feature described with respect to any one example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example in the examples or any combination of any other examples in the examples. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the present invention as defined in the appended claims.

Claims

1. A computer-implemented method for determining a location in target medical imaging data at which a given feature is represented, the target medical imaging data comprising an array of elements having respective values ​​and representing respective locations, the method comprising: obtaining an initial descriptor for an initial position in the target medical imaging data, the initial descriptor representing a value of an element of the target medical imaging data positioned relative to the initial position according to a first predefined pattern; determining an approximate location in the target medical imaging data at which the given feature is represented based on inputting data representing the initial descriptor to the trained machine learning model; Based on the approximate position, determining a plurality of candidate positions in the target medical imaging data; obtaining a candidate descriptor for each of the plurality of candidate positions, each candidate descriptor representing a value of an element of the target medical imaging data positioned relative to the candidate position according to a second predefined pattern; obtaining a reference descriptor for a reference location in reference medical imaging data, the reference medical imaging data comprising one or more sets of reference medical imaging data, the reference location being, for each of the one or more sets of reference medical imaging data, a location in the set of reference medical imaging data at which the given feature is represented, the reference descriptor representing, for each of the one or more sets of reference medical imaging data, a value of an element of the set of reference medical imaging data positioned relative to the reference location according to the second predefined pattern; For each of the plurality of candidate positions, comparing the reference descriptor with a candidate descriptor for the candidate position to obtain a similarity measure; selecting a candidate position from among the plurality of candidate positions based on the similarity measure; as well as A location in the target medical imaging data at which the given feature is represented is determined based on the selected candidate location.

2. The computer-implemented method of claim 1 , further comprising: determining a plurality of further candidate positions in the target medical imaging data based on the determined positions in the target medical imaging data at which the given feature is represented, wherein distances between the further candidate positions are smaller than distances between the candidate positions; obtaining a further candidate descriptor for each of the plurality of further candidate positions, each further candidate descriptor representing a value of an element of the target medical imaging data positioned relative to the further candidate position according to the second predefined pattern; for each of the plurality of further candidate positions, comparing the reference descriptor with a further candidate descriptor for the candidate position to obtain a further similarity measure; selecting a further candidate position from among the plurality of further candidate positions based on the further similarity measure; and A refined position in the target medical imaging data at which the given feature is represented is determined based on the selected further candidate position.

3. The computer-implemented method of claim 1 , further comprising: An image registration process is performed using the location in the target medical imaging data at which the given feature is represented.

4. The computer-implemented method of claim 3, further comprising: A location in the target medical imaging data at which a further feature is represented is determined, wherein performing the image registration process comprises performing the image registration process using the location in the target medical imaging data at which the further feature is represented.

5. The computer-implemented method of claim 1 , wherein determining the location at which the given feature is represented comprises: The selected candidate position is determined as the position in the target medical imaging data at which the given feature is represented.

6. The computer-implemented method of claim 1, wherein the one or more sets of reference medical imaging data include multiple sets of reference medical imaging data, and the reference descriptor is an average reference descriptor obtained from the multiple sets of reference medical imaging data.

7. The computer-implemented method of claim 1 , wherein determining the approximate location comprises: generating, based on input of data representing the initial descriptors to the trained machine learning model, an initial set of coordinates representing initial body positions in a template body, the initial body positions in the template body corresponding to body positions in a body at least partially represented by the target medical imaging data represented at the initial positions in the target medical imaging data; and The approximate position is determined based on the initial set of coordinates, a set of feature coordinates representing the position of a given feature in the template body, and the initial position.

8. The computer-implemented method of claim 7, wherein determining the approximate location comprises: calculating a vector between the initial coordinate set and the feature coordinate set; and calculating the approximate position based on the initial position and the vector.

9. The computer-implemented method of claim 8, wherein calculating the approximate location comprises: Add the vector to the vector representation of the initial position.

10. The computer-implemented method of claim 7, further comprising performing a refinement process for refining the approximate position.

11. The computer-implemented method of claim 10, wherein the improving process comprises: Determining a direction based on the initial coordinate set and the feature coordinate set; determining a further initial position in the target medical imaging data based on the initial position and the direction; obtaining a further descriptor for the further initial position, the further descriptor representing a value of an element of the target medical imaging data positioned relative to the further initial position according to the first predefined pattern; generating a further initial set of coordinates representing a further initial body position in the template body based on input of data representing the further descriptor to the trained machine learning model, the further initial body position in the template body corresponding to a body position represented at the further initial position in the target medical imaging data in a body at least partially represented by the given medical imaging data; and The approximate position is determined based on the further initial set of coordinates, a set of feature coordinates representing the position of a given feature in the template body, and the further initial position.

12. The computer-implemented method of claim 11 , wherein determining the direction comprises: The direction of the vector between the initial coordinate set and the feature coordinate set is determined as the direction.

13. The computer-implemented method of claim 7, wherein the trained machine learning model has been trained by a training method comprising: providing a machine learning model configured to generate a set of coordinates representing body locations in the template body based on an input of data representing a given descriptor representing a value of an element of the given medical imaging data positioned relative to a given location in the given medical imaging data according to the first predefined pattern; providing training data comprising a plurality of training descriptors, each training descriptor representing a value of an element of a given set of medical imaging data positioned relative to a given position in the given set of medical imaging data according to the first predefined pattern, the training data further comprising, for each training descriptor, a set of real coordinates associated with a corresponding body position in the template body; as well as The machine learning model is trained based on the training data to minimize a loss function between a set of coordinates generated by the machine learning model based on the training descriptors and a corresponding set of true coordinates.

14. The computer-implemented method of claim 13, wherein providing the training data comprises obtaining one of the plurality of training descriptors by: obtaining a template descriptor for a reference position in the template medical imaging data, the reference position in the template medical imaging data being a position in the template medical imaging data at which the corresponding body position is represented, the template descriptor representing a value of an element of the template medical imaging data positioned relative to the reference position according to the second predefined pattern; obtaining a candidate descriptor for each of a plurality of candidate positions in the given set of medical imaging data, each candidate descriptor representing a value of an element of the given set of medical imaging data positioned relative to the candidate position according to the second predefined pattern; for each of the plurality of candidate locations in the given set of medical imaging data, comparing the template descriptor with a candidate descriptor for the candidate location to obtain a template similarity measure; as well as The training descriptor is determined based on the candidate descriptors and the template similarity measure.

15. An apparatus for determining a location at which a given feature is represented in target medical imaging data, the target medical imaging data comprising an array of elements having respective values ​​and representing respective locations, the apparatus comprising: a non-volatile memory device for storing computer readable program code; as well as a processor in communication with the non-transitory memory device, the processor operating with the computer readable program code to perform steps comprising: obtaining an initial descriptor for an initial position in the target medical imaging data, the initial descriptor representing a value of an element of the target medical imaging data positioned relative to the initial position according to a first predefined pattern; determining an approximate location in the target medical imaging data at which the given feature is represented based on inputting data representing the initial descriptor to the trained machine learning model; Based on the approximate position, determining a plurality of candidate positions in the target medical imaging data; obtaining a candidate descriptor for each of the plurality of candidate positions, each candidate descriptor representing a value of an element of the target medical imaging data positioned relative to the candidate position according to a second predefined pattern; obtaining a reference descriptor for a reference location in reference medical imaging data, the reference medical imaging data comprising one or more sets of reference medical imaging data, the reference location being, for each of the one or more sets of reference medical imaging data, a location in the set of reference medical imaging data at which the given feature is represented, the reference descriptor representing, for each of the one or more sets of reference medical imaging data, a value of an element of the set of reference medical imaging data positioned relative to the reference location according to the second predefined pattern; For each of the plurality of candidate positions, comparing the reference descriptor with a candidate descriptor for the candidate position to obtain a similarity measure; selecting a candidate position from among the plurality of candidate positions based on the similarity measure; as well as A location in the target medical imaging data at which the given feature is represented is determined based on the selected candidate location.

16. The apparatus of claim 15, wherein the processor operates with the computer readable program code to perform further steps comprising: determining a plurality of further candidate positions in the target medical imaging data based on the determined positions in the target medical imaging data at which the given feature is represented, wherein distances between the further candidate positions are smaller than distances between the candidate positions; obtaining a further candidate descriptor for each of the plurality of further candidate positions, each further candidate descriptor representing a value of an element of the target medical imaging data positioned relative to the further candidate position according to the second predefined pattern; for each of the plurality of further candidate positions, comparing the reference descriptor with a further candidate descriptor for the candidate position to obtain a further similarity measure; selecting a further candidate position from among the plurality of further candidate positions based on the further similarity measure; and A refined position in the target medical imaging data at which the given feature is represented is determined based on the selected further candidate position.

17. The apparatus of claim 15, wherein determining the location at which the given feature is represented comprises: The selected candidate position is determined as the position in the target medical imaging data at which the given feature is represented.

18. The apparatus of claim 15, wherein the one or more sets of reference medical imaging data comprises a plurality of sets of reference medical imaging data, and the reference descriptor is an average reference descriptor obtained from the plurality of sets of reference medical imaging data.

19. The apparatus of claim 15, wherein determining the approximate location comprises: generating, based on input of data representing the initial descriptors to the trained machine learning model, an initial set of coordinates representing initial body positions in a template body, the initial body positions in the template body corresponding to body positions in a body at least partially represented by the target medical imaging data represented at the initial positions in the target medical imaging data; and The approximate position is determined based on the initial set of coordinates, a set of feature coordinates representing the position of a given feature in the template body, and the initial position.

20. One or more non-transitory computer-readable media embodying instructions executable by a machine to perform operations for determining a location at which a given feature is represented in target medical imaging data, the target medical imaging data comprising an array of elements having respective values ​​and representing respective locations, the operations comprising: obtaining an initial descriptor for an initial position in the target medical imaging data, the initial descriptor representing a value of an element of the target medical imaging data positioned relative to the initial position according to a first predefined pattern; determining an approximate location in the target medical imaging data at which the given feature is represented based on inputting data representing the initial descriptor to the trained machine learning model; Based on the approximate position, determining a plurality of candidate positions in the target medical imaging data; obtaining a candidate descriptor for each of the plurality of candidate positions, each candidate descriptor representing a value of an element of the target medical imaging data positioned relative to the candidate position according to a second predefined pattern; obtaining a reference descriptor for a reference location in reference medical imaging data, the reference medical imaging data comprising one or more sets of reference medical imaging data, the reference location being, for each of the one or more sets of reference medical imaging data, a location in the set of reference medical imaging data at which the given feature is represented, the reference descriptor representing, for each of the one or more sets of reference medical imaging data, a value of an element of the set of reference medical imaging data positioned relative to the reference location according to the second predefined pattern; For each of the plurality of candidate positions, comparing the reference descriptor with a candidate descriptor for the candidate position to obtain a similarity measure; selecting a candidate position from among the plurality of candidate positions based on the similarity measure; as well as A location in the target medical imaging data at which the given feature is represented is determined based on the selected candidate location.

Images