Body Expression

Abstract

A method for training a machine learning system to represent a patient's body includes obtaining a plurality of stored medical imaging datasets, each medical imaging dataset representing at least a portion of a respective patient, wherein a first medical imaging dataset of the plurality of stored medical imaging datasets represents a different portion of the patient's body than a second medical imaging dataset of the plurality of stored medical imaging datasets. Estimating a plurality of landmarks in the stored medical imaging datasets, and aligning each stored medical imaging dataset to a predefined pose using the plurality of landmarks to generate an aligned medical imaging dataset, sampling a plurality of points in the aligned medical imaging datasets, and training a machine learning system based on at least the plurality of points to configure parameters of the machine learning system. The parameters of the machine learning system are then stored and used in a method for inferring a body representation.

Description

Technical Field

[0001] The present invention relates to processing medical image data to determine a representation of the body and in particular to improving the accuracy of such a representation. Background Art

[0002] Medical imaging techniques can be used to obtain information about the human or animal body. For example, X-ray images can be used to detect and classify bone fractures. In another example, magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound imaging can be used to image body tissues and monitor changes over time.

[0003] The image data captured by each of these modalities can be used to detect abnormalities in bone and / or tissue and enable snapshots to be taken at a given time. These snapshots can be analyzed to locate and classify abnormalities. In some examples, it may be desirable to accurately aim the imaging. This is particularly important where the imaging modality involves exposing the patient to radiation to capture image data. In such examples, in order to provide the desired image quality, it is desirable to optimize the amount of radiation each patient is exposed to. This can be achieved by determining an accurate body representation and for other purposes such as estimating the patient's weight or the volume of an organ or bone.

[0004] Therefore, in some imaging modalities, a representation of the human body may be used to target radiation to desired areas.

[0005] It is an object of the present invention to address at least some of the difficulties of providing accurate body representation. Summary of the invention

[0006] According to a first aspect of the present invention, there is provided a computer-implemented method for training a machine learning system for representing a patient's body, comprising obtaining a plurality of stored medical imaging datasets, each medical imaging dataset representing at least a portion of a corresponding patient, wherein a first medical imaging dataset of the plurality of stored medical imaging datasets represents a different portion of the patient's body compared to a second medical imaging dataset of the plurality of stored medical imaging datasets, estimating a plurality of landmarks in the stored medical imaging datasets, aligning each of the stored medical imaging datasets to a predefined pose using the plurality of landmarks to generate an aligned medical imaging dataset, sampling a plurality of points in the aligned medical imaging datasets, training a machine learning system based on the plurality of points to configure parameters of the machine learning system; and storing at least the parameters of the machine learning system.

[0007] Training a machine learning system for representing a patient's body based on aligned medical imaging data enables the patient's body shape and size to be quickly and efficiently determined based on existing medical imaging data sources. This enables an estimate of a representation of the patient's body that allows accurate imaging of the patient. In some examples where the imaging modality involves exposing the patient to potentially harmful radiation, accurate targeting of the radiation quanta required for imaging reduces the patient's overall exposure to radiation. It should be appreciated that other uses may be apparent, some of which are described below.

[0008] Preferably, the stored medical imaging data set is at least one of computed tomography data, magnetic resonance imaging data, x-ray imaging data, and ultrasound imaging data. This enables the machine learning system to be trained using different types of imaging data, as each imaging device may require the patient to be positioned differently when imaging the patient.

[0009] Preferably, sampling a plurality of points comprises determining, for each of the plurality of points, a distance to a nearest surface represented in the medical imaging dataset. This enables information associated with the point to be determined and used to assess the authenticity of the neural network output.

[0010] Training the machine learning system includes adjusting at least one body representation based on at least one of the plurality of points and a corresponding distance to a nearest surface. This enables evaluation of the plurality of points and enables the machine learning system to adjust the body representation.

[0011] The at least one body representation may be an n-dimensional vector. This enables information about the body representation that is adjusted as part of the training process of the machine learning system to be efficiently stored for analysis by one or more algorithms.

[0012] Optionally, estimating the plurality of landmarks comprises applying at least one landmark detection algorithm to the stored medical imaging dataset.This enables efficient processing of the medical imaging dataset to obtain landmark positions based on previously analyzed medical imaging datasets.

[0013] Each landmark detection algorithm can be a machine learning algorithm trained to identify one or more landmarks of the human body. This enables the use of different landmarks to be identified using different algorithms dedicated to detecting each specific landmark, thereby improving efficiency.

[0014] The landmarks may be three-dimensional anatomical landmarks. The use of three-dimensional anatomical landmarks enables more accurate positioning and orientation of medical imaging data to a predefined pose.

[0015] Optionally, aligning the stored medical imaging data sets includes: identifying at least one first medical data landmark in the first medical imaging data set; identifying at least one second medical data landmark in the second medical imaging data set, wherein the first medical data landmark and the second medical data landmark represent corresponding landmarks; and aligning the first medical imaging data set and the second medical imaging data set based on the first medical data landmark and the second medical data landmark. This enables multiple sets of medical imaging data sets (sometimes representing different parts of a patient's body) to be aligned before being combined and aligned to a predefined pose, thereby providing additional detail for use during the training process.

[0016] The machine learning system (440) includes at least one neural network. This enables training the neural network based on the plurality of points and measured distances to optimize the network and body representation.

[0017] According to a second aspect of the invention, there is provided a computer-implemented method for inferring a human body representation, the method comprising: receiving image data captured using an imaging modality; estimating a plurality of landmarks in the image data representing at least a portion of a patient; aligning the image data to a predefined pose using the plurality of landmarks to generate aligned image data; sampling a plurality of points in the aligned image data; analyzing the plurality of points using a trained machine learning system, the machine learning system being configured to represent the patient's body; inferring a human body representation based on the analysis; and outputting the inferred human body representation.

[0018] Inferring a body representation based on a trained machine learning system enables inferring an accurate representation of the body, thereby enabling accurate imaging of the patient. In some examples where the imaging modality involves exposing the patient to potentially harmful radiation. Accurate targeting of the radiation quanta required for imaging reduces the patient's overall exposure to radiation. It should be appreciated that other uses may be apparent, some of which are described below.

[0019] Optionally, the image data is captured by an imaging device, the image data being at least one of computed tomography data, magnetic resonance imaging data, x-ray imaging data, ultrasound data, point cloud data, and depth data. This enables inferring a body representation for different types of image data, as each imaging device may require the patient to be positioned differently when imaging the patient.

[0020] Sampling a plurality of points may include, for each of the plurality of points, determining a distance to the nearest surface represented in the image data, thereby determining a plurality of distances. This enables information associated with the point to be determined and used to assess the authenticity of the neural network output.

[0021] Preferably, inferring the body representation comprises iteratively providing the body representation and the plurality of points to a trained machine learning system, and comparing a corresponding output of the machine learning system with a plurality of distances associated with the plurality of points. This enables the output of the machine learning system to be validated based on the plurality of points and based on the associated distances measured from the image data of the provided body representation.

[0022] Optionally, the method may further include adjusting the body representation based on a comparison between the corresponding output and a plurality of distances associated with the plurality of points. This enables the body representation to be adjusted based on the output of the machine learning system to determine whether a more accurate body representation is available based on the plurality of points and the corresponding measured distances.

[0023] The body representation may be an n-dimensional vector. This enables information about the body representation to be stored efficiently for analysis by one or more algorithms.

[0024] Preferably, estimating a plurality of landmarks comprises applying at least one landmark detection algorithm to the image data.This enables efficient processing of the image data to obtain landmark positions based on previously analyzed medical imaging data.

[0025] Each landmark detection algorithm can be a machine learning algorithm trained to identify one or more landmarks of the human body. This enables the use of different landmarks to be identified using different algorithms dedicated to detecting each specific landmark, thereby improving efficiency.

[0026] The landmarks may be three-dimensional anatomical landmarks. The use of three-dimensional anatomical landmarks enables more accurate positioning and orientation of medical imaging data to a predefined pose.

[0027] Optionally, aligning the image data includes identifying at least one first data landmark in the received image data, identifying at least one second data landmark associated with a predefined gesture, wherein the first data landmark and the second data landmark represent corresponding landmarks, and aligning the first image data and the predefined gesture based on the first data landmark and the second data landmark. This enables multiple sets of image data to be aligned before being combined and aligned to the predefined gesture, thereby providing additional detail in inferring the body representation.

[0028] Preferably, the machine learning system is trained according to the first aspect.

[0029] According to a third aspect of the present invention, there is provided an apparatus for inferring body representation, the system comprising: a training interface configured to obtain stored medical imaging data; a training unit configured to train at least one machine learning system based on the stored medical imaging data; an inference interface configured to receive image data captured using an imaging modality; and an inference unit configured to use the trained machine learning system to infer body representation based at least on the captured image data.

[0030] Inferring a body representation based on a trained machine learning system enables inferring an accurate representation of the body, thereby enabling accurate imaging of the patient. In some examples where the imaging modality involves exposing the patient to potentially harmful radiation. Accurate targeting of the radiation quanta required for imaging reduces the patient's overall exposure to radiation. It should be appreciated that other uses may be apparent, some of which are described below.

[0031] According to a fourth aspect of the present invention, a system for inferring a body representation is provided, the system comprising: an imaging device configured to capture image data; a storage device for storing a medical imaging data set; and at least one processor arranged to perform the methods described above with respect to the first and second aspects, and arranged to obtain the stored medical imaging data set from the storage device, as well as the image data captured by the imaging device.

[0032] According to a fifth aspect of the present invention, there is provided a non-transitory computer-readable storage medium, comprising a computer-readable instruction set stored thereon, which, when executed by at least one processor, enables the processor to perform the method described above with respect to the first and second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above-mentioned properties, features and advantages of the present invention and the manner of achieving the same will become more apparent and understandable with the following description of embodiments of the present invention with reference to the accompanying drawings, in which the same reference numerals are used to designate the same features.

[0034] Figure 1 is a schematic representation of a medical imaging apparatus 1 according to an example;

[0035] Figure 2 is a flow chart illustrating a method of training a machine learning system for representing a patient's body according to an example;

[0036] Figure 3 is a flow chart illustrating a method for inferring body representation from examples;

[0037] Figure 4 is a schematic representation of an apparatus according to an example;

[0038] Figure 5 is a representation of a machine learning system according to an example; and

[0039] Figure 6 is a schematic representation of a system according to an example. DETAILED DESCRIPTION

[0040] Figure 1 A medical imaging device 100 according to an aspect of the invention is shown, comprising a data processing unit 135 .

[0041] As an example, a computer tomography apparatus for a medical imaging apparatus 100 is shown without limiting the general inventive concept.

[0042] The imaging device 100 has a stage 120. The stage 120 has a fixed support frame 121. The stage 120 has a rotor 124 mounted so that it can rotate by a pivot bearing device. The imaging device 100 has an image recording area 104 formed by a tunnel-shaped opening 109. The area of ​​the object to be imaged can be arranged in the image recording area 104.

[0043] The imaging device 100 has a patient positioning device 110. The patient positioning device 110 has a positioning table 111 for positioning a patient 113 and a transfer plate 112. The transfer plate 112 is arranged on the positioning table 111 so that it can be moved relative to the positioning table 111 so that the transfer plate 112 can be introduced into the image recording area 104 in the longitudinal direction of the transfer plate 112.

[0044] Radiation projection devices 126, 128 are arranged on the rotor 124. The radiation projection devices 126, 128 have a radiation source 126 designed to emit radiation quanta and a detector 128 designed to detect the radiation quanta. Radiation quanta 127 can pass from the radiation source 126 to the area to be imaged and, after interacting with the area to be imaged, hit the detector 128. In this way, a projection profile of the area to be imaged can be detected.

[0045] By rotating the radiation projection devices 126, 128 around the image recording region, at least one projection profile can be detected for different arrangements of the radiation source 126 and the detector relative to the region of the object to be imaged. A plurality of projection profiles can form a projection data set. A medical image, in particular a tomographic medical image, of the region to be imaged can be reconstructed based on the projection data provided by the image reconstruction unit 136.

[0046] The imaging device 100 has a control device 130 for controlling the imaging device 100. The imaging device 100 also has an input unit 138 for inputting control information, such as imaging parameters and examination parameters, and an output unit 139 for outputting control information and images, in particular, a medical image with a segmentation result obtained by a method according to an aspect of the present invention.

[0047] The control device 130 may be, for example, a computer and / or a data processing system. The control device 130 includes a computer-readable medium 132 , a processor 134 , a data processing unit 135 and an image reconstruction unit 136 .

[0048] Aiming the radiation source 126 at an area of ​​interest, such as an abnormality, reduces the radiation quanta delivered to the patient 113. To do this, the control device 130 needs to know the positioning and body shape of the patient 113. Typically, the control device 130 uses a rough but body representation based on body data obtained from a variety of sources. For example, this rough body representation can be formed by a plurality of cylinders, such as a large cylinder for the body and four smaller cylinders for the patient's limbs, each of which can be adjusted by parameters such as the patient's age, height, and weight.

[0049] In other examples, a rough body representation is estimated based on a statistical body shape model, such as shape completion and animation of a model of a person, which is based on SCAPE's data-driven human shape model. However, such models do not accurately represent clinical settings. For example, while such models are based on a human body in a standing position, when a patient (such as Figure 1 This is not the case for the patient 113 as shown in FIG. 1 . Figure 1 As shown in the patient 113 in FIG, the patient typically lies with his or her back flat on a table, whereas the model typically used represents a standing body representation with the back somewhat bent. Although some adjustments can be made to simulate bone changes and fat tissue deformation, the adjustments are prone to error due to the uncertainty inherent in these simulations.

[0050] Given the accuracy required during imaging, in particular the accuracy of the target radiation quantum, or when performing other functions related to the imaging of the above-mentioned patient, it is desirable to ensure that the most accurate body representation is used when aiming and setting other system parameters (such as pulse frequency in MRI). This facilitates image acquisition and mitigates unwanted side effects. The body representation can be a representation of the human or animal body. There is a large amount of high-quality medical imaging data available, which is associated with many patient characteristics. Importantly, the medical imaging data also represents the patient in the required positioning, thereby improving the accuracy of the body representation generated therefrom. However, such medical imaging data is partial and only represents a part / segment of the body. Therefore, it cannot be used alone to generate a mesh representing the entire body or a part of the body representation as required. Further, in the case of assuming that the data represents a part of the body, the point cloud representation will be undesirable in view of its unstructured nature because it has an undefined topology and the associated post-processing will be complex. Similarly, given the high resolution of such data to generate representations using voxels and / or grids, the complexity and required processing power are increased.

[0051] In order to generate high-quality and accurate body representations from such medical imaging data, in an embodiment of the present invention, machine learning techniques are used to estimate patient body representations. This provides the benefits of efficiency, accuracy, and scalability. Such high-quality and representative medical imaging data can be used to train a machine learning system for representing the patient's body, which can then be used to estimate the patient's body shape. The body representation adjusted based on the training of the machine learning system and inferred using the machine learning system can be a model or template of a specific type of body. For example, the body representation can be associated with specific characteristics of the patient, such as body type, gender, weight, height. In some examples, the type of patient can be a characteristic, i.e., whether the patient is a human or an animal. This estimate can then be used in treatment, such as to accurately aim the radiation quantum.

[0052] Figure 2 It is shown that a machine learning system trained to represent a patient's body is used - such as the following reference Figure 5 A flow chart of the method 200 of the machine learning system 440 is described. At box 210, multiple medical imaging data sets are obtained. The medical imaging data sets may be data captured from multiple patients over time using one or more imaging devices of the same or different imaging modalities. Further, the medical imaging data sets may represent different parts of the patient's body. The medical imaging data set may be a collection of previously captured medical imaging data captured using different imaging modalities. For example, one medical imaging data set may be CT imaging data of a specific patient with a specific type of body.

[0053] Further, the medical imaging data can be grouped by patient characteristics, such as age, gender, weight, height, so that multiple sets within the medical imaging data set are grouped together and used when training a machine learning system based on those patient characteristics. The data in the medical imaging data set can be represented in point coordinates and distance to the nearest surface format, which allows the use of different kinds of data, whether they represent a part of the patient's body or the entire patient's body, regardless of the original imaging modality. Further, the medical imaging data set can be obtained from a CT scanner, an MRI scanner, an x-ray product, and / or ultrasound, however, it should be appreciated that other imaging devices and modalities can be used to collect such medical imaging data. The medical imaging data represents a plurality of different patient bodies, and as mentioned above, may represent only a portion of the patient, such as the abdomen, head, or other body regions. Although it should be appreciated that the medical imaging data can be obtained from a single type of imaging device, such as a CT scanner, it should also be appreciated that the medical imaging data can be obtained from a plurality of different types of imaging devices. The medical imaging data can be obtained from a storage device of a control system, or from a remote server via a network such as the Internet. For example, medical imaging data may be obtained from a data management system, such as an EMR (electronic medical record), a PACS (picture archiving and communication system), a LIS (laboratory information system), a RIS (radiology information system), and / or a CVIS (cardiovascular information system).

[0054] Once the medical imaging data has been acquired, the method 200 proceeds to block 220 where a plurality of landmarks are estimated in the stored medical imaging data. The landmarks may be associated with the portion of the body representation represented by the medical imaging data and may represent internal or external landmark points. For example, the location of the identifier in the organ shape may be estimated based on the CT medical imaging data, however, it should be appreciated that other landmarks may be used based on the portion represented by the medical imaging data. Such landmarks may be determined based on identifiable points of the anatomical object and the imaging modality used.

[0055] To detect the locations of the first and second landmark points, a landmark detection machine learning algorithm can be applied to the medical imaging data. In some examples, the landmark detection machine learning algorithm is based on a deep reinforcement learning model that is trained to detect specific landmarks within the medical imaging data based on the imaging modality used. An artificial intelligence agent associated with the landmark detection machine learning algorithm is arranged to detect landmark points and can be trained via the deep reinforcement learning model. The artificial intelligence agent can be trained to learn anatomical structures present in the medical imaging data and navigate towards landmarks.

[0056] A deep reinforcement learning model used to train an artificial intelligence agent can be represented by a tuple {S, A, T, r}, where:

[0057] S is the state of the agent. The state of the agent may correspond to a portion of the medical imaging data centered at point p. p represents the current position of the agent within the medical imaging data.

[0058] A is the set of actions performed by the agent. Since the agent is trained to locate landmarks within medical imaging data, A may represent different displacements along each Cartesian axis, x-axis and y-axis (and z-axis when the medical imaging data is three-dimensional) associated with the medical imaging data.

[0059] t is , which represents the random transition function, describing the probability of arriving in a given state after a given action. For example, if the agent moves according to action A, what is the probability of reaching the landmark.

[0060] r is the reward collected by the agent by interacting with the environment. r can be defined as:

[0061]

[0062] where p c represents the current position of the agent, p gt is the ground truth position of the landmark it is pursuing, p n is the next position of the agent after performing the action. As such, the closer the agent moves towards the goal, the higher the reward, and if the agent moves away from the goal, the reward has a negative value.

[0063] The agent can be represented by a fully convolutional neural network with a given number of layers. Given a specific path within the medical imaging data, the network can output six actual values ​​corresponding to Q values ​​representing the long-term rewards of the actions. In this example, the Q value represents the evaluation of the actions in the action set A. The maximum Q value returned indicates the direction to move in the next step. Therefore, assuming that the agent has been trained on the correct anatomy, in order to collect the maximum reward, the agent will always move closer to the location of the target landmark and then stop at the location of the target landmark. The above example is an example of how landmarks can be detected. It should be appreciated that other methods of detecting landmarks can be used, including using other deep learning methods such as convolutional neural networks.

[0064] Once a plurality of landmarks have been estimated in the medical imaging data, the method 200 proceeds to box 230, where the medical imaging data is aligned to a predefined pose based on the estimated landmarks. In some examples, where there are multiple medical imaging data sets representing the same body part, before both the landmarks in the first item of medical imaging data and the landmarks in the second item of medical imaging data are aligned to the predefined pose, the plurality of representations may then be aligned so that the two are aligned. A predefined pose is a general representation of a body in a desired pose. A predefined pose may be a canonical pose, such as a pose that a patient is expected or required to adopt when being imaged. For example, in the case of CT imaging, a canonical pose may be a pose adopted by a patient lying on a table. Each of the estimated landmarks is aligned with a corresponding landmark on the predefined pose so that each item of the medical imaging data is positioned in a desired orientation and position. For example, a predefined pose may have a landmark that represents a given feature of a person, such as the position of an eye. Thus, a landmark representing an eye in the medical imaging data may be identified and used to orient and position the medical imaging data, thereby aligning the medical imaging data to the predefined pose. Since each set of medical imaging data represents a portion of a desired representation, multiple sets of medical imaging data may be combined to represent a predefined gesture.

[0065] Assuming that different medical imaging data sets can be used to construct a body representation of a patient, each medical imaging data set may not be at the same scale, and / or multiple medical imaging data sets may represent the same or substantially similar parts of a patient. Therefore, in some examples, it is desirable to apply an affine transformation to the medical imaging data. This involves rotating, scaling, and / or translating the medical imaging data relative to a predefined posture so that landmarks are aligned with predefined locations associated with the predefined posture. Further, it may be necessary to scale the medical imaging data by normalizing the data so that related medical imaging data sets can be aligned to a single body representation based on patient characteristics, and / or in order to provide information associated with different types of patients.

[0066] Normalizing the medical imaging data includes determining the point farthest from the center of an object represented in the medical imaging data and setting that distance to 1. The distance from the center to all other points can be represented based on that distance.

[0067] In view of this, in cases where there are multiple medical imaging datasets for a given portion of a body representation, the alignment of the medical imaging datasets may not be substantially perfect. In such an example, averaging may be applied such that in cases where the medical imaging datasets have similar landmarks representing the same point on a predefined posture - which landmarks are not substantially perfectly aligned - then the landmark locations may be averaged. Additionally or alternatively, in cases where the alignment of landmarks within the medical imaging datasets is substantially imperfect, a closest fit / best match alignment may be chosen with the goal of reducing the error between the predefined posture and the detected landmarks.

[0068] After the medical imaging data is aligned with the predefined pose, in box 240, points are sampled. Sampling the points includes randomly selecting multiple points for comparison against the aligned medical data. The points can be represented as [x, y, z] and a signed distance from the point to the nearest surface. The signed distance can represent the location of the point, such that a positive distance represents a point outside the surface boundary and a negative distance represents a point within the surface boundary. In this way, the sign of the distance will change at the surface boundary. As mentioned above, assuming that multiple sets of medical imaging data are aligned with the predefined pose, it is possible to sample points across multiple medical imaging data, such as by obtaining three-dimensional coordinates of the points, which are based on the medical imaging data that can be used to train a machine learning system.

[0069] After sampling, the machine learning system 440 is trained at block 250, such as with reference to Figure 5The machine learning system described herein is a machine learning system. The machine learning system is trained based on sample points and the corresponding distance from each point to the nearest body surface. The machine learning system may include: a deep neural network that can be used, which is trained using the points and the corresponding distances to adjust parameters and a body representation that is initialized in a random manner and adjusted throughout the training process. Instead of a neural network, or together with a neural network, it should be appreciated that the machine learning system may include any regression model, such as a random forest or any other appropriate algorithm or method. The machine learning system can be initialized with a random body representation, which together with the parameters of the machine learning system will be refined / configured based on the sample points and the corresponding distances to the nearest body surface. The random body representation can be an n-dimensional representation of the body. During training, the machine learning system is optimized by modifying one or more parameters and optimizing the random body representation based on the sample points and the associated distances. By refining the random body representation, the machine learning system is able to receive the sample points and output a signed distance value to the nearest surface. The signed distance value can be verified based on a known distance value associated with each sample point. Based on the success or failure of the verification, one or more parameters of the machine learning system can be adjusted accordingly. The parameters may be individual weights and / or biases associated with neurons of a neural network of the machine learning system, which are refined along with the body representation with the goal of providing a closer match between the measured distance value and the calculated distance value output by the machine learning system. This may be repeated for each sampling point, such that the body representation and parameters of the machine learning system are refined. The trained machine learning system may then be used to determine the body representation based on the received data, as will be described below in conjunction with Figure 3 Describing.

[0070] Once the machine learning system has been trained, the adjusted parameters of the machine learning system are stored in a storage device associated with a system configured for training the machine learning system.

[0071] Figure 3 is a flow chart illustrating a method 300 for inferring a body representation according to an example. In box 310, image data is received. The image data may be medical data captured by a medical imaging device such as a CT scanner, or may be depth data or point cloud data. It should be appreciated that the image data may be any three-dimensional data that can indicate the positioning and shape of a patient, and in some examples may indicate the internal anatomical structure of a patient. The image data in this example is captured using an imaging modality via an imaging device, such as a CT scanner, an MRI scanner, an X-ray product, and / or ultrasound. However, it should be appreciated that other imaging devices and modalities may be used to collect such image data. The image data may be three-dimensional medical imaging data.

[0072] At block 320, the received image data is analyzed to estimate the location of one or more landmarks. Much like block 220 in method 200, the estimation of the above-described landmarks may be performed using a machine learning algorithm such as a deep reinforcement learning model. The landmarks may be internal landmarks representing points within the patient's body, or they may be external landmarks representing points on the patient's body. At block 330, the estimation of a plurality of these landmarks enables alignment of the image data received from the imaging device to the image used to generate the above-described image of the patient's body. Figure 2 Further, at block 340, a plurality of points are sampled on the image data to determine point coordinates and signed distances to the nearest body surface in the image data.

[0073] At block 350, once a plurality of points have been sampled, an analysis is performed based on the sampled points and a trained machine learning system. The trained machine learning system may be a system using the above reference Figure 2 The method 200 described herein generates a machine learning system in which previously recorded medical imaging data is analyzed to determine a plurality of landmarks and then aligned to a predefined pose. The machine learning system may be as described below with reference to Figure 5 A machine learning system 440 is described. Figure 2 The method 200 is used to train a machine learning system based on sampling points and body representations that are refined during a training process. The analysis of the sampling points includes analyzing the sampling point coordinates calculated as part of the sampling process, determining signed distances and body representations using the trained machine learning system. The trained machine learning system can be a deep learning network, a random forest, or a regression model, although it should be appreciated that any machine learning system can be utilized. The trained machine learning system receives the sampling point coordinates and the randomized body representation, the machine learning representation uses the sampling points and the randomized body representation to determine the signed distances, and refines the body representation based on the signed distances output by the machine learning system. If the signed distances calculated based on the sampling points and the medical imaging are substantially similar to the signed distances determined by the machine learning system, the current body representation input to the machine learning algorithm is selected. Otherwise, the body representation is adjusted, and the process is repeated until the output of the machine learning system is substantially similar to the determined signed distances. In this manner, the process can be iterated until the best body representation is selected. Once the analysis has been performed, at box 360, an estimated body representation associated with the patient is inferred by, for example, selecting a body representation that causes the machine learning system to output signed distances based on sample points that are closely aligned with the calculated signed distances associated with the sample points.

[0074] The inferred body representation may be output to a control device or an application on a control device, such as the one described above with respect to Figure 1The inferred body representation may be used as input to one or more of a number of different implementations, including a further machine learning system arranged to determine a variety of information used when imaging the patient. Figure 2 The body representation generated by the training process described is used to train further machine learning systems. For example, one application is to use the inferred body representation within the imaging process to accurately target the radiation quantum, because the patient's body representation accurately reflects their characteristics, such as height, weight, and positioning during the medical imaging process in particular. The body representation is an n-dimensional vector that captures relevant information and can be used during the imaging process to guide the imaging process as described above. This improves the efficiency of any medical imaging performed subsequently. Other example applications include automatic and accurate ISO calculation in CT imaging; automatic scan range estimation in CT, MR and X-ray imaging; automatic patient registration based on height, weight and posture in CT, MR and X-ray imaging; automatic and accurate specific absorption rate parameter estimation in MR imaging; automatic and accurate radiation dose calculation in CT and X-ray imaging; and pre-operative relative to intra-operative data registration for augmented / virtual reality applications.

[0075] In a further example, the image data obtained in block 310 may be fed back to Figure 2 Method 200 to further refine the machine learning system.

[0076] Figure 4 is a schematic representation of an apparatus 400 according to an example. The apparatus 400 may form part of a system, as will be described below with respect to Figure 6 described above, and can achieve the above Figure 2 and Figure 3 Method 200 and method 300 described in .

[0077] The apparatus 400 is arranged to receive stored medical imaging data 410, which, as described above, is a plurality of sets of medical imaging data previously captured and representing different types of patients. The stored medical imaging data 410 may be classified according to patient characteristics. The stored medical imaging data 410 is obtained through a training interface 420, which in some examples may be connected to a storage device (not shown) storing the stored medical imaging data 410. The training interface 420 may be configured to communicate with a storage device that may be external to the apparatus 400 via a wired or wireless connection.

[0078] The apparatus 400 further includes a training unit 430 configured to train at least one machine learning system based on the stored medical imaging data 410 obtained via the training interface. The training unit 430 is configured to perform the above Figure 2 The method 200 described herein is performed to train at least one machine learning system 440. The trained machine learning system 440 may be stored in a storage device (not shown), which may be the same storage device as the storage device configured to store the stored medical imaging data 410.

[0079] In addition, the apparatus 400 is also configured to receive image data 450 captured using an imaging modality. The imaging modality may be associated with an imaging device (not shown) that is configured to capture image data using the imaging modality. The image data 450 is received by an inference interface 460. Once received, the imaging data 450 is processed by an inference unit 470 that is configured to apply the above-described Figure 3 The method 300 is described, and uses a machine learning system 440 trained by a training unit 430 to infer a body representation based on at least the captured imaging data 450.

[0080] Although described above and Figure 4 The apparatus 400 shown in FIG. 4 shows that both training and inference are performed in the same apparatus, but it should be appreciated that training and inference can be performed by separate apparatuses. Similarly, although the apparatus 400 shows separate generation interface 420 and training interface 460, it should be appreciated that these can be a single interface configured to interact with a storage device (not shown) and an imaging device (not shown).

[0081] Figure 5 is a representation of a machine learning system 440 according to an example. The machine learning system 440 can be trained to provide a body representation. Figure 2 As mentioned in method 200, a machine learning system 440 is trained based on a plurality of points 442 sampled from a plurality of medical imaging data sets. Each of the plurality of points 442 has a corresponding measured signed distance (not shown) based on the relationship of the point to the medical imaging data set (i.e., a ground truth). For example, the measured signed distance can be the distance to the nearest surface of an object represented in the medical imaging data set. The measured signed distance can be measured or calculated, for example, using one or more image processing methods, such as a fast marching method. Further, although Figure 5The example of shows a point 442 comprising 3D Cartesian coordinates, but it will be appreciated that other coordinate systems may be used. The point 442 is passed to a trainable portion of a machine learning system 442, such as a deep neural network 446. It should be appreciated that the trainable portion may be any trainable machine learning algorithm, such as a convolutional neural network, a random forest, or a regression model.

[0082] The neural network 446 or other trainable portion includes a plurality of parameters 446a that are adjustable during a training process. The neural network also includes a representation 446b of a body. The representation 446b may be an n-dimensional vector representing a body shape / type. During a training process as described above with respect to method 200, the representation 446b is randomly initialized and, as part of the training process, refined along with the parameters 446a. The parameters may be individual weights and / or biases associated with neurons of the neural network of the machine learning system. Training the machine learning system 440 includes providing a plurality of points 442 and adjusting the parameters 446a and the representation 446b such that an estimated signed distance value 444 is output. If the estimated signed distance value output by the neural network or other trainable portion substantially corresponds to a measured signed distance based on a medical imaging dataset (not shown), the parameters 446a and the representation 446b may be stored. The values ​​of the parameters 446a and the representation 446b may then be stored in a format as described above with respect to the method 200. Figure 3 The method 300 described uses a trained machine learning system 440 during inference of a portion of a body representation.

[0083] As mentioned above about Figure 3 During the inference of the body representation described in method 300, a plurality of sample points 442 based on received image data and associated measured distance values ​​444 may be provided to a trained portion 446, which may be a trained neural network. Based on trained parameters 446a of the trained portion 446, a body representation 446b may be inferred based on the plurality of points 442 provided as input and the signed distances expected as output. This is accomplished by optimizing the values ​​within the body representation 446b using the trained portion 446. The inferred body representation 446b may then be stored for use in further processing associated with a plurality of tasks, such as calculating SAR, estimating patient weight, adjusting radiation dose, or estimating scan parameters, as described above.

[0084] As mentioned above, the machine learning system may include a deep neural network, which may be based on the deep neural network described in Section 4 of reference [1], Section 3 of reference [2], or Section 4 of reference [3].

[0085] Figure 6is a schematic representation of a system 500 according to an example. The system 500 includes at least one processor 510, which is configured to execute the above Figure 2 and Figure 3 The methods 200, 300 described above may be configured in some examples with respect to Figure 4 The processing unit described. Thus, processor 510 may include multiple processing units, such as an image signal processor (ISP), a graphics processing unit (GPU), a general-purpose central processing unit (CPU), or a neural processing unit (NPU). The use of such a processor can optimize system 500 to utilize the described neural network. This is because, as should be appreciated, such a processor can utilize multi-threaded processing and handle a large number of threads simultaneously.

[0086] Furthermore, the processor 510 may be present alongside other components such as a memory 520 for storing instructions, so that the processor 510 can implement the Figure 2 and Figure 3 The method 200, 300 for training a machine learning system and inferring a body representation described herein. The memory 520 is further configured to store medical imaging data used during training of the machine learning system. The storage device 520 may also be configured to store instructions for the neural network described with respect to blocks 220 and 320 for identifying landmarks within the medical imaging data. Alternatively, the one or more neural networks, the instructions, and the stored medical imaging data may thereafter be supplied (e.g., in the form of a computer program product) with the aid of a computer-readable storage medium, such as a compact disk (CD), a digital versatile disk (DVD), a hard drive, a solid-state drive, a flash memory device, or the like. Alternatively, the one or more neural networks and the instructions may be downloaded to the storage medium 430 via a data communications network (e.g., the Internet).

[0087] The system 500 also includes an imaging device 530 configured to acquire image data, such as a CT imaging device, an MRI imaging device, an x-ray imaging device, an ultrasound probe, or a depth sensor. In some imaging devices, a reconnaissance scan can be performed. A reconnaissance scan is a relatively fast scan of a patient and, in some examples such as CT / X-ray imaging, limits patient exposure to low-dose / low-energy radiation. As such, the system 500 may include an image acquisition machine configured to obtain image data using a desired imaging modality.

[0088] The components 510, 520, 530 of the system 500 may be interconnected with other components of the system 500, for example using a system bus, although it should be appreciated that one component may be directly connected to another component such that the output of the first component is directly connected to the input of the second component in a pipeline. This allows data to be efficiently transferred between the various components without first storing the output of the first component.

[0089] Although the present invention has been illustrated and described in detail in the context of specific embodiments, the present invention is not limited to the disclosed embodiments. Other variations may be deduced by those skilled in the art without departing from the scope of the claimed invention.

[0090] In summary, a method, apparatus, system, and computer-readable storage medium for training a machine learning system for representing a patient's body are disclosed, comprising: obtaining a plurality of stored medical imaging data, wherein a first medical imaging data set in the plurality of stored medical imaging data sets represents a different portion of the patient's body than a second medical imaging data set in the plurality of stored medical imaging data sets; and estimating a plurality of landmarks in the stored medical imaging data. Each of the stored medical imaging data is aligned to a predefined pose using the plurality of landmarks, and a plurality of points in the stored medical imaging data are sampled. The sampled points are then used to train the machine learning system by configuring parameters of the machine learning system that are then stored.

[0091] Also disclosed is a method, apparatus, system, and computer-readable storage medium for inferring a body representation, comprising receiving image data representing at least a portion of a patient captured using an imaging modality, and estimating a plurality of landmarks in the image data. The medical imaging data is aligned to a predefined pose using the plurality of landmarks, and a plurality of points in the image data are sampled. The plurality of points are then analyzed using a trained machine learning system, the machine learning system being configured to represent the patient's body, which in turn is used to infer a body representation. The inferred body representation is then output.

[0092] References

[0093] [1] "DeepSDF: Learning Continuous Signed Distance Functions for Shape Representations" by Park et al., arXiv.org, 2019.

[0094] [2] "Occupancy Networks: Learning 3D Reconstruction in Function Space" by Mescheder et al., arXiv.org, 2019.

[0095] [3] Chen Z, Zhang H, "Learning Implicit Fields for Generative Shape Modeling", arXiv.org, 2019.

Claims

1. A computer-implemented method for training a machine learning system for representing a patient's body, comprising: obtaining a plurality of stored medical imaging data sets, each medical imaging data set representing at least a portion of a respective patient, wherein a first medical imaging data set of the plurality of stored medical imaging data sets represents a different portion of the patient's body than a second medical imaging data set of the plurality of stored medical imaging data sets; estimating a plurality of landmarks in a stored medical imaging dataset; aligning each of the stored medical imaging data sets to a predefined posture using the plurality of landmarks to generate a plurality of aligned medical imaging data sets, wherein the predefined posture is a posture that a patient is expected or required to assume when being imaged; sampling a plurality of points in the aligned medical imaging dataset; training the machine learning system based on at least the plurality of points to configure parameters of the machine learning system; and Store at least configuration parameters of the machine learning system.

2. The computer-implemented method of claim 1 , wherein the stored medical imaging dataset is at least one of: Computed tomography data; Magnetic resonance imaging data; X-ray imaging data; and Ultrasound imaging data.

3. The computer-implemented method of claim 1 or 2, wherein sampling a plurality of points comprises determining, for each of the plurality of points, a distance to a nearest surface represented in the medical imaging dataset.

4. The computer-implemented method of claim 3, wherein training the machine learning system comprises adjusting at least one body representation based on at least one of the plurality of points and a corresponding distance to a nearest surface.

5. The computer-implemented method of claim 4, wherein the at least one body representation is an n-dimensional vector.

6. A computer-implemented method according to claim 1 or 2, wherein estimating multiple landmarks includes applying at least one landmark detection algorithm to the stored medical imaging data set, wherein the landmark detection algorithm is a machine learning algorithm trained to identify one or more landmarks of the body.

7. The computer-implemented method of claim 1 or 2, wherein aligning the stored medical imaging dataset comprises: identifying at least one first medical data landmark in a first medical imaging dataset; identifying at least one second medical data landmark in the second medical imaging data set, wherein the first medical data landmark and the second medical data landmark represent corresponding landmarks; and The first medical imaging dataset and the second medical imaging dataset are aligned based on the first medical data landmark and the second medical data landmark.

8. The computer-implemented method of claim 1 or 2, wherein the machine learning system comprises at least one neural network.

9. A computer-implemented method of inferring a body representation, the method comprising: receiving image data representing at least a portion of a patient captured using an imaging modality; estimating multiple landmarks in image data; aligning the image data to a predefined posture using the plurality of landmarks to generate aligned image data, wherein the predefined posture is a posture that a patient is expected or required to assume when being imaged; sampling a plurality of points in the aligned image data; analyzing the plurality of points using a trained machine learning system configured to represent a body of the patient and trained using a plurality of stored medical imaging datasets, wherein a first medical imaging dataset of the plurality of stored medical imaging datasets represents a different portion of the body of the patient than a second medical imaging dataset of the plurality of stored medical imaging datasets; inferring body representations based on analysis; and Outputs the inferred body representation.

10. The computer-implemented method of claim 9, wherein the image data is captured by an imaging device, the image data being at least one of: Computed tomography data; Magnetic resonance imaging data; X-ray imaging data; Ultrasound data; Point cloud data; and Depth data.

11. A computer implemented method according to claim 9 or claim 10, wherein sampling a plurality of points comprises, for each of the plurality of points, determining a distance to a nearest surface represented in the image data, thereby determining a plurality of distances.

12. A computer-implemented method according to claim 11, wherein inferring the body representation comprises iteratively providing the body representation and a plurality of points to a trained machine learning system and comparing a corresponding output of the machine learning system with a plurality of distances associated with the plurality of points.

13. The computer-implemented method of claim 12, further comprising adjusting the randomized body representation based on a comparison between the corresponding output and a plurality of distances associated with a plurality of points.

14. The computer-implemented method of claim 9 or 10, wherein the body representation is an n-dimensional vector.

15. The computer-implemented method of claim 9 or 10, wherein estimating a plurality of landmarks comprises applying at least one landmark detection algorithm to the medical imaging data, the landmark detection algorithm being a machine learning algorithm trained to identify one or more landmarks of the body.

16. The computer-implemented method of claim 9 or 10, wherein aligning the image data comprises: identifying at least one first data landmark in the received image data; identifying at least one second data landmark data associated with a predefined gesture, wherein the first data landmark and the second data landmark represent corresponding landmarks; and Based on the first data landmark and the second data landmark, the received image data and the predefined gesture are aligned.

17. A computer-implemented method according to claim 9 or 10, wherein the machine learning system is trained according to any one of claims 1 to 8.

18. An apparatus for inferring a body representation, the apparatus comprising: a training interface configured to obtain a stored medical imaging dataset; a training unit configured to train at least one machine learning system based on stored medical imaging datasets, wherein a first medical imaging dataset of the stored medical imaging datasets represents a different part of a patient's body than a second medical imaging dataset of the stored medical imaging datasets; an inference interface configured to receive image data captured using an imaging modality; An inference unit configured to apply the method according to any one of claims 9 to 17.

19. A system for inferring a body representation, the system comprising: an imaging device configured to capture image data; A storage device for storing a medical imaging data set; and At least one processor arranged to perform the method of any one of claims 1 to 17 and arranged to obtain from a storage device a stored medical imaging data set and image data captured by an imaging device.

20. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, which, when executed by at least one processor, causes the processor to perform the method of one of claims 1 to 17.

Images