Model processing equipment and method

Abstract

A model processing device and method. The shape and / or position of an organ may change based on changes in the patient's body shape and / or posture. This document describes systems, methods, and apparatus for automatically determining the shape and / or position of an organ using an artificial neural network (ANN) based on a human body model that reflects the patient's body shape and / or posture. The ANN can be trained to learn the spatial relationship between the organ and the patient's body shape or posture. Then, at inference time, the ANN can be used to determine the relationship based on a first patient model and a first representation (e.g., a point cloud) of the organ, so that thereafter, given a second patient model, the ANN can automatically determine the shape and / or position of the organ corresponding to the patient's body shape or posture indicated by the second patient model.

Description

Technical Field

[0001] The present application relates to the field of human body model processing, specifically to the representation of human organs, model processing equipment and methods. Background Art

[0002] Determining organ shape and position is a crucial aspect of clinical applications. For example, preoperative planning and radiotherapy require precise knowledge of the physical characteristics of the target organ, such as its orientation, contour, and volume. Modern medical imaging techniques provide the means to acquire this knowledge. However, because the physical characteristics of an organ can vary depending on the patient's body shape and / or posture, a medical scan of an organ acquired at a previous moment (e.g., before treatment) may not reflect the organ's characteristics at the current moment (e.g., during treatment). Consequently, it is often necessary for the patient to maintain the same posture or position during different medical procedures. When this is not possible, additional imaging may be required to account for changes in the patient's body shape and / or posture. The dependence of organ shape and / or position on the patient's body shape and / or posture can also pose challenges for comparative organ studies. For example, because medical scans of an organ acquired at different times may inherently differ depending on the patient's body shape and / or posture, it can be difficult to isolate pathological changes in the organ from non-pathological changes caused by changes in the patient's body shape and / or posture.

[0003] Therefore, it may be highly desirable to have systems and methods for automatically determining the shape and / or position of an organ based on the patient's body shape and / or posture. These systems and methods may be used, for example, to facilitate treatment and preoperative planning, improve the accuracy and effectiveness of surgical procedures, avoid or reduce unnecessary medical scans, reduce radiation exposure to patients, enable comparative clinical research and / or diagnosis, and the like. Summary of the Invention

[0004] This document describes systems, methods, and apparatus for automatically determining the geometric features of an organ based on a patient's body shape and / or posture. A device configured to perform this task may include one or more processors configured to receive a first model of the patient and a representation of the organ. The first model may indicate the patient's body shape or posture, while the representation of the organ may indicate the geometric features (e.g., shape and / or position) of the organ corresponding to the body shape or posture indicated by the first model. Based on the model and the representation of the organ, the one or more processors of the device may be configured to determine a relationship between the geometric features of the organ and the patient's body shape or posture using an artificial neural network (ANN). This relationship may, for example, be represented by multiple parameters indicating a spatial relationship between one or more points of the organ and one or more points of the first model. When determining the relationship, the one or more processors of the device may receive a second model of the patient, the second model indicating that at least one of the patient's body shape or posture has changed from the body shape or posture indicated by the first model. The one or more processors may determine the geometric features of the organ corresponding to the patient's body shape or posture indicated by the second model based on the second model and the determined relationship between the organ and the patient's body shape or posture.

[0005] The ANN described herein can be trained based on multiple patient training models and multiple training representations of an organ to learn the relationship between the geometric features of the organ (e.g., shape and / or position) and the patient's body shape or posture. As described above, this relationship can be reflected by multiple parameters that the ANN can learn during training. An example training process for the ANN may include one or more of the following steps. For each of the multiple patient training models, the ANN can obtain a representation of the organ corresponding to the body shape and / or posture of the patient represented by the patient training model from the multiple training representations. The ANN can estimate the values ​​of the above-mentioned multiple parameters based on the training model and the representation of the organ. The ANN can then obtain a second training model of the patient and generate an estimated representation of the organ based on the estimated values ​​of the multiple parameters and the second training model. The ANN can then compare the estimated representation it has generated with the training representation of the organ corresponding to the second patient model (e.g., as a gold standard representation) and adjust the execution parameters (e.g., weights) of the ANN based on the difference between the gold standard representation and the representation predicted by the ANN (e.g., gradient descent associated with the difference).

[0006] In an example, the ANN described herein may include one or more encoders and one or more decoders. The one or more encoders may be trained to determine a relationship (e.g., a plurality of parameters reflecting the relationship) between a geometric feature of an organ and a body shape or posture of the patient based on a first model of the patient and a first representation of the organ. The one or more decoders may be trained to construct a representation of the organ corresponding to the body shape or posture of the patient indicated by the second model based on a second model of the patient and the relationship determined by the encoder.

[0007] In an example, each representation of an organ described herein may include a point cloud (e.g., a three-dimensional point cloud) that may be obtained based on at least a scanned image of the organ taken when the patient is in a body shape or posture indicated by the corresponding patient model. In an example, such a point cloud may be obtained by aligning the scanned image of the organ with the corresponding patient model and determining the point cloud based on the alignment. In an example, each patient model described herein may include a corresponding parametric model of the patient, and the patient model may be generated based on a corresponding image of the patient captured by one or more sensing devices. In an example, an apparatus described herein may include one or more sensing devices.

[0008] The techniques described herein for automatically determining the geometry of an organ based on a patient's body shape and / or posture can be used to serve multiple clinical purposes. For example, when determining the geometry of an organ corresponding to a second patient model (e.g., indicating a second patient shape or posture), a scan image taken while the patient was in a first shape or posture (e.g., indicated by the first patient model) can be manipulated to align with the second patient model. This can not only eliminate the need for additional scans of the organ but also allow diagnostic studies and treatment planning to be performed based on changes in the patient's body shape and / or posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Examples disclosed herein can be understood in more detail from the following description given by way of example in conjunction with the accompanying drawings.

[0010] Figure 1 is a simplified diagram illustrating an example environment associated with one or more embodiments described herein.

[0011] Figure 2 is a simplified diagram illustrating example operations associated with automatic determination of organ shape and / or organ position in accordance with one or more embodiments described herein.

[0012] Figure 3 is a simplified diagram illustrating the training of a neural network according to one or more embodiments described herein.

[0013] Figure 4is a simplified flowchart illustrating example operations that may be associated with training a neural network in accordance with one or more embodiments described herein.

[0014] Figure 5 is a simplified diagram illustrating an example of human body mesh recovery according to one or more embodiments described herein.

[0015] Figure 6 is a block diagram illustrating example components of a device that may be configured to perform the tasks described in one or more embodiments provided herein. DETAILED DESCRIPTION

[0016] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0017] Figure 1 1 is a simplified diagram illustrating an example environment 100 associated with one or more embodiments described herein. Environment 100 may be part of a medical facility, such as a scanning room (e.g., magnetic resonance imaging (MRI), X-ray, computed tomography (CT), etc.) or an operating room (OR), a rehabilitation facility, a fitness center, etc. Environment 100 may be equipped with one or more sensing devices (e.g., 102a, 102b, 102c), such as one or more digital cameras, configured to capture images (e.g., two-dimensional (2D) images) of a patient 104 within environment 100. Sensing devices 102a-c may be communicatively coupled to a processing unit 106 and / or other devices of environment 100 via a communication network 108. Each sensing device 102a-c may include one or more sensors, such as one or more 2D vision sensors (e.g., 2D cameras), one or more 3D vision sensors (e.g., 3D cameras), one or more red, green, and blue (RGB) sensors, one or more depth sensors, one or more RGB plus depth (RGB-D) sensors, one or more thermal sensors (e.g., infrared (FIR) or near infrared (NIR) sensors), one or more radar sensors, and / or other types of image capture devices or circuits.

[0018] Each of the sensing devices 102a-c may include a functional unit (e.g., a processor) configured to process images captured by the sensing device and / or generate (e.g., construct) a human body model, such as a 3D human body mesh model of the patient, based on the images. Such a human body model may include a plurality of parameters that indicate the patient's body shape and / or posture when the patient is within the environment 100 (e.g., during an MRI, X-ray, or CT procedure). For example, the parameters may include a body shape parameter β and a posture parameter θ, which may be used to determine a plurality of vertices associated with the patient's body (e.g., 6,890 vertices based on 82 body shape and posture parameters) and construct a visual representation (e.g., a 3D mesh) of the patient model, for example, by connecting the vertices with edges to form polygons (e.g., triangles), connecting the plurality of polygons to form surfaces, determining a 3D shape using the plurality of surfaces, and applying textures and / or shading to the surfaces and / or shapes.

[0019] The patient model described above may also be generated by the processing unit 106. For example, the processing unit 106 may be communicatively coupled to one or more of the sensing devices 102a-c and may be configured to receive images of the patient from these sensing devices (e.g., in real time or based on a predetermined schedule). Using the received images, the processing unit 106 may construct a patient model, for example, in a manner similar to that described above. It should be noted that even if the processing unit 106 is Figure 1 Although shown as separate from the sensing devices 102a-c, any of the sensing devices 102a-c can also be configured to operate as a processing unit 106 (e.g., using one or more functional units or processors included in the sensing device). For example, the sensing devices 102a-c can be interconnected via a communication link 108 and exchange images with each other. One sensing device can be configured to perform the model building tasks described herein based on images received from other sensing devices.

[0020] The sensing devices 102a-c or the processing unit 106 may also be configured to automatically determine geometric features of the patient's organs based on the patient's body shape and / or posture indicated by the patient model. The organ may be, for example, the patient's spleen, liver, heart, etc., and the geometric features may include, for example, the shape and / or position of the organ corresponding to the patient's body shape and / or posture indicated by the patient model. In an example, the sensing devices 102a-c or the processing unit 106 may be configured to automatically determine these geometric features of the organ using a machine learning model that may indicate a correlation (e.g., a spatial relationship) between the geometric features of the organ and the patient's body shape and / or posture. In an example, such a machine learning model may take as input a patient model and a representation of the organ (e.g., a three-dimensional (3D) point cloud) and generate an output (e.g., multiple parameters) indicating how the geometry (e.g., shape and / or position) of the organ may change based on changes in the patient's body shape and / or posture. It can be seen that, using the machine learning model, the sensing devices 102a-c or the processing unit 106 can determine the correlation between the geometric features of the organ and the patient's body shape and / or posture based on the first patient model and the first representation of the organ, and when obtaining a second patient model indicating that the patient's body shape and / or posture has changed, automatically determine the geometric features of the organ corresponding to the body shape and / or posture indicated by the second patient model.

[0021] The techniques described herein can be used for a variety of purposes. For example, based on the automatically determined shape and / or position of an organ corresponding to a second patient model (e.g., which can indicate a second body shape and / or a second posture of the patient), a scanned image of an organ associated with a first patient model (e.g., which can indicate a first body shape and / or a first posture of the patient) can be manipulated to align with the second patient model. The aligned scanned image and patient model can then be used to determine changes in the structure and / or functional state of the organ, regardless of potential changes in the patient's body shape and / or posture, and without requiring additional scans of the organ. Having the ability to automatically determine the geometry (e.g., shape and / or position) of an organ corresponding to a second body shape and / or posture of the patient can also allow medical procedures (e.g., surgery, radiation therapy, etc.) planned based on the first body shape and / or posture to be adapted to changes in the patient's body shape and / or posture.

[0022] Information about the automatically determined geometry (e.g., shape and / or position) of an organ and / or a patient model described herein can be provided in real time to a downstream application or device (e.g., a surgical robot). This information can also be saved (e.g., as metadata associated with the scanned image) to a repository (e.g., Figure 1The repository may be communicatively coupled to the sensing devices 102a-c and / or the processing unit 106 to receive this information. Such metadata may then be used to align the medical scan associated with the metadata with other models of the patient (e.g., if the patient's shape and / or posture changes in such models) or with other medical scans of the patient (e.g., medical scans captured by different imaging modalities).

[0023] The patient models described herein can be derived independently of one another (e.g., based on different images of the patient taken at different times of the year), or one patient model can be derived based on another patient model and / or a given protocol (e.g., such a protocol may indicate that the second patient model may share the same features of the first patient model, except for the patient's pose). In an example, medical scans of a patient corresponding to different patient models (e.g., at different body shapes and / or poses at different times) can be aligned to determine changes in the patient's organs. For example, a patient may undergo multiple scanning sessions over the course of a year, and during / after each scanning session, the scan images can be linked to a parametric model (e.g., a 3D mesh) representing the patient's body shape and / or pose during the scanning session. Subsequently, by manipulating the corresponding body shape and / or pose parameters of the parametric model, the models obtained during the different scanning sessions can be updated to reflect the same body shape and / or pose of the patient, so that the scan images can also be aligned with the same body shape and / or pose of the patient. In this way, the scan images (e.g., the segmentation masks associated with the scan images) can be compared and evaluated to determine how the patient's organs may have evolved over time.

[0024] In an example, a first patient model can be generated based on the patient's position in a scanning room, and a second patient model can be generated based on the patient's position in an operating room (e.g., based on data collected by sensing devices in the operating room). By aligning scan images associated with the first patient model with the second patient model (e.g., based on automatically determined organ shapes and / or positions corresponding to the second patient model), the aligned scan images can be used for surgical planning, surgical guidance, or patient diagnosis. In an example, given a treatment, surgery, or procedure plan designed based on the first patient model, the second patient model and / or automatically determined organ shapes and / or positions can be used to correct or update the plan. In an example, the patient model and / or automatically determined organ shapes and / or positions can be used to optimize scan parameters to target treatment areas, adjust radiation dose, and the like. In an example, a patient model can be generated based on information captured at the injury scene (e.g., images of the patient). By aligning a medical scan of a patient with such a patient model, a more accurate assessment of the injury can be achieved.

[0025] Figure 2is a simplified diagram illustrating example operations that may be performed to automatically determine geometric features of an organ. These example operations may be performed by, for example, Figure 1 The processing unit 106 or the sensing device 102a, 102b or 102c shown in FIG. 1 is executed by a device or apparatus. For ease of description, such a device or apparatus may be referred to herein as an organ geometry estimator. Figure 2 As shown, the organ geometry estimator 200 can be configured to obtain a first model 202 of a patient and a representation 204 of an organ of the patient (e.g., spleen, liver, heart, etc.). The first model 202 can include a parametric model of the patient, a two-dimensional (2D) or three-dimensional (3D) outline of the patient, a 3D mesh of the patient, a 3D point cloud representing the patient's body shape and / or posture, a 2D or 3D skeletal representation of the patient, descriptors of one or more 2D or 3D joint positions of the patient, a set of measurements indicating physical features of the patient, and / or other types of representations that can indicate the patient's body shape and / or posture when the patient is in a certain posture (e.g., standing in front of a scanning device, lying on a surgical table, etc.). Using a parametric model as an example, the first model 202 can include a plurality of parameters, such as a plurality of posture parameters θ (e.g., 72 posture parameters associated with the patient's joints) and / or a plurality of shape parameters β (e.g., 10 coefficients of a principal component analysis (PCA) space), which can be used to determine the patient's body shape and / or posture (e.g., via the 3D mesh). The first model 202 may be generated by the organ geometry estimator 200 or a different device or apparatus based on the information generated by the sensing devices described herein (e.g., Figure 1 If generated by a device other than the organ geometry estimator 200, the first model 202 may be provided (e.g., parameters of the first model 202 may be provided) to the organ geometry estimator 200 for execution. Figure 2 The operation shown.

[0026] Figure 2The depicted representation 204 may indicate one or more geometric features (e.g., shape and / or position) of an organ corresponding to the patient's body shape and / or posture represented by the first model 202. The representation 204 may be obtained in various forms, including, for example, a 3D point cloud of the organ, a parametric model of the organ (e.g., a 3D parametric model), and the like. The representation 204 may be generated, for example, based on one or more scanned images of the organ (e.g., taken while the patient was in the body shape and / or posture indicated by the first model 202) and a statistical shape model of the organ. The statistical shape model may include a mean shape of the organ (e.g., a mean point cloud indicating the shape of the organ) and a principal component matrix that may be used to determine the shape of the organ depicted by the one or more scanned images (e.g., as a variation of the mean shape) based on features extracted from the one or more scanned images. The statistical shape model may be predetermined, for example, based on sample scanned images of the organ collected from a particular population or cohort and segmentation masks of the organ corresponding to the sample scanned images. The segmentation masks may be registered to each other via an affine transformation, and the registered segmentation masks may be averaged to determine a mean point cloud representing the mean shape of the organ. Based on the average point cloud, corresponding point clouds can be derived in the image domain of each sample scan image, for example, by inverse deformation and / or transformation. The derived point clouds can then be used to determine a principal component matrix, for example, by extracting the main variation modes of the average shape.

[0027] It should be noted that the representation 204 (e.g., point cloud) may be derived by the organ geometry estimator 200 or by a different device or apparatus. In the latter case, the representation 204 may be provided to the organ geometry estimator 200 in order to perform the example operations described herein. Figure 2 As shown, the organ geometry estimator 200 may include an artificial neural network (ANN) 206 that is trained to determine a correlation (e.g., a spatial relationship) between the geometric characteristics (e.g., shape and / or position) of the organ and the patient's body shape and / or posture based on the patient's model 202 and the organ's representation 204. This correlation can be represented, for example, by a plurality of parameters (e.g., referred to herein as α), which can indicate how the geometric characteristics of the organ may change based on a change in the patient's body shape and / or posture (e.g., from a first body shape and / or first posture to a second body shape and / or second posture).

[0028] In an example, the ANN 206 may include a point cloud feature encoder 206a trained to extract features from the representation 204 of the organ. The point cloud feature encoder 206a may include a convolutional neural network (CNN) having multiple layers, such as one or more convolutional layers, one or more pooling layers, and / or one or more fully connected layers. Each convolutional layer may include multiple convolution kernels or filters configured to extract features from the representation 204. The convolution operation may be followed by batch normalization and / or linear or nonlinear activation, and the features extracted by the convolution layer may be downsampled (e.g., using a 2×2 window and a stride of 2) by a pooling layer and / or a fully connected layer to reduce the redundancy and / or size of the features (e.g., by a factor of 2), thereby obtaining a representation of the downsampled features, for example, in the form of a feature map or feature vector (e.g., a PTC or a point cloud vector).

[0029] In an example, the ANN 206 may include an encoder 206b that is trained to encode the features of the representation 204 extracted by the point cloud feature encoder 206a, the body shape parameters β, and / or the posture parameters θ into a plurality of parameters α, the plurality of parameters representing a correlation (e.g., a mapping or spatial relationship) between the geometric features (e.g., shape and / or position) of the organ and the body shape or posture of the patient. In an example, the encoder 206b may include a multi-layer perceptron (MLP) neural network having multiple layers (e.g., an input layer, an output layer, and one or more hidden layers) with linear or nonlinear activation nodes (e.g., perceptrons), the neural network being trained to infer the correlation between the geometric features of the organ and the body shape or posture of the patient and generating the parameters α to represent the correlation. In an example, the parameters α may include a vector of floating point numbers (e.g., float32 numbers), which may be used to determine the positions (e.g., coordinates) of one or more points on the representation 204 (e.g., in the image domain) based on the positions (e.g., coordinates) of the one or more points on the first model 202 (e.g., in the image domain). Subsequently, given a second model 210 of the patient (e.g., compared to the first model 202) indicating a new (e.g., different) body shape and / or posture of the patient, the organ geometry estimator 200 may generate (e.g., estimate or predict) a representation 208 (e.g., a point cloud) based on the parameter α to indicate the geometric features (e.g., shape and / or position) of the organ at the new body shape and / or posture indicated by the second model 210.

[0030] In an example, the ANN 206 may include a point cloud decoder 206c trained to generate a representation 208 (e.g., a point cloud) based on the parameter α and the model 210 of the patient. In an example, the point cloud decoder 206c may include one or more non-pooling layers and one or more transposed convolution layers. Through the non-pooling layers, the point cloud decoder 206c may upsample the features of the point cloud 204 extracted by the point cloud encoder 206a and encoded by the encoder 206b, and further process the upsampled features through one or more transposed convolution operations to derive a dense feature map (e.g., upscaled by a factor of 2 from the original feature map generated by the point cloud encoder 206a). Based on the dense feature map, the point cloud decoder 206c may recover the representation 208 of the organ to reflect changes in the geometric features of the organ (e.g., changes in the shape and / or position of the organ) caused by changes in the body shape and / or posture of the patient as indicated by the second model 210.

[0031] Figure 3 is an example of training a neural network (e.g., neural network 306, which may be Figure 2 2 is a simplified diagram of an example of an ANN 206 (illustrated in FIG2 ) for automatically determining geometric features (e.g., shape and / or position) of an organ based on a patient's body shape and / or posture. Training can be performed using multiple patient models (e.g., parametric human body models such as skinned multi-person linear (SMPL) models) and multiple representations of the organ (e.g., 3D point clouds), which can be obtained from a publicly available training dataset. Each of the multiple representations of the organ can be associated (e.g., paired) with a corresponding one of the multiple patient models (e.g., the training representation can depict the shape and / or position of the organ when the corresponding patient is in the posture indicated by the patient model). As described herein, each patient model used for training can include multiple posture parameters θ (e.g., 72 posture parameters), multiple body shape parameters β (e.g., 10 coefficients of PCA space), and / or multiple vertices (e.g., (6890, 3) vertices) that can be derived based on the posture parameters θ and the body shape parameters β. Similarly, each representation used for training may include multiple parameters (eg, (512, 3) vertices of a 3D point cloud) that indicate geometric features (eg, shape and / or position) of the organ.

[0032] During the training process, the neural network 306 can obtain a first patient training model 302 and a corresponding first training representation 304 of the organ. Using a point cloud encoder 306a, the neural network 306 can extract features from the first training representation 304 and provide the extracted features, along with shape parameters β and pose parameters θ of the first training model 304, to an encoder (e.g., an MLP encoder 306b) to estimate parameters α. As described herein, parameters α can represent a correlation or mapping (e.g., a spatial relationship) between the geometric features of the organ (e.g., as reflected by the representation 304) and the shape and / or pose of the patient in image space (e.g., as reflected by the first training model 302). The neural network 306 can then obtain a second patient training model 310 and a corresponding second training representation 312 of the organ. Using a point cloud decoder 306c, the neural network 306 can estimate a representation 308 (e.g., a point cloud) of the organ based on the parameters α predicted by the MLP encoder 306b and the shape parameters β' and / or pose parameters θ' of the second training model 310. The neural network 306 can then compare the representation 308 with a second training representation 312 (e.g., a gold standard representation) and determine a loss associated with the encoding and / or decoding operations described above. This loss can be determined based on various loss functions, including, for example, mean squared error (MSE), L1 norm, L2 norm, structural similarity index (SSIM), etc. Once the loss is determined, the neural network 306 can adjust its parameters (e.g., weights associated with various filters or kernels of the point cloud encoder 306a, MLP encoder 306b, and point cloud decoder 306c) by backpropagating the loss through the neural network 306 (e.g., gradient descent based on the loss).

[0033] Figure 4 This illustrates the methods that can be used in training the neural networks described herein (e.g., Figure 2 2 ). For example, at 402, parameters of a neural network (e.g., weights associated with various filters or kernels of the neural network) may be initialized. The parameters may be initialized, for example, based on samples collected from one or more probability distributions or parameter values ​​of another neural network having a similar architecture. At 404, the neural network may receive a first training model of a patient (e.g., an SMPL model) and a first training representation (e.g., a 3D point cloud) of an organ (e.g., a spleen) of the patient. At 406, the neural network may extract features from the first training representation and estimate values ​​of a plurality of parameters α based on the extracted features and parameters of the first training model (e.g., body shape parameters β and / or posture parameters θ). As described herein, the parameters α may indicate a correlation or mapping (e.g., a spatial relationship) between the body shape and / or posture of the patient and geometric features (e.g., shape and / or posture) of the organ.

[0034] At 408, the neural network may receive a second trained model of the patient (e.g., an SMPL model), which may include second shape parameters β' and second posture parameters θ'. The neural network may predict a representation of the organ (e.g., a point cloud) based on the second trained model and the estimated parameters α. As described herein, this representation of the organ may depict geometric features of the organ corresponding to the patient's shape and / or posture as indicated by the second trained model (e.g., by the second shape parameters β' and / or the second posture parameters θ'). At 410, the neural network may compare the predicted representation of the organ with a gold standard representation (e.g., provided as part of the training data) to determine a loss associated with the prediction. As described herein, the loss may be determined based on MSE, L1-norm, L2-norm, SSIM, etc. Once determined, the loss may be used to determine, at 412, whether one or more training termination criteria have been met. For example, if the determined loss is below a predetermined threshold, if the corresponding change in loss between two training iterations (e.g., between consecutive training iterations) is below a predetermined threshold, etc., the training termination criteria may be considered met. If it is determined at 412 that the training termination criteria have been met, training may terminate. Otherwise, the neural network may adjust its parameters at 414 by backpropagating the loss through the neural network (e.g., based on gradient descent associated with the loss) before training returns to 406 (at 406, the neural network may make another prediction for α).

[0035] For simplicity of illustration, the training steps are depicted and described herein in a particular order. However, it should be understood that the training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training process are depicted and described herein, and not all illustrated operations need to be performed.

[0036] Figure 5 A simplified block diagram is shown illustrating how a neural network 504 may be used to recover a patient model 502 (e.g., a 2D image or a 2D+depth image) based on an image 506 (e.g., a 2D image or a 2D+depth image) of the patient. Figure 2 Patient model 202, 210 or Figure 3 The neural network 504 may be part of an organ geometry estimator as described herein (e.g., Figure 2206 is shown). The neural network 504 can also be a separate network trained to recover the patient model. In either case, the neural network 504 can include multiple layers, such as an input layer, one or more convolutional layers, one or more linear or nonlinear activation layers, one or more pooling layers, one or more fully connected layers, and / or an output layer. One or more of these layers can include corresponding filters (e.g., kernels), each of which can be designed to detect (e.g., learn) key points that collectively represent features or patterns. The filters can be associated with corresponding weights that, when applied to the input, produce an output indicating whether certain features or patterns have been detected. The neural network can learn the weights associated with the filters through a training process that includes: inputting a large number of images from one or more training data sets to the neural network, predicting an output (e.g., a set of features identified in each input training image), calculating the difference or loss resulting from the predictions (e.g., based on a loss function such as MSE, L1 norm, etc.), and updating the weights assigned to each filter (e.g., stochastic gradient descent based on the loss) to minimize the loss. Once trained (e.g., having learned to extract features from training images), the neural network can take an image at an input layer, extract and / or classify features or patterns from the image, and provide an indication of the identified features at an output layer. The identified features can be indicated, for example, by a feature map or feature vector.

[0037] Neural network 504 can also be trained, for example, to infer pose parameters θ and shape parameters β that can be used to restore patient model 502 based on features extracted from input image 506. For example, neural network 504 can be trained to determine the joint angles of the patient, as depicted in input image 506, based on a training dataset that covers a wide range of human subjects, human activities, background noise, body shape and / or pose variations, camera motion, and the like. The joint angles can be associated with, for example, the 23 joints included in a skeletal rig, as well as a root joint, and the pose parameters θ derived therefrom can include 72 parameters (e.g., 3 parameters for each of the 23 joints and 3 parameters for the root joint, where each parameter corresponds to an axis-angle rotation from the root orientation). Neural network 504 can be trained to determine shape parameters β for predicting the patient's mixed body shape based on image 506. For example, neural network 504 can learn to determine shape parameters β via PCA, where the shape parameters determined thereby can include multiple coefficients (e.g., the first 10 coefficients) in the PCA space. Once the posture and shape parameters are determined, a plurality of vertices (e.g., 6,890 vertices based on 82 body shape and posture parameters) can be obtained for constructing a visual representation of the patient's body (e.g., a 3D mesh). Each vertex can include its own position, normal, texture, and / or shading information. Using these vertices, a 3D mesh of the patient can be created, for example, by connecting the vertices with edges to form polygons (e.g., triangles), connecting the polygons to form surfaces, using the surfaces to determine a 3D shape, and applying textures and / or shading to the surfaces and / or shapes.

[0038] The systems, methods, and / or devices described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 6 6 is a block diagram illustrating an example device 600 that can be configured to automatically determine the shape and / or position of an organ according to one or more embodiments described herein. As shown, the device 600 may include a processor (e.g., one or more processors) 602, which may be a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other circuit or processor capable of performing the functions described herein. The device 600 may also include communication circuitry 604, a memory 606, a mass storage device 608, an input device 610, and / or a communication link 612 (e.g., a communication bus) through which one or more components shown in the figure can exchange information.

[0039] The communication circuit 604 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including a local area network (LAN), a wide area network (WAN), the Internet, a wireless data network (e.g., Wi-Fi, 3G, 4G / LTE, or 5G network). The memory 606 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions, which, when executed, causes the processor 602 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc. The mass storage device 608 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROM or DVD-ROM disks, etc., on which instructions and / or data may be stored to facilitate the operation of the processor 602. The input device 610 may include a keyboard, a mouse, a voice control input device, a touch-sensitive input device (eg, a touch screen), etc., for receiving user input of the device 600 .

[0040] It should be noted that the apparatus 600 can operate as a standalone device or can be connected (e.g., networked or clustered) with other computing devices to perform the functions described herein. Figure 6 Only one example of each component is shown in the figure, and those skilled in the art will also understand that the device 600 may include multiple examples of one or more components shown in the figure.

[0041] Although the present disclosure has been described in terms of certain embodiments and generally associated methods, variations and transformations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit the present disclosure. Other changes, substitutions, and variations are also possible without departing from the spirit and scope of the present disclosure. In addition, unless otherwise specifically stated, discussions utilizing terms such as "analyze," "determine," "enable," "identify," "modify," etc. refer to the actions and processes of a computer system or similar electronic computing device, which manipulate and transform data represented as physical (e.g., electronic) quantities within the registers and memories of a computer system into other data represented as physical quantities within the computer system memory or other such information storage, transmission, or display device.

[0042] It should be understood that the above description is intended to be illustrative, rather than restrictive. After reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the full scope of equivalents to which the appended claims and such claims are given.

Claims

1. A model processing device comprising: One or more processors configured to: obtaining a first model of a patient, wherein the first model indicates a body shape or posture of the patient; obtaining a representation of an organ of the patient, wherein the representation indicates geometric features of the organ corresponding to the body shape or posture of the patient indicated by the first model; determining a relationship between the geometric features of the organ and the body shape or posture of the patient using an artificial neural network (ANN), wherein the relationship is determined by the ANN based on the first model of the patient and the representation of the organ; obtaining a second model of the patient, wherein the second model indicates that at least one of the body shape or posture of the patient has changed in the second model compared to the first model; and Based on the second model and the relationship between the geometric features of the organ determined by the ANN and the body shape or posture of the patient, the geometric features of the organ corresponding to the body shape or posture of the patient indicated by the second model are determined.

2. The device according to claim 1, wherein The geometrical feature of the organ comprises at least one of a shape of the organ or a position of the organ.

3. The device according to claim 1, wherein The representation of the organ includes a point cloud associated with the organ, wherein the point cloud is obtained based on a scanned image of the patient taken when the patient is in the shape or posture indicated by the first model.

4. The device according to claim 1, wherein The one or more processors are configured to determine the relationship between the geometric features of the organ and the body shape or posture of the patient, including: the one or more processors are configured to determine a plurality of parameters indicating a spatial relationship between one or more points of the organ in the image space and one or more points of the first model in the image space.

5. The device according to claim 4, wherein The ANN is trained based on a plurality of patient training models and a plurality of training representations of the organ to determine the relationship between the geometric features of the organ and the body shape or posture of the patient, and wherein, during the training, the ANN is configured to: obtaining a first patient-trained model from the plurality of patient-trained models; obtaining a first trained representation of the organ from the plurality of training representations, wherein the first trained representation indicates the geometric features of the organ corresponding to a first patient shape or a first patient posture indicated by the first patient training model; estimating values ​​of the plurality of parameters based on the first patient training model and the first trained representation of the organ; obtaining a second patient-trained model from the plurality of patient-trained models; obtaining a second training representation of the organ from the plurality of training representations as a gold standard representation of the organ, wherein the second training representation of the organ indicates the geometric features of the organ corresponding to a second patient shape or a second patient posture indicated by the second patient training model; predicting a representation of the organ based on the estimated values ​​of the plurality of parameters and the second patient-trained model; and Execution parameters of the ANN are adjusted based on a difference between the predicted representation of the organ and the gold standard representation of the organ.

6. The apparatus according to claim 1, wherein The ANN includes one or more encoders and one or more decoders, the one or more encoders being configured to determine the relationship between the geometric features of the organ and the body shape or posture of the patient, and the one or more decoders being configured to generate an estimated representation of the organ corresponding to the body shape or posture of the patient indicated by the second model based on the second model of the patient and the relationship determined by the one or more encoders.

7. The apparatus according to claim 1, wherein At least one of the first model or the second model of the patient comprises a parametric model of the patient.

8. The apparatus according to claim 1, wherein The one or more processors are configured to obtain the first model of the patient, including: the one or more processors are configured to construct the first model based on the first image of the patient, and wherein the one or more processors are configured to obtain the second model of the patient, including: the one or more processors are configured to construct the second model based on the second image of the patient, wherein the device further includes a sensing device configured to capture the first image of the patient and the second image of the patient.

9. The apparatus according to claim 1, wherein The one or more processors are further configured to align the representation of the organ with the second model of the patient based on the geometric features of the organ, the geometric features of the organ corresponding to the body shape or posture of the patient indicated by the second model.

10. A model processing method comprising: obtaining a first model of a patient, wherein the first model indicates a body shape or posture of the patient; obtaining a representation of an organ of the patient, wherein the representation indicates geometric features of the organ corresponding to the body shape or posture of the patient indicated by the first model; determining a relationship between the geometric features of the organ and the body shape or posture of the patient using an artificial neural network (ANN), wherein the relationship is determined by the ANN based on the first model of the patient and the representation of the organ; obtaining a second model of the patient, wherein the second model indicates that at least one of the body shape or posture of the patient has changed in the second model compared to the first model; and Based on the second model and the relationship between the geometric features of the organ determined by the ANN and the body shape or posture of the patient, the geometric features of the organ corresponding to the body shape or posture of the patient indicated by the second model are determined.

Images