System and method for mapping the skin

Abstract

A system that measures the movement of a camera 106 moving over a live subject, referencing the images captured by the camera while measuring the subject's shape, posture, and movement. The system can segment each image into pixels containing the subject and pixels not containing the subject, and assign each pixel containing the subject to a precise location on the subject's body. In one implementation, a computer connected to a camera 106 and an Inertial Measurement Unit (IMU) 116 provides estimates of the camera's location, orientation, and speed by integrating the camera's movement with respect to the subject's environmental and surface features, corrected for the subject's movement. The system corrects errors accumulated in the camera's movement integration by recognizing the subject's body shape in the collected data.

Description

System and method for mapping the skin Technical field The present invention relates to mapping systems and methods for evaluating skin tissue. Background The detection and monitoring of some diseases benefit from photographic documentation of the skin. A baseline set of images collected during an initial imaging session is compared with the skin's appearance at a later time to enable the detection and measurement of skin changes. In patients at high risk of developing melanoma, these methods have been shown to allow for the detection of melanoma at an earlier stage than the standard of care, while requiring the excision of fewer benign lesions. Current practices for collecting, organizing, and using these images are costly and logistically complex, which limits their adoption. Traditionally, full-body photography (FBP) has been performed by a trained photographer using expensive cameras in a purpose-built studio. The photographer positions the subject in a series of reference poses, allowing photographs to be taken of as much of the body's skin as possible. The process of positioning the subject and taking the photographs can take twenty minutes or more, usually during a separate appointment for the patient. Using the resulting photographs to recognize skin changes is time-consuming and complicated: the dermatologist typically scans the photograph from one side to the other, trying to identify the change. The cost of FBP is justified only for patients at very high risk of melanoma. Several companies have developed products to simplify TBP collection. One class of these products takes the form of a booth equipped with dozens of high-quality cameras, each focused on a relatively small area of ​​skin. The patient enters the booth, assumes each reference position, the cameras take photographs, and the data are stored. These systems have proven effective in several studies, but the high capital cost of the system and the space required to house it have limited their adoption. They do not appear to reduce the cost of TBP. Other companies have developed smartphone apps to modernize the traditional TBP workflow. The app helps the photographer guide the patient into reference poses, take photographs, and store and organize the photos according to the pose and the body area covered. In this process, there is a trade-off between the image quality obtained and the number of photographs required to cover the body. Most apps take only 2-4 photographs to cover the body to limit the time required and the complexity of aligning each image. The resulting images are of insufficient quality for evaluating potential skin cancers: at the distance from the subject, the pixel density is too low to resolve important features of cancerous lesions.There hasn't been a good option for bringing the camera closer to the body, because it's difficult for the photographer to consistently capture the correct portion of the body from a close range, and the careful work of aligning each of a large number of images would be very time-consuming. An object of the invention described herein is to overcome these trade-offs and, in particular, to provide a method for tracking the camera's posture position relative to the subject's body despite body movement, so that a large number of close-up images can be quickly captured, each of which is mapped to its location on the subject's body. Existing techniques such as photogrammetry, visual odometry (VO), and visual-inertial odometry (VIO) require a static subject and 3D environment. Living subjects are not sufficiently static to reliably generate accurate VO, VIO, or photogrammetry results. Consequently, new methods must be developed to achieve the objectives of the present invention. U.S. provisional patent application 62 / 838,662, published under the Patent Cooperation Treaty as PCT / WO2020 / 219665, discloses the general approach of embedding mathematical models of scene motion in the static VIO problem to enable accurate estimation of camera posture in a moving 3D scene. The present application extends that work by specifying mathematical models of scene motion that describe moving subjects such as living beings. Documents FR 3094125 A1, US 9189886 B2, and US 10529137 B1 also disclose some similar methods. Summary of the invention The principles of the present invention provide a system that measures the position of a camera moving across a surface relative to the body of a moving subject, while simultaneously estimating the body's posture and shape. In one embodiment, a computer connected to a camera, an inertial measurement unit (IMU), a depth sensor, and software provide estimates of the camera's location, attitude, and speed, as well as the body's posture. The software works by integrating the camera's movement with respect to the subject's skin characteristics and within the 3D environment, correcting the subject's movement according to a model of the subject's kinematics and dynamics.In addition to providing an estimate of camera position and body posture, this implementation can repeatably map the location of each image pixel to a fixed coordinate system that flexes with the subject's body surface. Brief description of the drawings. The accompanying drawings, which are incorporated herein and form a part of this descriptive memorandum, illustrate embodiments of the invention and, together with the detailed description provided below, serve to explain various aspects of the invention. Figure 1 shows the system diagram of a typical embodiment of the invention. Figure 2 shows the parameterized body mesh that models the subject. Figure 3 shows the matching of detected body landmarks to the appropriate locations in the body mesh model through the body landmark objective function component. Figure 4 represents the objective function components associated with 3D depth sensors. Figure 5 shows the linear approximation of the 3D depth functions. Detailed description Visual-inertial odometry (VIO), photogrammetry, and related techniques often fail to produce accurate solutions when the field of view contains a living subject. Their mathematical basis relies on the assumption that image features are in static 3D locations. A living, breathing subject violates this assumption, resulting in significant errors in camera position estimates and in estimations of the subject's 3D shape. The principles of the present invention, as elaborated herein, overcome this drawback. A system constructed according to the principles of the present invention can enable the subject's movement by taking advantage of the known characteristics of the subject's shape and movement. Whereas traditional photogrammetry solves a series of mathematical optimization problems to find a point cloud representing the subject, and then fits a mesh to that point cloud, the embodiments disclosed herein begin with a parameterized mesh capable of matching the range of shapes the subject can assume. Therefore, these embodiments solve a series of optimization problems similar to those solved in VIO and photogrammetry, except that they directly solve for the time-varying parameters of the mesh.This approach leverages the knowledge that the body is a solid with a known topology, and with a roughly known size, shape, and motion characteristics. The ability of traditional VIO and photogrammetry to resolve more general shapes often produces meaningless results when presented with data containing a moving subject, specular reflections, or other common phenomena. Referring to Figure 1, one embodiment of the invention includes one or more processors 102 connected to a camera 106 and optional additional sensors 108, 110, 112, 114, 124, 126. This enables skin mapping through the integration of the body's kinematic and dynamic features into the camera's pose estimation calculations. A multitude of complementary input data types are supported and have been observed in a prototype implementation, including: 1) image feature locations tracked in each image frame 120; 2) inertial data produced by an inertial measurement unit (IMU) 116 rigidly attached to the camera 106; and 3) output from a depth sensor 108 such as a LiDAR or structured light depth camera.This embodiment of the invention also supports the use of the output of separate software modules, such as motion capture software 110, visual-inertial odometry (VIO) software 126, plane detection software 112, and body segmentation software. The visual fuser 102 outputs 118 estimated camera position and body posture associated with each frame. The visual reference unit 104 outputs 128 associated frames. These can be stored for further offline processing and later data review. The remainder of the following description is organized as follows. First, the means of parameterizing the mesh to take the shape of living subjects are discussed. Second, different forms of data that can serve as input are covered. Third, the formulation of an estimator for calculating the probable posture and shape of the subject, and the camera's posture over time, is explained. Parameterized mesh Body kinematics are approximated using one or more models of the subject, in which the locations of the bones of an internal skeleton are modeled. Referring to Figure 2, each bone of the skeleton is described by a coordinate transformation from global coordinates to bone coordinates, placing the bone in its position within the skeleton. A bone hierarchy also describes bone connections as a network of major and minor bones physically connected through joints. In one embodiment, the center of the pelvis is defined as the root bone, from which the spine extends upward as a line of children, and the hips and legs descend as other lines of children. The bones in the model need not correspond to physical bones in the subject's body, but they should represent the subject's range of motion.For example, an obese person may have abdominal adipose tissue that can be moved to obtain images of the underlying skin. Although the adipose tissue and other mobile soft tissues lack bones in the anatomical sense, adding virtual bones to the modeled skeleton allows for the representation of movement in these soft tissues. A collector mesh surface 130, consisting of vertices V 136 connected by edges E 138, is used to represent the subject's skin. The position of each mesh vertex is calculated using techniques commonly referred to as "skinning" in the computer graphics industry. In one embodiment, Linear Blend Skinning (LBS) is used. In LBS, the positions of vertex j V are calculated within each bone coordinate system i B in some reference posture. A weight matrix with elements wij captures the dependence of each vertex position 136 on the location and orientation of each bone 132. The location and orientation of each bone i can be captured through a 4x3 bone transformation matrix Ti, such that the position of each vertex is the weighted sum of its transformed positions from each bone coordinate system to global coordinates.Therefore, as the 132 bones move, their positions in the global coordinates shift, and each mesh vertex 136 moves according to the 132 bones it depends on with a non-zero weight. Other representations can also be used for coordinate transformation, such as a 3-element translation vector plus a 3-element rotation vector or a 4-element quaternion. A computational advantage of this type of approach is its linearity. In particular, the sensitivity of the position of any vertex in global coordinates to any of the elements of Ti can be easily calculated and used to quickly find solutions within standard convex optimization solvers. Similarly, the position of any point on the skin specified by barycentric coordinates varies in the same way. Variations in shape can also be captured through two additional techniques: additional hand-designed parameters and linear interpolation between a set of vertex-at-rest positions. An example of a hand-designed parameter is the bone-specific circumference scaler. This parameter multiplies the x and y position of each vertex in bone coordinates (adopting, without loss of generality, the convention that the z-dimension is along the length of the bone) to expand or reduce the circumference of the body part associated with that bone. Similarly, a bone-specific length scaler that multiplies the z-position of each vertex in bone coordinates lengthens or shortens each bone and its associated body parts. Even when such hand-designed parameters capture the body variations of interest, it may be appropriate to use a second technique: linear interpolation between a set of resting vertex positions.In this technique, a second weighting matrix with xik elements is used to express the similarity of each scanned body part to the corresponding part of each reference body in a library. In equation form, this approach can be useful for capturing the different ways bodies are shaped, representing each person's shape as a combination of the shapes of other people within a shape library. Input data - tracked entities VO and VIO software packages use a variety of approaches to find and track visual features of interest from image frame to image frame. For example, OpenCV has several built-in algorithms, such as goodFeaturesToTrack, to find points with strong contrast in two directions, and calcOpticalFlowPyrLK, to track those points frame by frame. Others have used neural network approaches. Regardless of the exact source, the xy image coordinates of these features in each frame are an optional input to the invention. If this input is not provided or calculated within the invention, the output of a standard VIO package may be used instead. Input data - inertial Data from an inertial measurement unit (IMU) rigidly attached to the camera is an optional input that enables enhanced performance. Providing tracked features alongside inertial data allows for deep integration of the two data types, as commonly done in VIO software, along with other considerations contained in other inputs. Without these, the output of a VO / VIO package can be used by the invention, a configuration that allows for the correction of VO / VIO errors based on consistency with the object model and other data. Input data - VO / VIO estimates Several VIO packages are widely available. In fact, some are now integrated into software available as standard on Android and iOS-based mobile devices and are carefully designed for efficiency on that hardware. To facilitate implementation, embodiments of the present invention can utilize the output of such packages. Specifically, the 6-degree-of-freedom camera posture at the time of each image capture can be used as input, which corrects the VO / VIO posture estimate based on the other data. Furthermore, some VO / VIO software provides 3D positions of tracked features, which can be input in the same way as depth sensor data. Input data - depth sensors A variety of depth sensors have been integrated into consumer mobile devices, and their data can help produce more accurate and robust results. Depth sensor data can be converted into 3D points in the sensor's coordinate system. The global coordinate positions of these 3D points can be calculated using the transform that expresses the sensor's coordinate system in global coordinates. Therefore, comparing the 3D data from the depth sensors with the modeled 3D shape of the subject and environment provides embodiments of the present invention with information related to the sensor's coordinate system, as well as information related to the 3D shape of the modeled subject and environment. Input data - labeled entities As the system collects image data, it can run a variety of feature labeling algorithms to uniquely identify features in the environment and on the body. Examples of such algorithms include scale-invariant feature transform (SIFT), accelerated robust feature transformation (SURF), and specialized skin lesion identification algorithms. The output of these algorithms—image coordinates and feature labels—is an optional input for some embodiments of the invention and is used to reduce errors that might otherwise accumulate in the system's estimates. Input data—detected planes Many augmented reality software systems on mobile devices can detect and locate flat surfaces within the environment. Embodiments of the present invention can use this type of information in several ways. First, the presence of a ground plane provides a lower boundary for subject positioning. If the subject is standing on the ground, the difference between the top of the subject and the ground plane provides the subject's height, which is particularly useful for initial scaling. If the subject is placed on the ground with space around it, segmenting the subject from other parts of the environment is simplified because ground-related data can be quickly identified as such by its 3D location. Once identified, floor plan data and other data not connected to the subject are easily segmented geometrically from the subject.The locations of the detected planes are therefore provided to the invention when they are available. Input data - detected body landmarks A variety of software systems are available for detecting and locating humans and other subjects within an image frame. Neural networks can recognize bodies and body landmarks, and systems provided in common mobile operating systems also provide the locations of inferred landmarks that are obscured or outside the field of view. The embodiments of the present invention utilize this data, when available, to align the modeled subject with the actual subject. Key body landmarks, such as skeletal joints, facial features, and the like, may be provided as 3D locations within a VIO reference frame, or they may be provided as 2D locations within each image frame. Input data - pre-labeled entities If the subject has been previously scanned, and the locations of previously labeled features have been mapped onto the subject's body, then the labels created during a new scan of the subject can be compared to the previously mapped labeled features. High-confidence matches between the previously labeled features and the new features are then treated as detected body landmarks. Posture estimation Using input data such as those described above, one embodiment of the present invention estimates the camera posture and the body shape and posture, i.e., the six variables for camera orientation and position, the variables comprising the bone coordinate transforms and other parameters of the modeled skeleton, and optionally, the xik weights. In one embodiment, this estimation is performed by finding the solution to a nonlinear optimization problem in which the objective function is a weighted sum of terms, each term expressing the error between an aspect of the modeled body and the observations contained in the input data. If M aspects of such error are included, the objective function can be expressed as: In this case, the decision variables q represent the parameters being optimized, i.e., camera posture parameters, body posture parameters, any weight parameter, etc. The function fm(.) is a cost function for the error aspect m, and pm(.) is a loss function as commonly used to improve robustness to outliers in nonlinear optimization. An important aspect of the invention lies in the cost functions fm(.) used for each type of error. These aspects are as follows. Body landmark positions: Figure 3 illustrates the use of body landmark positions in embodiments of the present invention. Landmarks 142 in the modeled parameterized mesh 130, corresponding to landmarks detected in the input data 140, are mapped onto the model and represented using the same weighted deformation scheme (e.g., linear blending deformation) as used to represent skin vertices. In this way, the 3D position of each landmark 142 on the modeled subject 130 is represented as a function of the optimization decision variables.If the 140 detected body landmarks are provided in 2D coordinates, the cost function measures the 2D distance from the detected landmarks to the projection of the modeled landmarks onto the image plane, where the projection conforms to the camera posture decision variables. If the detected body landmarks are provided in 3D coordinates, the cost function can (1) measure the 2D distance from the projection of the detected landmarks onto the image plane to the projection of the modeled landmarks onto the image plane, where the projection of the former is made according to the camera projection estimate used by the landmark detector, and the projection of the latter is made according to the camera posture decision variables, or (2) measure the 3D distance from the detected landmarks compensated by parameters representing a VIO drift estimate to the modeled landmarks. The option that works best depends on the method used internally by the body landmark sensor to geolocate the body landmarks. Depth sensor data: Embodiments of the present invention may use available depth sensor data, as depicted in Figure 4. Using this data, the system first segments the depth sensor data into on-body points, off-body points, and indeterminate points. This process relies on identifying the ground or other surface on which the subject is resting, using built-in horizontal plane detection algorithms or input from a horizontal plane detector 112. Once the system has tracked the body with a certain degree of accuracy, it performs segmentation by taking the points closest to the previously tracked body and placing them in the on-body point set. It then adds any neighbors of the on-body points to the on-body point set, and this process is repeated until there are no more neighbors to add.Neighbors to a point p in this process are defined as points within a distance d from point pd that are not close enough to the ground or other environmental features to potentially be part of the out-of-body or indeterminate point sets. The distance d can be chosen according to the sensor's characteristics: it must be greater than the spacing between points in the sensor array not projected onto the subject (potentially considerably greater if the depth sensor sometimes fails to detect points where they land on the body), and less than the distance between the subject and surrounding non-body objects. Points that comfortably exceed the distance d from the body when this process is complete are placed in the out-of-body point set. If the body has not yet been reliably tracked, another method is used to initialize the process. Using a body landmark detector, interacting with the device user, and detecting a cluster of points in the point cloud supported from the ground and of the correct proportions to be the subject are all methods that can be used in different circumstances. Once the segmentation is complete, in some embodiments of the invention, two types of data error can be measured, creating two cost functions. The first type of cost function penalizes solutions based on the distance of the modeled skin 154 from the observations of points 156 on the body. The second type of cost function penalizes solutions based on the degree to which the modeled body 130 intersects rays 164 from the sensor through non-body points 162 that are farther from the sensor than the body. In the first type of cost function, the nearest point on the modeled skin mesh to each detected 3D point is first found using a method such as an axis-aligned bounding box tree (ABB).To eliminate the possibility of overlapping surfaces obscured from the depth sensor's perspective, the AABB tree is constructed using only mesh faces with surface normals pointing toward the sensor. Referring to Figure 5, the natural cost function to use would then be the 3D distance 172 from the nearest point 154 on skin 130 to the associated 3D detected point 156. However, because the AABB tree operation is computationally expensive and complicated to integrate into nonlinear optimization solvers, it is advantageous to replace the skin point where the nearest point 154 was initially calculated before any solver iteration moves the skin by changing the bone transform decision variables.Therefore, the nearest point 154 is represented using the same weighted deformation scheme (e.g., linear blending deformation) as is used to represent skin vertices. In linear blending deformation, the row of the weighting matrix corresponding to this point can be found by averaging the rows of the weighting matrix corresponding to the vertices that define the face on which the nearest point lies, weighted according to the barycentric coordinates of the nearest point. As the solver iterations update the bone transform decision variables, the skin point 154, which was originally closest to the detected point 156, moves with the skin. Recalculating the closest point on the skin through the AABB tree would generally produce a different point 178, so that the detected point 156 lies along the normal 174 to the skin from point 178. Using the natural cost function that minimizes the 3D distance 172 would inhibit the tangential movement of the skin and the efficient solution of the optimization. One solution to this problem is through the use of a cost function that measures only the normal skin component 172 of the distance from the nearest point 154 to the observed point 156. This cost function approximates the skin surface as a plane tangent to the originally calculated nearest point 154 in order to calculate the distance of the observed point 156 from the skin. This local linearization allows the Jacobian of the cost function to be calculated, enabling fast iteration by the solver, and imposes no restrictions on the tangential movement of the skin. This choice of cost function drives the optimization toward solutions where a linear approximation 170 to the skin surface in the general neighborhood of the nearest point 154 moves toward the observed point 156. Referring again to Figure 4, the second type of cost function penalizes solutions based on the degree to which the modeled body 130 intersects the sensor rays 164 through the non-body points 162. For this cost function, an AABB tree can be used to find intersections between the modeled body mesh 130 and the 3D sensor rays 164 through the observed non-body points 162. Any such intersection indicates that a portion of the modeled mesh 130 is in a space where the physical subject is not, or that the modeled 3D sensor 150 is in a location where the physical sensor is not, or both. In the preferred embodiment, all intersections of the rays with the modeled body are found. These intersections occur in pairs: one intersection where the ray enters the body 158 and another intersection where it exits the body 160.The midpoints between the entry and exit intersections are found by linear interpolation and are, by design, internal to the modeled body. Next, the nearest point on the mesh to each of these midpoints is found in a direction normal to the intersection ray. The distance from the nearest points to the midpoints constitutes the cost function that must be minimized, triggering adjustments to the body to move it out of the path of the rays. Derived from VO / VIO If estimates from a separate VO / VIO system are provided as input in particular embodiments of the invention, those estimates will drift as the subject moves. The form of drift depends largely on the VO / VIO system used, the accuracy of the inertial information, the degree to which the VO / VIO system relies on static versus moving features in any given situation, and a variety of related issues. Including two VIO drift elements in the cost function helps drive solutions to incorporate the available information into the VIO estimates without being deviated from the correct solution by VIO drift errors. The two VIO drift elements that can be used in embodiments of the present invention are (1) incremental VIO drift with respect to the previous VIO drift estimate, and (2) total VIO drift. For VIO systems with sufficiently accurate angular inertial frames, it may be sufficient to use only a 3-DOF translational drift component as the cost function. Other systems may require a 6-DOF drift estimate. The different drift dimensions should generally be weighted differently. Angular drift and translational drift certainly have very different characteristics and should have very different weights. Similarly, pitch and roll drift are usually eliminated from estimates by the VIO algorithm using gravity measurements, whereas heading generally lacks such a reliable reference and grows over the course of navigation if left uncorrected. The VIO drift decision variables related to a frame can be added to the 3D translation component of the VIO camera pose estimation for the frame (in the case of 3-DOF drift), or to the 6D translation + rotation vector representation of the VIO camera pose for the frame in the case of 6-DOF drift. Internal VO / VIO cost function If VO / VIO is performed internally in an embodiment of the invention, traditional elements of the VO / VIO system cost functions, such as reprojection error and embedded inertial discrepancies, can be used and added to the optimization objective function. The use of the keyframe approach greatly improves computational performance. Subject movement cost functions If sensors are placed on the subject, the discrepancy between the detected positions, accelerations, etc. and the modeled body posture is a useful cost function. With or without instrument placement, a subject attempting to maintain a posture justifies two cost functions: (1) the difference between the target posture and the modeled body posture, and (2) the speed of movement of the body posture. Small differences and small amounts of movement are unavoidable; therefore, the weights on these target coefficients must be set appropriately. Skeletal integrity cost function The location of a secondary bone in the primary bone coordinate frame should generally not change. The joint connecting a primary bone to a secondary bone is rigidly attached to the primary bone when both bones are part of an anatomical skeleton. Including the difference in 3D position between the primary bone's coordinates at the start of the joint and its coordinates throughout the solution process can prevent unrealistic separation of the skeletal components. Use of optimization result To enable fast, real-time solutions, the optimization problem for each frame can be formulated as an independent problem with initial conditions established by the solution to the previous frame's problem. Alternatively, a keyframe approach can be used, in which a number of recent frames and previous keyframes are solved together, with initial conditions set according to the solution determined before the problem's time horizon begins. After a scan is completed, solutions can be re-optimized in post-processing across the entire set of frames simultaneously. Whether used in real time or post-processing, solutions to the modeled body's position and camera pose provide an initial indication of which pixels in each image belong to which parts of the modeled skin. This initial indication can be used in several ways. It can be used to shade the body model according to the captured pixel density of the skin, or according to the sharpness or other captured image quality measures of the skin, or a combination thereof. It can also be used to texture the model's skin according to the best images obtained for each skin area. Furthermore, it can serve as a starting point for further refining the body model using local displacement textures and / or dense model reconstruction with a higher feature count, where mismapping is minimized by adhering to the initial solution.

Claims

1. A system (126) for capturing and biorecording images of a living subject, comprising: a. a camera (106), b. a processing unit configured to receive images of a skin surface of the subject from the camera (106), the processing unit controlling the camera (106) to reveal images that include a plurality of images taken at close range of the subject, each image capturing less than the entire body of the subject in its field of view, the processing unit further causing images to be taken at different times, where movement occurs between those different times causing deformation of the subject's skin, c.a memory containing a parameterized mathematical model that characterizes the subject and its dynamics and kinematics of movement, including a memory representation of a body coordinate system of reference fixed to the subject's body and flexing with the body surface (158, 160) as the subject moves, and d.A program in the processing unit for calculating parameter estimates of the mathematical model characterizing the subject and its movement and the posture and movement of the camera, wherein the processor (102): determines a location within the reference body coordinate system of pixels in each collected image that captures the subject's skin, and minimizes a distance (172) between the pixel coordinates of more than one image belonging to a common skin feature, as measured in the reference body coordinate system or a projection thereof onto each image, such that the subject's movement between points (162) at the time the images were taken is consistent with the characterization of the mathematical model of the subject's movement.

2. The system (126) of claim 1 further comprising a depth sensor (108) aligned with at least part of the camera view. 3.The system of claim 1, further comprising wherein the program in the processing unit implements a feature detection algorithm that finds and / or tracks features within the 3D scene, providing their 2D image locations through at least two images.

4. The system (126) of claim 1, wherein the program in the processing unit implements a neural network-based artificial intelligence algorithm to detect the locations of body landmarks (158, 160) comprising one or more joints (134) and facial features.

5. The system (126) of claim 1, wherein the program in the processing unit implements a visual-inertial odometry (VIO) algorithm that estimates the position and orientation of the camera (106) from feature tracking in a static scene. 6.The system (126) of claim 1, wherein the parameterized mathematical model of the subject includes the location and orientation of the skeletal structure from which the location of the vertices of a mesh (130) representing the subject's skin is calculated through linear blending deformation.

7. The system (126) of claim 6, wherein the parameterized mathematical model of the subject includes additional parameters representing adjustments to measurements of the subject's height, weight, or circumference.

8. The system (126) of claim 7, wherein the mesh vertex displacements used in the linear blending deformation are determined by a set of parameters that linearly weight the mesh vertex displacements from a library of body meshes (158, 160) representing different shapes of the subject. 9.The system (126) of claim 8, wherein the processing unit calculates the camera (106) and subject states by solving an optimization problem that minimizes the discrepancy between the detected data (140) and the data (140) expected to be detected as a result of applying the subject and camera (106) motion models to the states.

10. The system (126) of claim 9, wherein a factor of the cost function is a mathematical function of the normal displacement to the mesh surface (130) of 3D points (162) detected by a depth sensor (108) aligned with at least part of the camera view.

11. The system of claim 9, wherein a factor of the cost function is the distance (172) by which the subject model occludes 3D observations of an environment behind the subject produced by a depth sensor (108) aligned with at least part of the camera view. 12.The system (126) of claim 9, wherein a cost function factor is the 3D displacement of the detected body landmarks (158, 160) from their normal positions on the grid (130).

13. The system of claim 9, wherein a cost function factor is the average velocity of movement of the subject's skeletal joints.

14. The system of claim 9, wherein a cost function factor is the difference in the location of a joint (134) between a primary bone and a secondary bone, expressed in the coordinate system of the primary bone, from its location on the initial reference skeleton.

15. The system (126) of claim 9, wherein the objective function weights are adjusted according to the degree of uncertainty in the measurements.

Images