Systems and methods for mapping skin
By combining the inertial measurement unit and depth sensor, using the kinematic model of the object, the accurate estimation of the camera posture and body shape on the living object is achieved, solving the problem of measurement difficulties in the prior art, and improving the efficiency of the skin image mapping system and accuracy, reducing costs.
Patent Information
- Application Number
- CN202380020296.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-04
- Filing Date
- 2023-01-04
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to accurately measure and track camera posture on living objects, resulting in high cost and inefficiency in detecting skin changes, especially in patients with high risk of melanoma.
By combining inertial measurement units, depth sensors and software computers, using kinematic and dynamic models of objects, we provide a system that can track camera poses and map image pixels to the object's body surface to achieve accurate postures and shapes of moving objects using kinematics and dynamics models. estimate.
It realizes rapid capture of large amounts of close-up images on living objects, improves image quality and efficiency, reduces the cost and complexity of the system, and can detect skin changes early.
Smart Images

Figure CN118922857A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to mapping systems and methods for evaluating skin tissue. Background Art
[0002] The detection and management of some diseases benefit from photographic documentation of the skin. A baseline set of images collected in an initial imaging session is compared to the appearance of the skin at a later time to allow detection and measurement of skin evolution. In patients at high risk for melanoma, these approaches have been shown to allow detection of melanoma at an earlier stage than standard treatment, while requiring the removal of fewer benign lesions. Current methods of collecting, organizing, and using images are costly and logistically complex, limiting their adoption.
[0003] Traditionally, total body photography (TBP) is performed by a trained photographer in a studio set up for this purpose using an expensive camera. The photographer positions the subject in a number of reference poses to allow as many photos of the body's skin as possible to be taken. The process of positioning the subject and taking the photos can require 20 minutes or more, often requiring a separate appointment for the patient. Using the resulting photos to identify changes in the skin is time consuming and complicated, with dermatologists typically switching their eyes back and forth between the photos and the patient's skin in an attempt to detect changes. The cost of TBP is only available to patients at very high risk for melanoma.
[0004] Several companies have developed products to simplify the collection of TBP. One class of products takes the form of a booth equipped with dozens of high-quality cameras, each focused on a relatively small area of skin. The patient enters the booth, assumes each reference pose, the camera takes a picture, and the data is stored. These systems have been shown to be effective in several studies, but the high capital cost of the system and the space required to house it have limited their adoption. They do not appear to have reduced the cost of TBP.
[0005] Other companies have developed smartphone apps to modernize the traditional TBP workflow. The apps help the photographer guide the patient into a reference pose, take the photos, and store and organize the photos according to pose and area of the body covered. In this process, there is a trade-off between the quality of the image obtained and the number of photos required to cover the body. Most apps only take 2-4 photos to cover the body to limit the time required and the complexity of aligning each image. The resulting image quality is not adequate for assessing potential skin cancer: at the distance from the subject at which the photos were taken, the pixel density is too low to resolve important features of cancerous lesions. There is no good option to bring the camera close to the body because it is difficult for the photographer to consistently capture the correct part of the body from a close distance, and carefully aligning each of the large number of images would be time-consuming.
[0006] It is an object of the invention described herein to overcome these trade-offs and, in particular, to provide a method for tracking the pose position of a camera relative to a subject's body independent of body motion, so that a large number of close-up images can be captured quickly, each of the images being mapped to a location on the subject's body.
[0007] Existing techniques, such as photogrammetry, visual odometry (VO), and visual-inertial odometry (VIO), require static 3D objects and environments. Living objects are not static enough to reliably generate accurate VO, VIO, or photogrammetry results. Accordingly, new methods must be developed to achieve the goals of the present invention.
[0008] U.S. Provisional Patent Application No. 62 / 838,662 (published under the Patent Cooperation Treaty as PCT / WO2020 / 219665) discloses a general approach to embedding a mathematical model of scene motion into a static VIO problem to allow accurate estimation of camera poses in a moving 3D scene. The present application extends that work by specifying a mathematical model that describes scene motion of moving objects (e.g., living objects). Summary of the invention
[0009] The principles of the present invention provide a system for measuring the pose of a camera traveling on a surface relative to a moving object body, while also measuring the pose and shape of the body. In one embodiment, a computer connected to a camera, an inertial measurement unit (IMU), a depth sensor, and software provides an estimate of the position, attitude, and velocity of the camera and the pose of the body. The software operates by integrating the motion of the camera relative to the features on the skin of the object and in the 3D environment, and it corrects the motion of the object according to the kinematic and dynamic models of the object. In addition to providing an estimate of the pose of the camera and the pose of the body, the embodiment can repeatedly map the position of each image pixel to a coordinate system that is fixed to the surface of the object body and bends with it. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the detailed description given below, serve to explain aspects of the invention.
[0011] Figure 1 A system diagram showing an exemplary embodiment of the present invention.
[0012] Figure 2 A parameterized body mesh modeling the object is shown.
[0013] Figure 3 Matching detected body landmarks to appropriate locations in the body mesh model via a body landmark objective function component is shown.
[0014] Figure 4 Depicted are objective function components associated with a 3D depth sensor.
[0015] Figure 5 A linear approximation of the 3D depth function is shown. DETAILED DESCRIPTION
[0016] Traditional visual inertial odometry (VIO), photogrammetry, and related techniques often fail to produce accurate solutions when living objects are included in the field of view. Their mathematical foundations include the assumption that image features are in static 3D positions. Living, breathing objects violate this assumption, leading to significant errors in camera pose estimation and object 3D shape estimation.
[0017] The principles of the present invention set forth herein overcome this shortcoming. Systems constructed in accordance with the principles of the present invention can allow object motion by utilizing known properties of the shape and motion of an object. Traditional photogrammetry solves a series of mathematical optimization problems to find a point cloud representing an object, and then fits a mesh to the point cloud, while the embodiments disclosed herein start with a parameterized mesh that can match the range of shapes that an object can present. Then, these embodiments solve a series of optimization problems, such as those solved in VIO and photogrammetry, except that it directly solves the time-varying parameters of the mesh. The method utilizes the knowledge that an object is a solid with a known topology, with a roughly known size, shape, and motion characteristics. The ability of traditional VIO and photogrammetry to solve more general shapes usually produces meaningless results when presenting data containing moving objects, specular reflections, or other common phenomena.
[0018] refer to Figure 1 , an embodiment of the present invention includes one or more processors 102 connected to a camera 106 and optional additional sensors 108, 110, 112, 114, 124, 126. It allows mapping the skin by integrating body kinematics and dynamics into the camera pose estimation calculation. A variety of complementary input data types are supported and have been demonstrated in prototype implementations, including: 1) the position of image features tracked in each image frame 120; 2) inertial data generated by an inertial measurement unit (IMU) 116 rigidly attached to the camera 106; and 3) the output of a depth sensor 108 such as a lidar or structured light depth camera. This embodiment of the present invention also supports the use of outputs of separate software modules, such as motion capture software 110, visual inertial odometry (VIO) software 126, plane detection software 112, and body segmentation software.
[0019] The vision fuser 102 outputs 118 the estimated camera pose and body pose associated with each frame. The visual reference unit 104 outputs 128 the associated frames. These can be stored for further offline processing and later review of the data.
[0020] The rest of the description is organized as follows. First, the means of parameterizing meshes to represent the shape of living objects are discussed. Second, various forms of data that can be used as input are introduced. Third, the formulas for the estimator are explained, which are used to calculate the possible poses and shapes of objects and the poses of the camera over time.
[0021] Parameterized Meshes
[0022] The kinematics of the body are approximated using a model of one or more objects, where the positions of the bones of the internal skeleton are modeled. Figure 2 , each bone b∈B 132 of the skeleton is described by a coordinate transformation from world coordinates to bone coordinates, thereby placing the bone at its location in the skeleton. The bone hierarchy also describes the connections of the bones as a network of parent bones and child bones that are physically connected by joints 134. In one embodiment, the center of the pelvis is defined as a root bone, from which the spine extends upward as one child bone line, and the hips and legs extend downward as other child bone lines. The bones in the model do not need to correspond to physical bones in the subject's body, but they should represent the subject's range of motion. For example, an obese person may have abdominal pannus that can be moved to image the underlying skin. Although pannus and other movable soft tissues lack bones in an anatomical sense, the addition of virtual bones in the modeled skeleton allows for representation of the motion of these soft parts.
[0023] A manifold mesh surface 130 consisting of vertices V 136 connected by edges E 138 is used to represent the skin of the object. The position of each vertex of the mesh is calculated using a technique commonly referred to as "skinning" in the computer graphics industry. In one embodiment, linear blend skinning (LBS) is used. In LBS, the position of a vertex j∈V is calculated in the coordinate system of each bone i∈B at some reference pose: With element w ij The weight matrix is such that Capture the dependency of the position of each vertex 136 on the position and orientation of each bone 132. The position and orientation of each bone i can be determined by the 4×3 bone transformation matrix T i is captured so that the position of each vertex is the weighted sum of its position transformed from each bone coordinate system to the global coordinate system, i.e. Therefore, as bone 132 moves, The position of also moves, and each mesh vertex 136 moves according to the bone 132 with non-zero weight on which it depends. Other representations of coordinate transformations may also be used, such as a 3-element translation vector plus a 3-element rotation vector or a 4-element quaternion.
[0024] The computational advantage of this approach is that it is linear. In particular, the position of any vertex in world coordinates is proportional to the position of any element T i The sensitivities of can be easily computed and used to quickly find solutions within a standard convex optimization solver. Similarly, the position of any point on the skin specified via centroid coordinates varies in the same way.
[0025] Shape deformation can also be captured by two additional techniques: additional hand-crafted parameters and linear interpolation between sets of stationary vertex positions.
[0026] An example of a hand-crafted parameter is a bone-specific perimeter scaler. This parameter multiplies the x and y positions of each vertex in bone coordinates (without loss of generality, adopting the convention that the z dimension is along the length of the bone), enlarging or reducing the perimeter of the body part associated with the bone. Similarly, a bone-specific length scaler multiplies the z position of each vertex in bone coordinates, thereby lengthening or shortening each bone and its associated body part.
[0027] Even in cases where such hand-crafted parameters capture the body variations of interest, it may be appropriate to use a second technique: linear interpolation between sets of vertex rest positions. In this technique, a second weighting matrix x with elements ik to express the similarity of each part of the scanned body to the corresponding part of each reference body in the library. Expressed in equation form, This approach can be used to capture the different ways in which bodies are shaped, representing each person's shape as a combination of the shapes of other people in a library of shapes.
[0028] Input Data - Tracked Features
[0029] VO and VIO packages use various methods to find and track visual features of interest on a per-image frame basis. For example, OpenCV has several built-in algorithms, such as goodFeaturesToTrack, for finding points with strong contrast in two directions, and calcOpticalFlowPyrLK, for tracking these points on a per-frame basis. Others have used neural network approaches. Regardless of the exact source, the xy image coordinates of these features in each frame are an optional input to the present invention. If this input is not provided or calculated within the present invention, the output of the standard VIO package can be used instead.
[0030] Input Data - Inertia
[0031] Data from an Inertial Measurement Unit (IMU) rigidly attached to the camera is an optional input that allows for improved performance. Providing tracked features and inertial data allows for deep integration of both types of data, as is typically performed in VIO software, along with other considerations included in other inputs. In the absence of these, the present invention can use the output of the VO / VIO package, a configuration that allows for correction of VO / VIO errors based on consistency with object models and other data.
[0032] Input Data - VO / VIO Estimation
[0033] Several VIO packages are widely available. In fact, some are now built into software available in standard installations on Android and iOS based mobile devices, and are carefully designed to be efficient with the hardware. For ease of implementation, embodiments of the present invention may utilize the output of such packages. Specifically, at the time of each image, the 6 degrees of freedom camera pose may be used as input, which corrects the VO / VIO pose estimate based on other data. Additionally, some VO / VIO software provides the 3D position of tracked features, which may be input as well as data from a depth sensor.
[0034] Input Data - Depth Sensor
[0035] Various depth sensors have been integrated into commodity mobile devices and their data can help produce more accurate and robust results. The data from the depth sensor can be converted to 3D points in the sensor coordinate system. The world coordinate positions of these 3D points can be calculated using a transformation that expresses the sensor coordinate system in world coordinates. Therefore, comparing the 3D data from the depth sensor with the modeled 3D shape of the object and environment provides embodiments of the present invention with information about the sensor coordinate system and information about the modeled 3D shape of the object and environment.
[0036] Input data - Label characteristics
[0037] As the system collects image data, it can run various feature labeling algorithms to uniquely identify features in the environment and on the body. Examples of such algorithms are Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and specialized skin lesion recognition algorithms. The outputs of these algorithms, image coordinates and labels of features, are optional inputs to some embodiments of the invention to reduce errors that might otherwise accumulate in the system's estimates.
[0038] Input data - detected planes
[0039] Many augmented reality software systems that exist on mobile devices can detect and locate planar surfaces within an environment. Embodiments of the present invention can use such information in several ways. First, the presence of a floor plane provides a lower limit for the location of an object. If the object is standing on the floor, the difference between the top of the object and the floor plane provides the object height, which is particularly useful for initial scaling. If the object is located on the floor and there is space around him / her / it, segmentation of the object from the rest of the environment becomes simple because data about the floor can be quickly identified by its 3D position. Once identified, the floor data and other data not related to the object are easily geometrically segmented from the object. Therefore, the location of the detected plane is provided to the present invention when available.
[0040] Input data - detected body features
[0041] Various software systems can be used to detect and locate the position of humans and other objects in image frames. Neural networks can recognize bodies and body landmarks, and systems provided in common mobile operating systems also provide the positions of inferred landmarks that are occluded or out of view. Embodiments of the present invention utilize this data (when available) to align modeled objects with real objects. They can be provided as 3D positions of key body landmarks (e.g., bone joints, facial features, etc.) within the VIO reference frame, or they can be provided as 2D positions within each image frame.
[0042] Input data - previously labeled features
[0043] If the object has been previously scanned, and if the locations of previously labeled features have been mapped to the object's body, the labels created during a new scan of the object can be compared to the previously mapped labeled features. A high confidence match between the previously labeled features and the new features is then considered to be the same as the detected body landmarks.
[0044] Pose Estimation
[0045] Using the input data as described above, one embodiment of the present invention estimates the camera pose as well as the shape and pose of the body, namely the six variables of camera orientation and position, the variables constituting the bone coordinate transformation and other parameters of the modeled skeleton, and the optional weights x ik In one embodiment, the estimation is performed by finding a solution to a nonlinear optimization problem, where the objective function is a weighted sum of terms, each term expressing the error between one aspect of the modeled body and the observations contained in the input data. If M aspects of such error are included, the objective function can be expressed as:
[0046] min q ∑ m∈1…M ρ m (||f m (qm1 ,…,q mk )∣| 2 )
[0047] Here, the decision variable q represents the parameters to be optimized, i.e., camera pose parameters, body pose parameters, any weighting parameters, etc. Function f m (.) is the cost function of the error in terms of m, and m (.) is a loss function, which is usually used to improve the robustness of outliers in nonlinear optimization. An important aspect of the present invention is that the cost function f for each type of error m (.) These aspects are as follows.
[0048] Body landmark location: Figure 3 Depicted is the use of body landmark locations in an embodiment of the present invention. Prior to this, landmarks 142 on a modeled parameterized assembly mesh 130 corresponding to landmarks detected in input data 140 are mapped into the model and represented via the same weighted skinning scheme (e.g., linear blend skinning) as used to represent skin vertices. In this way, the 3D position of each landmark 142 on the modeled object 130 is represented as a function of an optimization decision variable. If the sensed body landmark 140 is provided in 2D coordinates, the cost function measures the 2D distance from the sensed landmark to the projection of the modeled landmark on the image plane, where the projection is a function of the camera pose decision variable.
[0049] If the sensed body landmarks are provided as 3D coordinates, the cost function can either (1) measure the 2D distance from the projection of the sensed landmark on the image plane to the projection of the modeled landmark on the image plane, where the former projection is made according to the camera projection estimate used by the landmark detector and the latter projection is made according to the camera pose decision variables, or (2) measure the 3D distance from the sensed landmark to the modeled landmark shifted by parameters representing the VIO drift estimate. The option that performs best depends on the method used internally by the body landmark sensor to geolocate the body landmarks.
[0050] Depth sensor data: Embodiments of the present invention may use available depth sensor data, such as Figure 4As depicted. Using this data, the depth sensor data is first segmented into points on the body, points outside the body, and undetermined points. This process relies on identifying the floor or other surface supporting the object using a built-in horizontal plane detection algorithm or input from the horizontal plane detector 112. When the system has tracked the body with a certain degree of accuracy, it performs segmentation by taking the points closest to the previously tracked body and placing them in the set of points on the body. It then adds any neighbors of the points on the body to the set of points on the body, repeating this process until there are no more neighbors to add. In this process, the neighbors of point p are defined as points within a distance d from point p that are not close enough to the floor or other elements of the environment to potentially be part of the set of points outside the body or undetermined points. The distance d can be selected based on the characteristics of the sensor: it should be larger than the spacing between points in the sensor array that are not projected onto the object (potentially significantly larger if the depth sensor sometimes fails to detect points that they fall on the body), and smaller than the distance between the object and surrounding non-body objects. When this process is completed, points that comfortably exceed the distance d from the body are placed in the set of points outside the body.
[0051] If the body is not already reliably tracked, another method is used to initialize the above process. Using body landmark detectors, interaction with the user of the device, and detecting groupings of points in the point cloud supported from the floor and the correct scale of the object are all methods that can be used in different situations.
[0052] Once the segmentation is complete, in some embodiments of the invention, two types of errors can be measured from the data, creating two cost functions. The first type of cost function penalizes solutions based on the distance between the modeled skin 154 and the point observations 156 on the body. The second type of cost function penalizes solutions based on the degree to which the modeled body 130 intersects with non-body points 162 where rays 164 from the sensor pass farther than the body.
[0053] In the first type of cost function, the closest point on the modeled skin mesh to each sensed 3D point is first found via a method such as an axis-aligned bounding box (AABB) tree. To remove the possibility of matching occluded surfaces from the perspective of the depth sensor, the AABB tree is built using only faces of the mesh where the surface normal points toward the sensor. Figure 5, the natural cost function to use would be the 3D distance 172 from the closest point 154 on the skin 130 to the associated sensed 3D point 156. However, because AABB tree operations are computationally expensive and complex to embed in a nonlinear optimization solver, it is advantageous to replace the skin point at the initially calculated closest point 154 before any solver iteration moves the skin by changing the bone transformation decision variables. Therefore, the closest point 154 is represented using the same weighted skinning scheme (e.g., linear blend skinning) as is used to represent the skin vertices. In linear blend skinning, the row of the weight matrix corresponding to the point can be found by averaging the rows of the weight matrix corresponding to the vertices of the face defining the closest point, and weighted according to the barycentric coordinates of the closest point.
[0054] As the solver iteratively updates the bone transform decision variables, the point 154 on the skin that was originally closest to the sensed point 156 moves with the skin. Recalculating the closest point on the skin via the AABB tree typically produces a different point 178, so that the sensed point 156 lies on the normal 174 from the point 178 to the skin. Using a natural cost function that minimizes the 3D distance 172 will suppress tangential movement of the skin and optimize a valid solution.
[0055] The solution to this problem is by using a cost function that measures only the skin normal component 172 of the distance from the closest point 154 to the sensed point 156. The cost function approximates the skin surface as a plane tangent to the originally calculated closest point 154 in order to calculate the distance of the sensed point 156 from the skin. This local linearization allows the Jacobian matrix of the cost function to be calculated, thereby allowing the solver to iterate quickly, and it does not impose any restrictions on the tangential movement of the skin. This choice of cost function drives the optimization towards a solution in which the linear approximation 170 of the skin surface approximately near the closest point 154 moves towards the observation point 156.
[0056] refer to Figure 4, a second type of cost function penalizes solutions based on the degree to which the modeled body 130 intersects the ray 164 from the sensor through the non-body point 162. For this cost function, an AABB tree can be used to find intersections between the modeled body mesh 130 and the ray from the 3D sensor 164 through the observed non-body point 162. Any such intersection indicates that a portion of the modeled mesh 130 is located in space where the physical object is not, or that the modeled 3D sensor 150 is located where the physical sensor is not, or both. In a preferred embodiment, all intersections of the ray with the modeled body are found. These intersections occur in pairs: one intersection is where the ray enters the body 158 and the other intersection is where the ray leaves the body 160. The midpoints between the entry and exit intersections are found by linear interpolation, and by construction, they are located inside the modeled body. The point on the mesh that is closest to each midpoint along the normal direction of the intersecting ray is then found. The distance from the closest point to the midpoint constitutes the cost function to be minimized, thereby driving adjustments to the body to move it out of the path of the ray 164.
[0057] VO / VIO drift
[0058] If, in certain embodiments of the invention, estimates from a separate VO / VIO system 126 are provided as input, these estimates will drift as the object moves. The manner of drift depends largely on the VO / VIO system used, the accuracy of the inertial information, the degree to which the VO / VIO system relies on static versus moving features in any given situation, and various related issues. Including both elements of VIO drift in the cost function helps drive the solution to include the information available in the VIO estimates, rather than deviating from the correct solution due to VIO drift errors.
[0059] Two elements of VIO drift that can be used in embodiments of the present invention are (1) incremental VIO drift relative to the previous VIO drift estimate, and (2) total VIO drift. For VIO systems with a sufficiently accurate angular inertial system, it is sufficient to use only the 3-degree-of-freedom (DOF) translation drift component as the cost function. Other systems may require 6-DOF drift estimates. Different dimensions of drift should typically have different weights. Of course, angular drift and translation drift exhibit very different characteristics and should have very different weights. Similarly, pitch and roll drift are typically removed from the estimate by the VIO algorithm using gravity measurements, while heading typically lacks a reliable reference and, if not corrected, it will grow during navigation.
[0060] The frame-dependent VIO drift decision variable can be added to the 3D translation component of the frame's VIO camera pose estimate (in the case of 3-DOF drift), or to the 6D translation+rotation vector representation of the frame's VIO camera pose in the case of 6-DOF drift.
[0061] Internal VO / VIO cost function
[0062] If VO / VIO is performed internally in an embodiment of the present invention, traditional elements of the VO / VIO system cost function, such as reprojection error and integrated inertia difference, can be used and added to the optimization objective function. Using a keyframe approach can greatly improve computational performance.
[0063] Object motion cost function
[0064] If the object is equipped with sensors, the difference between the sensed position, acceleration, etc. and the modeled body pose is a useful cost function.
[0065] With or without the instrument, the subject attempts to maintain a pose justifying two cost functions: (1) the difference between the target pose and the modeled body pose, and (2) the rate at which the body pose moves. Small differences and small amounts of movement are unavoidable, so the weights of these objective coefficients should be set appropriately.
[0066] Skeletal integrity cost function
[0067] The position of a child bone in the parent bone coordinate system should generally not move. When both are part of the anatomical skeleton, the joint connecting the parent and child bones is firmly attached to the parent bone. Including the 3D position difference between the original parent bone coordinates of the joint and its coordinates during the solution process can prevent unrealistic separation of bone components.
[0068] Use of optimization results
[0069] To allow fast solving in real time, the optimization problem for each frame is formulated as an independent problem whose initial conditions are set by the solution of the problem for the previous frame. Alternatively, a keyframe type of approach can be used, where multiple recent frames and previous keyframes are solved together and the initial conditions are fixed based on the solution determined before the start of the problem time horizon.
[0070] After the scan is complete, the solution can be re-optimized for the entire collection of frames simultaneously in post-processing.
[0071] As a result of real-time or post-processing use, the solution to the modeled body position and camera pose provides an initial indication of which pixels of each image belong to which parts of the modeled skin. This initial indication can be used in several ways. It can be used to color the body model according to the captured skin pixel density, or according to the clarity of the captured skin or other image quality measures, or according to a combination of them. It can also be used to texture the model's skin based on the best image obtained for each area of the skin. In addition, it can be used as an initial solution to further refine the body model using local displacement textures and / or dense model reconstruction with a large number of features, where false correspondences are minimized by requiring compliance with the initial solution.
Claims
1. A system for capturing and biometrically registering images of a living subject, comprising: a. Camera, b. a processing unit configured to receive an image from said camera, c. a memory containing a parameterized mathematical model characterizing the object and its possible motion, and d. A program on the processing unit for computing estimates of the subject's body pose and motion and the camera's pose and motion, such that the position of each pixel in the collected images on the surface of the subject's body can be determined.
2. The system of claim 1 further comprising a depth sensor aligned with at least a portion of the camera view.
3. The system according to claim 2, wherein: The depth sensor is a lidar depth sensor.
4. The system of claim 1, further comprising an inertial measurement unit (IMU) rigidly attached to the camera.
5. The system of claim 1 further comprising a compass or magnetometer rigidly attached to the camera.
6. The system according to claim 1, further comprising, wherein, The program on the processing unit executes a feature detector algorithm that finds and / or tracks features within a 3D scene, providing a 2D image position of the feature on at least 2 images.
7. The system according to claim 1, wherein: The program on the processing unit executes an artificial intelligence algorithm to detect the locations of body landmarks, including one or more of joints and facial features.
8. The system according to claim 1, wherein: The program on the processing unit executes a visual inertial odometry (VIO) algorithm that estimates the position and orientation of the camera based on tracking features in a static scene.
9. The system according to claim 1, wherein: The parameterized mathematical model of the object comprises the position and orientation of a skeletal structure, from which the positions of the vertices of a mesh representing the skin of the object are calculated by linear blend skinning.
10. The system according to claim 9, wherein: The parameterized mathematical model of the subject includes additional parameters representing adjustments for the subject's height, weight, waist circumference, and / or similar measurements.
11. The system according to claim 10, wherein: The mesh vertex offsets used in linear blend skinning are determined by a set of parameters that linearly weight the mesh vertex offsets from a library of body meshes representing different object shapes.
12. The system according to claim 11, wherein: The processing unit computes the camera and object states by solving an optimization problem that minimizes the difference between the sensed data and the data predicted to be sensed due to applying the object and camera motion models to the states.
13. The system according to claim 12, wherein: One factor in the cost function is a mathematical function of the mesh surface normal displacement of the 3D points sensed by the depth sensor.
14. The system according to claim 12, wherein: One of the factors in the cost function is the normal displacement of the mesh surface of the 3D points estimated by the VIO algorithm.
15. The system of claim 12, wherein: One factor in the cost function is the surface normal component of the reprojection error.
16. The system of claim 12, wherein: One factor in the cost function is the distance that the object model occludes the 3D view of the environment behind the object, as produced by the depth sensor.
17. The system of claim 12, wherein: One factor in the cost function is the 3D displacement of the sensed body landmarks from their normal position on the grid.
18. The system of claim 17, wherein: The normal position of the body landmark is determined by the position of the landmark photographically corresponding to the current landmark in a previously completed body scan.
19. The system of claim 17, wherein: The normal position of the body landmark is determined by the position of the landmark that corresponds photographically to the current landmark captured earlier in the current body scan.
20. The system of claim 17, wherein: The normal position of the body landmark is determined by the position of the object's anatomical features, such as bone joints or facial features, on the parameterized mathematical model.
21. The system of claim 12, wherein: One factor in the cost function is the amount of incremental drift and / or total drift contained in the estimate produced by the VIO algorithm.
22. The system of claim 12, wherein: One factor in the cost function is the average rate of motion of the subject's skeletal joints.
23. The system of claim 12, wherein: One factor in the cost function is the difference between the subject's bone position and the desired reference pose.
24. The object according to claim 12, wherein: One factor in the cost function is the difference between the position of the joint between the parent and child bones, expressed in the parent bone's coordinate system, and its position in the initial reference skeleton.
25. The system of claim 12, wherein: The objective function weights are adjusted according to the degree of uncertainty in the measurements.
Citation Information
Patent Citations
System and method for navigating over water
WO2020219665A1
Methods of generating personalized 3d head models or 3d body models
CN107924579A
Image generation device, image generation method, and program
JP2008186219A
Object design support program and device
JP2009140025A
Image generation device, image generation method and program
JP2009146103A
