Method for 3D reconstruction of dynamic objects using a mobile camera

Abstract

This paper describes a method for flexible 3D motion reconstruction of moving objects with multiple cameras on drones outdoors. A multi-drone capture system is used to remove the limitations of dedicated virtual reality filming locations "hot seats", allowing actors / objects to perform active or long-distance activities outdoors. The 3D pose parameters of the object are estimated from multi-view images captured by the drones, and the sequence of poses becomes the motion in time to control the animation of a pre-constructed 3D model. This method can be directly integrated into existing virtual reality / augmented reality (VR / AR) production chains, and can extend the scope to animals that are difficult to accommodate in VR camera mesh spaces.

Description

[Technical field]

[0001] The present invention relates to three-dimensional reconstruction, and more particularly to three-dimensional reconstruction using a mobile camera. [Background technology]

[0002] While the visual effects are fantastical and ubiquitous today, they are not efficient in filmmaking: the costs are prohibitive and the development times are prohibitively long. Existing systems for 3D motion capture require motion to be constrained to a space enveloped by a fixed camera mesh. Summary of the Invention

[0003] This paper describes a method for 3D motion reconstruction of moving objects with high outdoor flexibility, featuring multiple cameras on drones. A multi-drone capture system is used to remove the limitations of dedicated virtual reality filming locations "hot seats", allowing actors / objects to perform active or long-distance activities. The object's 3D pose parameters are estimated from multi-view images captured by drones, and a sequence of poses results in temporal motion, controlling the animation of a pre-built 3D model. The method can be directly integrated into existing virtual reality / augmented reality (VR / AR) production chains, and can extend the scope of objects to animals that are difficult to accommodate in a VR camera mesh space.

[0004] In one aspect, the method includes obtaining a 3D model of an object generated using a multi-view capture system, capturing motion of the object while it is moving with multiple drones, estimating pose parameters using the motion captured from the multiple drones, and applying the pose parameters to animate the 3D model. The method further includes positioning the object within the field of view of the multiple drones by performing 3D positioning directly. The method further includes positioning the object within the field of view of the multiple drones by performing 3D positioning indirectly from 2D. The method further includes using the prediction to predict a future position of the object to determine where to position the multiple drones. The method further includes collecting the motion captured from the multiple drones at a ground control station. The ground control station receives video, positions, and timestamps from the multiple drones and sends any control or correction commands to the multiple drones. The method further includes controlling a drone formation of the multiple drones by the ground control station. The method further includes controlling the drone formation of the multiple drones with a tracking drone of the multiple drones. Each camera of each drone of the multiple drones is configured to broadcast an absolute position to all other cameras.

[0005] In another aspect, the apparatus includes a non-transitory memory that stores an application that performs the following: obtaining a 3D model of an object; receiving motion of the object captured from a plurality of drones while the object is moving; estimating pose parameters of the object using the motion captured from the plurality of drones; and applying the pose parameters to animate the 3D model; and a processor coupled to the memory and configured to process the application. The application is further configured to position the object within the field of view of the plurality of drones by directly performing 3D positioning. The application is further configured to position the object within the field of view of the plurality of drones by indirectly performing 3D positioning from 2D. The application is further configured to predict a future position of the object and use the prediction to determine where to position the plurality of drones. The apparatus receives video, positions, and timestamps from the plurality of drones and sends any control or correction commands to the plurality of drones. The application is further configured to control a drone formation of the plurality of drones.

[0006] In another aspect, a system includes a plurality of drones configured to capture motion of an object while the object is moving, and a ground control station configured to obtain a 3D model of the object generated using a multi-view capture system, estimate pose parameters of the object using the motion captured from the plurality of drones, and apply the pose parameters to animate the 3D model. Each drone of the plurality of drones is equipped with at least one RGB camera device, and the camera orientation is controllable by a gimbal attached to each drone of the plurality of drones. The plurality of drones is configured to position the object within the field of view of the plurality of drones by directly performing 3D positioning. The plurality of drones is configured to position the object within the field of view of the plurality of drones by indirectly performing 3D positioning from 2D. The plurality of drones is configured to use the prediction to predict a future position of the object and determine where to position the plurality of drones. The ground control station is configured to collect visual data from the plurality of drones. The ground control station is configured to receive video, positions, and timestamps from the plurality of drones, and is configured to send any control or correction commands to the plurality of drones. The ground control station is configured to control a drone formation of the plurality of drones. A tracking drone of the multiple drones is configured to control a drone formation of the multiple drones, and each camera of each drone of the multiple drones is configured to broadcast an absolute position to all other drones. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a method of 3D model controlled by multi-drone capture, according to some embodiments. [Diagram 2] FIG. 1 is a diagram of a drone and a coordinate system, according to some embodiments. [Diagram 3] FIG. 1 is a diagram of a drone and a coordinate system, according to some embodiments. [Figure 4]FIG. 1 is a diagram of a drone and a coordinate system, according to some embodiments. [Diagram 5] 1 is a flowchart of a method for multi-drone tracking and capture, according to some embodiments. [Figure 6] 13 is an example of an embodiment for indirectly locating Psub(t) in 3D from 2D. [Figure 7] FIG. 1 is an example diagram of multiple drones positioned at different angles, according to some embodiments. [Figure 8] FIG. 1 is a diagram of camera fine tuning, according to some embodiments. [Figure 9] FIG. 1 illustrates an implementation of a gimbal control system according to some embodiments. [Figure 10] FIG. 1 is a block diagram of an example computing device configured to implement a drone-based 3D motion reconstruction method, in accordance with some embodiments. [Figure 11] FIG. 1 is a diagram of a system configured to implement a drone-based 3D motion reconstruction method, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A method for drone-based 3D motion reconstruction of flexible moving objects outdoors with multiple cameras on a drone is described herein. A multi-drone capture system is used to remove the limitations of dedicated virtual reality filming locations "hot seats" so that actors / objects can perform agile or long-distance activities outdoors. The 3D pose parameters of the object are estimated by multi-view images captured by the drone, and the sequence of poses results in the motion in time and controls the animation of a pre-built 3D model. The method can be directly integrated into existing virtual reality / augmented reality (VR / AR) production chains, and the object can be expanded to animals that are difficult to accommodate in the VR camera mesh space.

[0009] The drone-based 3D motion reconstruction method involves obtaining or generating a 3D model of an object using a stationary multi-view capture system that uses multiple drones to capture the motion of a moving object and applies the captured motion to animate the 3D model. Multiple cameras on an unmanned aerial vehicle (e.g., a drone) or another mobile device are used to control the 3D model. A drone swarm (e.g., a set of multiple drones), each equipped with a camera, is configured to track, follow, and capture the object from multiple views. The object can move and perform freely, either indoors or outdoors. The object can also interact with gadgets not allowed in the current camera mesh space. Each drone can determine its next pose in real time using only its own broadcasted object position. The entire swarm formation ensures that no drones face each other to prevent 3D pose estimation of an incorrect pose.

[0010] FIG. 1 illustrates a diagram of a method of 3D model controlled by multi-drone capture, according to some embodiments. In step 100, offline object 3D modeling is performed to obtain a 3D model of the object and other individual characteristics such as biomechanical conditions. In step 102, multi-drone tracking and capture is performed. At least two drones are used to capture a video of the object from different viewpoints. Although the drone swarm is described herein as a mobile platform of cameras, the mobile platform may be replaced with other groups of mobile devices that meet the prerequisites for communication and control. In step 104, object pose parameter estimation is performed by using visual data from all drones to estimate 3D object pose parameters. In step 106, 3D reconstruction with model animation is performed. The 3D object pose parameters are for rendering the object in AR / VR. In some embodiments, fewer or additional steps are performed. In some embodiments, the order of the steps is modified.

[0011] A ground control station (GCS) collects data from each drone and performs object pose parameter estimation for 3D model control. Drone formation control can be performed by the GCS or specific tracking drones, which calculate and send data or commands to position each drone. This is discussed in U.S. Patent Application No. 16 / 918,013, filed June 30, 2020, entitled "SYSTEM OF MULTI-DRONE VISUAL CONTENT CAPTURING," and is hereby incorporated by reference in its entirety for all purposes.

[0012] Wireless communication is established between the drones and the GCS, but is not limited to any particular wireless protocol. The GCS receives drone video, location, and timestamps, and sends any control or correction commands to any drone.

[0013] Absolute positioning of the drone can be achieved by a real-time kinematic GPS (RTK-GPS) and an internal measurement unit (IMU), or by additional devices such as those using ultra-wideband (UWB) for GPS-deprived environments. Given the target position and expected drone camera looking direction, the drone can move its position and rotate its gimbal to meet the expected camera position and optical axis orientation.

[0014] Each drone has at least one RGB camera (or another type of camera) for video capture. The orientation of the camera can be controlled by a gimbal attached to the drone. Any additional hardware can be implemented to enable movement of the drone and / or the camera. Posing the RGB cameras includes all steps from positioning the drone to rotating the gimbal. Each camera can broadcast its absolute position to all other cameras / drones at any given time. Drone positioning for positioning multiple drones given a known target location is discussed in U.S. patent application Ser. No. 16 / 917,671, filed June 30, entitled "METHOD OF MULTI-DRONE CAMERA CONTROL," and hereby incorporated by reference in its entirety for all purposes.

[0015] FIG. 2 illustrates a diagram of a drone and coordinate systems, according to some embodiments. An exemplary coordinate system is North, East, and Down (NED). Global coordinates are fixed 3D coordinates in the real world. This can be any predefined or customized coordinates. Drone coordinates are defined on the drone and are dynamic with respect to the global coordinates. Camera coordinates are defined on a camera mounted on a rotatable gimbal, which is fixed to the drone. Thus, the camera coordinates are fixed relative to the drone coordinates. There are also 2D image coordinates defined on the camera sensor, which are fixed to the camera coordinates. The image coordinates can be fully derived from the camera coordinates. In some embodiments, the coordinates are not fixed and are dynamic in nature.

[0016] FIG. 3 illustrates a diagram of a drone and a coordinate system, according to some embodiments. The position of the drone in global coordinates is available via GPS or any existing drone positioning method. The direction is It can be expressed by three rotation angles with respect to TIFF2025510757000002.tif7150, TIFF2025510757000003.tif7150 is the rotation angle around each of the X, Y and Z axes. 3D rigid body transformation from drone coordinates to global coordinates at any time t TIFF2025510757000004.tif7150 is derived from the above position and orientation.

[0017] 4 illustrates a diagram of a drone and a coordinate system, according to some embodiments. TIFF2025510757000005.tif7150 is fixed and time invariant. Camera orientation relative to the drone TIFF2025510757000006.tif7150 is controlled by rotation around the gimbal axis. Camera optical axis direction in drone coordinates TIFF2025510757000007.tif13150 is This gives the global pose of the camera at time t in the drone's local coordinates: TIFF2025510757000009.tif7150 is The file is TIFF2025510757000010.tif7150. Given TIFF2025510757000011.tif7150, TIFF2025510757000012.tif13150 It can be converted to global coordinates as TIFF2025510757000013.tif13150, TIFF2025510757000014.tif13150 Global pose of camera as TIFF2025510757000015.tif7150 Can be converted to TIFF2025510757000016.tif7150. TIFF2025510757000017.tif13150 is normalized to have unit length, Determine TIFF2025510757000018.tif7150.

[0018] FIG. 5 illustrates a flowchart of a method for multi-drone tracking and capture, according to some embodiments. In step 500, object localization is performed on the drone. The object is localized in 3D, either by performing 3D localization directly or indirectly from 2D. Object localization includes framing the object in the field of view of the drone's "camera" so that the drone can capture the object's content. In step 502, object motion prediction is performed. Object motion prediction can be performed on the drone and / or on the GCS. Prediction can be performed by continuous pose estimation. For example, the next object pose is predicted and then the next suitable camera position is determined / predicted. By performing object prediction, the drone flight can be smoother and the object can be more central in the video. In step 504, camera positioning is performed. Camera positioning (or calibration) is performed on / between the drones. Real camera positioning includes moving the drone to the predicted camera position. The position of each camera is fine-tuned by the residual between the captured object image and the expected object image. Each camera and / or GCS knows the location of the other cameras in real time. In step 506, each drone captures content of interest (e.g., video) from its respective angle. In some embodiments, fewer or additional steps are performed. In some embodiments, the order of the steps is changed.

[0019] The positioning of the target is performed on each drone. The position is expressed as the target’s position in global coordinates at time t The file is TIFF2025510757000019.tif7150. TIFF2025510757000020.tif7150 is defined by a reference point of the object. Examples of reference points include, but are not limited to, a facial feature such as a nose, a body feature such as a waist, a feature of a gadget attached to the object such as a marker or glass sphere on a helmet, a 3D center of gravity point of the object's volume or head volume, or a point of a sensor or marker that has a particular spatial relationship to the object.

[0020] 3D by each drone Direct positioning embodiments of TIFF2025510757000021.tif7150 include a RTK-GPS receiver worn by the subject, a wireless (e.g., UWB) transmitter worn by the subject, radar or laser infrared radar positioning methods, or an RGB-D or stereo camera. The wireless transmitter periodically broadcasts a beacon, allowing more than three receivers to determine the 3D position of the transmitter.

[0021] 2D indirectly leads to 3D Embodiments for locating TIFF2025510757000022.tif7150 include using a full body 2D detector or a 2D part detector and a tracker, and can use either a marker-based or marker-less method. In a marker-based method, a marker is placed on the subject's body that can be detected by an imaging device on the drone. The marker can be located in such a method in the 2D image. In a marker-less method, the location is detected either of the subject's whole body, or a part of the subject's body, and the area / location of that part in the 2D image is reported. Projected into 2D TIFF2025510757000023.tif7150 is defined as some specific location such as the neck joint, a specific marker, or the center of mass of the whole body in image coordinates. The face location is an example of a 2D part detector and tracker.

[0022] FIG. 6 illustrates a 2D to 3D indirection in accordance with some embodiments. 1 illustrates an example embodiment for registering TIFF2025510757000024.tif7150. As noted above, examples include marker-based, full-body, body part, markerless, and facial registration.

[0023] 2D to 3D There are several implementations or steps in TIFF2025510757000025.tif7150. The 2D position estimates from each drone and the current camera pose are sent to the GCS and / or to the specific tracked drone. The TIFF2025510757000026.tif7150 can be estimated by optimizing the 2D reprojection error on each drone camera image, or by a deep neural network (DNN) regression model given the 2D position and camera pose. The value of TIFF2025510757000027.tif7150 can be used for prediction.

[0024] Tracking and prediction for 2D and 3D object positioning includes 2D tracking on each drone for faster detection and 3D tracking on the GCS and / or specific tracked drones for object position prediction.

[0025] For 2D tracking on each drone, options include visual trackers such as optical flow and template matching, statistical filtering such as Kalman filters and its variants or particle filters, learning-based methods such as correlation kernel filters (CKF) and its variants. In any approach, 2D localization at time t uses a priori the predicted position at time t-1.

[0026] 3D tracking and predictions on GCS or specific tracked drones are up-to-date Contains TIFF2025510757000028.tif7150 Using the history of TIFF2025510757000029.tif7150, statistical filtering tracking and prediction are performed, such as a Kalman filter or its variants or a particle filter, and the prediction can also be performed by a DNN model trained for position prediction, in particular a variant of the RNN sequential model. At the end of the tracking, t←t+1 is updated and the prediction TIFF2025510757000030.tif7150 is broadcast to all drones, so that the drones can smoothly control their own motion to follow the target.

[0027] A key aspect of multi-drone tracking and capture is camera positioning. Given TIFF2025510757000031.tif7150, multiple drones equipped with cameras are positioned. As described herein, the application entitled "Method of Multi-Drone Camera Control" TIFF2025510757000032.tif7150 discusses a method for positioning multiple drones with given cameras. In some embodiments, a centralized camera positioning implementation of multiple drones is used, in some embodiments a distributed camera positioning implementation of multiple drones is implemented, in some embodiments three or more drones are implemented, in some embodiments a specific high speed formation for subject pose capture is implemented.

[0028] A fast formation method can be used for target pose capture. Drones can be positioned above the target (called the "top camera"), with more drones to the side of the target. For each drone, the GCS or the predicted pose from the top drone is used. TIFF2025510757000033.tif13150 is received. For the purposes of calculation, assume that a total of N drone cameras are positioned. The process is to position the top camera (first camera), Starting with TIFF2025510757000034.tif7150. TIFF2025510757000035.tif13150 is the top camera Determine TIFF2025510757000036.tif7150, where (x,y,z) are in exemplary global NED coordinates. (x,y) are constant offsets to comply with safety requirements for flying over targets. z<0 is also a preset constant, and |z| TIFF2025510757000037.tif7150, so that the camera image can cover the complete area of ​​the subject.

[0029] The positioning of the first side camera (second camera) is TIFF2025510757000038.tif13150, TIFF2025510757000039.tif13150 is the image of this camera. Determine TIFF2025510757000040.tif13150. The file is TIFF2025510757000041.tif13150. In TIFF2025510757000042.tif13150, d is the distance between the camera and the object that is pre-submitted so that the image of the camera can cover the complete area of ​​the object. The combination of TIFF2025510757000043.tif7150 can be arbitrary or can be determined by certain facing criteria. TIFF2025510757000044.tif7150 or TIFF2025510757000045.tif7150, and the camera's image can cover the entire area of ​​the object. For an additional nth camera, n >= 3, TIFF2025510757000046.tif7150, The file is TIFF2025510757000047.tif26150. TIFF2025510757000048.tif13150 TIFF2025510757000049.tif13150 and this camera Determine TIFF2025510757000050.tif13150. The side cameras are positioned to avoid exactly the opposite (180°) viewing direction in the XY plane.

[0030] 7 illustrates an example diagram of multiple drones positioned at different angles, according to some embodiments. Depending on the number of drones in use, the angles between the drones are modified.

[0031] In some embodiments, the formation transforms in global coordinates. TIFF2025510757000051.tif13150 where δ is a predetermined height difference between the camera and the subject's head, and the image captured by the camera can cover the entire subject's body.

[0032] In some embodiments, the head guidance is based on the second camera TIFF2025510757000053.tif13150 is used to specify the face orientation. Any face pose estimation method can be used to identify the face orientation in 3D. The method can be 2D face detection using eye and nose positioning on each drone's camera, so that the eye and nose positions are triangulated in 3D by multiple drones. The 3D face orientation can be estimated by facial feature points. The method can also use 3D face pose estimation directly from each single 2D face image by some 2D to 3D face pose inference model, so that the pose is optimized among all drones' pose estimates in global coordinates. After the face pose is estimated, the unit vector of the face orientation in global coordinates is It is defined as TIFF2025510757000054.tif13150, TIFF2025510757000055.tif13150 TIFF2025510757000056.tif13150, where d is defined here.

[0033] The pose capture formation described herein is With minimal communication between the drone or GCS, TIFF2025510757000057.tif7150 is sufficient. 3D tracking with target position prediction to obtain TIFF2025510757000058.tif7150 can be done on the top drone. The top drone can act as a specific tracking drone because its camera has an aerial view that is more accurate for tracking targets on the XY (ground) plane than the side drone cameras (e.g., by flying 10 meters above the target). The top drone can also keep tracking targets accurately in environments where GPS is not available. In some embodiments, for ease of handling, the ground plane is assumed to be parallel to the XY plane of the global coordinate system. If the ground is inclined, the XY positions of all side cameras can be adjusted to the same position for flight safety. It can be tilted by changing the tilt angle around the Z axis centered at TIFF2025510757000059.tif7150.

[0034] The drone formation emphasizes effective viewpoints for triangulation of 3D body parts rather than volumetric coverage of interest. Each drone is fine-tuned. Fine-tuning of the drones (or, more specifically, the cameras on the drones) is performed as described in U.S. Patent Application No. 16 / 917,671, filed June 30, 2020, and entitled "METHOD OF MULTI-DRONE CAMERA CONTROL." At time t, the camera position is TIFF2025510757000060.tif7150, and the true position of the object is TIFF2025510757000061.tif13150. The 3D vector from the camera to the object in global coordinates is TIFF2025510757000062.tif13150 and The control policy Gim for a set of gimbal parameters at time t is TIFF2025510757000064.tif13150 It is set to be close to TIFF2025510757000065.tif13150, that is, The file is TIFF2025510757000066.tif13150.

[0035] FIG. 8 illustrates a diagram of camera fine tuning, according to some embodiments.

[0036] The control policy causes the camera to be moved by the gimbal in such a way that it always keeps (or at least tries to keep) the target in the center of the image. The implementation of the gimbal control policy depends on the mechanical structure of each gimbal.

[0037] 9 illustrates a diagram of an implementation of a gimbal control system, according to some embodiments. The gimbal control system centers the subject in the image.

[0038] Object pose parameter estimation is performed using offline object 3D modeling. Offline object 3D modeling is used to obtain a 3D model of the object and other individual characteristics such as biomechanical constraints. Object pose parameter estimation is related to 2D part position estimation and 2D object pose estimation. 2D part positioning is implemented on the drone and / or GCS. 3D object pose estimation is performed on the GCS.

[0039] The 2D part positions can be used to perform or aid in the implementation of 3D object pose estimation. Part positions (e.g., joints, eyes, hands) are determined for each camera image. If this task is performed on drones, each drone sends back to the GCS the 2D part positions and the latest timestamp received from all other drones. If the task is performed on a GCS, each drone sends back to the GCS each frame and the timestamp received for this frame from all other drones. In both implementations, the camera positions are sent back to the GCS.

[0040] 3D object pose estimation involves skeleton lengths, camera positions, 2D part positions in each camera image, and 2D / 3D spatio-temporal smoothness constraints.

[0041] For subject pose parameter estimation, an offline subject model construction is implemented. To achieve high accuracy, subject body parameters are measured before capture. Body parameters may include the skeletal lengths between the joints and the biokinetic extremes of each joint that the subject can perform. Typically, 20-40 major body joints are used to control the avatar, the more measurements the better. The subject's surface model is also modeled with the subject's texture and shape. Given the surface model, it is possible to calculate or generate detailed expressions or actions of the avatar.

[0042] For subject pose parameter estimation, an exemplary implementation is described in U.S. Patent Application No. 62 / 782,862, entitled "PHOTO-VIDEO BASED SPATIAL-TEMPORAL VOLUMETRIC CAPTURE SYSTEM FOR DYNAMIC 4D HUMAN FACE AND BODY DIGITIZATION," which is hereby incorporated by reference in its entirety for all purposes. The exemplary implementation is a system that constructs a human skeletal and surface model and captures space-time variations.

[0043] The estimation of the pose parameters of each 2D image can be carried out using a CNN-based multi-object method. The estimation can be combined with a prediction model for better tracking. When the calculation is performed on the GCS, the 2D images from each camera are sent to the CGS.

[0044] Embodiments of the pose parameters can be for sparse or dense parts that are compatible with the avatar model. For sparse parts, one can use methods such as OpenPose or Mask R-CNN to position each major joint of each object in the image in real time. When the calculation is performed on the drone, the 2D joint positions are sent to the GCS. For dense parts, one example is DensePose which positions the surface of the elastic part within the image. When the calculation is performed on the drone, all output images or partial parameters are sent to the GCS. In DensePose, these are patches, U images, and V images.

[0045] For the estimation of the pose parameters of an object, the input is a set of model parameters M of the object (e.g., the lengths of different body parts of the object, the surface model of the object, or the biomechanical motion limits of individual parts), the current time t, the total number of cameras N, the intrinsic parameters K of camera c c , the pose of the camera in global coordinates TIFF2025510757000067.tif7150, where j indicates the partial index, and the 2D partial position of the object within the image coordinates of the camera having the timestamp t c TIFF2025510757000068.tif13150, t c <all for t including the history of TIFF2025510757000069.tif13150.

[0046] The output of the target pose parameter estimation includes the pose parameter set A(t) of the 3D target at time t. For the avatar with sparse partial control, the model A(t) is the 3D position and rotation of the six degrees of freedom of each joint. For the avatar with dense partial control, the model A(t) is the vertex position of the 3D body surface mesh model.

[0047] In some examples, the view of the target is blocked or closed, and the 3D pose cannot be generated from the acquired data. The time-constrained 3D pose estimation using multiple drones is used for such conditions. The estimation sets the 3D pose of the target so that it is estimated in the GCS at the period q to which the tolerance belongs. The minimum number of cameras for triangulation is m (m≥2). At the estimation time, the linear camera C is set to C = Φ. For each drone camera c, the latest received TIFF2025510757000070.tif13150 is t - t c <∈, then C = C ∪ c. After all cameras are checked, if |C| < m, extrapolation is performed on A*(t), which is the 3D parameter set at time t, and A*(t) is output for avatar control. Otherwise, if |C| ≥ m, for each drone camera c ∈ C, t - t c In this case, extrapolation is performed on each 2D parameter at time t, and the estimated TIFF2025510757000071.tif13150 is formed.

[0048] The 2D to 3D estimation is K = ∪K C and TIFF2025510757000072.tif13150. The target is to find the optimal A*(t) that minimizes the loss function L TIFF2025510757000073.tif13150. More specifically, L is TIFF2025510757000074.tif13150 and is defined as E geo、 E kin and E smis an energy function of the 3D to 2D partial reprojection error, the deviation due to biodynamic statistics, and the 3D time smoothness of the partial trajectories. W geo、 W kin and W sm are the corresponding weights of the energy terms. A his is the history of A earlier than t.

[0049] Regarding data output, A*(t) is the output for avatar control. A*(t) is A his The oldest A(t) is added to A if it is not used by future extrapolations. his A*(t) and t are broadcast to all drones, and each drone can then use this information in its predictive model for 2D part localization.

[0050] A*(t) can be explained by optimization methods, usually iterative, such as gradient descent, Gauss-Newton, or variants of quasi-Newton methods such as L-BFGS. A*(t) may also be explained by end-to-end DNN-based methods by training a regression model with an output head containing A*(t). In inference, the computation may be done in one cycle without iterations.

[0051] An example implementation can be found in U.S. Patent Application No. 63 / 003,097, entitled “ML-BASED NATURAL HUMAN FACE / BODY ANIMATION USING VOLUMETRIC CAPTURE SYSTEM + MESH TRACKING,” which is hereby incorporated by reference in its entirety for all purposes.

[0052] FIG. 10 illustrates a block diagram of an exemplary computing device configured to implement a drone-based 3D motion reconstruction method, according to some embodiments. The computing device 1000 may be used to obtain, store, compute, process, communicate, and / or display information, such as images and videos, including 3D content. The computing device 1000 may perform any encoding / decoding aspects. In general, a hardware configuration suitable for implementing the computing device 1000 includes a network interface 1002, a memory 1004, a processor 1006, an I / O device 1008, a bus 1010, and a storage device 1012. The choice of processor is not important as long as a suitable processor of sufficient speed is selected. A GPU may also be included. The memory 1004 may be any conventional computer memory known in the art. The storage device 1012 may include a hard drive, a CDROM, a CDRW, a DVD, a DVDRW, a high definition disk / drive, an ultra-HD drive, a flash memory card, or any other storage device. The computing device 1000 may include one or more network interfaces 1002. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O devices 1008 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. A drone-based 3D motion reconstruction application 1030 used to implement the drone-based 3D motion reconstruction method may be stored in the storage device 1012 and memory 1004 and processed as applications are typically processed. More or fewer components shown in FIG. 10 may be included in the computing device 1000. In some embodiments, drone-based 3D motion reconstruction hardware 1020 is included.10 includes an application 1030 and hardware 1020 for drone-based 3D motion reconstruction implementation, the drone-based 3D motion reconstruction method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the drone-based 3D motion reconstruction application 1030 is programmed into a memory and executed using a processor. In another example, in some embodiments, the drone-based 3D motion reconstruction hardware 1020 is programmed with programmable hardware logic that includes gates specifically designed to implement the drone-based 3D motion reconstruction method.

[0053] In some embodiments, the drone-based 3D motion reconstruction application 1030 includes multiple applications and / or modules. In some embodiments, the modules in turn include one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0054] Suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., DVD writer / player, high definition disc writer / player, ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., a self-driving car), a drone, or any other suitable computing device.

[0055] FIG. 11 illustrates a diagram of a system configured to implement a drone-based 3D motion reconstruction method according to some embodiments. In some embodiments, the system includes a set of drones 1100 and a GCS 1102. Each of the drones 1100 includes a camera 1104. The drones 1100 can include other functions / components, including additional cameras, sensors, gimbals, gyroscopes, and / or other components. As described herein, the drones 1100 are configured to track, predict, and be positioned with respect to the target and each other to capture the target's content (e.g., images / video) from many different angles. In some embodiments, at least one of the drones 1100 is positioned directly above the target (or above within an offset), and this drone is a control drone with additional functionality, such as additional software functionality that allows the drone to communicate and process commands and information to the other drones. The GCS 1102 is configured to communicate with the drones 1100 and process content received from the drones 1100. The drones 1100 are similarly configured to communicate with each other. As described herein, the drone 1100 and / or the GCS can perform the steps of the drone-based 3D motion reconstruction described herein. In some embodiments, instead of utilizing a drone, a separate set of mobile cameras is used.

[0056] To utilize the drone-based 3D motion reconstruction method, multiple drones obtain images and videos of an object from various angles. The multiple drones are configured to establish a position and track the object. The drone-based 3D motion reconstruction method can be performed with user assistance or automatically (e.g., by utilizing artificial intelligence) without user involvement.

[0057] In operation, the drone-based 3D motion reconstruction method and system can track and follow the motion of a subject. The method and system removes the limitations of past motion capture systems, including space to capture, location to capture, and human-only subjects. For example, the limitation of a dedicated VR filming location "hot seat" is removed, so that actors / subjects can perform active or long-distance activities. The resulting pose parameters are used to manipulate existing surface / volumetric VR actor models. The method can be directly integrated into existing VR / AR production chains. The method and system does not require significant construction efforts required for past motion capture systems. Moreover, full-scale site surveys and planning, robust camera deployment, complex wiring, and area marking capture are not required. The method and system are utilized with studio-level VR / AR for film or TV production, remote training of professional athletes, or dancers. The method and system are also used for video conferencing, virtual Youtube, games, and motion replays (space-time albums).

[0058] Some embodiments of the method for 3D reconstruction of dynamic objects by mobile camera (Section 1) Obtaining a 3D model of an object generated using a multi-view capture system; Using multiple drones to capture the motion of a target while it is moving, Estimating pose parameters of a subject using motion captured from multiple drones; Apply pose parameters to animate 3D models A method comprising: (Section 2) The method of claim 1 , further comprising: locating the object within the field of view of the multiple drones by directly performing 3D positioning. (Section 3) The method of claim 1 , further comprising: positioning the object within the field of view of the multiple drones by performing 3D positioning indirectly from 2D. (Section 4) 10. The method of claim 1, further comprising using prediction to predict a future position of the object to determine where to position the multiple drones. (Section 5) The method of claim 1 , further comprising collecting the captured motion from the multiple drones at a ground control station. (Section 6) 6. The method of claim 5, wherein the ground control station receives video, positions, and timestamps from the multiple drones and sends any control or correction commands to the multiple drones. (Section 7) The method of claim 1 , further comprising controlling a drone formation of a plurality of drones by a ground control station. (Section 8) The method of claim 1 , further comprising controlling a drone formation of the plurality of drones by a tracking drone of the plurality of drones. (section 9) The method of claim 1 , wherein each camera of each drone in the plurality of drones broadcasts its absolute position to all other cameras. (Section 10) obtaining a 3D model of the object; receiving, from a plurality of drones, motion of a subject captured while the subject is moving; estimating pose parameters of a subject using motion captured from multiple drones; and applying the pose parameters to animate the 3D model; a non-transitory memory for storing an application that executes a processor coupled to the memory and configured to process an application; 13. An apparatus comprising: (Section 11) The apparatus of claim 10 , wherein the application is further configured to position the object within the field of view of the multiple drones by directly performing 3D positioning. (Section 12) The apparatus of claim 10 , wherein the application is further configured to position the object within the field of view of the multiple drones by performing 3D positioning indirectly from 2D. (Section 13) The apparatus of claim 10 , wherein the application further comprises using the prediction to predict a future position of the object to determine where to position the multiple drones. (Section 14) The apparatus of claim 10 , wherein the apparatus receives video, positions and timestamps from multiple drones and sends any control or correction commands to the multiple drones. (Section 15) The apparatus of claim 10 , wherein the application is configured to control a drone formation of a plurality of drones. (Section 16) a plurality of drones configured to capture motion of the subject while the subject is moving; Obtaining a 3D model of the object using a multi-view capture system; Estimating pose parameters of a subject using motion captured from multiple drones; and Applying pose parameters to animate 3D models a ground control station configured to execute Including, the system. (Section 17) 20. The system of claim 16, wherein each drone of the plurality of drones is equipped with at least one RGB camera device, and the orientation of the camera is controllable by a gimbal attached to each drone of the plurality of drones. (Section 18) 20. The system of claim 16, wherein the multiple drones are configured to position the object within the field of view of the multiple drones by directly performing 3D positioning. (Section 19) 17. The system of claim 16, wherein the multiple drones are configured to position an object within a field of view of the multiple drones by performing 3D positioning indirectly from 2D. (section 20) 20. The system of claim 16, wherein the multiple drones are configured to use the predictions to predict future positions of the object to determine where to position the multiple drones. (Section 21) 20. The system of claim 16, wherein the ground control station is configured to collect visual data from the multiple drones. (section 22) 20. The system of claim 16, wherein the ground control station is configured to receive the video, location and time stamps and send any control or correction commands to the multiple drones. (Section 23) 20. The system of claim 16, wherein the ground control station is configured to control a drone formation of a plurality of drones. (Section 24) 20. The system of claim 16, wherein a tracking drone of the plurality of drones is configured to control a drone formation of the plurality of drones. (Section 25) 20. The system of claim 16, wherein each camera of each drone of the plurality of drones is configured to broadcast an absolute position to all other cameras.

[0059] The present invention has been described with respect to specific embodiments incorporating details to facilitate an understanding of the principles of the structure and operation of the invention. Such references in this document to specific embodiments and their details are not intended to limit the scope of the claims appended hereto. Those skilled in the art will readily recognize that various other modifications may be made in the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. Obtaining a 3D model of an object generated using a multi-view capture system; capturing, by a plurality of drones, motion of the subject while the subject is moving; estimating pose parameters of the subject using the motion captured from the multiple drones; and applying the pose parameters to animate the 3D model; A method comprising:

2. The method of claim 1 , further comprising: locating the object within a field of view of the multiple drones by directly performing 3D positioning.

3. The method of claim 1 , further comprising: positioning the object within a field of view of the multiple drones by performing 3D positioning indirectly from 2D.

4. The method of claim 1 , further comprising using prediction to predict a future position of the object to determine where to position the plurality of drones.

5. The method of claim 1 , further comprising collecting the motion captured from the multiple drones at a ground control station.

6. 6. The method of claim 5, wherein the ground control station receives video, positions, and timestamps from the plurality of drones and sends any control or modification commands to the plurality of drones.

7. The method of claim 1 , further comprising controlling a drone formation of the plurality of drones by a ground control station.

8. The method of claim 1 , further comprising controlling a drone formation of the plurality of drones by a tracking drone of the plurality of drones.

9. The method of claim 1 , wherein each camera of each drone in the plurality of drones broadcasts an absolute position to all other cameras.

10. A non-transient memory, comprising: obtaining a 3D model of the object; receiving, from a plurality of drones, motion of the object captured while the object is in motion; estimating pose parameters of the subject using the motion captured from the multiple drones; and applying the pose parameters to animate the 3D model; a non-transitory memory for storing an application that executes the a processor coupled to the memory and configured to process the application; 13. An apparatus comprising:

11. The apparatus of claim 10 , wherein the application is further configured to position the object within a field of view of the multiple drones by directly performing 3D positioning.

12. The apparatus of claim 10 , wherein the application is further configured to position the object within a field of view of the multiple drones by performing 3D positioning indirectly from 2D.

13. The apparatus of claim 10 , wherein the application further comprises using predictions to predict a future position of the object to determine where to position the multiple drones.

14. The apparatus of claim 10 , wherein the apparatus receives video, positions and timestamps from the plurality of drones and sends any control or modification commands to the plurality of drones.

15. The apparatus of claim 10 , wherein the application is configured to control a drone formation of the plurality of drones.

16. a plurality of drones configured to capture motion of an object while the object is moving; obtaining a 3D model of the object using a multi-view capture system; estimating pose parameters of the subject using the motion captured from the multiple drones; and applying the pose parameters to animate the 3D model. a ground control station configured to execute Including, the system.

17. 20. The system of claim 16, wherein each drone of the plurality of drones is equipped with at least one RGB camera device, the camera orientation being controllable by a gimbal attached to each drone of the plurality of drones.

18. 20. The system of claim 16, wherein the multiple drones are configured to position the object within a field of view of the multiple drones by directly performing 3D positioning.

19. 20. The system of claim 16, wherein the multiple drones are configured to position the object within a field of view of the multiple drones by performing 3D positioning indirectly from 2D.

20. 20. The system of claim 16, wherein the plurality of drones are configured to use predictions to predict future positions of the object to determine where to position the plurality of drones.

21. 20. The system of claim 16, wherein the ground control station is configured to collect visual data from the multiple drones.

22. 20. The system of claim 16, wherein the ground control station is configured to receive the video, location and time stamps, and to send any control or modification commands to the plurality of drones.

23. 20. The system of claim 16, wherein the ground control station is configured to control a drone formation of the plurality of drones.

24. 20. The system of claim 16, wherein a tracking drone of the plurality of drones is configured to control a drone formation of the plurality of drones.

25. 20. The system of claim 16, wherein each camera of each drone of the plurality of drones is configured to broadcast an absolute position to all other cameras.

Images