Determining camera control points for virtual fabrication

Abstract

Systems, apparatuses, and methods for rendering content based on control points of a camera device are disclosed. In an example, a marker is attached to a camera device, and its pose is tracked over time. An offset between the marker and the control point is determined based on a camera model of the camera device. The attitude of the tracked marker may be translated and / or rotated according to the offset to estimate the attitude of the control point. Rendering of the content is adjusted over time based on the estimated attitude of the control point. When content is rendered in a real-world space (e.g., on a display component located therein), a camera device may capture the content as well as other objects in the real-world space and generate a video stream thereof.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 390,255, filed on July 18, 2022, entitled “DETERMINING A CAMERA CONTROLPOINT FOR VIRTUAL PRODUCTION,” the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The field of the invention relates to virtual production and camera control technology. Background Art

[0004] The background description includes information that may be used to understand the subject matter of the present invention. It is not an admission that any of the information provided herein is prior art or that the applicant has admitted prior art, or that it is relevant to the presently claimed subject matter, or that any publication explicitly or implicitly referenced is prior art or that the applicant has admitted prior art.

[0005] Virtual production typically involves a virtual stage that presents content associated with the scene, a camera rig that generates cinematographic data by capturing video of people, objects, and content, and a motion capture system that tracks the camera, people, and / or objects. The content can be dynamic (e.g., video content that changes over time), and / or its presentation can be adjusted based on the tracking.

[0006] All publications identified herein are incorporated by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. If a definition or use of a term in an incorporated reference is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.

[0007] In some embodiments, the digits representing the amount or units of data for example for describing and claiming certain embodiments of the subject of the present invention should be understood as being modified by the term "approximately" in some cases. Therefore, in some embodiments, the numerical parameters set forth in the written description and the appended claims are approximate values, which can vary according to the desired properties sought to be obtained in a particular embodiment. In some embodiments, the numerical parameters should be interpreted according to the number of significant figures reported and by applying common rounding techniques. Although the wide range of numerical ranges and parameters of some embodiments of the subject of the present invention are approximate values, the numerical values ​​set forth in the specific embodiments are reported as accurately as possible. The numerical values ​​presented in some embodiments of the subject of the present invention may include certain errors caused by the standard deviation found in their corresponding test measurements.

[0008] Unless the context indicates otherwise, all ranges set forth herein should be interpreted as including their endpoints, and open-ended ranges should be interpreted as including only commercially practical values. Likewise, all lists of values ​​should be interpreted as including intermediate values, unless the context indicates otherwise.

[0009] As used in the specification herein and throughout the claims, the meanings of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. Similarly, as used in the specification herein, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

[0010] The description of value ranges herein is intended only to be used as a shorthand method for individually referring to each individual value falling within the range. Unless otherwise indicated herein, each individual value is incorporated into the specification as if it were individually described herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradictory with context. The use of any and all examples or exemplary language (e.g., "such as") provided with respect to some embodiments herein is intended only to better illustrate the subject matter of the present invention, rather than to limit the scope of the subject matter of the present invention claimed in addition. Any language in the specification should not be interpreted as indicating that any unclaimed element is essential for the practice of the subject matter of the present invention.

[0011] The grouping of the alternative elements or embodiments of the subject matter of the present invention disclosed herein should not be construed as limiting. Each group member may be mentioned and claimed individually, or mentioned and claimed in any combination with other members of the group or other elements found herein. For convenience and / or patentability reasons, one or more members in the group may be included in the group or deleted from the group. When any such inclusion or deletion occurs, this specification is considered to contain the modified group in this article, thereby satisfying the written description of all Markush groups used in the appended claims.

[0012] It should be understood that many of the basic technical features provided in the following description are presented in order to enable a compact examination of the disclosed inventive subject matter. Although some of the basic technical features described herein may appear obscure, in many cases such features can be considered to be within the scope of understanding of those skilled in the art. Therefore, the presentation of such background technology should not be considered limiting. Summary of the invention

[0013] The present subject matter provides systems, apparatuses, and methods for rendering content based on control points of a camera device. Embodiments of the present disclosure relate particularly to determining the pose of a control point (eg, its focus) of a camera device.

[0014] One embodiment of the present invention includes a method comprising determining, via at least one processor, a first pose of a marker in a real-world space, wherein the marker is attached to a camera device. The method may further comprise determining, via the at least one processor, an offset between the marker and a control point of the camera device based on the first pose and at least one calibration parameter of the camera device. Additionally, the method may comprise setting, in a computer-readable memory, via the at least one processor, a second pose of a virtual camera in a virtual space based on the offset, wherein the virtual camera represents the camera device in the virtual space, and wherein the virtual space represents the real-world space. Further, the method may comprise rendering, via the at least one processor, content based on the second pose of the virtual camera on a display, wherein the content is to be captured by the camera device when presenting the content in the real-world space.

[0015] Various objects, features, aspects and advantages of the present subject matter will become more apparent from the following detailed description of preferred embodiments and the accompanying drawings, in which like reference numerals represent like components. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An example of a virtual production system according to an embodiment of the present disclosure is illustrated.

[0017] Figure 2An example of a tracking camera device according to an embodiment of the present disclosure is illustrated.

[0018] Figure 3 An example of tracking error according to an embodiment of the present disclosure is illustrated.

[0019] Figure 4 An example of a camera calibration system configured to determine the focus of a camera device according to an embodiment of the present disclosure is illustrated.

[0020] Figure 5 Illustrated are examples of objects that may be used in a camera calibration procedure according to an embodiment of the present disclosure.

[0021] Figure 6 An example of generating pose data and image data of an object during a camera calibration procedure according to an embodiment of the present disclosure is illustrated.

[0022] Figure 7 An example of gesture data and image data associated with an object according to an embodiment of the present disclosure is illustrated.

[0023] Figure 8 An example of a camera model determined based on a camera calibration procedure according to an embodiment of the present disclosure is illustrated.

[0024] Fig. 9 An example of a process for rendering content based on a determination of a focus of a camera device according to an embodiment of the present disclosure is illustrated.

[0025] Fig.10 An example of a process for determining an offset between a focus of a camera device and a marker attached to the camera device according to an embodiment of the present disclosure is illustrated.

[0026] Fig.11 An example of a process for associating pose data with image data in a camera calibration procedure according to an embodiment of the present disclosure is illustrated.

[0027] Fig.12 Illustrated are example components of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] It should be noted that any language for computers should be interpreted to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing devices operating alone or together. It should be understood that the computing device includes at least one processor configured to execute software instructions stored on a tangible non-transitory computer-readable storage medium (e.g., a hard drive, FPGA, PLA, solid-state drive, RAM, flash memory, ROM, etc.). The software instructions or software instruction sets configure or program the computing device or its processor to provide roles, responsibilities, or other functions, as discussed below with respect to the disclosed apparatus or system. Further, the disclosed technology can be embodied as a computer program product, which includes a non-transitory computer-readable medium storing software instructions or software instruction sets, which cause one or more processors to perform the disclosed steps associated with the implementation of a computer-based algorithm, process, method, or other instruction. In some embodiments, various servers, systems, databases or interfaces use standardized protocols or algorithms to exchange data, which may be based on HTTP, HTTPS, TCP, UDP, FTP, SNMP, IP, AES, public-private key exchange, web services or RESTful APIs, known financial operation protocols or other electronic information exchange methods. Data exchange between devices can be carried out through the following networks: packet switching networks, the Internet, LAN, WAN, VPN or other types of packet switching networks; circuit switching networks; cell switching networks; or other types of networks (wired or wireless).

[0029] As used in the specification herein and throughout the claims that follow, when a system, engine, server, agent, device, module, or other computing element is described as being configured to perform or execute functions on data in a memory, the meaning of "configured to" or "programmed to" is defined as one or more processors or cores of the computing element being programmed by a set of software instructions stored in the memory of the computing element to perform a set of functions on target data or data objects stored in the memory. It should be understood that the combination of software and hardware working in concert can create a set of specialized physical real-world structures that provide one or more users with utility that does not exist outside the physical, real-world asset.

[0030] It should be appreciated that the disclosed techniques provide a number of advantageous technical effects, including improved calibration of a camera device and improved use of control points of a camera device given its calibration. For example, content may be rendered on a set of displays based on a pose of a control point, where the pose may be a position and / or orientation of the control point in real-world space. Calibration includes estimating an offset between a control point and at least one marker attached to the camera device. Given the offset, the pose of the control point may be tracked over time in an accurate manner by tracking the marker(s), thereby improving the quality of content rendering.

[0031] Embodiments of the present disclosure relate particularly to determining the pose of a control point (e.g., its focus) of a camera device. In an example, at least one marker is attached to the camera device, and the pose of the marker in real-world space (e.g., its position, orientation, etc.) can be tracked over time. Given that the marker cannot be mounted to the same orientation as the control point of the camera device due to the physics of the camera, the pose of the marker may not accurately represent the pose of the control point. However, some applications rely on the pose of the control point to generate output (e.g., content to be rendered), especially output to be rendered from the perspective of the control point. Therefore, inaccuracies in the estimation of the pose of the control point may affect the accuracy of the output (e.g., causing quality issues related to the rendered content, inconsistencies between real items and virtual items, etc.).

[0032] To improve the estimation of the pose of the control point, a camera model including external parameters of the camera device, internal parameters and / or a distortion model is used. Given the camera model and the pose of the marker, an offset between the pose of the marker and the pose of the control point can be determined, and the offset can include a rotation offset and / or a position offset. Instead of setting the pose of the control point to the pose of the marker, the pose of the control point can be estimated by offsetting the pose of the marker by the rotation offset and / or the position offset (if applicable). Therefore, based on the pose of the marker, the pose of the control point can be more accurately estimated and tracked over time, thereby improving the accuracy of the output (e.g., alleviating quality issues related to the rendered content).

[0033] The camera model can be determined according to a camera calibration procedure. In an example, the camera calibration procedure involves tracking the pose of an object in real-world space while the camera device generates images of the object in different poses (e.g., in its video stream, in still images, etc.). Each pose is associated with at least one image. The camera model can be initialized and used to determine the pose of the camera device by at least projecting the pose of the object onto an image plane and determining a projection error. In particular, the pose of the object (e.g., with three-dimensional coordinates, etc.) is projected onto the image plane, and the projection is compared with the pixel coordinates of the object in the associated image to determine the corresponding projection error relative to the real-world pose. The projection error is determined in different pose-image pairs, and the camera model can be updated by reducing or mitigating the projection error using the fitted model.

[0034] For illustration, consider an example of a virtual production use case. A camera device is used to generate a video frame of a scene in a real-world space. The content is presented on a display component that typically operates as a background plate and forms part of the scene. The content itself and / or its presentation can be adjusted based on the pose of the focus of the camera device (which is an example of a control point) so that the background content is rendered on the display component so that the background content looks natural from the focus perspective. Using the above-mentioned technology, a motion capture system is used in a camera calibration procedure to generate a camera model, thereby attaching a marker to the camera device and tracking it, and thereby determining the offset between the pose of the marker and the pose of the focus. In a virtual production use case, a virtual camera device can represent a camera device in a virtual space, which in turn represents a real-world space. The pose of the virtual camera in the virtual space is set to correspond to the pose of the focus. The game engine can render content based on the pose of the virtual camera, and the rendered content is presented on the display component. In a further description, the camera model includes a distortion model. As part of rendering, the game engine may distort part of the content (or all of the content) to be presented on a specific sub-region of the display component (or on the entire display component) based on the pose of the virtual camera and the distortion model.

[0035] For clarity of explanation, various embodiments of the present disclosure are described in conjunction with virtual production use cases (e.g., film production using virtual backgrounds, production of augmented reality content, etc.). However, the embodiments are not limited thereto, but are equally applicable to other use cases, such as virtual reality, augmented reality, mixed reality, content projection (e.g., home theater, cinema or building, etc.), performance stage (e.g., concert), etc. In general, embodiments of the present disclosure allow for improvements in the output of an application or system that relies on control points of a camera device to generate output. A control point may be a physical point of a camera device, wherein the physical point has a corresponding virtual point in a camera model of the camera device, and wherein the virtual point may be used to control various aspects of the camera device itself or the use of the camera device (e.g., for rendering content). The pose of the control point may be along one or more dimensions and cover multiple degrees of freedom (DOF) (e.g., at least six DOF) for operating the camera device. For example, the pose may indicate the position and orientation of the camera device along each axis of a three-dimensional system, and / or may correspond to the movement of the camera over time (in which case the movement may be estimated based on velocity, acceleration, state information, etc. to predict the current, past, and / or next pose). The control point may, but does not necessarily have to, be converted into an actual dimensionless point. In contrast, embodiments may approximate control points as dimensionless points and may be used to control volumes (e.g., elliptical volumes, etc.). Thus, embodiments allow optimization based on specific properties of camera devices. Various embodiments are also described herein using focal points as examples of control points. However, embodiments are not limited thereto, but are equally applicable to other types of control points, such as control points corresponding to the center point (or principal point) of the image plane of the camera model, or, for example, points along the focal length between the center point and the focal point. In addition, the camera device may also have multiple lenses (e.g., a stereo camera) and / or a camera array that may be multiple individual cameras. In this case, the control point may be associated with multiple lenses (e.g., it may be the geometric center of the focus of the lens) and / or a camera array (e.g., it may be the geometric center of each control point of each camera).

[0036] Figure 1An example of a virtual production system 100 according to an embodiment of the present disclosure is illustrated. As shown, the virtual production system 100 includes a display component 110, a camera device 120, motion capture devices 130A, 130B, 130C (generally referred to by the reference numeral "130"), and a computer system 140, etc. The display component 110 can be configured to present content 112 and define a virtual stage of a volume (multiple such camera devices 120 are also possible) in which the camera device 120 is located. The presentation of the content 112 can be controlled by a game engine (e.g., UNITY game engine, Unreal (UNREAL) game engine, Blender, etc.) executed on the computer system 140. For example, the content 112 itself and / or the parameters of the presentation (e.g., translation, angulation, skew, distortion, etc.) can be controlled based on a variety of factors. These factors include the posture of the camera device 120 in the volume. The motion capture device 130 can generate motion capture data that is processed to determine not only the pose of the objects 150 (150A, 150B, 150C, generally referred to by the reference numeral "150", and which can be actors, stage furniture, set furniture, facilities, etc.), but also the pose of the camera device 120. This motion capture data can be processed and used to control the content 112 presented on the display assembly 110.

[0037] In an example, the display assembly 110 includes a plurality of displays arranged to form a content presentation screen or "wall". The content presentation screen can have different shapes (e.g., curved, flat intersecting surfaces, etc.) and can surround the area where the object 150 is located, thereby defining a volume. Although the content presentation screen is illustrated as being fixedly mounted, in some embodiments, the screen or individual displays can be more dynamic, possibly moving around the production volume. The content presentation screen can be used to present interactive, dynamic scenes. Objects 150 can interact with such scenes, and the content 112 can be updated based on the interactions (e.g., actors 150A can interact with virtual objects presented in the content 112). Although the display assembly 110 is illustrated as having a vertical position (e.g., arranged as a curved wall), the display assembly 110 (or a second display assembly) can be additionally or alternatively positioned in other positions (e.g., horizontal positions for defining a ceiling or floor, different geometric shapes, 4π steradian coverage, etc.). Further, the volume can include a set of sliding displays that can be moved in and out of the volume to define a specific geometric shape. For example, the volume can be shaped as a horseshoe, and the slide group can be positioned to close the open portion of the horseshoe. In this way, the camera device 120 in the volume can be surrounded by a full 360-degree display. Other shapes are also possible, whereby the volume can be, for example, a semicircle, a spherical (i.e., 4π steradians), a cylinder, a cube. The display assembly 110 can form a volume for virtual production. The volume can include approximately 2,230 m 2 A virtual production space of FIG. 1 is provided in which the display assembly 110 forms a curved LED wall that is approximately 16 m wide by 20 m long and 270 degree elliptical (with a possible 360 ​​degree configuration). In a specific illustrative use case, each display of the display assembly 110 is a BLACK PEARL 2 display available from ROECREATIVE DISPLAY. In this illustrative use case, the screen is a flat panel having dimensions of 500×500×90 mm (height×width×depth), a resolution of 176×176 (horizontal×vertical), a pixel pitch of 2.84 mm, an LED surface mount diode (SMD) configuration, and a magnesium frame with a magnetic connector and locking system.

[0038] The camera device 120 may be a movie camera mounted on a support (floor and / or ceiling mount) that can be repositioned in the volume, or a movable support (e.g., a tripod, a pan / tilt, etc.). In this manner, the camera device 120 may be configured to capture a scene (particularly from more than one angle) by generating video data (and optionally audio data) showing one or more objects 150 and / or part or all of the content 112 presented on the display assembly 110. The camera device 120 can have a high resolution (e.g., 4K, 6K, 8K, 12K, etc.) and can be obtained from, for example, BLACKMAGIC (e.g., URSAMINIPRO 12K, STUDIO CAMERA4K PLUS, STUDIO CAMERA4K PRO, URSABROADCASTG2, etc.), ARRI (e.g., ALEXAMINI LF, ALEXA LF, ALEXA MINI, ALEXASXT W, AMIRA, AMIRA LIVE, ARRI MULTICAM SYSTEM, etc., equipped with ARRI SIGNATURE PRIME 35mm T1.8 lens, ARRI SIGNATUREPRIME 75mm T1.8 lens, etc.), etc.

[0039] The marker 122 can be attached to the camera device 120 (e.g., releasably mounted to the top surface of the camera device) and can be tracked by the motion capture system. Tracking can involve determining the pose of the marker 122. The pose can correspond to the pose of the camera device 120. The marker 122 can include multiple tracking points (each tracking point corresponds to a separate marker) so that the marker 122 is a marker assembly including a rigid body to which the tracking points are attached. In an example, the rigid body includes a base with an arm extending therefrom, wherein each tracking point is connected to the end of the arm. Six tracking points can be installed, each of which can be a separate marker from VICON (e.g., a pearl hard marker) or OPTITRACK (e.g., an M3 marker, an M4 marker, etc.).

[0040] The motion capture device 130 may be a motion capture camera (e.g., an infrared camera) and / or other types of motion sensors (e.g., a depth sensor) that are part of a motion capture system configured to track motion in a volume. The motion capture system may be obtained, for example, from VICON (e.g., using VANTAGE, VERO, VUE, VIPER, VIPERX cameras, etc. and SHOGUN software, etc.) or OPTITRACK (e.g., using PRIME-X 41, PRIME, SLIM-X, SLIM, FLEX cameras, etc. and UNREAL PLUGIN, UNITY PLUGIN, MOTION BUILDER PLUGIN, OPTICAL MOTION CAPTURE software, MAYAPLUGIN software, etc.). Although the motion capture system uses infrared technology, any trackable marker may be used alone or in combination, including, for example, radio frequency markers (using radio frequency technology), visual markers (e.g., using computer vision technology), etc. Optionally, the motion of the object may be tracked by using a motion tracker attached to the object (the marker 122 may be used to track the motion of the camera device 120). Tracking may involve locating an object in a volume by determining the position and rotation of the object over time.The coordinate system of the motion capture system (eg, a Cartesian coordinate system or any other coordinate system) may be defined relative to any origin in the volume.

[0041] The computer system 140 may be configured to process at least a portion of the motion capture data and optionally a portion of the video data. For example, a game engine may render content 112 using a virtual model of the display component 110, a virtual camera simulating the camera device 120, and the pose of the camera device 120. In an example, the pose of the virtual camera is set to the pose of the camera device 120. As further described below, the pose of the camera device 120 may be determined by the pose of the offset marker 122. Rendering may involve synthesizing multidimensional images and / or image frames (e.g., two-dimensional, three-dimensional, etc.) and / or applying visual transformations (e.g., distortion, trapezoid, skew, and / or inverse transformations thereof) thereto to present on the display component as content 112. The computer system 140 may also be configured for predictive processing. For example, given the motion history of the camera device 120, the computer system 140 may perform extrapolation to predict the next set of poses (e.g., future motion) of the camera device 120. Such predictions may also be based on data indicating a sequence of events (e.g., choreography data) in a scene (which may be included as part of an executable script). Based on the prediction, the computer system 140 can calculate (or pre-calculate) the required transformation (e.g., based on the offset between the camera marker 122 and the focus of the camera device 120) to render the content. By doing so, the delay between the actual pose and the content rendering adjustment can be reduced because the computer system 140 does not actually need to observe the actual pose first and then calculate the required transformation because such transformation is predicted a priori based on the actual pose.

[0042] Figure 2 An example of a tracking camera device 210 according to an embodiment of the present disclosure is illustrated. A marker 212 is attached to the camera device 210 so that the pose of the camera device 210 can be tracked over time by a motion capture system including a motion capture device 220. The camera device 210 generates image data 214 and sends it to a computer system 230. The image data 214 represents a video stream captured by the camera device 210. The computer system 230 also receives motion capture data 222 indicating the pose of the camera device 210. A video stream synchronization process 232 can be executed on the computer system 230 (e.g., as a process of a game engine, some other rendering engine, etc.) to synchronize the rendering of content with the pose of the camera device 210. When presented (e.g., on a display component, a separate display, a set of displays, etc.), the content is captured in the video stream of the camera device 210. The camera device 210, the marker 212, the motion capture device 220, and the computer system 230 are respectively Figure 1 1 . An example of a camera device 120, a marker 122, a motion capture device 130, and a computer system 140 in FIG.

[0043] In an example, the motion capture system implements a specific motion capture technology, such as any one or a combination of infrared (IR) technology, radio frequency (RF) technology, and / or computer vision (CV) technology (e.g., available from a programming library such as the Open Source Computer Vision (OpenCV) library). IR technology can rely on active and / or passive tags that emit (e.g., transmit and / or respond to) IR signals that can be detected by the motion capture system. RF technology can rely on RF beacons and / or RF signal triangulation. CV technology can rely on one or more machine learning (ML) models (e.g., available from a programming library such as the Scikit-Learn library) for image processing and object detection.

[0044] In the example, the marker 212 is a rigid body that implements the motion capture technology according to the motion capture system. For example, in the case of IR technology, the marker 212 may include one or more infrared emitting (active or passive) points (each point using a different infrared frequency). In general, the greater the number of points, the more accurate the pose estimation will be. In the example, the marker 212 includes a single point that can be detected by an infrared motion capture camera. In this case, at least three infrared motion capture cameras are required to detect the pose of the marker 212. In particular, each of the three cameras will generate a two-dimensional image that shows the marker position in two dimensions. Because the position, orientation, and field of view of each camera are known, the three-dimensional vector at which the marker 212 is located can be determined based on the three sets of two-dimensional positions. In another example, the marker 212 includes multiple points detected by an infrared motion capture camera. In this case, a single infrared motion capture camera may be sufficient to detect the pose of the marker 212. In particular, the relative poses of these points are known a priori, and this knowledge is used to process the images generated by the infrared motion capture camera. In virtual production, a large number of motion capture cameras can be used depending on the setting of the virtual production. Return to reference Figure 1 An example virtual production of may use more than a dozen motion capture cameras, including fifty to sixty motion capture cameras or more.

[0045] Of course, technologies other than infrared can also be used. For example, using RF technology, the tag 212 can be implemented as a group of RF beacons and / or a group of active and / or passive RF identification (RFID) tags. The RF signal can be received and processed by the motion capture device 220 to determine the range and direction, and its intersection can indicate the posture of the tag 212. In the case of CV technology, a two-dimensional visual tag whose size is encoded can be used, wherein an optical sensor operating within the visible wavelength range of humans can generate an image showing the two-dimensional visual tag. The posture of the two-dimensional visual tag can be determined by decoding the size and applying geometric reconstruction to the image. Further, in some cases, the CV technology may not rely on the two-dimensional visual tag. On the contrary, the CV technology can detect a group of features of the camera device 210 and track the posture of the group. In this case, the tag 212 is a group of features, rather than a rigid body that can be releasably attached to the camera device 210. This technology can be used alone or in various combinations to reduce errors or improve accuracy.

[0046] Regardless of the underlying motion capture technology, the pose of the marker 212 is tracked over time and represents the motion of the marker 212. The motion can be tracked by defining a tracking point 213 on the marker 212 (e.g., the center of a rigid body, the center of one of the emission points of the marker 212, etc.) and determining the position and rotation of the tracking point 213 over time in the coordinate system of the motion capture system. The origin of the coordinate system can be a point in real world space (e.g., Figure 1 ). The motion capture data 222 may indicate position and rotation in a coordinate system and may be generated based on a particular rate (eg, 24 frames per second (FPS), 60 FPS, 144 FPS, etc.).

[0047] like Figure 2 As shown, camera device 210 may include focus 216. Figure 8 As further described in , the focus 216 may be an aperture point at a distance from the image plane of the camera device 210, wherein the distance is equal to the focal length of the camera device 210. Generally, the camera model of the camera device 210 includes external parameters that may indicate the focus 216. Further, in some embodiments, the focus may change as operating parameters of the camera change (e.g., changes in focal length, lens, etc.). The disclosed techniques are robust to such changes.

[0048] There is an offset 240 between the tracking point 213 on the marker 212 and the focus 216 of the camera device 210. The offset 240 may include any one or a combination of a positional offset (e.g., indicating a relative distance between the tracking point 213 and the focus 216) and / or a rotational offset (e.g., indicating a relative orientation between the tracking point 213 and the focus 216).

[0049] The video stream synchronization process 232 can rely on the motion of the camera device 210 to synchronize the content rendering with the video stream captured by the camera device 210, wherein the synchronization can be performed according to an acceptable threshold. There are different options for determining the motion. In one option, the pose of the camera device 210 is set to be the same as the pose of the marker 212. In other words, the offset 240 is ignored, resulting in the focus 216 being assumed to be the same as the tracking point 213. Figure 3 As further described in , this option may cause errors that affect the output quality of the video stream synchronization process 232. In another option, the offset 240 is determined and used to transform the pose of the marker 212 to the pose of the camera device 210 (e.g., each pose of the tracking point 213 is converted to the corresponding pose of the focus point 216 based on the position offset, rotated based on the rotation offset, etc.). This option can reduce or even eliminate the errors, thereby improving the output quality of the video stream synchronization process 232.

[0050] Figure 3 An example of tracking error according to an embodiment of the present disclosure is illustrated. The left side shows a left projection view 300 along the YZ plane. The right side shows a top projection view 350 along the XZ plane. The camera device 310 moves from a first pose 302 to a second pose 304. A marker 312 is attached to the camera device 310 to determine the first pose 302 and the second pose 304. The camera device 310 also includes a focus 314, wherein the pose of the focus 314 can be used for content rendering synchronization. The camera device 310 and the marker 312 are respectively Figure 2 An example of a camera device 210 and a marker 212 in FIG.

[0051] As shown in the left projection view 300, when the camera device 310 moves from the first posture 302 to the second posture 304, the marker 312 moves downward and the focus 314 moves upward. Also as shown in the top projection view 350, when the camera device 310 moves from the first posture 302 to the second posture 304, the marker 312 moves to the right and the focus 314 moves to the left.

[0052] Therefore, if the gesture of the marker 312 is used to track the movement of the camera device 310, without considering the offset between the marker 312 (e.g., the tracking point thereon) and the focus 314, there will be a gesture error. For example, referring back to the two gestures (the first gesture 302 and the second gesture 304), the camera device 310 will be assumed to move downward and to the right. However, its focus 314 is actually moving upward and to the left. Therefore, the video stream synchronization process will mistakenly determine it as a lower right movement, but it should actually be determined as an upper right movement, and it is used to synchronize the rendering of the content with the video stream of the content captured by the camera device 310. This difference can cause the rendering of the background content on the screen to be unrealistic, thereby disrupting the viewer's experience. By determining the gesture of the marker 312 and transforming the gesture of the marker into the gesture of the focus 314 based on the offset between the marker 312 and the focus 314, the gesture error can be reduced or even eliminated.

[0053] There are challenges in determining the offset between the focus of a camera device and a marker attached to the camera device. Typically, the focus is not marked on the camera device (because the focus is usually located inside the camera device). Further, the marker positioning is not predefined, but can be flexible. For example, markers of different types or configurations can be used (e.g., with different emission points, different sizes, etc.). Once a marker is selected, the attachment of the marker to the camera device involves a quick installation mechanism. Alternatively, in the case of using CV technology to detect and use a set of features on the surface of the camera device as markers, the set of features may not be predefined. Tools (e.g., rulers, protractors, etc.) can be used to measure the offset between the focus and the marker. However, these measurements may not be accurate enough (especially considering that there is no marker or other way to identify the focus on the camera device). Instead, as further described in this article below, the offset can be determined based on the camera model of the camera device. The camera calibration program can be used to determine any or all external parameters, internal parameters, and distortion models of the camera model.

[0054] Figure 4An example of a camera calibration system 400 configured to determine the focus of a camera device 410 according to an embodiment of the present disclosure is illustrated. The camera calibration system 400 includes a motion capture system that includes a motion capture device 420, a computer system 430, and an object 440. The camera calibration system 400 performs a camera calibration procedure at least in part by processing motion capture data 442 generated by the motion capture system and image data 414 generated by the camera device 410. Markers (not shown) may be attached to the object 440 to track the movement of the object by the motion capture system. The motion capture data 442 indicates the movement of the object 440 over time, while the image data 414 includes an image showing the pose of the object 440 (e.g., a video stream presenting the movement of the object 440). The processing may be performed by a camera calibration process 432 executed on the computer system 430 that receives the motion capture data 442 and the image data 414, so as to subsequently calibrate the camera model of the camera device 410. The camera device 410, the motion capture device 420, and the computer system 430 are respectively Figure 2 An example of a camera device 210, a motion capture device 220, and a computer system 230 in FIG.

[0055] Marker 412 may be attached to camera device 410 to track the movement of camera device 410 before starting a camera calibration procedure, during the procedure, or after the procedure is completed. Figure 2 An example of mark 212 in FIG.

[0056] In an example, the camera device 410 remains stationary (e.g., not moving) during the camera calibration procedure. In this case, the marker 412 does not need to be attached before the camera calibration procedure is completed, or if the marker is attached, it may not be used. After the camera calibration procedure is completed, the object 440 can be removed and no longer used. The camera calibration process 432 can output an indication of the focus 416 of the camera device 410 based on the calibrated camera model (e.g., the position and / or rotation of the focus 416 in a coordinate system (such as the coordinate system of the camera device 410 if the marker 412 is not used, or the coordinate system of the motion capture system if the marker 412 is used)).

[0057] Thereafter, the camera device 410 may be moved, thereby changing the pose of its focus 416. The marker 412 may be attached (if not already attached) to the camera device 410, and the offset of the focus 416 may be estimated so that the motion (e.g., sequence of poses) of the focus 416 may be determined by tracking the motion of the marker 412. In particular, the motion capture system generates motion capture data 422 indicating the pose of the marker 412 over time. The camera calibration process 432 may receive the motion capture data 422, determine the pose of the camera device 410 using the determined pose of the marker 412 (e.g., each video frame captured from the motion capture data 422), and determine the offset between the marker 412 (e.g., a tracking point thereon) and the focus 416. The offset may be output to a video stream synchronization process 434 executing on a computer system. Based on the motion capture data 422 and the offset, the video stream synchronization process 434 determines the motion (e.g., pose) of the focus 416 by translating and / or rotating (if applicable) the motion (e.g., pose) of the marker 412. Video stream synchronization process 434 may then synchronize content rendering with the video stream captured by camera device 410 based on the movement of focus point 416 .

[0058] It may be sufficient to determine the offset once (e.g., based on an initial pose of the marker 412, which may correspond to the pose of the camera device 410 during a camera calibration procedure). However, the offset may be determined each time the pose of the marker 412 changes, or at some predefined rate. The predefined rate may be related to the FPS rate used by the motion capture system and / or camera device. For example, the predefined rate may be once per second, corresponding to one hundred and forty-four motion capture frames when a 144 FPS motion capture rate is used. Alternatively, the predefined rate may be once every forty milliseconds, corresponding to approximately twenty-four video frames when a 24 FPS video frame rate is used.

[0059] Return to reference Figure 1 System 100 and Figure 4 The same motion capture system used in virtual production can also be used to calibrate the camera device (e.g., camera device 120 or 410). Before starting the virtual production or shooting the scene, the object 440 can be placed in the volume and a calibration procedure can be performed. Thereafter, the object 440 is removed and the virtual production can be started and / or the scene can be shot by using the camera device and adjusting the content 112 according to the movement of the focus 416.

[0060] exist Figure 4In the embodiment of the present invention, during the camera calibration procedure, the object 440 is moved while the camera device 410 remains stationary. However, the embodiments of the present disclosure are not limited to this. Instead, the camera device 410 may be movable additionally or alternatively. For example, the object 440 may remain stationary while the camera device is moved. In this case, the camera calibration process 432 uses the motion capture data 422 of the camera device 410 instead of the motion capture data 442 of the object 440 to calibrate the camera model. In another illustration, the object 440 and the camera device 410 are moved during the camera calibration procedure, and the camera calibration process 432 uses both the motion capture data 422 and the motion capture data 442 when calibrating.

[0061] As described above, the offset can be determined once or repeatedly based on the change of the posture. The change of the posture can be an actual observed change or a predicted change (e.g., a change expected to occur with a probability exceeding a predefined probability threshold). By predicting the change, the offset can be pre-calculated before the actual observed posture, thereby reducing the delay associated with when the offset can be determined and available for use by the video stream synchronization process 434. In addition, in both cases (e.g., one-time calculation or repeated calculation of the offset), the video stream synchronization process 434 can calculate the transformation used in content rendering, where the transformation can be calculated based on the offset. In order to reduce the delay associated with rendering content, the video stream synchronization process 434 can be configured for predictive processing. For example, the computer system 430 can store (e.g., in its memory) the motion history of the camera device 410. The video stream synchronization process 434 can perform extrapolation on the historical data to predict the next set of postures (e.g., future motion) of the camera device 410. This prediction can also be based on scene data (which can also be stored in the memory) indicating a sequence of events in the scene (e.g., choreography data). Given the prediction, the video stream synchronization process 434 can predict the required transformations (e.g., based on the offsets) to render the content. By doing so, the delay between the actual pose and the content rendering adjustment can be reduced. In addition to or as an alternative to extrapolating based on the motion history, the computer system 140 can store the transformation history (e.g., in a memory), and the video stream synchronization process 434 can predict the next set of transformations before observing the next set of actual poses, wherein such prediction can involve extrapolation of the transformation history.

[0062] Figure 5 An example of an object 510 that can be used in a camera calibration procedure according to an embodiment of the present disclosure is illustrated. The object 510 is Figure 4 In general, object 510 includes a surface having a pattern 520 on which features appear, wherein the surface having pattern 520 can be imaged by a camera device. Pattern 520 can be indexed (e.g., Figure 5 As shown, row index 530 uses numbers and column index 540 uses letters, but other types of indexing are possible and may depend on pattern 520). A marker is attached to the surface so that the pose of object 510 can be tracked over time. Figure 5 In the illustration of , the markers include three individual markers (e.g., IR emission points, identifiable markers, etc.) 550A, 550B, and 550C located at different corners of the pattern 520 (although different numbers of individual markers and / or distributions of such markers are possible). A local coordinate system 560 (e.g., a Cartesian coordinate system, etc.) can be defined for the object 510, and the origin of the coordinate system can be set to a point on the surface (e.g., the origin is the root rigid point). Figure 5 In the illustration, the origin is set to the upper left corner point of the pattern 520 and corresponds to the center of the individual mark 550A (but a different point may be used for the origin, such as the center of the pattern 520).

[0063] Object 510 can have different shapes and / or sizes. In general, object 510 is large enough so that its pattern 520 can be accurately detected by the implementation of the image processing algorithm, but small enough so that object 510 can be easily moved. The surface of object 510 can be planar (e.g., flat) or non-planar (e.g., curved). Pattern 520 can be a checkerboard pattern, but other patterns are also possible. In the checkerboard pattern, the feature can be a set of rectangular corners with a specific color (e.g., black and white), and can be a SIFT feature, SURF feature, ORB feature, etc. of a computer vision algorithm. Other configurations of features (e.g., shape, color, etc.) are also possible. In general, it is necessary to detect the feature based on the implementation of the imaging processing algorithm (e.g., OpenCV algorithm). In order to accurately detect and process, the size of the pattern (e.g., its cross section) needs to be substantially the same or comparable to the size of the marker (e.g., the cross section of a single emission point) attached to the camera device being calibrated. Substantially the same or comparable means that the difference between the two sizes is within a predefined size margin of each other (eg, + / - 10%). For example, the cross section of the marking is in the range of 200 to 300 millimeters (mm), while the size of the pattern is 215.9 mm by 279.4 mm.

[0064] A virtual object (e.g., a three-dimensional object) may model an object 510 and may include a pattern 520 and a virtual representation of features. For example, the pattern 520 may be represented as a grid, and each feature may be represented as a grid point whose location is defined in a local coordinate system 560 as follows: ij =I×cell_size,Y ij= J × cell_size, and in the case of a planar surface, Z ij =0, where "I" corresponds to column index 540 and "J" corresponds to row index 530. The grid (e.g., a virtual representation of pattern 520) can be used to project object 510 to the image plane based on the detected pose of object 510. The grid points (e.g., a virtual representation of features) can be used to determine the projection error of the projection.

[0065] Figure 6 An example of generating pose data and image data of an object 610 during a camera calibration procedure according to an embodiment of the present disclosure is illustrated. Figure 4 As explained, in addition to the object 610, the camera calibration procedure also involves the camera device 620 and the motion capture system including the motion capture device 630. Figure 6 In the illustration of , during the camera calibration procedure, the object 610 is movable while the camera device 620 remains stationary. The motion capture system is used to generate motion capture data of the object 610, which shows the motion of the object 610 over time. The camera device 620 generates image data of a video stream representing the motion. The motion capture frame rate and the video frame rate may be different. For example, a motion capture frame rate of 144 FPS and a video capture frame rate of 24 FPS may be used. The pose indicated by the motion capture data may be associated with an image (e.g., a video frame) based on timing, such as Figure 7 Further illustrated in FIG.

[0066] like Figure 6 As shown, object 610 is placed in a first pose 602 for a certain amount of time (e.g., a half-second time window, or a time window of some other length). While in the first pose 602, first motion capture data 632 is generated by the motion capture system at a motion capture frame rate and indicates the first pose 602. While also in the first pose 602, first image data 622 is generated by camera device 620 at a video frame rate and shows object 610 in the first pose 602.

[0067] Then, the object 610 is moved to the second pose 604. It may take a certain amount of time (e.g., several milliseconds, such as 50 ms, etc.) to place the object in the second pose 604. During this period, motion capture data and image data can still be generated. Figure 7 As explained, these motion capture data and image data can be identified and discarded.

[0068] The object 610 maintains the second pose 604 for a certain amount of time (e.g., another half-second time window or a time window of some other length) before being moved to the next pose. While in the second pose 604, second motion capture data 634 is generated by the motion capture system at the motion capture frame rate and indicates the second pose 604. While also in the second pose 604, second image data 624 is generated by the camera device 620 at the video frame rate and shows the object 610 in the second pose 604.

[0069] The above process of moving the object 610 to a different pose after each time window (e.g., every half second) may be repeated to generate motion capture data indicating the poses of the object 610 over time and image data representing a video stream showing the poses. In an example, determining a minimum number of poses of the object 610 (e.g., about four poses) may be sufficient to calibrate the camera device 620. However, a larger number of poses (e.g., in the range of thirty to sixty or more) may improve the accuracy of the camera calibration.

[0070] Moving the object 610 can be performed by an operator. The operator can be a human. Alternatively, the operator can be a robotic system. For example, the robotic system can include a movable body (e.g., a body mounted on wheels) including a controller (e.g., a set of processors) and a memory, a robotic arm extending from the movable body and controllable by the controller, and an end effector (effector) that can also be controlled by the controller and has one end permanently attached to the robotic arm and the other end releasably attached to the object 610. The memory can store instructions executed by (multiple) processors to control the robotic arm and the end effector. The memory can further store a motion path for controlling the robotic arm and the end effector, wherein the motion path indicates a sequence of postures in which the object 610 is to be placed.

[0071] As shown above, a gesture indicated by a portion of motion capture data can be associated with an image indicated by a portion of image data. The association can be based on timing (e.g., the timing of the gesture matches the timing of the image). However, other types of associations can also be used. For example, visual associations are also possible. In particular, referring back to the robotic system, the motion path can include an identifier for each gesture. The robotic system can also include a display that presents the identifier for each gesture. When imaging the gesture, the camera device 620 generates an image, wherein each image not only shows the object 610 in a specific gesture, but also shows the identifier of the gesture presented on the display. Therefore, each image is associated with a corresponding gesture, wherein the gesture is identified in the motion path.

[0072] Figure 7An example of pose data 710 and image data 720 associated with an object according to an embodiment of the present disclosure is illustrated. The pose data 710 may include motion capture data generated by a motion capture system (e.g., as combined with Figure 4 and Figure 6 ). In general, the gesture data 710 indicates the gesture of the object over time, wherein a sequence of gestures represents the motion of the object. A gesture may be defined as a position and rotation in a coordinate system (e.g., a coordinate system of a motion capture system). Thus, the portion of the gesture data 710 corresponding to a gesture may indicate the position and rotation by including x, y, and z coordinates and x, y, z rotations (although non-Cartesian coordinate systems may also be used). The image data 720 may include images generated by a camera device, wherein a sequence of images shows the motion of the object over time.

[0073] For the sake of clarity, Figure 7 , a graph of x-coordinates with values ​​that can be obtained from posture data 710 is shown. The horizontal axis of the graph corresponds to time, and the vertical axis corresponds to the value. However, the y and z coordinates and the x, y and z rotations can be plotted similarly. The posture data 710 changes over time. The magnitude or slope of the change can indicate a transition from one posture to the next. For example, two x-coordinates (or the average of two sets of x-coordinates) can be compared to determine the change in the x-coordinate (e.g., the change in the slope of the line connecting the x-coordinates). If the change is less than a threshold, the x-coordinate has not changed significantly, indicating that the object is still in the same posture. Otherwise, the x-coordinate has changed significantly, indicating that the object has been moved to the next posture.

[0074] refer to Figure 7 , during a first time window "TW0" (e.g., a 500ms time window) defined by a start time "t0" and an end time "t1", the corresponding x-coordinate 712 does not change significantly, indicating that the object is in a first posture between time "t0" and "t1". During a transition time window (e.g., a time window of approximately 50ms) starting at time "t1" and ending at time "t2", a significant change occurs, indicating a transition to a next (second) posture. During a second time window "TW1" (e.g., a 500ms time window) defined by a start time "t2" and an end time "t3", the corresponding x-coordinate 716 does not change significantly, indicating that the object is in a second posture between time "t2" and "t3". During a transition time window (e.g., a time window of approximately 50ms) starting at time "t3" and ending at time "t4", a significant change occurs, indicating a transition to a next (third) posture.

[0075] The first gesture is associated with the first time window "TW0". To determine the x-coordinate of the first gesture, some or all of the x-coordinates 712 belonging to the first time window "TW0" may be used. In particular, a statistical measure (e.g., an average function) may be applied to the associated x-coordinates 712 to calculate the x-coordinate of the first gesture. This type of calculation may be repeated over different time windows and z- and y-axes to determine different gestures.

[0076] Image data 720 includes images corresponding to gestures (e.g., first image 722 and second image 724). For example, first image 722 corresponds to a first gesture, and second image 724 corresponds to a second gesture. Of course, the number of images for each gesture may depend on the video frame rate and the length of time the object remains in the gesture (e.g., the length of time of each time window "TW0", "TW1", etc.). If more than one image is available for a gesture, a single image (or a subset of these images) may be randomly selected for further processing.

[0077] Figure 8 An example of a camera model 810 determined based on a camera calibration procedure according to an embodiment of the present disclosure is illustrated. The camera model 810 includes a focal point 812 and an image plane 814 separated by a distance representing a focal length 816. The focal point 812 may be the center of an aperture through which light rays from an object 850 pass and are projected onto the image plane 814 as an inverted image.

[0078] The camera model 810 may characterize the corresponding camera device by including camera extrinsic parameters 820, camera intrinsic parameters 822, and a distortion model 824. The camera extrinsic parameters 820 represent the orientation of the camera device in the real world space. The orientation may correspond to the pose of the focus 812. The camera intrinsic parameters 822 represent the optical center and focal length of the camera. For example, the camera intrinsic parameters 822 include the focal length 816 and the optical center (also called the principal point), and may be represented as a camera matrix: Where "f" represents the focal length 816 along the X, Y, and Z axes, and "c" represents the optical center along the X, Y, and Z axes. For example, when the image axes are not perpendicular (the axes form an acute angle "β" (e.g., the pixels have a parallelogram shape instead of a square shape)), the skew coefficient "s" can also be included in the camera matrix. In this case, s = f xtanα, where “α” is the angle difference between ninety degrees and “β”. During rendering, the skew factor “s” can be used in a de-skew operation so that the rendered pixels are adjusted to have a vertical axis (e.g., have a square shape). Camera extrinsic parameters 820 represent a rigid transformation from a three-dimensional coordinate system in real world space (e.g., a coordinate system used by a motion capture system) to a three-dimensional coordinate system local to the camera device. Camera intrinsic parameters 822 represent a projective transformation from the three-dimensional coordinates of the camera device to two-dimensional image coordinates.

[0079] The camera matrix does not account for lens distortion. To accurately represent the camera device, the distortion model 824 includes radial lens distortion and tangential lens distortion. Radial distortion occurs when light bends more at the edge of the lens than it does at the optical center. The smaller the lens, the greater the distortion. The radial distortion coefficient models this type of distortion. The distorted point is represented by (x 畸变 ,y 畸变 ):x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ) and y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ), where x, y represent the undistorted pixel positions (x and y are in normalized image coordinates; the normalized image coordinates are calculated by translating the pixel coordinates to the optical center and dividing by the pixel focal length), k1, k2, and k3 represent the radial distortion coefficients of the lens, and r 2 =x 2 +y 2 .

[0080] Tangential distortion occurs when the lens and image plane are not parallel. The tangential distortion coefficient models this type of distortion. The distortion point is represented by (x 畸变 ,y 畸变 ):x 畸变 =x+[2*p1*x*y+p2*(r 2 +2*x 2 )] and y 畸变 =y+[p1*(r 2 +2*y 2 )+2*p2*x*y], where x, y also represent the undistorted pixel positions, p1 and p2 are the tangential distortion coefficients of the lens, and r 2 It is also equal to x 2 +y 2 .

[0081] As further described in the following figures, each of the camera extrinsic parameters 820, the camera intrinsic parameters 822, and the distortion model 824 may be determined based on a camera calibration procedure. Alternatively, the camera intrinsic parameters 822 and / or the distortion model 824 may be predefined (e.g., data available from the manufacturer of the camera device or a third party and defined for the device model, or a device serial number, etc.). If the camera intrinsic parameters 822 or the distortion model 824 are available, the camera calibration procedure may only determine the camera extrinsic parameters 820 without updating the predefined camera intrinsic parameters 822 and / or the distortion model 824 (if applicable). In this case, alternatively, in addition to determining the camera extrinsic parameters 820, the camera calibration procedure may initialize the camera model 810 to use the predefined camera intrinsic parameters 822 and / or the distortion model 824 (if applicable), and then update the predefined camera intrinsic parameters 822 and / or the distortion model 824.

[0082] Figures 9 to 11 A process is described that is associated with determining and using one or more gestures of camera device focus. The operations of the process can be performed by a computer system, such as computer system 140 or 430. Some or all of the instructions for performing the operations can be implemented as hardware circuits and / or stored as computer-readable instructions on a non-transitory computer-readable medium of the computer system. As implemented, the instructions represent a component that includes circuits or codes that can be executed by (multiple) processors of the computer system. The use of such instructions configures the computer system to perform specific operations described herein. Each circuit or code in combination with (multiple) associated processors represents a device for performing (multiple) corresponding operations. Although these operations are illustrated in a particular order, it should be understood that the particular order is not required, and one or more operations may be omitted, skipped, performed in parallel, and / or reordered.

[0083] Fig. 9 An example of a process for rendering content based on the determination of the focus of a camera device according to an embodiment of the present disclosure is illustrated. The process may begin at operation 902, where a computer system determines a first posture of a tag attached to a camera device. In an example, a computer system receives posture data over time, such as motion capture data generated by a motion capture system, wherein the posture data indicates the movement of the tag and is defined for a tracking point on the tag (e.g., its rigid body point). A portion of the posture data may have a timestamp corresponding to the time when the camera device is in the first posture in real world space. The portion of the posture data may include x, y, and z coordinates and x, y, and z rotations, which indicate the position and rotation of the tag in a coordinate system (e.g., the coordinate system of the motion capture system).

[0084] At operation 904, the computer system determines an offset between the marker and the focus of the camera device based on the first posture and at least one calibration parameter of the camera device. In an example, the camera device is associated with a camera model that includes at least one calibration parameter, such as one or more camera extrinsic parameters and / or one or more camera intrinsic parameters. The (multiple) camera extrinsic parameters can locate the focus in the coordinate system of the camera device. The (multiple) camera intrinsic parameters can indicate an image plane parallel to a plane containing the focus, and can also indicate a focal length between the image plane and the focus. Based on the (multiple) camera extrinsic parameters and the (multiple) camera intrinsic parameters, a position offset and a rotation offset are calculated based on the position and rotation of the marker (e.g., its tracking point). The position offset and the rotation offset form an offset between the marker and the focus. The camera model can be predefined or determined partially or completely based on a camera calibration procedure.

[0085] At operation 906, the computer system sets a second pose of the virtual camera based on the offset. The virtual camera can be a virtual representation of a camera device in a virtual space, where the virtual space is in turn a virtual representation of a real-world space. In an example, the pose of the focus in the coordinate system of the real-world space (the coordinate system of the motion capture system that determines the first pose) is determined by translating the position of the marker by the position offset and by rotating the rotation of the marker by the rotation offset. The pose of the focus can be represented as the position and rotation of the marker in the coordinate system. Given a mapping between the coordinate system of the real-world space and the coordinate system of the virtual space, the position and rotation of the focus are mapped to a virtual position and a virtual rotation in the virtual space. The virtual position and the virtual rotation define the second pose of the virtual camera.

[0086] At operation 908, the computer system renders the content based on the second posture. For example, the computer system synthesizes images and / or image frames (two-dimensional or three-dimensional) to generate content. The synthesis can apply a transformation (e.g., skew, angle, distortion, etc.) given the second posture, so that the content itself and / or its presentation is adjusted to take into account the posture of the focus in the real world space when presenting the content (e.g., on a display component). In another example, the camera model may include a distortion model. The rendering can distort the content based on the second posture by using radial distortion and / or tangential distortion (or its (multiple) inverse distortions) to give special effects to the presentation of the content given the posture of the focus in the real world space. For example, the display component is modeled as a virtual display in a virtual space. The computer system determines a virtual display area (e.g., the entire virtual display or a portion thereof) based on the second posture (e.g., a virtual display area within the field of view of a virtual camera given the second posture). The computer system also determines a portion of the content associated with the virtual display area (e.g., content to be displayed in the display area of ​​the display component corresponding to the virtual display area). The computer system then distorts a portion of the content based on the distortion model (eg, by using radial distortion, tangential distortion, inverse distortion(s) thereof, etc.).

[0087] At operation 910, the computer system causes the presentation of content. For example, the computer system outputs the content to a display component, where the display component presents the content. While presenting the content, the camera device may generate a video stream showing the content.

[0088] Combined with a single pose of the camera device to describe Fig. 9 Some or all of the operations of the process may be repeated for different poses of the camera device, where the sequence of poses corresponds to the motion of the camera device. For example, the offset may be determined once and reused, or it may be calculated as the pose of the marker changes. In both cases, the second pose of the focus may be updated given the latest pose of the marker. The updated pose may be used to update the rendering of the content.

[0089] Fig.10 An example of a process for determining an offset between a focus of a camera device and a marker attached to the camera device according to an embodiment of the present disclosure is illustrated. The offset may be determined while performing a camera calibration procedure. Fig.10 The operations of the process can be implemented as Fig. 9 In the example, Fig.10The process of can begin at operation 1002, where the computer system determines a pose of an object. A pose sequence represents the movement of the object over time. In an example, the computer system receives pose data indicating a pose, such as motion capture data generated by a motion capture system and tracking a marker attached to the object. Given changes in the pose data, the computer system can determine the pose.

[0090] At operation 1004, the computer system receives an image showing a gesture. For example, the image is generated by a camera device, represents a video stream showing the motion of the object, and is received from the camera device as image data (eg, a video frame).

[0091] At operation 1006, the computer system associates the image with the gesture. For example, an image showing the gesture of the object is associated with the gesture. In an example, the association can be time-based. In particular, the timing of each image is determined (e.g., as a timestamp when the image was generated) and the timing of the image is matched with the timing of the gesture (e.g., matched with a time window associated with the gesture data from which the gesture was calculated; the match can correspond to a timestamp falling between the start time and the end time of the time window). Given the match, an association of the image to the gesture is established. Other types of associations can be used additionally or alternatively. For example, referring back to a robotic system that moves the object and displays an identifier for each gesture, the image showing the gesture can be processed so that the identifier of the gesture displayed by the robotic system can also be detected.

[0092] At operation 1008, the computer system calculates a camera model based on the image and the pose (and the association between the two). For example, the image and the pose are used in a camera calibration procedure to determine (if applicable) camera extrinsic parameters, camera intrinsic parameters, and / or a distortion model. In an example, the camera calibration procedure includes multiple steps.

[0093] In a first step, a subset of images is randomly selected (e.g., a minimum of four images, or more typically in the range of thirty to sixty images, etc.). Visual features on the object (e.g., as determined by Figure 5 The pattern 520 formed by the image is detected on each selected image as a two-dimensional coordinate on the image. Based on any recognizable pattern (e.g., a checkerboard pattern), any type of two-dimensional feature (e.g., SIFT, SURF, ORB, etc.) can be used.

[0094] In the second step, a correspondence is established between the three-dimensional features (e.g., their three-dimensional coordinates (in millimeters) in the coordinate system of the motion capture system (referred to herein as the global coordinate system)) and the two-dimensional features (e.g., their two-dimensional image coordinates (in pixels) in the coordinate system of the image (referred to herein as the image coordinate system). For example, if a checkerboard pattern is used, the position of any checkerboard grid point (e.g., any detectable feature) can be described in the local checkerboard pattern coordinate system as: ij =I×cell_size,Y ij = J × cell_size, and in the case of a planar surface, Z ij = 0, where “I” and “J” are the indices of the feature and cell_size is the size of the feature in millimeters.

[0095] Next, a camera calibration procedure can be performed on the extracted features. The motion capture data of the recognizable pattern is used as the initial value of the camera external parameters. Each selected image has six external parameters: three position coordinates and three Rodrigues angles. The camera internal parameters are the same for the selected images. According to user settings, multiple internal parameters may include focal length (in the X direction and Y direction) and principal point (X image coordinate and Y image coordinate), and the distortion model may include multiple distortion parameters ("k" (radial distortion) and "p" (tangential distortion) as described above.

[0096] In the example of a distortion model involving four distortion parameters (k8-k11, such as in the case of a thin prism model), the following expression may be used: x″=x′*(1+k1r 2 +k2r 4 +k3r 6 ) / (1+k4r 2 +k5r 4 +k6r 6 )+2p1x′y′+p2*(r 2 +2x′ 2 ), y″=y′*(1+k1r 2 +k2r 4 +k3r 6 ) / (1+k4r 2 +k5r 4 +k6r 6 )+p1(r 2 +2y′ 2 )+2p2x′y′, and r 2 = x′ 2 +y′ 2x' and y' are the normalized projection coordinates (normalized image coordinates) of the 3D point of the object. The distortion model can also include a thin prism model distortion. In this case, the additional offset in the 3D point projection can be described as X" + = k8 * r 2 +k9*r 4 And Y"+=k10*r 2 +k11*r 4 .

[0097] In the case of using the angle parameter (τ x and τ y In another example of a distortion model, such as also in the case of a thin prism model, the following expression (called a tilt matrix) can be used:

[0098]

[0099] Using the pose associated with each selected image, the computer system receives the three-dimensional coordinates of the features in the pose, wherein these coordinates are represented as: P ij_global =R 图案 *P ij_local +P 图案 .P ij_global and P ij_local Correspondingly, it is the three-dimensional position of any feature in the global coordinate system and the local coordinate system, R 图案 is the rotation matrix of the pattern, and P 图案 is the position of the pattern.

[0100] After determining the correspondence between the 3D coordinates (in the global coordinate system, in millimeters) and the 2D coordinates (in the image coordinate system, in pixels) of any image, the n-point perspective problem (PnP) can be solved to estimate the pose of the calibrated camera device from the “n” 3D to 2D feature correspondences. For any image, the pose of the received camera device (Rk 相机 , the rotation of the camera device for the kth image, and Pk 相机 , the position of the camera device for the kth frame).

[0101] The computer system can then rotate the initial camera projection plane (R 相机 ) and the initial camera focus position (P 相机 ) is estimated as the weighted average of the received posture data: R 相机 =Σ n k＝0 weight_k*Rk 相机 And P 相机 =Σ n k＝0 weight_k*Pk 相机The weights may be computed as normalized reprojection errors in the PnP solution for each image. The reprojection error may be computed by determining a projection of an object (or a feature thereof) in a certain pose on the image plane of the camera device and the error (e.g., distance error) between the projection and the two-dimensional coordinates of the object (or a feature thereof) in the image showing the pose.

[0102] Using the camera calibration parameters and the initial pose estimate, the internal parameters and the external parameters can be estimated by applying the fitting model to the camera calibration parameters (including any of the camera extrinsic parameters, camera intrinsic parameters and / or distortion parameters). In general, the fitting model is a data fitting model that iteratively estimates the camera calibration parameters so as to reduce or minimize the reprojection error. Different types of data fitting models are possible, such as Levenberg-Marquardt nonlinear least squares algorithm, chi-square test algorithm, curve fitting algorithm, weighted least squares fitting algorithm, polynomial regression algorithm, Gauss-Newton algorithm, shift-cut algorithm, gradient algorithm, Nelder-Mead (simplex) search algorithm, etc. When the camera device remains stationary during the camera calibration procedure, the number of variables in the fitting model may depend on the camera intrinsic parameters and the distortion model, and may not depend on the number of selected images, because the same camera device pose is expected to be used for all images. The reprojection error is used for the least squares curve fitting problem. Additionally or alternatively, the fitting model may include a machine learning model (such as a regression model or a convolutional neural network) that is trained using multiple known camera models, known object poses, and training images showing such object poses.

[0103] Further, the fit model can be called twice. First, the full set of selected images is used to estimate the camera model and processed by the fit model. Next, a subset of the selected images is used. The subset can be selected by selecting the images with the smallest reprojection error (e.g., if thirty to sixty images were selected, then a subset of ten to twenty images or some other number of images with the smallest reprojection error is selected). This subset can be reprocessed again by the fit model to further refine the parameters of the camera model.

[0104] At operation 1010, the computer system determines the pose of a marker attached to a camera. For example, the marker is attached before a camera calibration procedure begins. The pose of the marker can be determined from pose data generated, for example, by a motion capture system. During the camera calibration procedure, the camera can remain stationary. In this case, the pose data is substantially unchanged during the camera calibration procedure. Therefore, the position and rotation of the marker can be calculated in a global coordinate system as an average of its pose data.

[0105] At operation 1012, the computer system determines a rotational offset and a positional offset of the focus of the camera device based on the pose of the marker and the camera model. For example, the pose of the marker indicates a first rotation of the marker in a real-world space (e.g., in a global coordinate system, where the first rotation is represented as a rotation matrix R around the X, Y, and Z axes). 标记 ). The camera model (e.g., one or more camera extrinsic parameters thereof) may indicate a second rotation of the focus (e.g., where the second rotation is represented as a rotation matrix R around the X, Y, and Z axes 相机 The rotation offset may be determined based on the first rotation and the second rotation, such as by using the following expression: 偏移 =R 相机 *R 标记 -1 , where R 标记 -1 is the rotation matrix R of the marker 标记 The inverse matrix of .

[0106] The pose of the marker also indicates a first position of the marker in real-world space (e.g., in a global coordinate system, where the first position is represented as a position matrix T along the X, Y, and Z axes). 标记 ). The camera model (e.g., one or more camera extrinsic parameters thereof) may indicate a second position of the focal point (e.g., where the second position is represented as a position matrix T along the X, Y, and Z axes 相机 The position offset may be determined based on the first rotation, the first position, and the second position, such as by using the following expression: 偏移 =R 标记 -1 *(T 相机 -T 标记 ).

[0107] Fig.11 An example of a process for associating pose data with image data in a camera calibration procedure according to an embodiment of the present disclosure is illustrated. These associations are time-based. Fig.11 The operations of the process can be implemented as Fig. 9 The sub-operations of the process and / or Fig.10 In the example, Fig.11 The process may begin at operation 1102, where a computer system receives motion capture data indicating a pose of an object over time. For example, a motion capture system generates motion capture data at a specific frame rate (e.g., 144 FPS) and sends the data to a computer system.

[0108] At operation 1104, the computer system receives image data of a video stream representing a pose of an object. For example, a camera device generates image data as video frames at a specific frame rate (eg, 24 FPS) and sends the data to the computer system.

[0109] At operation 1106, the computer system determines the change in the motion capture data over time. The change can be the magnitude and / or slope of an x, y, or z coordinate or an x, y, z rotation.

[0110] At operation 1108, the computer system determines the pose of the object based on the change. The change can be compared to thresholds defined for each of the x, y, and z axes. If the change in all three axes is less than the threshold, a pose is detected and associated with the time window in which the change is less than the threshold. Otherwise, a transition between the current pose and the next pose is detected.

[0111] At operation 1110, the computer system associates the image with the gesture based on the timing of the gesture and the image. For example, each received image has a timestamp. Each gesture is associated with a time window. If the timestamp of the image falls within the time window of the gesture, the image is associated with the gesture, wherein the association indicates that the image shows the gesture.

[0112] Fig.12 Illustrated are example components of a computer system 1200 according to an embodiment of the present disclosure. Computer system 1200 is an example of computer system 140 or 430. Although components of computer system 1200 are illustrated as belonging to the same computer system 1200, computer system 1200 may also be distributed (eg, among multiple user devices).

[0113] The computer system 1200 includes at least a processor 1202, a memory 1204, a storage device 1206, an input / output peripheral (I / O) 1208, a communication peripheral 1210, and an interface bus 1212. The interface bus 1212 is configured to communicate, transport, and deliver data, control, and commands between the various components of the computer system 1200. The memory 1204 and the storage device 1206 include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage devices (e.g., Memory) and other tangible storage media. Any such computer-readable storage medium may be configured to store instructions or program codes embodying various aspects of the present disclosure. Memory 1204 and storage device 1206 also include computer-readable signal media. Computer-readable signal media include propagated data signals in which computer-readable program code is embodied. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media include any computer-readable media that is not a computer-readable storage medium and can transmit, propagate, or convey a program used in conjunction with computer system 1200.

[0114] Further, the memory 1204 includes an operating system, programs and applications. The processor 1202 is configured to execute stored instructions and includes, for example, a logic processing unit, a microprocessor, a digital signal processor and other processors. The memory 1204 and / or the processor 1202 can be virtualized and can be hosted in another computer system such as a cloud network or a data center. The I / O peripherals 1208 include user interfaces (such as keyboards, screens (e.g., touch screens), microphones, speakers, other input / output devices) and computing components (such as graphics processing units) serial ports, parallel ports, universal serial buses and other input / output peripherals. The I / O peripherals 1208 are connected to the processor 1202 via any port coupled to the interface bus 1212. The communication peripherals 1210 are configured to facilitate communication between the computer system 1200 and other systems through a communication network, and include, for example, a network interface controller, a modem, wireless and wired interface cards, antennas, and other communication peripherals.

[0115] It is obvious to those skilled in the art that, without departing from the inventive concept of this article, more modifications are possible in addition to those modifications that have been described. Therefore, the subject matter of the present invention is not limited except for the spirit of the appended claims. In addition, when interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprise" and "comprising" should be interpreted as referring to elements, parts or steps in a non-exclusive manner, indicating that the referenced elements, parts or steps may exist or be utilized, or be combined with other elements, parts or steps that are not explicitly referenced. When the specification or claim mentions at least one selected from the group consisting of A, B, C ... to N, the text should be interpreted as requiring only one element in the group, rather than A plus N, or B plus N, etc.

Claims

1. A computer-implemented method comprising: determining, via at least one processor, a first pose of a marker in a real-world space, wherein the marker is attached to the camera device; determining, via the at least one processor, an offset between the marker and a control point of the camera device based on the first pose and at least one calibration parameter of the camera device; setting, via the at least one processor, in a computer-readable memory, a second pose of a virtual camera in a virtual space based on the offset, wherein the virtual camera represents a camera device in the virtual space, and wherein the virtual space represents the real-world space; and Content is rendered, via the at least one processor, on a display based on a second pose of the virtual camera, wherein the content is to be captured by the camera device when presenting the content in the real-world space.

2. The computer-implemented method of claim 1, wherein: Determining the offset includes: determining a first rotation of the marker in the real-world space based on the first pose; determining a second rotation of the control point based on the at least one calibration parameter; and A rotation offset is determined based on the first rotation and the second rotation.

3. The computer-implemented method of claim 2, wherein: Determining the offset further includes: determining a first position of the marker in the real-world space based on the first pose; determining a second position of the control point based on the at least one calibration parameter; and A position offset is determined based on the first rotation, the first position, and the second position.

4. The computer-implemented method of claim 1, wherein: The second pose is set as a translation and rotation of the first pose based on the offset.

5. The computer-implemented method of claim 1, wherein: The at least one calibration parameter is determined from a camera model of the camera device, wherein the camera model includes a distortion model, and wherein rendering the content comprises: determining a virtual display area in the virtual space based on the second gesture; determining a portion of content associated with the virtual display area; and The portion of the content is distorted based on the distortion model.

6. The computer-implemented method of claim 1, wherein: The at least one calibration parameter is determined from a camera model of the camera device, wherein the camera model is determined based on a calibration procedure including: determining a plurality of poses of the object in the real-world space; receiving a plurality of images generated by the camera device, wherein the plurality of images represent a video stream showing the object in the plurality of poses; and A camera model of the camera device is determined based on the multiple postures and the multiple images, wherein the camera model includes camera extrinsic parameters, camera intrinsic parameters, and a distortion model.

7. The computer-implemented method of claim 1, wherein: The at least one calibration parameter is determined based on a camera calibration procedure including: determining a third pose of the object in the real-world space; receiving an image generated by the camera device, wherein the image shows the object when the object is in the third pose; and The at least one calibration parameter is determined based on the third pose and the image.

8. The computer-implemented method of claim 7, wherein: The camera calibration procedure is performed when one of the camera device or the object is stationary and the other one of the camera device or the object is movable.

9. The computer-implemented method of claim 7, wherein: The camera calibration procedure further includes: determining a plurality of poses of the object in the real-world space when the camera device is maintained at the same pose; receiving a plurality of images generated by the camera device, wherein the plurality of images represent a video stream showing the object in the plurality of poses; determining that the third posture is shown in the image; determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model includes the at least one calibration parameter; selecting an image subset of the plurality of images based on the projection error, wherein the image subset is associated with a pose subset of the plurality of poses; and The camera model is updated based on the image subset and the pose subset.

10. The computer-implemented method of claim 7, wherein: The camera calibration procedure further includes: determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model includes the at least one calibration parameter; and The camera model is updated based on a fitted model that reduces the projection error, wherein an output of the fitted model indicates one or more parameters of the camera model.

11. The computer-implemented method of claim 7, wherein: The camera calibration procedure further includes: determining a first estimated pose of the camera device based on a third pose of the object and the image; determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model includes the at least one calibration parameter; associating a first weight with the first estimated pose based on the projection error; determining a final estimated pose of the camera device based on the first estimated pose and the first weight; and The camera model is updated based on the fitted model using the final estimated pose and the projection error.

12. The computer-implemented method of claim 1, wherein: The at least one calibration parameter is determined from a camera model of the camera device, wherein the camera model is determined based on a camera calibration procedure including: receiving motion capture data representing a plurality of poses of an object in the real-world space over time while the camera device is maintained at a same pose; determining a timing of a third pose of the object based on the change in the motion capture data; and The at least one calibration parameter is determined based on the third pose.

13. The computer-implemented method of claim 12, wherein: The camera calibration procedure further includes: receiving image data representing a plurality of images generated by the camera device over time while the camera device is held at a same pose, wherein the plurality of images represent a video stream showing the object at the plurality of poses; and The third gesture is associated with an image in the plurality of images based on a timing of the third gesture and a timing of the image, wherein the at least one calibration parameter is further determined based at least in part on the timing.

14. A system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, configure the system to: determining a first pose of a marker in a real-world space, wherein the marker is attached to the camera device; determining an offset between the marker and a control point of the camera device based on the first pose and at least one calibration parameter of the camera device; setting, in the one or more memories, a second pose of a virtual camera in a virtual space based on the offset, wherein the virtual camera represents a camera device in the virtual space, and wherein the virtual space represents the real-world space; and Content is rendered on the display based on a second pose of the virtual camera, wherein the content is to be captured by the camera device when presenting the content in the real-world space.

15. The system of claim 14, wherein: The at least one calibration parameter is determined from a camera model of the camera device, wherein the camera model is determined based on a calibration procedure including: determining a plurality of poses of the object in the real-world space; receiving a plurality of images generated by the camera device, wherein the plurality of images represent a video stream showing the object in the plurality of poses; and The camera model is determined based on the plurality of poses and the plurality of images.

16. The system of claim 15, wherein: The object is placed in the plurality of poses by a robotic system.

17. The system of claim 15, wherein: The tag comprises a tag unit, wherein the object comprises a feature pattern, and wherein features of the feature pattern are substantially the same size as the tag unit of the tag.

18. One or more non-transitory computer-readable storage media storing instructions that, when executed on a system, cause the system to perform operations including: Determine the first pose of the marker in real-world space, where The marker is attached to the camera device; determining an offset between the marker and a control point of the camera device based on the first pose and at least one calibration parameter of the camera device; setting in a computer-readable memory a second pose of a virtual camera in a virtual space based on the offset, wherein the virtual camera represents a camera device in the virtual space, and wherein the virtual space represents the real-world space; and Content is rendered on the display based on a second pose of the virtual camera, wherein the content is to be captured by the camera device when presenting the content in the real-world space.

19. The one or more non-transitory computer-readable storage media of claim 18, wherein: Determining the offset includes: determining a first rotation and a first position of the marker in the real-world space based on the first pose; determining a second rotation and a second position of the control point based on the at least one calibration parameter; and A rotational offset is determined based on the first rotation and the second rotation, and a positional offset is determined based on the first rotation, the first position, and the second position.

20. The one or more non-transitory computer-readable storage media of claim 18, wherein: The second pose is set as a translation and rotation of the first pose based on the offset.

Images