Virtual fabrication based on display component pose and pose error correction

Abstract

Systems, apparatuses, and methods for rendering content based on display component gestures are disclosed. In an example, motion capture data is received for a display of a plurality of displays, the display moving from a first physical pose to a second physical pose. The motion capture data is processed to determine coordinates of the second physical pose. A transformation of a second physical pose of the display to a virtual pose of the display is generated. A virtual model of the plurality of displays is updated, the virtual model including virtual poses of the displays. And rendering the content on the updated virtual model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 390,252, filed on July 18, 2022, entitled “VIRTUAL PRODUCTION BASED ON POSE ERROR CORRECTION,” and U.S. Provisional Patent Application No. 63 / 458,412, filed on April 10, 2023, entitled “VIRTUAL PRODUCTION BASED ON DISPLAY ASSEMBLYPOSE,” the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The field of the invention relates to virtual production. Background Art

[0004] The background description includes information that may be used to understand the subject matter of the present invention. It is not an admission that any of the information provided herein is prior art or that the applicant has admitted prior art, or that it is relevant to the presently claimed subject matter, or that any publication explicitly or implicitly referenced is prior art or that the applicant has admitted prior art.

[0005] Virtual production, such as movie production, typically involves a virtual stage on which content associated with a scene is presented, a camera device that generates cinematographic data by capturing video of people, objects, and content, and a motion capture system that tracks the camera, people, and / or objects. The content can be dynamic (e.g., video content that changes over time), and / or its presentation can be adjusted based on the tracking.

[0006] All publications identified herein are incorporated by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. If a definition or use of a term in an incorporated reference is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.

[0007] In some embodiments, the digits representing the amount or units of data for example for describing and claiming certain embodiments of the subject of the present invention should be understood as being modified by the term "approximately" in some cases. Therefore, in some embodiments, the numerical parameters set forth in the written description and the appended claims are approximate values, which can vary according to the desired properties sought to be obtained in a particular embodiment. In some embodiments, the numerical parameters should be interpreted according to the number of significant figures reported and by applying common rounding techniques. Although the wide range of numerical ranges and parameters of some embodiments of the subject of the present invention are approximate values, the numerical values ​​set forth in the specific embodiments are reported as accurately as possible. The numerical values ​​presented in some embodiments of the subject of the present invention may include certain errors caused by the standard deviation found in their corresponding test measurements.

[0008] Unless the context indicates otherwise, all ranges set forth herein should be interpreted as including their endpoints, and open-ended ranges should be interpreted as including only commercially practical values. Likewise, all lists of values ​​should be interpreted as including intermediate values, unless the context indicates otherwise.

[0009] As used in the specification herein and throughout the claims, the meanings of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. Similarly, as used in the specification herein, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

[0010] The description of value ranges herein is intended only to be used as a shorthand method for individually referring to each individual value falling within the range. Unless otherwise indicated herein, each individual value is incorporated into the specification as if it were individually described herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradictory with context. The use of any and all examples or exemplary language (e.g., "such as") provided with respect to some embodiments herein is intended only to better illustrate the subject matter of the present invention, rather than to limit the scope of the subject matter of the present invention claimed in addition. Any language in the specification should not be interpreted as indicating that any unclaimed element is essential for the practice of the subject matter of the present invention.

[0011] The grouping of the alternative elements or embodiments of the subject matter of the present invention disclosed herein should not be construed as limiting. Each group member may be mentioned and claimed individually, or mentioned and claimed in any combination with other members of the group or other elements found herein. For convenience and / or patentability reasons, one or more members in the group may be included in the group or deleted from the group. When any such inclusion or deletion occurs, this specification is considered to contain the modified group in this article, thereby satisfying the written description of all Markush groups used in the appended claims.

[0012] It should be understood that many of the basic technical features provided in the following description are presented in order to enable a compact examination of the disclosed inventive subject matter. Although some of the basic technical features described herein may appear obscure, in many cases such features can be considered to be within the scope of understanding of those skilled in the art. Therefore, the presentation of such background technology should not be considered limiting. Summary of the invention

[0013] Embodiments described herein include a method for rendering content based on a display component posture. The method includes at least one processor receiving motion capture data of a display in a plurality of displays, the display moving from a first physical posture to a second physical posture. The at least one processor processes the motion capture data to determine the coordinates of the second physical posture. The at least one processor generates a transformation of the second physical posture of the display to a virtual posture of the display. The at least one processor updates a virtual model of the plurality of displays, the virtual model including the virtual posture of the display. The at least one processor renders content on the display based on the updated virtual model.

[0014] Embodiments may further include a system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, configure the system to receive motion capture data of a display in a plurality of displays, the display moving from a first physical pose to a second physical pose. The system may further process the motion capture data to determine coordinates of the second physical pose. The system may further update a transformation of the second physical pose of the display to a virtual pose of the display. The system may further update a virtual model of the plurality of displays, the virtual model including the virtual pose of the display. The system may further render content on the display based on the updated virtual model.

[0015] Embodiments may further include a non-transitory computer-readable medium including instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform operations including receiving motion capture data of a display in a plurality of displays, the display moving from a first physical posture to a second physical posture. The at least one processor processes the motion capture data to determine coordinates of the second physical posture. The at least one processor generates a transformation of the second physical posture of the display to a virtual posture of the display. The at least one processor updates a virtual model of the plurality of displays, the virtual model including the virtual posture of the display. The at least one processor renders content on the display based on the updated virtual model.

[0016] Embodiments may further include a computer-implemented method comprising the following steps: determining, by at least one processor, a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays; determining, by the at least one processor, a virtual model of the display assembly, wherein the virtual model is stored in a computer-readable memory and includes a virtual representation of each of the plurality of displays. Further, the method may include: determining, by the at least one processor, a transformation based on the first posture of the first display and a first virtual representation of the first display in the virtual model; and rendering, by the at least one processor, the content on at least some of the plurality of displays according to the transformation and the virtual model.

[0017] Additionally, an embodiment may include a system comprising one or more processors and one or more memories storing instructions that, when executed by the one or more processors, configure the system to determine a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays. The instructions may further configure the system to determine a virtual model of the display assembly, wherein the virtual model is stored in the one or more memories and includes a virtual representation of each of the plurality of displays. Further, the instructions may configure the system to: determine a transformation based on the first posture of the first display and a first virtual representation of the first display in the virtual model; and render the content on at least some of the plurality of displays according to the transformation and the virtual model.

[0018] Additionally, embodiments may include one or more non-transitory computer-readable storage media storing instructions that, when executed on a system, cause the system to perform operations including: determining a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays. The instructions may cause the system to determine a virtual model of the display assembly, wherein the virtual model is stored in a computer-readable memory and includes a virtual representation of each of the plurality of displays. The instructions may cause the system to: determine a transformation based on the first posture of the first display and a first virtual representation of the first display in the virtual model; and render the content on at least some of the plurality of displays according to the transformation and the virtual model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example of a virtual production system according to an embodiment of the present disclosure is illustrated.

[0020] Figure 2 An example of a display component and a virtual model thereof according to an embodiment of the present disclosure is illustrated.

[0021] Figure 3 An example of a motion path of a physical marker on a display assembly according to an embodiment of the present disclosure is illustrated.

[0022] Figure 4 An example of measured motion capture data that may be used to determine a physical pose of a display in a display assembly according to an embodiment of the present disclosure is illustrated.

[0023] Figure 5 An example of a presentation path of a virtual marker on a display component according to an embodiment of the present disclosure is illustrated.

[0024] Figure 6 An example of determining a transformation and displaying an updated virtual model of a component according to an embodiment of the present disclosure is illustrated.

[0025] Figure 7 An example of transformation-based position error correction according to an embodiment of the present disclosure is illustrated.

[0026] Figure 8 An example of a process for determining and using pose error corrections according to an embodiment of the present disclosure is illustrated.

[0027] Fig. 9 An example of a process for rendering content based on gesture error correction according to an embodiment of the present disclosure is illustrated.

[0028] Fig.10 An example of a process for processing motion capture data according to an embodiment of the present disclosure is illustrated.

[0029] Fig.11 An example of a process for determining a transformation based on a field of view of a camera device according to an embodiment of the present disclosure is illustrated.

[0030] Fig.12 An example of a virtual production system according to an embodiment of the present disclosure is illustrated.

[0031] Fig.13 An example of a virtual production system according to an embodiment of the present disclosure is illustrated.

[0032] Fig.14 is a graph of change point data of virtual production according to an embodiment of the present disclosure.

[0033] Fig.15 is an illustration of an example display assembly according to an embodiment of the present disclosure.

[0034] Fig.16 is an illustration of an example of determining a transformation and displaying an updated virtual model of a component according to an embodiment of the present disclosure.

[0035] Fig.17 is a table showing the difference between measured and calculated parameters.

[0036] Fig.18 An example of a process flow for determining an updated virtual model based on movement of a display according to an embodiment of the present disclosure is illustrated.

[0037] Fig.19 An example of a process flow for determining and initializing a motion capture system for virtual production according to an embodiment of the present disclosure is illustrated.

[0038] Fig. 20 An example flow for determining a transformation based on detecting movement of a display in a display assembly according to an embodiment of the present disclosure is illustrated.

[0039] Fig.21 Illustrated are example components of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] It should be noted that any language for computers should be interpreted to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing devices operating alone or together. It should be understood that the computing device includes at least one processor configured to execute software instructions stored on a tangible non-transitory computer-readable storage medium (e.g., a hard drive, FPGA, PLA, solid-state drive, RAM, flash memory, ROM, etc.). The software instructions or software instruction sets configure or program the computing device or its processor to provide roles, responsibilities, or other functions, as discussed below with respect to the disclosed apparatus or system. Further, the disclosed technology can be embodied as a computer program product, which includes a non-transitory computer-readable medium storing software instructions or software instruction sets, which cause one or more processors to perform the disclosed steps associated with the implementation of a computer-based algorithm, process, method, or other instruction. In some embodiments, various servers, systems, databases or interfaces use standardized protocols or algorithms to exchange data, which may be based on HTTP, HTTPS, TCP, UDP, FTP, SNMP, IP, AES, public-private key exchange, web services or RESTful APIs, known financial operation protocols or other electronic information exchange methods. Data exchange between devices can be carried out through the following networks: packet switching networks, the Internet, LAN, WAN, VPN or other types of packet switching networks; circuit switching networks; cell switching networks; or other types of networks (wired or wireless).

[0041] As used in the specification herein and throughout the claims that follow, when a system, engine, server, agent, device, module, or other computing element is described as being configured to perform or execute functions on data in a memory, the meaning of "configured to" or "programmed to" is defined as one or more processors or cores of the computing element being programmed by a set of software instructions stored in the memory of the computing element to perform a set of functions on target data or data objects stored in the memory. It should be understood that the combination of software and hardware working in concert can create a set of specialized physical real-world structures that provide one or more users with utility that does not exist outside the physical, real-world asset.

[0042] It should be appreciated that the disclosed technology provides many advantageous technical effects, including improved modeling of physical display components and improved rendering of content on at least some displays in the physical display components. For example, the accuracy of a virtual model representing a physical display component is improved by relying on gesture measurements of the physical display component. As the accuracy of the virtual model is improved, the rendering of content using the virtual model is also improved, whereby the position of content presented, for example, on a display is more accurate.

[0043] Embodiments of the present disclosure are particularly related to rendering content based on a virtual model representing the real-time posture of a display component. In an example, the display component includes a plurality of configurable displays, each of which is located within the display component with a specific posture (e.g., located within the display component with a specific physical position and a specific rotation). The posture of each display can be configurable to accommodate the desired virtual production. For example, the display can have six degrees of freedom and can, for example, be mechanically rotated, moved forward, moved backward, or tilted. In contrast, the virtual model includes a virtual representation of each display, wherein the virtual representation of the display within the virtual model indicates a real-time virtual posture corresponding to the real-time physical posture of the display. By updating the virtual model to reflect the current posture of each display in the display component, the quality of the virtual production can be improved.

[0044] By updating the virtual model to reflect the current pose of each display in the display assembly, the quality of the virtual production can be improved.

[0045] A display in the display assembly can move from a first physical posture to a second physical posture. For example, the display can be connected to an actuator, such as a winch, and can be repositioned from a first physical posture to a second physical posture. A virtual marker can be projected onto the display. Based on the detection of the repositioning, a motion capture system can be initialized to collect motion data using the virtual marker. The movement of the display can be detected by one or more image capture devices using the virtual marker.

[0046] Initialization of the motion capture system can be based on manual initialization or sensor response based initialization. Manual initialization can be performed by a user. For sensor based initialization, one or more sensors can be directed toward the display assembly and streaming data can be continuously collected. An algorithm, such as a change point detection algorithm based on a forgetting factor, can process the streaming data to determine that one or more parameters of the display assembly (e.g., rotation of the display) have changed.

[0047] A change point detection algorithm based on a forgetting factor can be used to detect change points in streaming media data. Some streaming media applications for change point detection may require the selection of two or more parameters for change point detection. However, parameter selection can be based on the predicted change size, and for streaming media data where multiple change sizes may occur, this selection may not be optimal. Therefore, a change point detection algorithm based on a forgetting factor that only requires the selection of a single parameter can be used.

[0048] Based on the motion data captured by the motion capture system, the computer system can determine a second physical pose. In some cases, the repositioning of the display can be performed during the shooting of the virtual production. In these cases, the second pose can be determined after the display has stopped moving, or the current real-time physical pose of the display can be continuously determined while the display moves.

[0049] In order to account for the shift of the position or orientation of the display in the display assembly, a fitting model is used to generate a transformation, wherein the current virtual posture is determined using the current physical posture (e.g., after being input to the fitting model after the shift). For example, the fitting model can execute one or more embodiments of the fitting algorithm; for example, the Levenberg-Marquardt nonlinear least squares algorithm, the chi-square test algorithm, the curve fitting algorithm, the weighted least squares fitting algorithm, the polynomial regression algorithm, the Gauss-Newton algorithm, the shift-cut algorithm, the gradient algorithm, or the Nelder-Mead (simplex) search algorithm. Different techniques can be used to determine the physical posture. In the example technology, a motion capture system is used to generate motion capture data for tracking a virtual marker presented on a moving display. The motion capture data is associated with the moving display (e.g., the first motion capture data as the first part of the motion capture data is associated with the moving display in the display assembly). The current posture of the display (e.g., the posture after the display moves) is derived from a portion of the motion capture data associated with the display.

[0050] Further, embodiments of the present disclosure are particularly related to rendering content based on correcting the posture errors (e.g., position errors and / or rotation errors) of a virtual model representing a display component. In an example, the display component includes a plurality of displays, each of which is located within the display component in a specific posture (e.g., located within the display component in a specific physical position and a specific rotation). In contrast, the virtual model includes a virtual representation of each display, wherein the virtual representation of the display within the virtual model indicates a virtual posture corresponding to the physical posture. Due to different factors (e.g., installation tolerances, incorrect installation, operating temperature, heat, thermal expansion, etc.), there may be a mismatch between the virtual posture and the physical posture. When rendering content based on the virtual model for presentation on the display component, this mismatch may cause quality issues. By correcting the posture errors, the mismatch can be reduced or even eliminated, thereby alleviating quality issues.

[0051] In order to correct the posture error, a fitting model is used to generate a transformation, wherein the physical posture and the virtual posture are input to the fitting model. For example, the fitting model can execute one or more embodiments of the fitting algorithm; for example, the Levenberg-Marquardt nonlinear least squares algorithm, the chi-square test algorithm, the curve fitting algorithm, the weighted least squares fitting algorithm, the polynomial regression algorithm, the Gauss-Newton algorithm, the shift-cut algorithm, the gradient algorithm, or the Nelder-Mead (simplex) search algorithm. Different techniques can be used to determine the physical posture. In one example technique, a motion capture system is used to generate motion capture data, which tracks physical markers placed at various positions on the display according to a predefined motion path. In another example technique, a motion capture system is used to generate motion capture data, which tracks virtual markers presented on the display according to a rendering path, rather than tracking physical markers. In both example techniques, the motion capture data is associated with the display (e.g., the first motion capture data as the first part of the motion capture data is associated with the first display in the display component, the second motion capture data as the second part of the motion capture data is associated with the second display in the display component, and so on). The physical pose of a display is derived from a portion of the motion capture data associated with the display by, for example, determining the coordinates and rotation of markers (physical or virtual) and including such data in the physical pose.

[0052] Different techniques can also be used to generate transformations. The first example technique (referred to as global determination in this article) uses a full set of physical gestures and a full set of virtual gestures as inputs to the fitting model. The second example technique (referred to as local determination in this article) instead uses a subset of physical gestures and a corresponding subset of virtual gestures. For example, the presentation of content on a display component can be based on one or more parameters (e.g., its gesture) of a camera device that generates video data showing the content. A subset of displays can be within the field of view of the camera device, while the remaining displays can be outside the field of view. In this illustration, only the physical gestures of the displays included in the subset and their corresponding virtual gestures are input to the fitting model. The physical gestures and virtual gestures of the remaining subsets are not input. In this way, the transformation is locally optimized by considering only relevant gesture data (e.g., position and / or rotation data of the display within the field of view). In general, the local determination technique uses a subset of displays, where the subset is localized based on one or more parameters of the camera device. For example, a subset can be defined in an XY plane by including "x" by "y" displays, where "x" is less than the total number of displays along the horizontal axis ("x 总 ”), and / or “y” is less than the total number of displays along the vertical axis (“y 总 ”). For example, a subset can be a vertical stack or a column of displays (e.g., “x” equals one “1” and “y” equals “y 总 ”), horizontal strips or a row of displays (e.g., “x” equals one “x 总 ” and “y” equals “1”), diagonal strip displays, “x” by “y” contiguous block displays, “x” by “y” non-contiguous block displays (e.g., the first display and the second display are part of the block, but the displays between the two displays are not part of the block), etc. By using a local subset of the displays, the transform can be fine-grained adjusted to optimize the transform to reduce pose error in specific dimensions that may correspond to specific camera work or movements.

[0053] For the sake of clarity of explanation, various embodiments of the present disclosure are described in conjunction with a virtual production use case, in which a virtual model based on a virtual stage presents content on the virtual stage. However, the embodiments are not limited thereto, but are equally applicable to other use cases, such as virtual reality, augmented reality, mixed reality, content projection (e.g., home theater, cinema or building, etc.), performance stage (e.g., concert), etc. In general, embodiments of the present disclosure allow for improvements in content presentation on a display component, wherein the presentation depends on a virtual model of the display component. In particular, embodiments disclose a technique for updating a virtual model of a virtual production based on the movement of a display element. Additionally or alternatively, embodiments disclose a technique for reducing the posture error between an actual physical posture of a display element in a display component and a virtual posture corresponding thereto. A display element may be an actual display represented as a rigid body in a virtual model. Additionally or alternatively, a display element may be a subdivision (e.g., a portion thereof) of a display, wherein the subdivision is also represented as a rigid body in a virtual model.

[0054] In addition, for the sake of clarity of explanation, various embodiments of the present disclosure are described in conjunction with position (in real world space and virtual space), position error, and corrected position error. However, these embodiments are not limited to this, but are equally applicable to rotation (in real world space and virtual space), rotation error, and corrected rotation error. These embodiments are also applicable to posture, where posture is a combination of position and rotation. (For example, in a posture) there may be position error and / or rotation error, and position error correction and / or rotation error correction may be performed.

[0055] Figure 1An example of a virtual production system 100 according to an embodiment of the present disclosure is illustrated. As shown, the virtual production system 100 includes a display component 110, a camera device 120, motion capture devices 130A, 130B, 130C (generally referred to by the reference numeral "130"), and a computer system 140, etc. The display component 110 can be configured to present content 112 and define a virtual stage of a volume (multiple such camera devices 120 are also possible) in which the camera device 120 is located. The presentation of the content 112 can be controlled by a game engine (e.g., UNITY game engine, Unreal (UNREAL) game engine, etc.) executed on the computer system 140. For example, the content 112 itself and / or the parameters of the presentation (e.g., translation, angulation, skew, etc.) can be controlled based on a variety of factors. These factors include the posture of the camera device 120 in the volume. The motion capture device 130 can generate motion capture data that is processed to determine not only the pose of objects 150A, 150B, 150C (generally referred to by the reference numeral "150" and can be actors, stage furniture, set furniture, facilities, etc.), but also the pose of the camera device 120 over time. Such motion capture data can be processed and used to control the presentation of content 112 on the display component 110.

[0056] In an example, the display component 110 includes a plurality of displays arranged to form a content presentation screen. Figure 2An example of such an arrangement is further described in . The position and orientation of each display may be configurable, so that a display or a group of displays in the display assembly may be moved from a first posture to a second posture. The content presentation screen may have different shapes (e.g., curved, flat intersecting surfaces, etc.), and may surround the area where the object 150 is located, thereby defining a volume. In some cases, one or more displays are moved to change the content presentation screen from having a first shape to having a second shape. The content presentation screen may be used to present an interactive, dynamic scene. The object 150 may interact with such a scene, and the content 112 may be updated based on the interaction (e.g., the actor 150A may interact with the virtual object presented in the content 112). Although the display assembly 110 is illustrated as having a vertical position (e.g., set as a wall), the display assembly 110 (or the second display assembly) may be additionally or alternatively positioned in other positions (e.g., a horizontal position for defining a ceiling or floor). Further, the volume may include a set of sliding displays that may be moved in and out of the volume to define a specific geometric shape. For example, the volume may be shaped as a horseshoe, and the sliding group may be positioned to close an open portion of the horseshoe. In this way, the camera device 120 in the volume may be surrounded by a full 360 degree display.Other shapes are also possible, whereby the volume may be, for example, a semicircle, a spherical body (ie 4π steradians), a cylinder, a cube.

[0057] The camera device 120 may be a movie camera mounted on a support (e.g., a floor and / or ceiling support) that can be repositioned in the volume, or a movable support (e.g., a tripod, a pan / tilt, etc.). In this manner, the camera device 120 may be configured to capture a scene (particularly from more than one angle) by generating video data (and optionally audio data) showing one or more objects 150 and / or part or all of the content 112 presented on the display assembly 110. The camera device 120 may have a high resolution (e.g., 4K, 6K, 8K, 12K, etc.), and may be obtained from, for example, BLACKMAGIC (e.g., URSA MINIPRO 12K, STUDIO CAMERA 4K PLUS, STUDIO CAMERA 4K PRO, URSA BROADCAST G2, etc.), ARRI (e.g., ALEXA MINI LF, ALEXALF, ALEXA MINI, ALEXA SXT W, AMIRA, AMIRA LIVE, ARRI MULTICAM SYSTEM, etc., equipped with ARRI SIGNATURE PRIME 35mm T1.8 lens, ARRI SIGNATURE PRIME 75mm T1.8 lens, etc.), etc.

[0058] The motion capture device 130 can be a motion capture camera (e.g., an infrared camera) and / or other types of motion sensors (e.g., a depth sensor) that are part of a motion capture system configured to track motion in a volume. The motion capture system can be obtained from, for example, VICON (e.g., using VANTAGE, VERO, VUE, VIPER, VIPERX cameras, etc. and SHOGUN software, etc.) or OPTITRACK (e.g., using PRIME-X 41, PRIME, SLIM-X, SLIM, FLEX cameras, etc. and UNREAL PLUGIN, UNITY PLUGIN, MOTION BUILDER PLUGIN, OPTICAL MOTION CAPTURE software, MAYA PLUGIN software, etc.). Optionally, the motion of the object can be tracked using a motion tracker attached to the object. Tracking can involve locating an object in the volume by determining the position and rotation of the object over time. The coordinate system of the motion capture system (e.g., a Cartesian coordinate system or any other coordinate system) can be defined relative to any origin in the volume.

[0059] The computer system 140 may be configured to process at least a portion of the motion capture data and optionally a portion of the video data. For example, a game engine may render content 112 using a virtual model of the display assembly 110 and positional data of the camera device 120. Rendering may include using a rendering engine and may involve synthesizing images and / or image frames (e.g., two-dimensional or three-dimensional), which are then presented on the display assembly 110 as content 112. In addition to displaying the content 112, one or more displays in the display assembly 110 may be configured to generate virtual light to illuminate one or more production elements. The rendering of the content may be affected by the physical posture of each display. Therefore, in order to make the content have the desired effect, the virtual model may be updated to include the current virtual posture of each display.

[0060] Figure 2 An example of a display assembly 110 and a virtual model 230 thereof according to an embodiment of the present disclosure is illustrated. The display assembly 110 may include an arrangement 210 of multiple displays. Each display 220 may be a panel having a specific shape and may be individually controllable to display content. The display assembly 110 may form a volume for virtual production. The volume may include approximately 2,230 m 2 A virtual production space of FIG. 1 , wherein the display assembly 110 forms a curved LED wall that is approximately 16 m wide, 20 m long, and 270 degree elliptical (with a possible 360 ​​degree configuration).

[0061] In an example, the display 220 is a light emitting diode (LED) panel having a flat screen that displays content. The flat screen may have a square shape and a specific pixel resolution (eg, Figure 2 20 is shown as a grid on the display 220, where each cell in the grid represents a pixel). However, other curvatures and / or shapes of the screen and underlying technologies of the display 220 (e.g., liquid crystal display LCD) are possible. In a specific illustrative use case, the display is a BLACK PEARL 2 display available from ROECREATIVE DISPLAY. In this illustrative use case, the screen is a flat panel having dimensions of 500×500×90 mm (height×width×depth), a resolution of 176×176 (horizontal×vertical), a pixel pitch of 2.84 mm, an LED surface mount diode (SMD) configuration, and a magnesium frame with a magnetic connector and locking system. Although each display 220 has a strong, well-defined geometry, mounting a large number of displays 220 to form the display assembly 110 along multiple degrees of freedom according to a desired shape (e.g., 6 DOF forming a 270 degree curved wall that is 16 m wide by 20 m long) may result in pose deviations that need to be accounted for and corrected in the virtual model of the display assembly.

[0062] Each display 220 may be further coupled to an actuator (e.g., a winch system, a motor, a robotic arm) for moving the display 220 from a first physical pose to a second physical pose. During virtual production, one or more displays may sometimes be moved in order to create a desired effect for the content displayed on the display assembly. For example, the shape of the display assembly may be changed to achieve a desired shot for the virtual production. In other examples, a single display may be moved to create a desired effect for a shot. In each case, moving the display 220 may change the visual parameters of the content displayed on or around the display 220. Thus, in order to display the desired content with the desired effect, the computer system (e.g., Figure 1 The computer system 140) can use a virtual model that includes virtual poses that are accurate representations of the current physical pose of each display.

[0063] Arrangement 210 may include a stack of displays to form a particular geometry of display assembly 110. For example, the displays are mounted adjacent to each other to form a desired curvature, height, and length of display assembly 110. Each display has a physical pose (i.e., an actual, real-world pose) in arrangement 210. The physical pose may be defined for a point on the display (e.g., the top left corner, the center, etc.) relative to an origin in the production volume (e.g., the origin of a coordinate system used by a motion capture system).

[0064] The virtual model 230 may include a three-dimensional object that represents the display assembly 110 as a rigid body in the game engine. The three-dimensional object may also represent each display as a rigid body by including a virtual representation of the display (e.g., as a three-dimensional sub-object). In this way, the virtual model 230 may not only be a virtual representation of the display assembly 110, but also a virtual representation of the displays that form the display assembly. As part of the virtual representation, the virtual model 230 may indicate a virtual shape, virtual size, and virtual pose (e.g., position and rotation) for each display within the three-dimensional object. Figure 2 In the illustration of , the virtual representation is shown as a curved grid simulating the display assembly 110 , where each cell in the grid represents one of the displays.

[0065] The virtual model 230 may be matched to the arrangement 210, wherein the virtual pose of the virtual representation of the display in the virtual model 230 matches the actual physical pose of the display in the arrangement 210. However, due to many factors (e.g., mounting tolerances, improper mounting, operating temperature, heat, thermal expansion, human interaction, natural forces, etc.), there may be a mismatch between the virtual position and the physical position and / or between the virtual rotation and the physical rotation. Such a mismatch may result in alignment errors when rendering content on the display, especially from the perspective of the camera.

[0066] In some cases, the current physical posture of the display is determined after the display stops moving. In these cases, the static physical posture of the display 220 can be determined after the display 220 stops moving. Once the static physical posture is determined, the current position and orientation of the display 220 can be determined using the fitting model. A transformation function can be applied to update the virtual model to include a virtual posture of the display that represents the current static physical posture of the display 220. In other cases, the current physical posture of the display 220 can be determined in real time when the display 220 moves. In these cases, the current physical parameters of the display 220 can be continuously updated and input into the fitting model. The fitting model can continuously output a virtual model of the current virtual posture of the display 220 that includes the mobile. In these cases, once the display 220 stops moving, or after a short time interval (e.g., a few seconds) after the display 220 stops moving, the current physical parameters can stop being used as input.

[0067] Figure 3An example of a motion path 330 of a physical marker 320 on a display assembly 310 (e.g., display assembly 110) according to an embodiment of the present disclosure is illustrated. In particular, the physical marker 320 can be placed over different displays in the display assembly 310 at different times according to the motion path 330. A motion capture system can track the physical marker 320, and the resulting motion capture data can be processed to determine the physical position and / or physical rotation of the display. The motion capture system can be the same as the motion capture system used in the virtual production system 100, such as a system available from VICON or OPTITRACK.

[0068] The arrangement of displays in display assembly 310 may be indexed by rows 350 and columns 340. Motion path 330 is Figure 3 30 is illustrated as starting from the right side of the bottom row of display assembly 310 (e.g., the display with index (C, 3)), moving horizontally to the left to the end of the bottom row, moving up one row, moving horizontally to the right, and so on. Of course, other types of motion paths are also possible (e.g., an "S" shaped path or even a zigzag path starting from the upper left, upper right, or lower left). On each display, a physical marker 320 is placed at a position corresponding to a point used to define the physical posture of the object, such as at the upper left corner point of the display (or any other point, such as the center, lower right corner point, etc.). The physical marker 320 remains at this position for a predefined time period (e.g., one second, two seconds, etc.) and then moves to the next position according to the motion path 330. A time of two seconds is considered acceptable.

[0069] The physical marker 320 can be a rigid body that implements motion tracking technology based on a motion capture system. For example, in the case of an infrared motion capture camera, the physical marker 320 can include one or more infrared emission (active or passive) points (each point using a different infrared frequency). Generally speaking, the greater the number of points, the more accurate the position estimate will be. In the above example of placing the physical marker 320 at the upper left corner point of the display, the upper left infrared red emission point of the marker can be placed above that position of the display and can be used as a reference point (e.g., the root of a rigid body) in the motion capture data to determine the physical position and / or physical rotation of the display. The motion capture system can be the same motion capture system used during virtual production involving the display component 310.

[0070] In the example, the physical marker 320 includes a single point that can be detected by an infrared motion capture camera. In this case, at least three infrared motion capture cameras may be required to detect the position of the physical marker 320. In particular, each of the three cameras will generate a two-dimensional image that shows the marker position in two dimensions. Because the position, orientation, and field of view of each camera are known, the three-dimensional vector at which the physical marker 320 is located can be determined based on the three sets of two-dimensional positions. In another example, the physical marker 320 includes multiple points detected by an infrared motion capture camera. In this case, one infrared motion capture camera may be sufficient to detect the position of the physical marker 320. In particular, the relative positions of these points are known a priori, and this knowledge is used to process the images generated by the camera. Of course, technologies other than infrared can also be used. For example, a two-dimensional visual marker that encodes its size can be used, where an optical sensor operating in the human visible wavelength range can generate an image. The pose of the visual marker can be determined by decoding the size and applying geometric reconstruction to the image.

[0071] At a certain point in time (e.g., before starting production, during virtual production, etc.), an operator (e.g., a person or a machine (such as a robot, an unmanned vehicle, etc.)) may move the physical marker 320 according to the motion path 330. For example, the operator initially places the physical marker 320 (e.g., its upper left infrared emission point) on the upper left corner point of the (C, 3) display for two seconds, then repositions the physical marker 320 (e.g., by aligning its upper left infrared emission point) to the upper left corner point of the (B, 3) display for two seconds, and so on.

[0072] The motion capture data may be processed according to motion path 330 to determine the physical pose of the display. Next, examples of processing are further described herein.

[0073] Figure 4 An example of measured motion capture data 410 that can be used to determine the physical pose of a display in a display assembly (e.g., display assembly 310) according to an embodiment of the present disclosure is illustrated. In particular, a graph 400 is illustrated that shows the change in the x-coordinate (vertical axis) of a physical marker (e.g., physical marker 320) over time (horizontal axis). In other words, Figure 4 The motion capture data 410 shown in corresponds to the movement of the physical marker along the X-axis. The motion capture data 410 can be used to determine the posture of each display along the X-axis.

[0074] Motion capture data for physical markers may be captured along other axes (e.g., Y and Z axes). For clarity of explanation, the x-coordinate is described herein. However, embodiments are equally applicable to other coordinates of the physical marker to determine the position and rotation of the physical marker in three-dimensional space (including X, Y, and Z coordinates and rotations). Embodiments are also equally applicable to non-Cartesian coordinate systems (e.g., if a polar coordinate system is used instead, ray and angular coordinates may be tracked and processed to determine the pose in three-dimensional space).

[0075] In an example, the motion capture data 410 is generated at a specific frame rate (e.g., 144 frames per second (FPS)), and thus a single x-coordinate is available at a specific frame rate (e.g., approximately once every 7 milliseconds (ms)). Further, the predefined motion path may indicate a timing for stationary positioning of a physical marker at a location above the display (e.g., approximately every 2 seconds), which may be related to an index of the display (e.g., back to reference Figure 3 , between approximately 0 and 2 seconds, the index is (C, 3), between approximately 2 and 4 seconds, the index is (B, 3), and so on). Processing of the motion capture data 410 may be performed based on the frame rate and the predefined motion path to determine the physical position of the display.

[0076] In particular, the second x-coordinate is subsequent to the first x-coordinate (or two subsequent sets of x-coordinates may be used to compare values ​​and determine a change in x-position). The value of the second x-coordinate may be compared to the value of the first x-coordinate. If the difference between the two values ​​is less than a predefined threshold, the smaller difference indicates that the x-position remains substantially unchanged. If the difference between the two values ​​is greater than a predefined threshold, the larger difference indicates that the x-position has changed. Different types of comparisons may be performed, such as comparing magnitudes, comparing slope changes, etc.

[0077] This type of comparison-based determination Figure 44. The physical marker is shown in FIG. 4 using reference numeral 420. In particular, between time "t1" and time "t2", a change 420 is determined, wherein the change 420 is greater than a predefined threshold. Therefore, between time "t1" and "t2", the physical marker is repositioned from a first position to a second position. The time window "TW0" between time "t0" and "t1" corresponds to the first position. The time window "TW0" has a time length of "t1-t0", and the time length can typically be about 2 seconds. Between time "t1" and "t2", repositioning of the physical marker occurs, as shown by the change 420. The next time window "TW1" starts approximately at "t2" (where the change 420 is no longer observed), corresponds to the second position, and its time length should be about 2 seconds. The 2-second time window provided herein is for illustrative purposes. Different time lengths may be used (e.g., approximately 1 second, approximately 5 seconds, etc.), and / or different time windows need not have approximately the same time length (e.g., one time window may be approximately 2 seconds, while another time window may be approximately 5 seconds).

[0078] Given the motion path, the first time window "TW0" corresponds to the first display index (C, 3). Similarly, the second time window "TW1" corresponds to the second display index (B, 3). Next, in order to determine the x position of the physical marker during the first time window "TW0", and equivalently determine the x position of the display with the first display index (C, 3), a portion of the motion capture data 410 with timing between time "t0" and "t1" (referred to herein as "first motion capture data" for clarity) is processed. For example, all first motion capture data starting at time "t0" and ending at time "t1", a specific percentage (e.g., 60%) of the first motion capture data, or a subset thereof starting after time "t0" and ending before time "t1" (e.g., 25ms after time "t0" to 30ms before time "t1") are statistically analyzed to determine a statistical value (e.g., an average) of the x position. Similar processing can be applied to the second motion capture data corresponding to the second time window "TW2", and so on. Similar processing can also be applied to determine the x and z positions and x, y and z rotations.

[0079] Figure 5An example of a rendering path 530 of a virtual marker 520 on a display component 510 (e.g., display component 110) according to an embodiment of the present disclosure is illustrated. Unlike using a physical marker 320, a virtual marker 520 is used here to determine the physical posture of a display included in the display component 510. The rendering path 530 can be similar to the motion path 330, whereby the virtual marker 520 is presented on different displays according to a predefined order (e.g., an "S" shaped order starting from the right side of the bottom row, a zigzag order starting from the left side of the top row, etc.). Optionally, the display index 522 of each display is also presented on the display in parallel with the presentation of the virtual marker 520 on the display. In this way, the rendering path 530 does not need to be predefined and can be random, as long as the virtual marker 520 is presented on different displays over time.

[0080] In an example, the virtual marker 520 can be a multi-dimensional model of a rigid body (e.g., a two-dimensional model, a three-dimensional model, etc.). The virtual marker can be presented at a specific position of the display (e.g., the center as shown, but other positions are also possible, such as the upper left corner point). Unlike physically moving between different positions in the case of the physical marker 320, the presentation of the virtual marker 520 can stay on the display at a specific position for a certain time window (e.g., two seconds), then stop on the display, and start at the next display (which can be but does not have to be an adjacent display) at the same time or shortly thereafter. The presentation of the display index can be parallel, and therefore also follows the presentation path 530. The display index 522 is typically displayed at a display position other than the virtual marker 520 (e.g., displayed at the lower right corner point, while the virtual marker 520 is presented at the center).

[0081] In general, the virtual tag 520 does not use infrared technology unless each display is capable of emitting light in the infrared range. Instead, the virtual tag 520 may include one or more virtual points that emit light in the human visible wavelength range, and a camera operating in the wavelength range may be used to capture one or more images of the virtual tag 520 when it is presented. The camera may, but need not be, a motion capture camera. Similar to the physical tag 320, the virtual tag 520 may include at least three points, each with a different color and / or a different shape, or may even be unique to a specific display (e.g., a barcode, a QR code, a unique shape, etc.), so a single camera may be sufficient to generate an image of the virtual tag 520, and the image may be processed to determine the corresponding presentation gesture on the display. Alternatively, the virtual tag 520 may include a single point, and three or more cameras may be used to generate an image of the virtual tag 520, and these images may be processed to determine the corresponding presentation gesture on the display. Alternatively, the virtual tag 520 may be a virtual visual tag that encodes its size, and a single camera may be sufficient to generate an image of the visual tag, and these images may be processed to determine the corresponding presentation gesture on the display. Alternatively, the virtual marker 520 may have an asymmetric shape (e.g., a rectangular prism rather than a square cuboid). The presentation of the virtual marker on the display may change its orientation (e.g., rotation, angle, etc.), and at least one image showing its changes may be captured and processed together with other images showing other changes to determine the presentation posture of the virtual marker 520 on the display. Regardless of which technique is used, the presentation posture of the virtual marker 520 on the display corresponds to the physical posture of the display.

[0082] Alternatively, different virtual markers may be presented on the display simultaneously or non-simultaneously, rather than repositioning the presentation of the virtual marker 520 between displays. The virtual markers may have different shapes, wherein each shape may be associated with a display index. In this way, the processing of the image may include identifying the shape and then associating each virtual marker with the corresponding display. In another example, the virtual markers may have the same shape, and when the virtual markers are presented on the display, the display index is also presented on the display. In this way, the processing of the image may also include identifying the display index. In an illustrative use case, virtual markers (different shapes, or the same shape with a display index) are presented simultaneously on the display component. One or more images are generated and processed to determine the posture of each virtual marker presented on the display, and the posture is associated with the physical posture of the display.

[0083] Once the images of the virtual marker 520 are generated, they can be processed to determine its presentation pose on each display, and thus equivalently determine the physical pose of each display. Different processing techniques are possible. In one example, when the display index is not presented but a predefined presentation path is used instead, a combination of Figure 4 The processing described. In particular, motion capture data is derived from the images by determining the presentation gesture in each image, and the data is processed to determine changes over time and associate the unchanged time window with a specific display index, thereby giving the timing indicated by the predefined presentation path. In another example, the display index is presented. When processing the image to detect the presentation gesture of the virtual marker 520 therein, the display index shown in the image can also be detected (for example, by using optical character recognition and / or object detection machine learning models). Therefore, the presentation gesture can be associated with the display index, and therefore can represent the physical gesture of the corresponding display.

[0084] A combination of techniques for collecting motion capture data (or more generally, image data) using physical markers and virtual markers is possible. In one example, a virtual marker is presented on a display as a placement instruction. The operator can then place the physical marker at the presented location, covering the virtual marker. In another example, the virtual marker may or may not be presented. However, a display index is presented on the display. In this way, in addition to generating motion capture data corresponding to a physical marker placed at a location above the display, image data can also be generated in parallel to capture the display index. The motion capture data can be processed to determine the physical posture of the physical marker, and the image data can be processed to detect the display index. Given a match between the timing of the motion capture data and the timing of the image data, the physical posture is associated with the display index.

[0085] Figure 6 An example of determining a transformation 610 and an updated virtual model 620 of a display component (display component 110) according to an embodiment of the present disclosure is illustrated. In the example, a virtual model 602 (e.g., virtual model 230) represents a display component. The transformation 610 includes a set of functions (e.g., rotation and / or translation, distortion, twist, bend, randomization, etc., along each dimensional axis at each point) that can be applied to the virtual model 602, and the application produces an updated virtual model 620. The updated virtual model 620 corrects (e.g., reduces or even eliminates) the pose error between the virtual model 602 and the display component (in other words, the pose error of the updated virtual model 620 is much smaller than the pose error of the virtual model 602, if any). The following figure illustrates an example of position error correction.

[0086] To generate the transformation 610, the physical pose 604 of the display included in the display assembly is determined and input to the fitting model 630 along with the virtual model 602 (or more specifically, the corresponding virtual position). The output of the fitting model 630 includes the parameters (e.g., coefficients) of the transformation 610. The physical pose 604 can be based on, for example, Figure 3 to Figure 4 Motion capture data as described in Figure 6 The positioning may be derived from image data (which may include motion capture data) and / or other positioning techniques described in.

[0087] Fitting model 630 can be a data fitting model that iteratively estimates the parameters of transformation 610 so that the transformed virtual position best fits the physical position. Different types of data fitting models are possible, such as data fitting models based on Levenberg-Marquardt nonlinear least squares algorithm, chi-square test algorithm, curve fitting algorithm, weighted least squares fitting algorithm, polynomial regression algorithm, Gauss-Newton algorithm, shift-cut algorithm, gradient algorithm, Nelder-Mead (simplex) search algorithm, or other types of fitting algorithm implementations. Additionally or alternatively, a machine learning model (such as a regression model or a convolutional neural network) can be trained using multiple known virtual models and corresponding display components to output transformation parameters. Once trained, virtual model 602 and physical posture 604 can be input to the machine learning model, which then outputs the parameters of transformation 610.

[0088] As further described in the following figure, the pose error may not be constant and may vary depending on the sub-region of the display assembly (for example, the pose error of a display located at the lower left corner point of the display assembly may be very different from the pose error of a display located at the center of the display assembly, and similarly may be very different from the pose error of a display located at the upper right corner of the display assembly). In order to optimize the pose error variation, a local determination technique is used instead of a global determination technique. The global determination technique involves inputting the entire set of virtual poses of the virtual model 602 and the entire set of physical poses 604 into the fitting model 630. In this way, a single transformation is generated and used to correct the pose error when rendering content on the entire display assembly.

[0089] In contrast, local determination techniques involve dividing the display assembly into sub-regions. Each sub-region contains a subset of displays. Each subset generates a transformation and uses it to correct posture errors when rendering a portion of the content, where the portion will be presented on the subset of displays. The first transformation associated with the first display subset may be different from the transformation of the second display subset. Generating the first transformation may include posture data (e.g., virtual position and physical position) of the first display subset and exclude posture data of the second display subset. In particular, to generate the first transformation, a subset of virtual postures and a corresponding subset of physical postures 604 are input to the fitting model 630. These two subsets are associated with the first display subset. Another subset of virtual postures and another subset of physical postures 604 can be input to the fitting model 630 to generate another transformation, and so on. As described above, for example, a subset can be defined in the XY plane by including "x" by "y" displays, where "x" is less than the total number of displays along the horizontal axis ("x 总 ”), and / or “y” is less than the total number of displays along the vertical axis (“y 总 ”). The subset can be selected based on a number of factors. For example, a subset corresponds to displays that are within the field of view of a camera. In another illustration, content is to be rendered on a subset of displays in a particular manner (e.g., with special effects), where the accuracy of the content presentation (e.g., including how visually perceptible the special effects are) depends on the positional errors of these displays. In this case, the subset is the subset used in the local determination technique. In yet another illustration, a coarse estimate of the transform can be used, for example, to reduce computational overhead or processing latency. In this illustration, every other display or some other selection pattern (e.g., a random selection distributed across the display components) can be used to define the subset. In a further illustration, a multi-granularity approach can be used, starting with a coarse, fast calculation of the transform, followed by a more targeted calculation (e.g., selecting displays within the field of view and / or performing a particular special effect presentation).

[0090] In the context of virtual production, different transformations (e.g., each transformation is associated with a sub-region of a display component, such as by generating a transformation to correct the posture error of positioning to the sub-region) can be generated offline and used during virtual production as needed. Alternatively, each transformation is generated in real time based on need. In particular, during virtual production, environmental factors may cause the posture of certain displays to change, and (multiple) transformations can be calculated in real time to correct the resulting posture errors. For example, environmental factors may include an increase in temperature in or around the volume, or an accidental collision of facilities / personnel with the display component. In the illustration, virtual production involves a camera device (e.g., camera device 120), thereby controlling the content presented on the display component and / or the presentation of the content based at least in part on the posture of the camera device. The posture (which can be tracked with a motion capture system) can indicate that the camera is at a certain distance from the display component and is oriented in a specific direction, and the resulting camera field of view includes a sub-region of the display component. In this case, a subset of the display contained in the sub-region can be determined. For example, a sub-region is defined as a projection of the field of view on the display component, wherein the projection is determined based on the orientation of the camera device and the distance from the display component. Given a display index, the display belonging to the projection is identified, and the corresponding virtual and physical poses are retrieved and input to the fitting model 630 to generate the transformation for content rendering in real time.

[0091] Figure 7 An example of position error correction based on transformation according to an embodiment of the present disclosure is illustrated. Figure 6 In particular, Figure 7 A graph 700 is illustrated in which the horizontal axis corresponds to the x-coordinate, the vertical axis corresponds to the y-coordinate, and the points in the graph correspond to the two-dimensional positions of the displays defined by the (x, y) coordinates in the XY plane. Three types of two-dimensional positions ((x, y) coordinates) are shown: circles correspond to virtual positions of the virtual model of the display assembly, cross marks correspond to physical positions of the displays in the display assembly, and triangles correspond to generated corrected virtual positions. Each virtual position corresponds to a physical position and a corrected virtual position, wherein the corrected virtual position is generated by applying a transformation to the virtual position. Each triplet of virtual position, physical position, and corrected virtual position corresponds to a single display. The bottom row of triplets in graph 700 corresponds to the first row of displays at a first height (e.g., Figure 3 ), while the top row of triplets in graph 700 corresponds to another row of displays at a second height (e.g., Figure 3 Instead, each column of triplets corresponds to a column of displays of different heights.

[0092] If there were no position errors at the outset (e.g., the transform was null, the transform was diagonally uniform, etc.), then each virtual position would match the corresponding physical position (e.g., both positions would have the same (x, y) coordinates). However, as explained above, this is not always the case due to different factors. Figure 7 The difference between the virtual position and its corresponding physical position is the position error (e.g., in the XY plane). Figure 7 As shown, position errors are reduced and in some cases eliminated, whereby the corrected virtual positions are more closely matched to the corresponding physical positions (some even overlap). In the example, actual measurements were taken of display components including the Black Pearl 2 display available from ROE CREATIVE DISPLAY. Each such display has a 500×500 front flat panel (height×width). The display components form a 270-degree curved wall that is 16m wide by 20m long. Without using an embodiment of the present disclosure, position errors along the X, Y, and Z axes vary between approximately 150 to 250 mm, 20 to 50 mm, and 175 to 225 mm, respectively. By implementing the global determination techniques of the embodiments, these position errors can be significantly reduced to approximately 30 to 60 mm, 5 to 15 mm, and 40 to 56 mm, respectively.

[0093] Although in Figure 7 It is not explicitly shown in , but the position error may not be consistent on different displays. As explained above, in the case where the position error is not uniformly distributed, a local determination technique can be used to generate multiple transformations corresponding to subsets of the displays respectively to optimize the correction of the position error.

[0094] Figures 8 to 11 Processes are described related to determining pose and correcting pose errors in the context of content rendering that relies on a virtual model of a display component on which the content is to be displayed. The operations of the process can be performed by a computer system (e.g., at least one processor, at least one computer-readable memory, etc.), such as Figure 1 The computer system 140 of the present invention is executed. Some or all of the instructions for performing the operations may be implemented as hardware circuits and / or stored as computer-readable instructions on a non-transitory computer-readable medium of the computer system. As implemented, the instructions represent a component including circuits or codes that can be executed by (multiple) processors of the computer system. The use of such instructions configures the computer system to perform specific operations described herein. Each circuit or code combined with (multiple) related processors represents a device for performing (multiple) corresponding operations. Although these operations are illustrated in a particular order, it should be understood that the particular order is not required, and one or more operations may be omitted, skipped, performed in parallel, and / or reordered.

[0095] Figure 8 An example of a process for determining and using position error correction according to an embodiment of the present disclosure is illustrated. The process may begin at operation 802, where a computer system collects motion capture data. For example, the motion capture data is generated by a motion capture system and sent to the computer system. The motion capture data may track the pose of a physical marker on a display component over time, where the pose changes according to a predefined motion path. Additionally or alternatively, the motion capture data may track the pose of a virtual marker presented on the display component. As shown in conjunction with Figure 5 As described above, in addition to or in lieu of motion capture data, image data may be generated when virtual markers are used, and the image data may similarly be collected by a computer system.

[0096] At operation 804, the computer system processes the motion capture data to determine the physical pose of the display. The processing may depend on the collection technique. In an example using a physical marker or a virtual marker, the change in the motion capture data over time is used to determine a time window, and a predefined motion path is used to associate the time window with a display index. The motion capture data with timing within the time window is used to determine the physical pose of the display with the corresponding display index (e.g., as a statistical measure applied to the data, such as an average). In another example using a virtual marker, the image data is processed to determine the pose of the virtual marker, and the timing of the image data is used to determine the corresponding display index according to the rendering path and / or the display index is also directly rendered and identified from the image.

[0097] At operation 806, the computer system accesses a virtual model representing the display component. For example, the virtual model is loaded from a memory of the computer system or retrieved from a remote data storage device.

[0098] At operation 808, the computer system generates a transformation. In one example, the virtual pose and the physical pose of the virtual model are input to a fitting model, which then outputs parameters of the transformation, wherein the transformation is associated with the display component. In another example, a subset of the virtual poses and a corresponding subset of the physical poses are input to the fitting model, which then outputs a transformation, wherein the transformation is associated with a sub-region of the display component.

[0099] At operation 810, the computer system renders the content by correcting the pose error of the virtual model based on the transformation. For example, an updated virtual model is generated by applying the transformation to the virtual model (or to a portion thereof corresponding to a subset of the virtual position). The updated virtual model is used by a game engine executed on the computer system to render the content, wherein the rendered content is then displayed by a display component.

[0100] Fig. 9An example of a process for rendering content based on posture error correction according to an embodiment of the present disclosure is illustrated. The operation of the process may be implemented as follows: Figure 8 In the example, Fig. 9 The process of the embodiment of the present invention may begin at operation 902, where the computer system determines a first posture (e.g., a first position and / or a first rotation) of a first display among a plurality of displays included in a display assembly. The display assembly is configured to display content on the plurality of displays. The first posture may be a physical posture of the display, and the physical posture may be based on processing such as Figures 3 to 5 The motion capture data and / or image data are used to determine the physical pose. The physical pose can be defined by using a point on the display (eg, the upper left corner point) relative to, for example, the origin of a coordinate system of the motion capture system.

[0101] At operation 904, the computer system determines a virtual model of the display assembly. The virtual model includes a virtual representation of each of the multiple displays. For example, the virtual representation of the display includes a multi-dimensional object that represents the display and its arrangement relative to other displays and indicates a virtual pose of the multi-dimensional object. The virtual pose can also be defined by using a point of the multi-dimensional object relative to the origin of the coordinate system (e.g., the upper left corner point).

[0102] At operation 906, the computer system determines a transformation based on the first posture of the first display and the first virtual representation of the first display in the virtual model. For example, the first virtual posture indicated by the first virtual representation and the first posture are input to the fitting model. The input to the model may include virtual postures and physical postures associated with other displays, depending on whether a global determination technique or a local determination technique is used. The fitting model can then output parameters of a function (e.g., rotation, translation) that defines the transformation.

[0103] At operation 908, the computer system renders content on at least some displays in the display assembly based on the transformation and the virtual model. For example, according to the parameters of the transformation, each, some or all virtual points of the virtual model are translated from the virtual model and / or a virtual object formed by multiple virtual points of the virtual model is rotated to generate an updated virtual model. The game engine can use the virtual model together with other data (such as the pose of the camera device) to render the content.

[0104] Fig.10 An example of a process for processing motion capture data according to an embodiment of the present disclosure is illustrated. The process corresponds to a motion capture data processing technique that relies on a predefined motion path along which physical markers are repositioned between different locations. The operations of the process may be implemented as follows: Figure 8 In the example, Fig.10The process of can start at operation 1002, at which the computer system receives motion capture data. The motion capture data can be generated by a motion capture system at a specific frame rate (e.g., 144fps).

[0105] At operation 1004, the computer system determines a change in the motion capture data. The motion capture data can be multi-dimensional. A change in each dimension can be determined. For example, the motion capture data includes x-coordinates, where each x-coordinate is generated at a specific frame rate (e.g., approximately every 7 ms). Therefore, the values ​​of the two x-coordinates (or the average of the two x-coordinate ranges) can be compared to determine a difference, and the difference is compared to a predefined distance threshold. If the difference is greater than the threshold difference, a change is detected, and the computer system determines that the change corresponds to a transition of the physical marker from one position above one display to another position above another display.

[0106] At operation 1006, the computer system determines a timing associated with the change. For example, the timing may be obtained from a timestamp of the motion capture data and may be matched to the end or beginning of a time window (e.g., a 2 second time window) during which the physical marker is expected to be substantially stationary at a location above the display. Each time window may be indicated by a predefined motion path.

[0107] At operation 1008, the computer system determines that the first motion capture data corresponds to the first display. For example, a change in the motion capture data is detected. Two consecutive changes correspond to the beginning and the end of the time window. The timing of the change is related to the display index of each predefined motion path. Thus, the portion of the motion capture data between the beginning and the end is the first motion capture data. And the first motion capture data can be associated with the display index.

[0108] At operation 1010, the computer system determines a pose of the first display based on the first motion capture data. For example, a statistical measure (e.g., averaging) is applied to the first motion capture data to determine the position. In some cases, the statistical measure may be applied to a subset of the first motion capture data (e.g., a percentage thereof, or a portion that begins a few milliseconds after the start of the time window and ends a few milliseconds before the end of the time window) to calculate the pose.

[0109] Fig.11 An example of a process for determining a transformation based on a field of view of a camera device according to an embodiment of the present disclosure is illustrated. The process corresponds to a local determination technique. The operations of the process may be implemented as follows: Figure 8 In the example, Fig.11The process may begin at operation 1102, where a computer system determines a subset of displays within a field of view of a camera device. In an example, content presented on a display component and / or the presentation of content may change based on a pose of the camera. A motion capture system may output motion capture data tracking the camera device, and the computer system may process the data to determine a pose. Given a pose, a projection of the camera's field of view onto the display component may be used to determine a sub-region of the display component within the field of view. Displays belonging to the sub-region are identified, and their display indexes are included in the subset.

[0110] At operation 1104, the computer system determines the physical posture of the displays, where the displays are in the field of view. Figures 3 to 5 The motion capture data of the described physical tracker and / or virtual tracker and / or the image data of the virtual tracker are used to determine the pose. In particular, the display indexes of these displays are used to find and retrieve the corresponding physical position data.

[0111] At operation 1106, the computer system determines a virtual pose of a virtual representation of the display in the virtual model. For example, the virtual representation (e.g., a multi-dimensional object representing the display) is also indexed with the same display index. Thus, the display index is used to find and retrieve corresponding virtual position data.

[0112] At operation 1108, the computer system generates a transformation. For example, the virtual pose and the physical pose are input to a fitting model, which then outputs parameters of the transformation.

[0113] Fig.12 An example of a virtual production system 1200 according to an embodiment of the present disclosure is illustrated. As shown, the virtual production system 1200 includes a display component 1210, a camera device 1220, motion capture devices 1230A and 1230B (generally referred to by the reference numeral "1230"), and a computer system 1240. The display component 1210 may include a peripheral wall 1250, a rear wall 1260, and a top 1270. The display component 1210 may be configured to present content and define a virtual stage of a volume where the camera device 1220 is located.

[0114] The top 1270 may include a display 1280 that can be moved from a first physical posture to a second physical posture. As shown, the display 1280 is arranged in a first physical posture, that is, the display is flush with the surface of the top 1270. The display 1280 may be coupled to an actuator that can move the display 1280 from the first physical posture to the second physical posture. The actuator 1290 may include, for example, a winch system, a motor, a lever, a pulley system, or a mechanical arm. In some cases, for example, the display 1280 can be moved from a first physical posture to a second physical posture to create a desired effect for the display 1280. This can be performed by moving the display 1280 using the actuator 1290. Additionally or alternatively, one or more other displays that may be located in the top 1270 and / or any other wall 1250 to 1260 may be moved, resulting in a change in posture. The posture change of any display may be updated in a virtual model (e.g., virtual model 230).

[0115] Fig.13 An example of a virtual production system 1300 according to an embodiment of the present disclosure is illustrated. As shown, the virtual production system 1300 includes a display component 1310, a camera device 1320, motion capture devices 1330A and 1330B (generally referred to by reference numeral "1330"), and a computer system 1340.

[0116] Fig.13 Can be different from Fig.12 , because display 1350 (corresponding to display 1280) is arranged in the second physical posture. Fig.12 In FIG. 1 , the display 1280 is arranged in a first physical posture, that is, the display 1280 is flush with the surface of the top 1270 . Fig.13 , display 1350 has been moved to a second physical posture, i.e., display 1350 is rotated away from top 1270. Computer system 1340 can still display content on display 1350. However, to generate the desired visual effect, the virtual model can be updated to include a virtual posture of display 1350 that represents the current physical posture of display 1350.

[0117] As indicated above, the process for updating the virtual model can be manually started or started based on the response to the sensor measurement. For manual initialization, the computer system 1340 can be operably communicated with the switch 1370. The switch 1370 may include one or more keys on the keyboard, a command prompt, a physical switch outside the computer system 1340, or other appropriate switches 1370. The user can use the switch to start the initialization display posture estimation process. The virtual mark 1380 can be displayed on the display 1350. The virtual mark 1380 may include a pattern, such as a synthetic mark (e.g., CharUCo or ArUco mark). As shown, the virtual mark 1380 spans the entire surface of the display 1350. However, in some cases, the virtual mark 1380 can be displayed on a portion of the display surface. The virtual mark can have a presentation gesture corresponding to the physical gesture of the display 1350.

[0118] By using switch 1370, the motion capture system including motion capture devices 1330A and 1330B can begin to capture motion data using virtual markers. In some cases, motion capture devices 1330A and 1330B can be configured with the coordinates of display 1350 to focus on the display. In other cases, motion capture devices 1330A and 1330B can capture the entire display assembly 1310. Motion capture devices 1330A and 1330B can continuously capture motion data frames as display 1350 moves. Motion capture devices 1330A and 1330B can further transmit the motion data to computer system 1340.

[0119] The computer system 1340 can use the final frame (e.g., a frame captured after the display 1350 stops moving) to determine the posture data of the display 1350. The posture data can include the physical coordinates of the display 1350. The computer system 1340 can generate input based on the posture data and the virtual model to feed into the fitting model. The fitting model can use a transformation function to convert the current physical posture of the display 1350 into the current virtual posture of the display 1350. The computer system 1340 can further update the virtual model to include the current virtual posture of the display 1350. The computer system 1340 can use the updated model to display the content on the display component 1310.

[0120] As an alternative to using switch 1370 to initiate the pose estimation process, a sensor-based response trigger can be used. Virtual production system 1300 may include one or more sensors 1390 configured to collect data (speed, acceleration, six-degree-of-freedom coordinates) from display assembly 1310. Sensor 1390 may include, for example, a proximity sensor, a light sensor, a pressure sensor, an infrared sensor, an ultrasonic sensor, or any other suitable sensor. Sensor 1390 may be in operative communication with computer system 1340 and provide the collected data to computer system 1340.

[0121] The sensor-based data may be presented in the form of a time series, and the computer system 1340 may use a change point detection algorithm to detect change points in the sensor data. Change point detection is the process of detecting changes in properties represented by the time series (e.g., physical coordinates of a display). The computer system 1340 may use a change point detection algorithm (e.g., a forgetting factor-based change point detection algorithm, window-based segmentation, binary segmentation, bottom-up segmentation, trimmed exact linear time, and exact segmentation dynamic programming) to identify boundaries between changes in the time series. Fig.14 A graph of the first detection change point and the second detection change point is provided in FIG.

[0122] In some cases, the first detected change point may indicate that the display 1350 begins to move, and the second detected change point may indicate that the display 1350 ends to move. Thus, the time interval between the first detected change point and the second detected change point may indicate the time interval that the display is in motion.

[0123] Based on detecting the first change point, a virtual marker may be displayed on the moving display 1350. Computer system 1340 may further initialize motion capture devices 1330A and 1330B. Initializing motion capture devices 1330A and 1330B may include determining the position and orientation of each motion capture device and configuring the device for capturing the motion of the display (e.g., configuring the frame rate). Motion capture devices 1330A and 1330B may capture frames of the display as the display 1350 moves. Motion capture devices 1330A and 1330B may continue to capture these frames until computer system 1340 detects a second change point indicating that the display 1350 has stopped moving. Computer system 1340 may further send a signal to motion capture devices 1330A and 1330B to stop collecting motion data of display 1350. In some cases, motion capture devices 1330A and 1330B continue to capture one or two frames after the display 1350 has stopped moving.

[0124] The computer system 1340 can use the collected sensor data to determine the physical posture (e.g., physical coordinates) of the display 1350 after it has stopped moving. The computer system 1340 can analyze the characteristics of the virtual marker to determine the presentation posture of the marker (e.g., the coordinates of the virtual marker corresponding to the physical coordinates of the display). These characteristics can include, for example, checkerboard corners, center points, and other image features. The computer system can further use an algorithm to determine the physical posture of the display 1350. For example, the computer system 1340 can use an n-point perspective (PnP) posture calculation algorithm to evaluate the physical posture of the display in the desired coordinate system (e.g., the coordinate system of the camera of the motion capture system). The computer system can further generate input for fitting the model based on the virtual model (e.g., the virtual model generated before the display 1350 moves) and the physical posture. The fitting model can output an updated virtual model, which includes an updated virtual posture of the display 1350, which represents the physical posture of the display after it has stopped moving.

[0125] It should be understood that in some embodiments, the computer system 1340 may begin determining the physical pose of the display 1350 as it moves, rather than waiting for the display 1350 to stop moving. For example, in response to detecting the first change point, a virtual marker may be displayed on the display 1350, and the computer system may initialize the motion capture devices 1330A and 1330B to begin capturing motion data using the virtual marker. The computer system 1340 may continuously generate inputs for fitting the model using a virtual model (e.g., a virtual model generated before the display 1350 moves) and the current physical pose of the display 1350 as it moves. The fitting model may use these inputs to transform the virtual pose of the display 1350 before it moves to the virtual pose of the display 1350 as it moves. For example, if the frame rate of the motion capture devices 1330A and 1330B is ten frames per second, and the time interval between the displays moving is ten seconds, the computer system 1340 may generate one hundred updated virtual poses for one hundred iterations of the updated virtual model.

[0126] Fig.14 1400 is a graph of change point data of virtual production according to an embodiment of the present disclosure. Graph 1400 includes time on the x-axis and a value on the y-axis. The value may be based on sensor-based data. It should be understood that the collected sensor-based data may require preprocessing. For example, sensor-based data may include abnormal values, missing values, and damaged values. The computer system may preprocess the data before performing analysis on the data. The value on the y-axis may represent the sensor-based data after cleaning.

[0127] As shown, the computer system has used a change point detection algorithm to detect a first change point 1410 at time 50 and a second change point 1420 at time 150. Data from time "0" to time "50" and data from time "150" and later can represent the time when the display in the display assembly has not moved. Data from time "50" to time "150" can represent the time when the display moves. The computer system can use a change point detection algorithm to detect the first change point 1410. Based on the detection, a virtual marker can be displayed on the moving display. The computer system can further initialize a motion capture system to capture motion data of the display using a virtual marker. In response to detecting the second change point 1420, the computer system can send a signal to the motion capture system to stop collecting motion data. The computer system can then determine the physical posture of the display based on the motion data. The computer system can further use the virtual model generated before the display moves and the physical posture of the display to generate input for fitting the model. The fitting model can use the input to transform the virtual posture of the display into an updated virtual posture to reflect the physical posture of the display after the movement.

[0128] Fig.15 1 is a diagram of an example display assembly 1500 according to an embodiment of the present disclosure. Display assembly 1500 includes multiple displays. As shown, the display assembly includes nine displays, which are identified by a grid system for convenience. As shown, display 1510 located at position (A, 1) has moved from a first physical posture (e.g., a first posture indicated by a dotted line) to a second physical posture protruding from the surface of each other display in display assembly 1500. While in the first physical posture, the surface of display 1510 is flush with the surface of each other display on display assembly 1500, this is not the case in the second physical posture. For example, an actuator can be used to move display 1510 to protrude from the surface of each other display in display assembly 1500.

[0129] A virtual marker 1520 is displayed on a portion of or the entire display 1510. In the case where the virtual marker 1520 is displayed on a portion of the display, the rest of the display 1510 may display a portion of the scene. For example, the rest of the display 1510 may display a portion of the scene being displayed before displaying the virtual marker 1520. The portion may be predefined at a certain position (e.g., the center portion of the display 1510 or any other portion (e.g., the upper left corner portion)) and have a predefined size (e.g., a certain number of width and height pixels). The virtual marker may be displayed based on detecting that the display 1510 is moving. In an example, the virtual marker 1520 may represent a multidimensional model of a rigid body (e.g., a two-dimensional model, a three-dimensional model, etc.). The virtual marker may be presented at a specific position of the display (e.g., the center as shown, but other positions are also possible, such as the upper left corner) according to a specific size. In general, the virtual marker 1520 does not use infrared technology unless each display is capable of emitting light in the infrared range. Instead, the virtual marker 1520 may include one or more virtual dots that emit light within a human visible wavelength range, and a camera operating within the wavelength range may be used to capture one or more images of the virtual marker 1520 when it is presented. The camera may, but need not be, a motion capture camera. The virtual marker 1520 may include at least three dots, each of which has a different color and / or a different shape, or may even be unique to a particular display (e.g., a barcode, a QR code, a unique shape, etc.), so a single camera may be sufficient to generate an image of the virtual marker 1520, and the image may be processed to determine the corresponding physical gesture of the display. Alternatively, the virtual marker 1520 may include a single dot, and three or more cameras may be used to generate an image of the virtual marker 1520, and the images may be processed to determine the corresponding physical gesture of the display. Alternatively, the virtual marker 1520 may be a virtual visual marker that encodes its size, and a single camera may be sufficient to generate an image of the visual marker, and the images may be processed to determine the corresponding physical gesture of the display. Alternatively, the virtual marker 1520 may have an asymmetric shape (e.g., a rectangular prism instead of a square cuboid). Of course, a combination of various techniques is also possible. For example, the virtual marker 1520 may include at least three points, and multiple cameras may be used to capture its image, so that the gesture determination may have higher accuracy. In another example, the virtual marker 1520 may be updated, and one or more cameras may be used to generate one or more images corresponding to each update. In particular, the virtual marker 1520 may initially be presented as including at least three points, and then updated to present a single point that multiple cameras may image.

[0130] The motion capture system may further use virtual marker 1520 to identify display 1510 and capture motion data representing the movement of display 1510. It should be understood that although Fig.15 The movement of display 1510 is illustrated, but other (multiple) displays in display assembly 1500 may also be configured to move. In addition, a group of displays may be configured to move. For example, the display positions at positions (B, 1), (B, 2), (C, 1), and (C, 2) may be configured to move. In this case, a virtual marker may be displayed on one or more than one display. If multiple displays move, the virtual markers may be the same or different. Using different virtual markers may facilitate tracking the position changes of each display and enable the use of the same camera system. Nevertheless, even when the same virtual marker is presented on different mobile displays, tracking may be performed using the same camera system. In this case, the initial posture of each display is known, and tracking may be performed relative to the initial posture, so each final posture may be associated with the initial posture, which in turn is associated with the display.

[0131] Fig.16 1600 is an illustration of an example of determining a transformation 1610 and an updated virtual model 1620 of a display assembly according to an embodiment of the present disclosure. In the example, the virtual model 1630 represents the display assembly before the display in the display assembly is moved. The transformation 1610 includes a set of functions (e.g., rotation and / or translation of each point along each dimensional axis, distortion, twist, bend, randomization, etc.) that can be applied to the virtual model 1630, and the application produces an updated virtual model 1620. The updated virtual model 1620 can represent the physical pose of each display in the display assembly.

[0132] To generate the transformation 1610, a physical pose 1640 of a display included in the display assembly is determined and input to a fitting model 1650 together with a virtual model 1620 (or more specifically, a corresponding virtual position). As shown, the physical pose 1640 includes the physical pose of the display 1660 before the display 1660 has moved and the physical pose of the display after the display 1660 has moved. In some cases, the physical pose only includes the physical pose of the display 1660 after it has moved. The output of the fitting model 1650 includes parameters (e.g., coefficients) of the transformation 1610. The physical pose 1640 can be derived based on motion capture data, image data (which can include motion capture data), and / or other positioning techniques.

[0133] Fitting model 1650 can be a data fitting model that iteratively estimates the parameters of transformation 1610 so that the transformed virtual position best fits the physical position. Different types of data fitting models are possible, such as data fitting models based on Levenberg-Marquardt nonlinear least squares algorithm, chi-square test algorithm, curve fitting algorithm, weighted least squares fitting algorithm, polynomial regression algorithm, Gauss-Newton algorithm, shift-cut algorithm, gradient algorithm, Nelder-Mead (simplex) search algorithm, or other types of fitting algorithm implementations. Additionally or alternatively, a machine learning model (such as a regression model or a convolutional neural network) can be trained using multiple known virtual models and corresponding display components to output transformation parameters. Once trained, virtual model 1630 and physical posture 1640 can be input to the machine learning model, which then outputs the parameters of transformation 1610.

[0134] Fig.17 1700 is a table representing the difference between the measured parameters and the calculated parameters. Table 1700 includes a test data column comprising the value obtained by actual measurement and a calculated value (CV) data column comprising the value obtained using the technology described herein. As shown in the figure, the test data value of the distance between the two postures of the display is 2395.117210 millimeters (mm), and the CV of the distance between the two postures of the display is 2396.601829mm. The difference between these two values ​​is 1.484619mm. In addition, the test data value of the relative rotation between the two postures of the display is -0.482724 °, -41.687118 ° and 1.1130060 °. The calculated value of the relative rotation between the two postures of the display is 0.578781 °, -41.718646 ° and -0.956472 °.

[0135] Figures 18 to 20 A process is described that is related to updating a virtual model in the context of content rendering that relies on a virtual model of a display component on which content is to be displayed. The operations of the process can be performed by a computer system (e.g., at least one processor, at least one computer-readable memory, etc.), such as Figure 1The computer system 140 of the present invention is executed. Some or all of the instructions for performing the operations may be implemented as hardware circuits and / or stored as computer-readable instructions on a non-transitory computer-readable medium of the computer system. As implemented, the instructions represent a component including circuits or codes that can be executed by (multiple) processors of the computer system. The use of such instructions configures the computer system to perform specific operations described herein. Each circuit or code combined with (multiple) related processors represents a device for performing (multiple) corresponding operations. Although these operations are illustrated in a particular order, it should be understood that the particular order is not required, and one or more operations may be omitted, skipped, performed in parallel, and / or reordered.

[0136] Fig.18 An example of a process flow 1800 for determining an updated virtual model based on the movement of a display according to an embodiment of the present disclosure is illustrated. At operation 1802, a computer system may collect motion capture data of a display moving from a first physical posture to a second physical posture. For example, the motion capture data may be generated by a motion capture system associated with a virtual production and sent to a computer system. The motion capture data may track the posture of a virtual marker on a display that has moved from a first physical posture to a second physical posture. The computer system may use one or more characteristics of the virtual marker to determine the presentation posture of the virtual marker. The presentation posture may correspond to the physical posture of the display. For example, a production staff may use an actuator to move the display from a first posture to a second posture to generate a desired effect for content rendered on a display component.

[0137] At operation 1804, the computer system can process the motion capture data to determine the physical pose of the display that has been moved. The processing can depend on the collection technology. In the example of using virtual markers, the motion capture data can be used to determine the physical coordinates of the display in the coordinate system of the camera of the motion capture system. For example, the computer system can use a PnP pose algorithm to determine the physical pose of the display that has been moved (the physical coordinates in the coordinate system of the motion capture system).

[0138] At operation 1806, the computer system may generate a transformation of the second physical pose of the display to the virtual pose of the display. In one example, the virtual pose of the virtual model and the physical pose of the moved display are used to generate inputs to a fitting model, which then outputs parameters of the transformation, wherein the transformation is associated with the display component.

[0139] At operation 1808, the computer system may update the virtual model, which includes the virtual gesture. For example, the computer system may update the virtual model determined before the display is moved to include the virtual gesture of the display. The updated virtual model may include a virtual gesture corresponding to the second physical gesture of the display that has been moved.

[0140] At operation 1810, the computer system may use the updated virtual model to render content of the display component. The updated virtual model may be used by a game engine executing on the computer system to render content, wherein the rendered content is then displayed by the display component.

[0141] Fig.19 An example of a process flow 1900 for determining and initializing a motion capture system for virtual production according to an embodiment of the present disclosure is illustrated. In an example, the process 1900 may begin at operation 1902, where a computer system receives input for initializing a motion capture system. The input may be a manual input based on a user or an input based on a sensor. The manual input based on a user may be received based on a user operating a switch that generates a control instruction for initializing the motion capture system.

[0142] Alternatively, the input may be a sensor-based input. The virtual production may include one or more sensors that collect streaming data from the display assembly. The streaming data may be received by a computer system that may analyze the streaming data to determine whether there is an indication that one or more displays in the display assembly have moved. The indication may be determined based on detecting a change point in the streaming data. For example, the computer system may use a forgetting factor-based change point detection algorithm to detect a first change point and a second change point in the streaming data. The first change point may indicate an indication that the display has begun to move. The second change point may indicate an indication that the display has stopped moving.

[0143] At operation 1904, the computer system may initialize the motion capture system, which may include determining the position and orientation of each motion capture device of the system (e.g., based on a coordinate system of the motion capture system) and configuring the device for capturing display motion (e.g., configuring a frame rate).

[0144] In some embodiments, the motion capture system captures frames including the display after the display has moved. For example, if the input is a sensor-based input, the motion capture system may capture frames after the second point. In other embodiments, the motion capture system may capture images of the display from when the display begins to move to when the display stops moving.

[0145] Fig. 20An example process 2000 for determining a transformation based on detecting movement of a display in a display assembly according to an embodiment of the present disclosure is illustrated. In an example, the process can begin at operation 2002, where a computer system determines that a display in a display assembly has changed from having a first physical posture to having a second physical posture. For example, the display can be moved by an actuator based on the expectations of a production crew of a virtual production. The computer system can determine the change to the second posture based on processing motion data received from a motion capture system.

[0146] At operation 2004, the computer system may determine a second physical pose of the display that has been moved (e.g., physical coordinates in the coordinate system of the motion capture system). The second physical pose may be determined based on the motion data of the virtual tracker and / or the image data of the virtual tracker. In particular, the display indexes of these displays are used to find and retrieve the corresponding physical position data.

[0147] At operation 2006 , the computer system may determine a virtual gesture corresponding to a second physical gesture of the display.

[0148] At operation 2008, the computer system may generate a transformation. For example, the virtual gesture corresponding to the second physical gesture and the virtual model determined before the display is moved are input to a fitting model, which then outputs parameters of the transformation. The output parameters may be further used to update the virtual model.

[0149] Fig.21 2100 according to an embodiment of the present disclosure. Figure 1 21. Although components of computer system 2100 are illustrated as belonging to the same computer system 2100, computer system 2100 may also be distributed (eg, among multiple user devices).

[0150] The computer system 2100 includes at least a processor 2102, a memory 2104, a storage device 2106, an input / output peripheral (I / O) 2108, a communication peripheral 2110, and an interface bus 2112. The interface bus 2112 is configured to communicate, transport, and deliver data, control, and commands between the various components of the computer system 2100. The memory 2104 and the storage device 2106 include computer-readable storage media, such as RAM; ROM; electrically erasable programmable read-only memory (EEPROM); a hard disk drive; a CD-ROM; an optical storage device; a magnetic storage device; an electronic non-volatile computer storage device, for example, Memory; and other tangible storage media. Any such computer-readable storage media may be configured to store instructions or program codes embodying aspects of the present disclosure. Memory 2104 and storage device 2106 also include computer-readable signal media. Computer-readable signal media include propagated data signals in which computer-readable program code is embodied. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media include any computer-readable media that is not a computer-readable storage medium and that can transmit, propagate, or convey a program used in conjunction with computer system 2100.

[0151] Further, the memory 2104 includes an operating system, programs, and applications. The processor 2102 is configured to execute stored instructions and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. The memory 2104 and / or the processor 2102 can be virtualized and can be hosted in another computer system such as a cloud network or a data center. The I / O peripherals 2108 include a user interface, such as a keyboard; a screen (e.g., a touch screen); a microphone; a speaker; other input / output devices; and a computing component, such as a graphics processing unit; a serial port; a parallel port; a universal serial bus; and other input / output peripherals. The I / O peripherals 2108 are connected to the processor 2102 via any port coupled to the interface bus 2112. The communication peripherals 2110 are configured to facilitate communication between the computer system 2100 and other systems through a communication network, and include, for example, a network interface controller, a modem, wireless and wired interface cards, antennas, and other communication peripherals.

[0152] It is obvious to those skilled in the art that, without departing from the inventive concept of this article, more modifications are possible in addition to those modifications that have been described. Therefore, the subject matter of the present invention is not limited except for the spirit of the appended claims. In addition, when interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprise" and "comprising" should be interpreted as referring to elements, parts or steps in a non-exclusive manner, indicating that the referenced elements, parts or steps may exist or be utilized, or be combined with other elements, parts or steps that are not explicitly referenced. When the specification or claim mentions at least one selected from the group consisting of A, B, C ... to N, the text should be interpreted as requiring only one element in the group, rather than A plus N, or B plus N, etc.

Claims

1. A computer-implemented method comprising: receiving, by at least one processor, motion capture data of a display among the plurality of displays, the display moving from a first physical pose to a second physical pose; processing, by at least one processor, the motion capture data to determine coordinates of the second physical gesture; generating, by the at least one processor, a transformation of a second physical pose of the display to a virtual pose of the display; updating, by the at least one processor, a virtual model of the plurality of displays, the virtual model including virtual poses of the displays; as well as Content is rendered on the display based on the updated virtual model by the at least one processor.

2. The computer-implemented method of claim 1, wherein: The computer-implemented method further comprises: receiving input for initializing a motion capture system for generating the motion capture data, the motion capture system comprising a motion capture device; and The motion capture system is initialized based on the input, wherein initializing the motion capture system includes determining a position and orientation of the motion capture device.

3. The computer-implemented method of claim 2, wherein: The input is based on user input, and wherein the computer-implemented method further comprises: projecting a virtual marker on the display based on receiving the user-based input; determining a presentation gesture of the virtual marker, the presentation gesture corresponding to the second physical gesture; and Coordinates of the second physical gesture are determined based on the presentation gesture.

4. The computer-implemented method of claim 2, wherein: The input is a sensor-based input, and wherein the computer-implemented method further comprises: receiving streaming data from a sensor configured to collect data associated with the display; Detecting a first change point and a second change point from the streaming media data; Based on detecting the second change point, determining that the display is in the second physical posture; and Based on the determination, the motion capture data is processed to determine coordinates of the second physical gesture.

5. The computer-implemented method of claim 4, wherein: The first change point and the second change point are detected using a change point detection algorithm based on a forgetting factor.

6. The computer-implemented method of claim 4, wherein: Determining coordinates of the second physical pose includes using an n-point perspective (PnP) pose calculation algorithm.

7. The computer-implemented method of claim 4, wherein: Determining coordinates of the second physical gesture includes determining coordinates in a coordinate system of a motion capture device used to capture the motion capture data.

8. The computer-implemented method of claim 4, wherein: The computer-implemented method further comprises: Based on detecting the first change point, starting to receive the motion capture data; When the display is moved: Continue to receive the motion capture data; continuously processing the motion capture data to determine current coordinates of the display; generating a current transform using the display's current coordinates; and A virtual model of the plurality of displays is updated to include a current virtual pose of the display, the current virtual pose being associated with the current coordinates of the display.

9. A system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, configure the system to: receiving motion capture data of a display among a plurality of displays, the display moving from a first physical pose to a second physical pose; processing the motion capture data to determine coordinates of the second physical gesture; updating a transformation from a second physical posture of the display to a virtual posture of the display; updating a virtual model of the plurality of displays, the virtual model including a virtual pose of the display; as well as Content is rendered on the display based on the updated virtual model.

10. The system of claim 9, wherein: The instructions, when executed by the one or more processors, further configure the system to perform the following operations: receiving input for initializing a motion capture system for capturing the motion capture data, the motion capture system comprising a motion capture device; and The motion capture system is initialized based on the input, wherein initializing the motion capture system includes determining a position and orientation of the motion capture device.

11. The system of claim 10, wherein: The input is based on user input, and wherein the instructions, when executed by the one or more processors, further configure the system to: projecting a virtual marker on the display based on receiving the user-based input; determining a presentation gesture of the virtual marker, the presentation gesture corresponding to the second physical gesture; and Coordinates of the second physical gesture are determined based on the presentation gesture.

12. The system of claim 11, wherein: The motion capture system includes at least one motion capture device, and wherein the virtual marker includes a plurality of virtual points projected on the display, the at least one motion capture device being configured to capture motion capture data based on tracking the plurality of virtual points.

13. The system of claim 11, wherein: The motion capture system includes a plurality of motion capture devices, and wherein the virtual marker includes at least one virtual point, the plurality of motion capture devices being configured to capture motion capture data based on tracking the at least one virtual point.

14. The system of claim 11, wherein: The virtual marker covers a portion of the display, and wherein a remainder of the display projects a portion of a scene.

15. The system of claim 11, wherein: The virtual marker covers the entire display.

16. The system of claim 12, wherein: The instructions, when executed by the one or more processors, further configure the system to perform the following operations: Based on detecting the first change point, starting to receive the motion capture data; When the display is moved: Continue to receive the motion capture data; continuously processing the motion capture data to determine current coordinates of the display; Generates the current transformation using the display's current coordinates; as well as A current virtual model of the plurality of displays is generated that includes a current virtual pose of the display, the current virtual pose being associated with the current coordinates of the display.

17. One or more non-transitory computer-readable storage media storing instructions that, when executed on a system, cause the system to perform operations comprising: receiving motion capture data of a display among a plurality of displays, the display moving from a first physical pose to a second physical pose; processing the motion capture data to determine coordinates of the second physical gesture; generating a transformation from a second physical posture of the display to a virtual posture of the display; updating a virtual model of the plurality of displays, the virtual model including a virtual pose of the display; as well as Content is rendered on the display based on the updated virtual model.

18. One or more non-transitory computer-readable storage media as claimed in claim 17, wherein: The instructions, when executed by the one or more processors, further configure the system to perform operations comprising: receiving input for initializing a motion capture system for capturing the motion capture data, the motion capture system comprising a motion capture device; and The motion capture system is initialized based on the input, wherein initializing the motion capture system includes determining a position and orientation of the motion capture device.

19. The one or more non-transitory computer-readable storage media of claim 18, wherein: The input is based on user input, and wherein the instructions, when executed by the one or more processors, further configure the system to perform operations comprising: projecting a virtual marker on the display based on receiving the user-based input; determining a presentation gesture of the virtual marker, the presentation gesture corresponding to the second physical gesture; and Coordinates of the second physical gesture are determined based on the presentation gesture.

20. The one or more non-transitory computer-readable storage media of claim 18, wherein: The input is based on user input, and wherein the instructions, when executed by the one or more processors, further configure the system to perform operations comprising: receiving streaming data from a sensor configured to collect data associated with the display; Detecting a first change point and a second change point from the streaming media data; Based on detecting the second change point, determining that the display is in the second physical posture; and Based on the determination, the motion capture data is processed to determine coordinates of the second physical gesture.

21. A computer-implemented method comprising: determining, by at least one processor, a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays; determining, by the at least one processor, a virtual model of the display assembly, wherein the virtual model is stored in a computer-readable memory and includes a virtual representation of each of the plurality of displays; determining, by the at least one processor, a transformation based on a first pose of the first display and a first virtual representation of the first display in the virtual model; and The content is rendered, by the at least one processor, on at least some of the plurality of displays based on the transformation and the virtual model.

22. The computer-implemented method of claim 21, wherein: Determining the first posture first includes: First motion capture data of a physical marker placed at a first location above the first display is received by at least one processor, wherein the first gesture is determined based on the first motion capture data.

23. The computer-implemented method of claim 22, wherein: Determining the first posture further includes: receiving, by at least one processor, second motion capture data of a physical marker placed at a second location above a second display according to a predefined motion path of the physical marker on the display assembly; determining, by at least one processor, a change between the first motion capture data and the second motion capture data; determining, by at least one processor, based on the change, that the first motion capture data corresponds to the first position; determining, by at least one processor, a position and a rotation of the physical marker at the first location based on the first motion capture data; and A first gesture of the first display is generated, by at least one processor, by including the position and the rotation in the first gesture.

24. The computer-implemented method of claim 23, wherein: Determining the first posture further includes: determining, by at least one processor, a timing of the first motion capture data; and The timing is determined, by at least one processor, to correspond to the first position based on the predefined motion path.

25. The computer-implemented method of claim 21, wherein: Determining the first posture includes: displaying, by at least one processor, a virtual marker on the display; and Motion capture data of the virtual marker is received by at least one processor, wherein the first pose is determined based on the motion capture data.

26. The computer-implemented method of claim 25, wherein: Determining the first posture further includes: displaying, by at least one processor, a display identifier on the display; and A determination is made, by at least one processor, that the motion capture data corresponds to the display based on the display identifier.

27. The computer-implemented method of claim 21, further comprising: A second pose of a second display of the plurality of displays is determined, by at least one processor, wherein the transformation is determined further based on the second pose and a second virtual representation of the second display in the virtual model.

28. The computer-implemented method of claim 21, further comprising: Gestures corresponding to respective ones of the plurality of displays are determined by at least one processor, wherein the transformation is further determined based on the gestures and virtual representations of the plurality of displays in the virtual model.

29. The computer-implemented method of claim 28, wherein: The virtual representation indicates a virtual gesture corresponding to a display of the plurality of displays, respectively, wherein the transformation is determined at least by fitting the gestures of the plurality of displays to the virtual gesture indicated by the virtual model.

30. The computer-implemented method of claim 21, wherein: The plurality of displays includes a first subset of a first display within a field of view of the camera device, wherein the content is further rendered based on the field of view, and wherein the computer-implemented method further comprises: determining, by at least one processor, a first gesture corresponding to each of the first displays; and A first virtual representation corresponding to each of the first displays is determined from the virtual model by at least one processor, wherein the transformation is further determined based on the first gesture and the first virtual representation.

31. The computer-implemented method of claim 30, wherein: The plurality of displays includes a second subset of second displays that are outside of the field of view of the camera device, and wherein the transformation is determined independently of a second pose of the second display and a second virtual representation of the second display in the virtual model.

32. The computer-implemented method of claim 21, further comprising: At least one processor generates an updated virtual model by correcting a pose error of the virtual model using the transformation, wherein the content is rendered based on the updated virtual model.

33. A system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, configure the system to: determining a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays; determining a virtual model of the display assembly, wherein the virtual model is stored in the one or more memories and includes a virtual representation of each display in the plurality of displays; determining a transformation based on a first pose of the first display and a first virtual representation of the first display in the virtual model; and The content is rendered on at least some of the plurality of displays according to the transformation and the virtual model.

34. The system of claim 33, wherein: Execution of the instructions further configures the system to perform the following operations: Receive motion capture data from a motion capture system configured to track movement of a physical marker according to a predefined motion path on the display component, wherein the motion capture data includes first motion capture data corresponding to the physical marker being placed at a first position above the first display, and wherein the first posture is determined based on the first motion capture data.

35. The system of claim 34, wherein: The motion capture data includes second motion capture data corresponding to the physical marker being placed at a second position above a second display in the display assembly, wherein execution of the instructions further configures the system to perform the following operations: determining a change between the first motion capture data and the second motion capture data; determining, based on the change, that the first motion capture data corresponds to the first position; determining a position and a rotation of the physical marker at the first location based on the first motion capture data; and A first gesture of the first display is generated by including the position and the rotation in the first gesture.

36. The system of claim 35, wherein: Execution of the instructions further configures the system to perform the following operations: determining a timing of the first motion capture data; and The timing is determined to correspond to the first position based on the predefined motion path.

37. One or more non-transitory computer-readable storage media storing instructions that, when executed on a system, cause the system to perform operations comprising: determining a first posture of a first display among a plurality of displays included in a display assembly, wherein the display assembly is configured to display content on the plurality of displays; determining a virtual model of the display assembly, wherein the virtual model is stored in a computer-readable memory and includes a virtual representation of each of the plurality of displays; determining a transformation based on a first pose of the first display and a first virtual representation of the first display in the virtual model; and The content is rendered on at least some of the plurality of displays according to the transformation and the virtual model.

38. One or more non-transitory computer-readable storage media as recited in claim 37, wherein: The operations further include: Gestures corresponding to respective ones of the plurality of displays are determined, wherein the transformation is further determined based on the gestures and virtual representations of the plurality of displays in the virtual model.

39. The one or more non-transitory computer-readable storage media of claim 37, wherein: The plurality of displays includes a first subset of a first display within a field of view of the camera device, wherein the content is further rendered based on the field of view, and wherein the operations further include: determining a first gesture corresponding to each of the first displays; and A first virtual representation corresponding to each of the first displays is determined from the virtual model, wherein the transformation is further determined based on the first pose and the first virtual representation.

40. The one or more non-transitory computer-readable storage media of claim 37, wherein: The operations further include: An updated virtual model is generated by correcting a posture error of the virtual model using the transformation, wherein the content is rendered based on the updated virtual model.

Images