Camera pose conversion method and system of camera robot system

Abstract

The application discloses a camera pose conversion method and system of a camera robot system, and the method comprises the following steps: presetting parameters of a pose converter; receiving pose data of a camera in the camera robot; the pose data comprises position information and attitude information; converting the pose data of the camera into pose data conforming to a FREE-D protocol through the pose converter; and sending the pose data conforming to the FREE-D protocol to a virtual production platform. Through the processing scheme, accurate and high-frequency pose data conversion can be realized, and the virtual camera movement in the virtual environment is coherent and smooth, and lens pose jump errors do not occur.

Description

Technical Field

[0001] This invention relates to the field of pose conversion technology, and in particular to a camera pose conversion method and system for a photography robot system. Background Technology

[0002] Automation is a trend of the times. The main function of a camera robot is to achieve automated control of the camera's position and orientation.

[0003] Virtual production is a crucial technique in film and television production, primarily encompassing three categories: green screen virtual, LED virtual, and reality + AR (Augmented Reality). The main principle of virtual production is to synthesize diverse scene worlds from real-world footage using media such as green screens or large LED screens. Combined with various auxiliary special effects, the final film effect is output. The film can be streamed live or pre-recorded and streamed centrally. In virtual production, the commonly used technology platform is a real-time virtual live streaming system, with popular software including Unreal Engine and Unity. The system's timecode conforms to the SMPTE standard, and the camera pose format conforms to the Free-D protocol.

[0004] There are three main forms of integration between existing virtual production systems and photography automation systems.

[0005] The first combined system consists of a semi-automatic camera robot and an optical tracking system. The tracking markers of the optical tracking system are divided into a rigid cursor body and a light marker sticker.

[0006] The camera is mounted at the end of a semi-automatic jib arm. For the optical markers, which are rigid bodies, they are mounted at specific positions on the camera rig. The optical tracking camera captures the pose of the optical markers, thereby obtaining the camera's real-time position and orientation. For the optical marker stickers, multiple stickers need to be installed within the positioning camera's field of view according to a distribution requirement. The camera's pose is calculated by observing changes in the optical marker stickers.

[0007] This system can achieve real-time tracking of camera position translation and attitude rotation. However, in this mode, the camera pose control is mainly performed by the photographer. It cannot perform complex trajectory reproduction or high-precision, high-speed trajectory control.

[0008] The second type of integrated system is an all-in-one camera compatible with the FREE-D protocol, which is essentially a fully automatic pan-tilt camera.

[0009] Disadvantages: The camera and lens models are fixed. While they can provide stable 2-axis posture motion, and some models can provide stable 3-axis posture motion, they cannot provide real-time stable 3-axis position translation. These models are more suitable for studio virtual shooting. They cannot provide shooting trajectories with richer motion, thus reducing visual impact and limiting the freedom of artistic expression. Furthermore, the system's built-in embedded system is only compatible with pose conversion for specific manufacturer-specific camera models.

[0010] The third type is a robotic arm-based photography robot, typically composed of six or more motors connected in series. The camera is located at the end of the robot and can perform translational motion in three directions and rotational motion in three directions in Cartesian space.

[0011] Compared to semi-automatic photography robots and integrated cameras, the main feature of arm-type photography robots is the high-speed, high-precision trajectory movement and trajectory reproduction of complex camera trajectories, enabling them to complete various film and television special effects shooting tasks.

[0012] For high-precision, high-speed robotic arm-type photography robot systems, there are two existing pose transmission schemes.

[0013] The first method involves directly mounting the positioning element of the optical tracking system at a designated position on the end-effector camera of the photography robot. While this method can directly obtain the camera's pose, due to its optical principles, the camera's movement space must be within the optical capture range. Therefore, this approach significantly limits the camera's range of motion, reducing the robot's shooting flexibility and artistic expression. Furthermore, interference between the optical elements and the robotic arm during movement must be avoided.

[0014] The second method involves the camera robot directly transmitting its pose data to a host computer via Ethernet, where the program receives the data directly. Mainstream virtual production platforms used on the host computer include, but are not limited to, Unreal Engine and Unity. These programs run on top of a computer operating system and contain pose transformation functions. The camera robot's program sends pose data to the host computer at a certain frequency based on the robot's internal timer. The pose data format is determined by the camera robot and is typically translation and Euler angles, not conforming to the Free-D protocol. The host computer program uses the built-in Euler angle transformation algorithm in its function library to convert the camera robot's pose into the virtual camera pose of the virtual production platform. This is then sent to the virtual production platform (e.g., a real-time virtual live streaming system, or virtual production software like Unity and Unreal Engine, hereinafter referred to as the virtual production platform) via Ethernet.

[0015] If multi-screen rendering is not required, the camera robot can also directly send data to the virtual production platform. Both of these solutions require camera pose data format conversion to be implemented on the host computer.

[0016] While the aforementioned camera pose transformation method can achieve pose transformation to a certain extent, several shortcomings have been found in practical use, preventing it from achieving optimal performance. These shortcomings can be summarized as follows:

[0017] 1. The mainstream technology platform for virtual production uses the Free-D protocol, common in the film and television industry, to describe camera pose data. Camera robot pose data typically consists of six parameters, following the manufacturer's specifications; this format does not conform to the Free-D protocol. Currently, camera robots do not provide corresponding format conversion algorithms.

[0018] 2. Currently, there is camera pose conversion software based on general-purpose computers. This software runs on the operating system of a general-purpose computer. It can convert the camera pose on the robotic arm into the Free-D protocol format on the software. Then, the virtual camera pose offset is calculated in the virtual production software. The main disadvantages of this approach are that it cannot guarantee the real-time performance of data processing and cannot directly receive the SMPTE standard time synchronization signals commonly used on film and television shooting sets. Specifically:

[0019] a) Windows is a non-real-time operating system. When used as the host computer system, the functions for receiving and converting the pose data of the real camera cannot guarantee stable response and execution within a limited time. The protocol conversion may begin immediately upon receiving data, or it may take 100ms before it starts. This is a inherent limitation of non-real-time systems. Under this architecture, the virtual camera is prone to exhibiting disjointed movement, frame skipping, or delayed pose updates in virtual editing software.

[0020] b) Non-real-time versions of Linux systems can run virtual reality software (Unity and Unreal Engine). However, there are currently no successful cases of running mainstream virtual reality software on a real-time version of Linux and successfully adding real-time control. Additionally, this type of system platform is expensive and bulky.

[0021] c) Mainstream virtual production technology platforms have unified clock requirements, conforming to the SMPTE standard with frequencies such as 50Hz and 60Hz. Based on a unified clock frequency and timecode, time consistency is achieved for images, camera poses, and camera shooting parameters. The motherboard hardware interfaces of the two operating systems mentioned above cannot directly receive the SMPTE standard clock synchronization signals used in current film and television shooting environments.

[0022] 3. Existing pose data conversion algorithms directly convert the real camera pose data transmitted from the camera robot system into pose data conforming to the Free-D protocol, and then transmit it to the virtual production platform. On the virtual production platform, offset poses are superimposed, so that the virtual camera is in a virtual position and posture specified by the user.

[0023] However, in this approach, the offset pose is determined within a virtual production platform. This method is prone to pose data overflow errors that conform to the Free-D protocol in various situations.

[0024] Therefore, it is evident that the existing pose conversion methods described above still have inconveniences and shortcomings in use, and urgently need further improvement. How to create a new camera pose conversion method for a camera robot system has become a pressing goal for the industry. Summary of the Invention

[0025] In view of this, the present disclosure provides a camera pose conversion method for a photography robot system, which at least partially solves the problems existing in the prior art.

[0026] In a first aspect, embodiments of this disclosure provide a camera pose conversion method for a photography robot system, the method comprising the following steps:

[0027] Preset parameters for the pose converter;

[0028] Receive pose data from the camera in the photography robot; the pose data includes position information and posture information;

[0029] The pose data of the camera is converted into pose data conforming to the FREE-D protocol using a pose converter;

[0030] The pose data conforming to the FREE-D protocol is sent to the virtual production platform.

[0031] According to a specific implementation of an embodiment of this disclosure, the parameters of the preset pose converter include:

[0032] Configure the network communication parameters of the pose converter and associated devices, the clock source for the pose data sampling period, and the pre-bias algorithm.

[0033] According to a specific implementation of an embodiment of this disclosure, setting the pre-bias algorithm includes the following steps:

[0034] Let the pose data of the camera in the photography robot be v = [v x v y v z v a v b vc ] T ; where v x v y v z For camera position; v a v b v c Let T be the Euler angles of the camera pose, and T be the transpose.

[0035] The pose data v is converted into a standard homogeneous matrix M, i.e., M = f(v);

[0036] Let the pre-bias data of the camera be u = [u x u y u z u a u b u c ] T ; where u x u y u z The adjustment amount for the camera position; u a u b u c The Euler angles corresponding to the camera attitude adjustment;

[0037] The pre-biased data u is converted into a standard homogeneous matrix N, i.e., N = g(u);

[0038] The pre-biased data is superimposed on the camera pose data to obtain T. c That is, T c =N·M;

[0039] Let the pose of the virtual camera conforming to the Free-D protocol be w = [w x w y w z w a w b w c ] T Among them, w x w y w z For the virtual camera position; w a w b w c Euler angles for the virtual camera pose;

[0040] w = h(T) c )

[0041] The h function is used to convert the standard homogeneous pose matrix into position information and attitude Euler angles that satisfy the coordinate system definition of the virtual production platform and conform to the FREE-D protocol. The standard homogeneous matrix is ​​a 4×4 matrix, with the upper left 3×3 submatrix representing the attitude and the upper right 3×1 submatrix representing the position. According to a specific implementation of this disclosure, the conversion of the pose data into pose data conforming to the FREE-D protocol via the pose converter includes the following steps:

[0042] The camera pose data is converted into a standard homogeneous matrix;

[0043] The position and attitude information of the camera are fed into the pre-bias algorithm in the preset pose converter to generate a virtual camera pose.

[0044] Based on the coordinate system definition of the virtual production platform, the virtual camera pose is converted into a left coordinate system or / or a right coordinate system expression;

[0045] The virtual camera pose expressed in the left or / right coordinate system is converted into position information and attitude Euler angles conforming to the FREE-D protocol.

[0046] According to a specific implementation of this disclosure, receiving the pose data of the camera in the camera robot includes: sending a pose data request to the camera robot; wherein sending the pose data request to the camera robot includes sending the pose data request to the camera robot at a preset fixed frequency; or sending the pose data request to the camera robot at an external synchronization signal frequency; and

[0047] The camera robot continuously transmits camera pose data to the pose converter.

[0048] According to a specific implementation of an embodiment of this disclosure, when a pose data request is sent to the camera robot at a preset fixed frequency, a pose request is sent to the camera robot system at a preset frequency.

[0049] When a pose data request is sent to the camera robot at the frequency of the external synchronization signal, the external synchronization signal is monitored, and a pose request is sent to the camera robot system whenever a frame synchronization signal is detected.

[0050] According to one specific implementation of the present disclosure, the method is used to perform pose transformation on a photography robot system that satisfies translational motion in three directions and rotational motion in three directions in Cartesian space.

[0051] Secondly, embodiments of this disclosure provide a camera pose conversion system for a photography robot system, the system comprising:

[0052] The preset module is configured to preset the parameters of the pose converter;

[0053] The data receiving module is configured to receive pose data from the camera in the photography robot; the pose data includes position information and attitude information.

[0054] The data processing module is configured to convert the camera's pose data into pose data conforming to the FREE-D protocol via a pose converter;

[0055] The data transmission module is configured to send the pose data conforming to the FREE-D protocol to the virtual production platform.

[0056] Thirdly, embodiments of this disclosure also provide a camera robot system, characterized in that the system includes a camera pose conversion system as described in the second aspect of the camera robot system.

[0057] Fourthly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0058] At least one processor; and,

[0059] A memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, cause the at least one processor to perform a camera pose conversion method for the photographic robot system described in any of the first aspects or any implementations thereof.

[0061] Fifthly, embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions that, when executed by at least one processor, cause the at least one processor to perform the camera pose conversion method of the camera robot system in the first aspect or any implementation thereof.

[0062] In a sixth aspect, embodiments of this disclosure also provide a computer program product, the computer program product including a computing program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute the camera pose conversion method of the camera robot system in the first aspect or any implementation thereof.

[0063] The camera pose conversion method of the photography robot system in this embodiment, based on an embedded pose conversion device, can achieve real-time, accurate, and high-frequency camera pose data conversion, enabling the virtual camera in the virtual environment to move synchronously with the real camera on the photography robot. The motion is smooth and seamless, without any stuttering. Through a dedicated synchronization signal interface chip, it can be synchronized with external synchronization signals of the SMPTE protocol, ensuring that the camera pose of the photography robot and the video frames have the same time frequency and correspond one-to-one, guaranteeing the consistency of the acquisition time of multimodal data of all devices in the camera pose conversion system of the photography robot system. An overflow-pre-biasing algorithm is proposed, which pre-biases the pose of the real camera at the end of the photography robot to the initial position of the virtual camera in the virtual production platform, while avoiding the overflow of pose Euler angle data caused by fixed-point data format. This algorithm ensures that the camera in the virtual environment can continuously map the pose of the real camera regardless of its initial pose, without lens pose jump errors. Attached Figure Description

[0064] The above is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Figure 1 A schematic diagram of a virtual camera provided in an embodiment of this disclosure;

[0066] Figure 2 This is a schematic flowchart of a camera pose conversion method for a photography robot system provided in an embodiment of the present disclosure;

[0067] Figure 3 A flowchart illustrating a camera pose conversion method for a photography robot system provided in this embodiment of the present disclosure;

[0068] Figure 4 This is a schematic diagram of the data communication of a camera pose conversion system in a photography robot system provided in an embodiment of the present disclosure;

[0069] Figure 5 This is a schematic diagram of a pose converter structure provided in an embodiment of the present disclosure;

[0070] Figure 6 A schematic diagram of a camera pose conversion system for a photography robot system provided in this disclosure embodiment; and

[0071] Figure 7 A schematic diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0072] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0073] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0074] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0075] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0076] A virtual camera is a camera within a virtual development platform (such as a real-time virtual live streaming system, Unreal Engine, and Unity), primarily used to capture images of a virtual environment (such as...). Figure 1 (As shown).

[0077] Current technologies lack pose pre-biasing functionality. Specifically, pose pre-biasing needs to include features to prevent premature overflow of fixed-point pose data. In a camera robot, the reference frame for the camera's pose is the camera robot's base coordinate system. In the virtual world, the reference coordinate system for the virtual camera is the virtual world coordinate system.

[0078] The types of Euler angles used for real-world camera pose and virtual camera pose are generally different. Specifically, any rotation matrix can be parameterized using Euler angles. Generally, sequential rotations around the three coordinate axes can achieve any pose. Since multiplication of rotation matrices is not commutative, the order of rotations is important. There are 12 different Euler angles, including 6 asymmetric Euler angles: XYZ, XZY, YXZ, YZX, ZXY, and ZYX; and 6 symmetric Euler angles: XYX, XZX, YXY, YZY, ZXZ, and ZYZ.

[0079] In typical applications, the position and orientation of a real camera need to be adjusted to the initial pose of the virtual camera in the virtual world through translation and rotation, based on the pose data of the real camera. In the Free-D protocol, the fixed-point numbers representing the position have a large data range, and position offsets generally do not cause fixed-point overflow. The orientation is specified by three Euler angles, each represented by a fixed-point number. The fixed-point Euler angles corresponding to the real camera's orientation can completely represent the camera's orientation.

[0080] However, to move the camera to a specified initial pose in the virtual environment, it's necessary to overlay pose adjustments onto the real camera's fixed-point format pose data. When the Euler angles exceed the upper and lower limits that the fixed-point format can represent, an overflow error occurs, causing a sudden change in camera pose. For example, in the FREE-D protocol, the camera pitch angle is defined as expressed in 24-bit signed two's complement form, with an effective range from -90° (0xD30000) to +90° (0x2D0000). If the initial pose of the real camera robot's base is required to be +85° upwards from the ground—a requirement determined by artistic needs—the camera can move and shoot freely within the camera robot's reach. The zero-position pose of the camera robot's end effector is determined by the camera robot's base coordinate system; therefore, when the real camera is at a +85° angle to the ground, the transmitted pitch angle is 0°.

[0081] On the other hand, when the real camera is in a zero-position attitude, the initial attitude of the virtual camera should be upward at a +85° angle to the ground. If, at this point, 85° is directly added to the pitch angle transmitted by the camera robot as the pitch angle in the FREE-D protocol, the real camera lens can only be raised by a maximum of 5°, at which point the pitch angle in the FREE-D format has reached its limit of 90°. If the camera lens angle is raised by another 1°, the pitch angle in the FREE-D format becomes 91°, exceeding the protocol's limit. As the real camera lens continues to rise, the camera receiving the FREE-D format information may experience attitude jump errors at any time.

[0082] This invention focuses on a robotic arm-type camera robot system for controlling the position and posture of a camera, and provides a method for camera pose conversion in a camera robot system. By using pose conversion technology between the camera robot and a virtual production platform, the purpose of camera pose conversion in the camera robot system can be achieved.

[0083] Figure 2 This is a schematic diagram of the camera pose conversion method flow of the photography robot system provided in the embodiments of this disclosure.

[0084] Figure 3 To and Figure 2 The flowchart of the camera pose conversion method for the corresponding photography robot system.

[0085] First, the network communication interface of the pose converter is initialized.

[0086] like Figure 2 As shown, proceed to step S210.

[0087] In step S210, the parameters of the pose converter are preset.

[0088] The system parameter settings on the host computer configure the network communication parameters of the pose converter and associated devices, the clock source of the pose data sampling period, and the parameters of pre-displacement and attitude offset (or pre-offset pose).

[0089] In this embodiment of the invention, the parameters of the preset pose converter include: setting the network communication parameters of the pose converter and the associated device, the pose data sampling period clock source, and the pre-bias algorithm.

[0090] More specifically, the host computer sets system parameters via Ethernet, according to the requirements of the external shooting system, including network communication parameters of the pose converter itself and associated devices, pose data sampling period clock source, pre-displacement and attitude offset parameters (the position offsets x, y, z of the virtual camera correspond to the displacement vectors in the three principal axes of the virtual world coordinate system, and the attitude offsets a, b, c of the virtual camera correspond to the rotation vectors in the three principal axes of the virtual world coordinate system).

[0091] In this embodiment of the invention, after the parameters of the pose converter are set, as follows: Figure 4 The system parameter setting host computer shown can be removed from the system for parameter settings.

[0092] In this embodiment of the invention, the pre-bias algorithm within the preset pose converter is configured to have an overflow prevention function, including the following steps:

[0093] Let the pose data of the camera in the photography robot be v = [v x v y vz v a v b v c ] T ; where v x v y v z For camera position; v a v b v c Let T be the Euler angles of the camera pose, and T be the transpose.

[0094] When the pose of the camera in the photography robot is received, the pose data v is converted into a standard homogeneous matrix M, i.e., M = f(v);

[0095] Let the pre-bias data of the camera be u = [u x u y u z u a u b u c ] T ; where u x u y u z The adjustment amount for the camera position; u a u b u c Here, T represents the Euler angle corresponding to the camera pose adjustment, and T is the transpose.

[0096] When the pre-biased data is received, the pre-biased data u is converted into a standard homogeneous matrix N, i.e., N = g(u);

[0097] The pre-biased data is superimposed on the camera pose data to obtain T. c That is, T c =N·M;

[0098] Let the pose of the virtual camera conforming to the Free-D protocol be w = [w x w y w z w a w b w c ] T Among them, w x w y w z For the virtual camera position; w a w b w c Euler angles for the virtual camera pose;

[0099] w = h(T) c )

[0100] The h function is used to convert the standard homogeneous pose matrix into position information and attitude Euler angles that satisfy the coordinate system definition of the virtual production platform and conform to the FREE-D protocol. The standard homogeneous matrix is ​​a 4×4 matrix, with the upper left 3×3 submatrix representing the attitude and the upper right 3×1 submatrix representing the position.

[0101] In this algorithm, the secondary adjustment of the virtual camera pose is not directly superimposed on the Euler angles of the real camera pose transmitted by the camera robot. This invention simultaneously converts the pose in the camera robot format and the secondary pose adjustment values ​​into a standard 4×4 homogeneous matrix. Based on this, a preset pose homogeneous matrix is ​​used as the multiplicand, and the pose matrix transmitted by the camera robot is used as the multiplier. The preset pose homogeneous matrix is ​​multiplied by the pose matrix transmitted by the camera robot to obtain the final virtual pose homogeneous matrix. Finally, the inverse kinematics operation h is used to obtain the pose data conforming to the FREE-D format. This algorithm ensures that the final transmitted FREE-D data will not experience pose jump errors. The algorithm is recreated within the pose converter embedded system.

[0102] In this embodiment of the invention, receiving the pose data of the camera in the photography robot includes: sending a pose data request to the photography robot; wherein sending a pose data request to the photography robot includes sending a pose data request to the photography robot at a preset fixed frequency; or sending a pose data request to the photography robot at an external synchronization signal frequency; and the photography robot continuously transmitting the pose data of the camera to the pose converter.

[0103] The pose converter sends pose data requests to the camera robot at a preset frequency, or sends pose data requests to the camera robot based on the camera / system synchronization clock source.

[0104] More specifically, when a pose data request is sent to the camera robot at a preset fixed frequency, a pose request is sent to the camera robot system at a preset frequency; when a pose data request is sent to the camera robot at the frequency of an external synchronization signal, an external video signal is monitored, and a pose request is sent to the camera robot system whenever a frame synchronization signal is detected.

[0105] In this embodiment of the invention, the external synchronization signal includes, but is not limited to, the SMPTE standard signal, or other custom signals.

[0106] To address the two shortcomings of using general-purpose computers as data processing platforms—the inability to guarantee real-time data processing and the inability to decode dedicated film and television synchronization signals—this solution incorporates two synchronization modes: a fixed adjustable frequency mode and an external synchronization signal mode. The fixed adjustable frequency mode primarily relies on the pose converter's internal timer, while the external synchronization signal mode primarily relies on the LM1881 chip to decode external synchronization signals.

[0107] When using a fixed adjustable frequency clock source from the pose converter, the camera robot's internal timer can achieve high-precision, high-frequency transmission, such as 20ms, 10ms, or even shorter intervals, eliminating the need for pose requests in the workflow. The camera robot can periodically send camera pose data to the pose converter based on its internal timer.

[0108] In this embodiment of the invention, the digital processing chip can be any chip with the same function, not limited to LM1881.

[0109] In this embodiment of the invention, the synchronization clock source may be a system-level external clock source or a synchronization signal conforming to SMPTE, which does not constitute a limitation of the invention.

[0110] More specifically, we now proceed to step S220.

[0111] In step S220, the pose data of the camera in the photography robot is received; the pose data includes position information and posture information.

[0112] More specifically, the camera robot receives a request or automatically transmits the current pose of the end camera to the pose converter according to a preset frequency.

[0113] Next, proceed to step S230.

[0114] In step S230, the pose data of the camera is converted into pose data conforming to the FREE-D protocol by a pose converter.

[0115] The pose converter receives pose format data defined by the camera robot and converts it into FREE-D format pose data using a data conversion algorithm.

[0116] More specifically, the pose data of the camera is converted into pose data conforming to the FREE-D protocol using a pose converter, including the following steps:

[0117] The camera pose data is converted into a standard homogeneous matrix;

[0118] The position and attitude information of the camera are fed into the pre-bias algorithm in the preset pose converter to generate a virtual camera pose.

[0119] Based on the coordinate system definition of the virtual production platform, the virtual camera pose is converted into a left coordinate system or / or a right coordinate system expression;

[0120] The virtual camera pose expressed in the left or / right coordinate system is converted into position information and attitude Euler angles conforming to the FREE-D protocol.

[0121] In this embodiment of the invention, the pose conversion algorithm can also be applied to other systems that require real-time hybrid control of the real and virtual cameras. For example, in the camera robot control handle system, embedding the pose converter proposed in this invention can achieve the effect of simultaneously controlling the real camera and the virtual camera to move synchronously through the control handle.

[0122] Next, proceed to step S240.

[0123] In step S240, the pose data conforming to the FREE-D protocol is sent to the virtual production platform to drive the virtual camera to update its pose.

[0124] Next, proceed to step S250.

[0125] At step S250, repeat steps S220-S240 until the preset conditions are met.

[0126] In this embodiment of the invention, the camera pose conversion method for a photography robot system proposed in this invention is used to perform pose conversion on a photography robot system that satisfies translational motion in three directions and rotational motion in three directions in Cartesian space.

[0127] The camera pose conversion method for an embedded system photography robot system proposed in this invention achieves short-cycle, accurate, real-time camera pose data conversion using the Free-D protocol by designing an embedded system that conforms to the SMPTE protocol and meets the film and television-specific synchronization requirements. This method has the following characteristics:

[0128] 1. It can receive SMPTE protocol-compliant film and television dedicated clock synchronization signals to achieve synchronous acquisition of camera footage and camera pose data on the robotic arm, as well as synchronization with mainstream virtual production platforms (such as real-time virtual live streaming systems / virtual production software Unity and Unreal Engine).

[0129] 2. Achieve high-frequency, precise, real-time camera pose data conversion using the Free-D protocol. Time synchronization error is far less than 0.5ms.

[0130] 3. While ensuring real-time performance, reduce the size of the pose converter and lower hardware costs.

[0131] By using a pre-bias algorithm, when there is a significant difference between the initial poses of the virtual and real cameras, the secondary pose adjustment superimposed on the FREE-D format of the real camera pose will not overflow in the Euler angle data expressed in the fixed-point real number data structure in the FREE-D protocol, causing the virtual camera pose to jump incorrectly during synchronous movement.

[0132] The present invention can achieve the following beneficial effects:

[0133] 1. Based on embedded pose conversion devices, precise and high-frequency pose data conversion for photography robots can be achieved. Virtual camera movement in the virtual environment is smooth and fluid, without any lag.

[0134] 2. It can synchronize with external synchronization signals of the SMPTE protocol, ensuring that the camera pose of the photography robot and the video frames have the same time frequency and correspond one-to-one. This guarantees the consistency of the acquisition time of multimodal data of all devices in the camera pose conversion system of the photography robot system.

[0135] 3. A pre-biasing algorithm is proposed, which allows the pose of the real camera at the end of the camera robot to be pre-biased to the initial position of the virtual camera in the virtual production platform, while avoiding the overflow of Euler angle data caused by fixed-point data format. Furthermore, this algorithm ensures that the camera in the virtual environment can consistently map the pose of the real camera regardless of its initial pose, preventing lens pose jump errors.

[0136] like Figure 5 As shown Figure 4 A schematic diagram of a mid-position pose converter.

[0137] The pose converter consists of the following components: a power supply module, a digital processing module, a video synchronization signal processing module, and a communication module.

[0138] The function of the power module is to provide a stable DC power supply to the circuit board.

[0139] The functions of the digital processing module include:

[0140] 1. Pre-bias parameter settings.

[0141] 2. Optionally, the camera robot system can be configured to send pose data requests at a fixed (adjustable) frequency. Alternatively, the camera robot system can be configured to send pose data requests at the frequency of an external synchronization signal. Or, the camera robot can continuously transmit camera pose data to the pose converter, with the external synchronization signal triggering the transmission of converted pose data conforming to the FREE-D protocol to the virtual production platform.

[0142] 3. Receive the current pose data of the photography robot system.

[0143] 4. Convert the pose data defined by the camera robot based on Euler angles into a standard homogeneous matrix.

[0144] 5. Combine the pre-bias pose with the actual camera pose to form a virtual camera pose.

[0145] 6. Based on the coordinate system definition of the virtual production platform, convert the virtual camera pose to left / right.

[0146] Represented by a right coordinate system.

[0147] 7. Convert the virtual camera pose into position information and attitude Euler angles defined by the FREE-D protocol.

[0148] The video synchronization signal processing module functions as follows: it receives external video synchronization signals compliant with SMPTE, extracts video frame synchronization signals, and ensures that the camera's pose data acquisition is synchronized with the video frame synchronization signals. Each time a video frame synchronization signal is captured, the pose converter completes a pose data conversion for the camera on the camera robot and sends pose data compliant with the FREE-D protocol to the virtual production platform.

[0149] The main function of the communication module is to provide transmission support for various data transmissions within the digital processing module. The task of the pose converter is defined as follows:

[0150] a) Pose Request Task

[0151] When the selected clock source is set to a fixed adjustable frequency, the digital processing module generates an interrupt at a fixed frequency using an internal timer (the frequency is adjustable by setting different timer periods) to send a pose request to the camera robot system. Upon receiving the pose request, the camera robot system sends the camera's actual pose to the pose converter. When the selected clock source is set to an external synchronization signal, the video synchronization signal processing module of the pose converter monitors the external synchronization signal. Whenever a frame synchronization signal is detected, it triggers an interrupt in the digital processing module. The pose converter then sends a pose request to the camera robot system. Upon receiving the pose request, the camera robot system sends the camera's actual pose to the pose converter.

[0152] b) Pose transition task

[0153] When the pose converter receives pose data from the end-effector camera of the photography robot, it performs pose pre-biasing and pose conversion to Free-D format. It then transmits the virtual camera pose corresponding to the current video frame to the virtual production platform.

[0154] c) Pose converter parameter setting task

[0155] It communicates with the host computer for system parameter settings to complete the tasks of setting pre-bias parameters and communication parameters.

[0156] d) Main loop task

[0157] Primarily responsible for maintaining communication status.

[0158] Figure 6 The present invention illustrates a camera pose conversion system 600 for a photography robot system, comprising a preset module 610, a data receiving module 620, a data processing module 630, and a data transmission module 640.

[0159] The preset module 610 is used to preset the parameters of the pose converter;

[0160] The data receiving module 620 is used to receive the pose data of the camera in the photography robot; the pose data includes position information and posture information;

[0161] The data processing module 630 is used to convert the camera's pose data into pose data conforming to the FREE-D protocol through a pose converter;

[0162] The data transmission module 640 is used to send the pose data conforming to the FREE-D protocol to the virtual production platform.

[0163] In this embodiment of the invention, the invention also includes a camera robot system, the system comprising a camera pose conversion system for the camera robot system proposed in this invention.

[0164] See Figure 7 This disclosure also provides an electronic device 70, which includes:

[0165] At least one processor; and,

[0166] The memory is communicatively connected to the at least one processor; wherein,

[0167] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the camera pose conversion method of the camera robot system in the foregoing method embodiments.

[0168] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the camera pose conversion method of the camera robot system in the foregoing method embodiments.

[0169] This disclosure also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the camera pose conversion method of the photography robot system in the foregoing method embodiments.

[0170] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device 70 suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0171] like Figure 7 As shown, the electronic device 70 may include a processing unit 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 70. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0172] Typically, the following devices can be connected to I / O interface 705: input devices 706, including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 707, including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc., wherein input devices 706 and output devices 707 are optional; storage devices 708, including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 70 to communicate wirelessly or wiredly with other devices to exchange data. Although electronic device 70 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0173] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0174] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0175] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0176] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire at least two Internet Protocol (IP) addresses; send a node evaluation request including the at least two IP addresses to a node evaluation device, wherein the node evaluation device selects an IP address from the at least two IP addresses and returns it; and receive the IP address returned by the node evaluation device; wherein the acquired IP address indicates an edge node in a content delivery network.

[0177] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive a node evaluation request including at least two Internet Protocol (IP) addresses; select an IP address from the at least two IP addresses; and return the selected IP address; wherein the received IP address indicates an edge node in the content delivery network.

[0178] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0179] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0180] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0181] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0182] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for camera pose conversion in a photography robot system, characterized in that, The method includes the following steps: Preset parameters for the pose converter; Receive pose data from the camera in the photography robot; the pose data includes position information and posture information; The pose data of the camera is converted into pose data conforming to the FREE-D protocol using a pose converter; The pose data conforming to the FREE-D protocol is sent to the virtual production platform; The parameters of the preset pose converter include: Configure the network communication parameters of the pose converter and associated devices, the clock source for the pose data sampling period, and the pre-bias algorithm; Setting the pre-bias algorithm includes the following steps: Let the pose data of the camera in the photography robot be v = [v x v y v z v a v b v c ] T ; where v x v y v z For camera position; v a v b v c Let T be the Euler angles of the camera pose, and T be the transpose. The pose data v is converted into a standard homogeneous matrix M, i.e., M = f(v); Let the pre-bias data of the camera be u = [u x u y u z u a u b u c ] T ; where u x u y u z The adjustment amount for the camera position; u a u b u c The Euler angles corresponding to the camera attitude adjustment; The pre-biased data u is converted into a standard homogeneous matrix N, i.e., N = g(u); The pre-biased data is superimposed on the camera pose data to obtain T. c That is, T c =N·M; Let the pose of the virtual camera conforming to the Free-D protocol be w = [w x w y w z w a w b w c ] T Among them, w x w y w z For the virtual camera position; w a w b w c Euler angles for the virtual camera pose; w=h(T c ) The h function is used to perform inverse kinematics on the standard homogeneous pose matrix and convert it into position information and attitude Euler angles that satisfy the coordinate system definition of the virtual production platform and conform to the FREE-D protocol. The standard homogeneous matrix is ​​a 4×4 matrix, with the upper left 3×3 submatrix representing the attitude and the upper right 3×1 submatrix representing the position. The process of converting the pose data into pose data conforming to the FREE-D protocol using a pose converter includes the following steps: The camera pose data is converted into a standard homogeneous matrix; The camera's position and attitude information are fed into the pre-bias algorithm in the preset pose converter to generate a virtual camera pose. Based on the coordinate system definition of the virtual production platform, the virtual camera pose is converted into a left coordinate system or / or a right coordinate system expression; The virtual camera pose expressed in the left or / right coordinate system is converted into position information and attitude Euler angles conforming to the FREE-D protocol.

2. The camera pose conversion method for the photography robot system according to claim 1, characterized in that, Receiving the pose data of the camera in the photography robot includes: sending a pose data request to the photography robot; wherein sending the pose data request to the photography robot includes sending the pose data request to the photography robot at a preset fixed frequency; or sending the pose data request to the photography robot at an external synchronization signal frequency; and The camera robot continuously transmits camera pose data to the pose converter.

3. The camera pose conversion method for the photography robot system according to claim 2, characterized in that, When a pose data request is sent to the camera robot at a preset fixed frequency, a pose request is sent to the camera robot system at a preset frequency. When a pose data request is sent to the camera robot at the frequency of the external synchronization signal, the external video signal is monitored, and a pose request is sent to the camera robot system whenever a frame synchronization signal is detected.

4. The camera pose conversion method for the photography robot system as described in any one of claims 1-3, characterized in that, The method is used to perform pose transformation on a photography robot system that satisfies translational motion in three directions and rotational motion in three directions in Cartesian space.

5. A camera pose conversion system for a photography robot system, characterized in that, The system includes: The preset module is configured to preset the parameters of the pose converter; The data receiving module is configured to receive pose data from the camera in the photography robot; the pose data includes position information and attitude information. The data processing module is configured to convert the camera's pose data into pose data conforming to the FREE-D protocol via a pose converter; The data transmission module is configured to send the pose data conforming to the FREE-D protocol to the virtual production platform; The parameters of the preset pose converter include: Configure the network communication parameters of the pose converter and associated devices, the clock source for the pose data sampling period, and the pre-bias algorithm; Setting the pre-bias algorithm includes the following steps: Let the pose data of the camera in the photography robot be v = [v x v y v z v a v b v c ] T ; where v x v y v z For camera position; v a v b v c Let T be the Euler angles of the camera pose, and T be the transpose. The pose data v is converted into a standard homogeneous matrix M, i.e., M = f(v); Let the pre-bias data of the camera be u = [u x u y u z u a u b u c ] T ; where u x u y u z The adjustment amount for the camera position; u a u b u c The Euler angles corresponding to the camera attitude adjustment; The pre-biased data u is converted into a standard homogeneous matrix N, i.e., N = g(u); The pre-biased data is superimposed on the camera pose data to obtain T. c That is, T c =N·M; Let the pose of the virtual camera conforming to the Free-D protocol be w = [w x w y w z w a w b w c ] T Among them, w x w y w z For the virtual camera position; w a w b w c Euler angles for the virtual camera pose; w=h(T c ) The h function is used to perform inverse kinematics on the standard homogeneous pose matrix and convert it into position information and attitude Euler angles that satisfy the coordinate system definition of the virtual production platform and conform to the FREE-D protocol. The standard homogeneous matrix is ​​a 4×4 matrix, with the upper left 3×3 submatrix representing the attitude and the upper right 3×1 submatrix representing the position. The process of converting the pose data into pose data conforming to the FREE-D protocol using a pose converter includes the following steps: The camera pose data is converted into a standard homogeneous matrix; The camera's position and attitude information are fed into the pre-bias algorithm in the preset pose converter to generate a virtual camera pose. Based on the coordinate system definition of the virtual production platform, the virtual camera pose is converted into a left coordinate system or / or a right coordinate system expression; The virtual camera pose expressed in the left or / right coordinate system is converted into position information and attitude Euler angles conforming to the FREE-D protocol.

6. A photographic robot system, characterized in that, The system includes the camera pose conversion system of the photography robot system as described in claim 5.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by at least one processor, cause the at least one processor to perform the camera pose conversion method of the photographic robot system as described in any one of claims 1 to 4.

Images