Camera tracking system using event camera

Abstract

A camera tracking system according to one disclosed embodiment comprises: at least one event camera; a film camera to / from which the event camera can be attached / detached; and a processor for calculating a posture value of the film camera, wherein the processor extracts a coordinate value of a feature point on the basis of edge data of the event camera, calculates a posture value of the event camera on the basis of a localization algorithm and the extracted coordinate value of the feature point, calculates a relative posture value of the film camera on the basis of image data captured by the film camera, and calculates a posture value of the film camera on the basis of the relative posture value of the film camera and the posture value of the event camera.

Description

Camera tracking system using event cameras

[0001] The present invention relates to a system for estimating the position of a film camera when shooting image data used in a TV (TeleVision) broadcast, movie or video game, and to a method for generating positioning data of a film camera based on coordinate values ​​of feature points extracted from an event camera.

[0002] Many technological advancements have been made in the field of image processing, which involves capturing and processing footage for use in TV broadcasts, movies, and video games. Thanks to these advancements, not only major film production companies but also ordinary people can easily superimpose virtual images onto footage shot with film cameras.

[0003] In the aforementioned image processing, it's crucial to determine which part of the actual shooting environment the film camera is pointing at. Once the system knows where the film camera is pointing, it can easily add or modify the generated augmented objects. Therefore, the system needs to build a 3D map that includes the film camera and the location it's shooting from. The key is tracking technology, which uses this 3D map to track the movement of the film camera.

[0004] Conventional camera tracking systems have attempted to create 3D maps and camera tracking by attaching markers to the filming location and including them in each captured image frame (e.g., Lightcraft Technology). These marker-based technologies are highly inefficient and require manual intervention, extending filming times. Furthermore, markers must always be within the film camera's field of view (FOV), making them ineffective for outdoor use. This presents numerous limitations.

[0005] To overcome the shortcomings of these prior art techniques, there is a prior art technique that installs a separate image sensor on a film camera and mounts reflective markers on the ceiling to take pictures. Among these prior art techniques, Patent Document 1 discloses a technique that attempts to track a camera based on the positions of markers photographed by an image sensor by restoring a 3D map through the positions of reflective markers. Specifically, Patent Document 1 estimates the attitude value of a film camera by estimating the subtended angle of a pair of markers. However, this prior art technique has problems in that it uses only one camera, so the restoration of marker positions is not precise, the marker position estimation error is large, and it requires considerable effort to restore the precise positions of the markers. In addition, Patent Document 1 has a disadvantage in that the positions of the markers must not change, making it difficult to use in places where the shooting scene changes frequently.

[0006] To overcome these shortcomings of prior art, a technology has been developed that attempts camera tracking by providing at least two image sensors on a film camera and comparing the positions of 3D points (or feature points within a video image) within the image data. This technology utilizes at least two image sensors (e.g., Stereo Vision Cameras) that can detect each other's positions, triangulates the positions of feature points, and then uses these feature points to estimate the camera's position.

[0007] However, these conventional techniques have the problem of slowing down the processing speed of data collected from at least two stereo image sensors when the film camera moves quickly or when processing hardware including high-performance hardware is not provided. In other words, the conventional techniques cannot be used in dynamic shooting scenes that require rapid movement of the film camera, and they have been difficult to apply in situations where high-speed camera position estimation is required and image synthesis or image processing is required.

[0008] (Patent Document 1) US 10580153 B2

[0009] According to the disclosed embodiment, the present invention relates to a camera tracking system capable of rapidly estimating the position of a camera using only information of changing pixels rather than the entire image by calculating the final attitude value of a film camera through coordinate values ​​of feature points collected from an event camera.

[0010] A camera tracking system according to one embodiment of the disclosure includes: at least one event camera; a film camera capable of being attached or detached to the event camera; and a processor for calculating a posture value of the film camera, wherein the processor extracts coordinate values ​​of feature points based on edge data of the event camera, calculates a posture value of the event camera based on the coordinate values ​​of the extracted feature points and a localization algorithm, calculates a relative posture value of the film camera based on image data captured by the film camera, and calculates a posture value of the film camera based on the relative posture value of the film camera and the posture value of the event camera.

[0011] The processor may store a first feature point coordinate value extracted from edge data of the event camera, extract a second feature point coordinate value from edge data collected through a scan of the event camera after a certain period of time, match the first feature point coordinate value and the second feature point coordinate value based on a distance between vectors and a feature vector, and calculate the event detail value based on a pair of matched feature point coordinate values ​​and the localization algorithm.

[0012] The processor can select a feature point from a point cloud through a 3D range sensor, store the coordinate values ​​of the selected feature point in advance as first feature point coordinate values, extract the second feature point coordinate values ​​from the event camera, and match the pre-stored first feature point coordinate values ​​with the extracted second feature point coordinate values.

[0013] The above processor can select a marker attached to a shooting scene as a feature point in the point cloud, include the coordinate value of the selected feature point as the first feature point coordinate value, and store the coordinate value of the first feature point.

[0014] The event camera further includes an IR light attached thereto, and the processor can perform on / off control of the IR light and extract the second feature point coordinate value based on edge data captured by the event camera for the marker.

[0015] The above event camera may further include a bandpass filter including a wavelength range emitted by the IR light.

[0016] The film camera further includes an IMU sensor, and the processor can calculate a posture value of the event camera based on sensor data of the IMU sensor and a localization algorithm.

[0017] The processor can generate motion information of the film camera and calculate a relative attitude value of the film camera based on the motion information of the film camera and the attitude value of the event camera.

[0018] The film camera further includes an auxiliary camera provided to capture images in the same direction as the film camera, and the processor can calculate a relative attitude value of the auxiliary camera and the film camera based on the coordinates of the first feature point and image data captured by the auxiliary camera.

[0019] The above processor can calculate the relative attitude value of the film camera based on the relative attitude values ​​of the auxiliary camera and the event camera that are stored in advance and the calculated relative attitude values ​​of the auxiliary camera and the film camera.

[0020] The above processor may further include an output unit that displays the produced film camera detail value.

[0021] The processor can transmit the calculated attitude value of the film camera to the outside, or perform background rendering, 3D object mixing, or VFX or CGI image processing based on the attitude value of the film camera.

[0022] The camera tracking system according to the disclosed embodiment can estimate the position of the camera at high speed using only information of changing pixels rather than the entire image by calculating the final attitude value of the film camera through the coordinate values ​​of feature points collected from the event camera.

[0023] In addition, the disclosed camera tracking system can solve problems of conventional stereo vision cameras and can be used in a variety of ways in the field of visual effects (VFX).

[0024] In addition, the disclosed camera tracking system can be used in shooting locations where various types of markers are installed or where markers cannot be attached.

[0025] Figure 1 is a schematic diagram illustrating the configuration of a camera tracking system.

[0026] Figure 2 is a control block diagram of the disclosed camera tracking system and user terminal.

[0027] Figure 3 is a flowchart schematically illustrating the disclosed camera tracking method.

[0028] FIG. 4 is a flowchart for explaining a camera tracking method according to the disclosed first embodiment.

[0029] FIG. 5 is a flowchart for explaining a camera tracking method according to the disclosed second embodiment.

[0030] FIG. 6 is a flowchart for explaining a camera tracking method according to the disclosed third embodiment.

[0031] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. However, the technical concept of the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided to ensure that the disclosure is thorough and complete and to sufficiently convey the spirit of the present invention to those skilled in the art.

[0032] In this specification, when a component is referred to as being on another component, it means that it can be formed directly on the other component, or a third component may be interposed between them. Furthermore, in the drawings, shapes and sizes are exaggerated for the purpose of effectively explaining the technical contents.

[0033] Additionally, although terms such as first, second, and third have been used to describe various components in various embodiments of this specification, these components should not be limited by these terms. These terms are only used to distinguish one component from another. Thus, what is referred to as a first component in one embodiment may be referred to as a second component in another embodiment. Each embodiment described and illustrated herein also includes its complementary embodiments. Additionally, the term "and / or" has been used herein to mean including at least one of the components listed before and after.

[0034] In the specification, singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "comprise" or "have" are intended to specify the presence of a feature, number, step, component, or combination thereof described in the specification, and should not be construed as excluding the presence or addition of one or more other features, numbers, steps, components, or combinations thereof. In addition, the term "connection" is used in the present specification to mean both indirectly connecting multiple components and directly connecting them.

[0035] In addition, when describing the present invention below, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

[0036] Figure 1 is a schematic diagram illustrating the configuration of a camera tracking system.

[0037] Referring to FIG. 1, a camera tracking system (1) according to one embodiment disclosed includes a film camera (5) and at least one event camera (Event Camera, 2) that can be detachably attached or detached to the film camera (5). Both the film camera (5) and the event camera (2) can capture images of an object (Ob) and a surrounding environment (e.g., a studio, an outdoor filming location) including the object (Ob), thereby constructing a 3D map.

[0038] Although not specifically shown in Fig. 1, the event camera (2) may be positioned on the upper surface of the film camera (5), but the attachment location is not limited.

[0039] The event camera (2) is a camera that generates events based on changes in light. The event camera (2) records log intensity (a value obtained by taking the log of the intensity of light) over time, and if a sudden change in light occurs and exceeds a certain threshold, a signal indicating that an event has occurred is generated. This is a device that is clearly different from an image sensor that transmits a conventional bitmap. Although it is technically difficult to implement the position using edge data, the capacity of the generated edge data is small, enabling high-speed data transmission.

[0040] The signal generated by the event camera (2) includes coordinate values ​​(location, x, y), time information (t), and polarity (p). Here, polarity refers to information about whether the light brightness is increasing or decreasing. In other words, the event camera refers to a data-based sensor, and the more movements are observed, the more data is generated. Hereinafter, the event information generated by the event camera (2) is expressed as edge data.

[0041] In the disclosed embodiment, the event camera (2) may include all of the various event cameras disclosed, such as Sony event cameras, DVS event cameras, ATIS, DAVIS, etc., and may generate data based on individual events, or may also include cameras with various algorithms that generate data by group (or packet) of events.

[0042] Meanwhile, although the aforementioned embodiment and other embodiments below describe mounting a single event camera (2), this is not necessarily limited to the present invention. That is, the disclosed camera tracking system (1) may be equipped with at least two event cameras (2) on a film camera (5), and may include at least two event cameras (2) to increase the accuracy of triangulation during the map building process described below.

[0043] A camera tracking system (1) according to another disclosed embodiment may include not only a film camera (5) and an event camera (2), but also an IMU sensor (Inertial Measurement Unit, 3) provided together with the event camera (2). The IMU sensor (3) refers to a sensor composed of a gyroscope, an accelerometer, or a geomagnetic sensor, and may include not only a 6-axis sensor having only a gyroscope and an accelerometer, but also a 9-axis sensor including all of a gyroscope, an accelerometer, and a geomagnetic sensor. The IMU sensor (3) may be provided together with the event camera (2), but its location does not necessarily have to be located on the upper surface of the film camera (5). That is, the IMU sensor (3) may also be separated from the film camera (5) like the event camera (2).

[0044] A camera tracking system (1) according to another disclosed embodiment may include a film camera (5), an event camera (2), an IMU sensor (3), and an auxiliary camera (4) such as a CMOS sensor. The auxiliary camera (4) may be provided together with the event camera (2) and may be detachably or attachably to the film camera (5).

[0045] A film camera (5) refers to a general camera that shoots images used in TV broadcasts, movies, or video games, and may include a support (not shown) that can fix an event camera (2) including an IMU sensor (3) and an auxiliary camera (4). Image data shot by the film camera (5) is transmitted to a user terminal (10) via wired or wireless communication.

[0046] The user terminal (10) calculates the pose value of the event camera (2) based on data collected by the event camera (2), IMU sensor (3), or auxiliary camera (4). Once the pose value of the event camera (2) is calculated, the user terminal (10) finally calculates the pose value of the film camera (5) to perform camera tracking. A detailed description thereof will be provided later with reference to other drawings.

[0047] Unlike that shown in FIG. 1, the user terminal (10) may receive the attitude value of the event camera calculated by a small terminal (not shown) provided together with the event camera (2), and then calculate the attitude value of the film camera (5) through the attitude value of the event camera. That is, the user terminal (10) may directly receive the edge data generated by the event camera (2) to calculate the final attitude value of the film camera (5), or may calculate the attitude value of the film camera (5) based on the attitude value of the event camera transmitted by the event camera (2).

[0048] The user terminal (10) can be implemented as a computer that performs tracking of a film camera (5), performs image processing using data captured by the film camera (5), or a wearable device that can be carried at a shooting location. Here, a computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser, and a wearable device may mean, for example, a wireless communication device that ensures portability and mobility, such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone, etc., and may also mean all kinds of handheld-based wireless communication devices.

[0049] Figure 2 is a control block diagram of the disclosed camera tracking system and user terminal.

[0050] Referring to FIG. 2, the camera tracking system (1) includes a film camera (5), a sensor unit (2, 3, 4) that can be attached to the film camera (5), and a user terminal (10).

[0051] Here, the user terminal (10) may include a processor (6) that calculates the attitude value of the film camera based on the sensor data transmitted by the sensor units (2, 3, 4) and the image data transmitted by the film camera (5), an output unit (9) that displays the image-processed data of the processor (6) or outputs the current position of the film camera (5) through a map based on the calculated attitude value of the film camera (5), and a memory (9) that collects and stores various algorithms, maps, and various data required for the operation of the processor (6).

[0052] Specifically, the event camera (2) transmits the edge data it generates to the processor (6). As described above in Fig. 1, the edge data may include coordinate values ​​(location, x, y), time information (t), and information about Polarity (p). The processor (6) extracts a feature point corresponding to the center (x, y) of each point group through clustering from the received edge data, and performs matching with the given 3D feature point coordinates (x, y, z) based on the coordinates of the extracted feature point and the relative distances with other feature points in the vicinity, and then performs localization through this.

[0053] The IMU sensor (3) is a sensor provided to supplement the tracking section that cannot be estimated using only the edge data of the event camera (2), such as when the film camera (5) moves at high speed and fails to track a feature point or when the event camera temporarily captures a space without any feature points. In addition, the IMU sensor (3) can prevent a jumping phenomenon that may occur during attitude estimation due to noise generated when performing localization only with the event camera (2). The IMU sensor (3) generates sensing data that detects a change in gravity and transmits it to the processor (6).

[0054] The auxiliary camera (4) refers to a camera that includes a camera lens on a general image sensor, and may include various types of cameras. The auxiliary camera (4) may be configured to capture an object (Ob) captured by the film camera (5), i.e., capture in the same direction as the film camera (5).

[0055] The auxiliary camera (4) is a device for easily estimating the relative attitude value between the film camera (5) and the event camera (2), and the relative attitude value between the auxiliary camera (4) and the event camera (2) can be stored in advance in the memory (9) through calibration performed in advance.

[0056] The processor (6) is configured to control the overall operation of the user terminal (10), and performs an algorithm for controlling the operation of the memory (9) and the output unit (11), an algorithm required for camera tracking described below, and an algorithm for processing image data captured by a film camera (5). The processor (6) stores the aforementioned algorithm and the data required to implement the algorithm through the memory (9). In addition, the processor (6) may store the data collected by the sensor units (2, 3, 4), and then load the stored data from the memory (9) when camera tracking is required.

[0057] Meanwhile, the processor (6) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed as a code or command included in a program required for image processing and control. As an example of a data processing device built into hardware, the device may include a processing device such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a graphics processing unit (GPU).

[0058] Memory (9) may be implemented as at least one of non-volatile memory elements such as cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, or volatile memory elements such as RAM (Random Access Memory), or storage media such as a hard disk drive (HDD) or CD-ROM, but is not limited thereto.

[0059] Meanwhile, the memory (9) and the processor (6) may be implemented as separate chips as shown in Fig. 2, but are not limited thereto and may be implemented as a single chip.

[0060] The output unit (11) may include a display (12) that outputs image data of an image-processed film camera. The display (12) may be provided as a digital light processing (DLP) panel, a plasma display panel, a liquid crystal display (LCD) panel, an electroluminescence (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0061] In addition to the display (12), the output unit (13) may further include various hardware configurations that can communicate with the user.

[0062] In addition to the configuration illustrated in FIG. 2, the disclosed camera tracking system (1) may further include hardware configurations necessary to calculate the attitude value of the film camera (5), and is not necessarily limited to the configuration or the name of the configuration in FIG. 2.

[0063] Figure 3 is a flowchart schematically illustrating the disclosed camera tracking method.

[0064] Referring to FIG. 3, the disclosed camera tracking system (1) includes a data sensing step (20) for collecting various data from an event camera (2), an IMU sensor (3), and an auxiliary camera (4), a localization step (30) for calculating a posture value of the event camera (2) based on the sensed data, a calibration step (40) for extracting feature points based on edge data sensed through the event camera (2) and then correcting a relative posture value of the film camera (5) based on the extracted feature points, and a step (50) for calculating a final posture value of the film camera (5) and tracking it.

[0065] Specifically, the data sensing step (20) can be divided into a first embodiment that executes the aforementioned camera tracking method using an event camera (2) and a film camera (5), a second embodiment that executes the camera tracking method using an event camera (2), an IMU sensor (3), and a film camera (5), and a third embodiment that performs localization using an event camera (2), an IMU sensor (3), and a film camera (5) and performs calibration using the relative attitude values ​​of the auxiliary camera (4) and the event camera (2) through the auxiliary camera (4).

[0066] The localization step (30) uses the feature points extracted through edge data sensed from the event camera (2) and the locations of 3D feature points matching the feature points. Here, the feature points can be implemented by directly performing map building through the event camera (2), or feature points extracted through an external device such as a 3D range sensor, for example, Lidar or a total station can be used. At this time, the feature points can be natural feature points such as edges, corners, or walls with unique visual patterns of a structure, or can be markers with unique IDs such as randomly attached retroreflective markers or April Tags.

[0067] The localization step (30) uses the coordinate values ​​of feature points (hereinafter, first feature point coordinate values) that can be extracted from the edge data of the event camera (2) that captured the shooting space at a specific time and the coordinate values ​​of feature points (hereinafter, second feature point coordinate values) that can be extracted from the edge data of the event camera (2) that captured the shooting space again. In the localization step (30), the coordinate values ​​of the first feature point that are stored in advance and the coordinate values ​​of the extracted second feature point are compared with each other to determine a feature point pair, and the attitude value of the event camera (2) is determined through a localization algorithm such as a Kalman filter or batch optimization.

[0068] Specifically, the disclosed camera tracking system (1) can determine a matching (feature point pair) between a first feature point and a second feature point by using a vector that characterizes the mutual distance between feature points within each feature point data. That is, the camera tracking system (1) can define a feature (descriptor) vector of a specific feature point belonging to the first feature point, and find a feature point having a feature vector closest to the defined feature vector among the feature points belonging to the second feature point, thereby determining the feature point pair.

[0069] Meanwhile, the disclosed camera tracking system (1) can also perform localization by tracking the location of a new feature point based on the location of an existing feature point, rather than determining a match using a feature point vector each time.

[0070] The camera tracking system (1) can calculate the pose value of an event camera by utilizing a feature point pair.

[0071] Specifically, the camera tracking system (1) can use an algorithm that projects 3D into 2D based on constraints of edge projection errors to calculate the pose value of the event camera.

[0072] First, the camera tracking system (1) converts a 3D point into the coordinate system of the event camera. At this time, the current pose of the event camera is required, which can be obtained by an algorithm such as PnP (Perspective-n-Point) using the 2D positions of the 3D point and the feature point. Thereafter, the camera tracking system (1) projects the 3D point converted into the coordinate system of the event camera onto the image plane based on the camera intrinsic matrix. The camera tracking system calculates the image projection error based on the following mathematical expression 1, and calculates the pose value of the event camera at which the image projection error (Eproj) function is minimized.

[0073]

[0074] Here, Xj is the position coordinate of the jth 3D feature point, x j is the location of the feature point observed from the event camera (2), is the position of the event camera relative to the world coordinate system (W), π() is a function that projects a 3D point in the camera coordinate system (E) through the camera inner product matrix. represents the result of projecting 3D feature points onto the image plane.

[0075] Meanwhile, in the localization step (30) executed in the second and third embodiments, the sensor data of the IMU sensor (3) can be combined with the sensor data of the feature point pairs that compare the first feature point coordinate value and the second feature point coordinate value to perform a Kalman filter or batch optimization.

[0076] That is, the camera tracking system (1) according to the second embodiment calculates the pose value of the event camera at which the error function of the mathematical expression 5 below is minimized.

[0077]

[0078] Here is the relative rotation value obtained by pre-integrating (pre-integrating) the sensor data of the IMU sensor (3) between successive scans k and k+1, It represents the relative position movement value obtained by pre-integrating the sensor data of the IMU sensor (3).

[0079] The camera tracking system (1) can also calculate the optimal event camera pose value by combining the terms of the error function described through mathematical expressions 1 and 2 in various ways depending on the situation.

[0080] Specifically, according to the first embodiment, when only the edge data of the event camera (2) is used, an error function (E) for calculating the attitude value of the event camera total ) is as shown in mathematical formula 2 below.

[0081]

[0082] The 3D information for the 2D feature points used here can be collected via external 3D range sensors, Lidar, or total stations. For example, after scanning the space with a 3D range sensor (e.g., Lidar), the marker locations can be extracted using point intensities. Alternatively, 3D reconstruction can be performed on edge data using visual inertial methods.

[0083] The second embodiment using the IMU sensor (3) enables additional observation of motion through the IMU sensor (3), and thus, the stability and precision of the camera tracking system (1) can be improved through the error function through the mathematical expression 2 and the mathematical expression 3 that fuses the data of the IMU sensor (3).

[0084]

[0085] However, in the second embodiment using the IMU sensor (3), the edge data of the event camera (2) can be generated faster than the sensor data of the IMU sensor (3). In this case, the localization algorithm can estimate the pose of the event camera using only mathematical expression 3.

[0086] Meanwhile, β and γ are weights for balancing the importance of feature point alignment and constraints of the IMU sensor (3) and film camera (5), and can be changed by the user.

[0087] The disclosed camera tracking system (1) can calculate the final pose value of the event camera by applying the error function set in Equations 2 to 3 to the Extended Kalman Filter (EKF). In addition, the camera tracking system (1) can also calculate the pose value of the event camera by using a placement optimization method using a nonlinear least square method (Gauss-Newton or Levenberg-Marquardt).

[0088] The camera tracking system (1) performs a calibration step (40) in parallel with the localization step (30). The calibration step (40) extracts feature points (41), measures the motion of the film camera at the extracted feature points (42), and calculates (43) the relative pose values ​​of the event camera (2) and the film camera (5) based on the amount of movement of the event camera and the measured motion information of the film camera.

[0089] As an example for extracting feature points, the camera tracking system (1) extracts feature points corresponding to the center (x, y) of a point cloud through clustering formed from edge data collected from an event camera (2).

[0090] As another example for extracting feature points, the camera tracking system (1) can also extract feature points from data of a point cloud formed by scanning in advance with a 3D range sensor. Here, a retroreflective marker can be attached to a structure, etc. at the shooting location so that the feature points can be easily distinguished from other sensing data of the 3D range sensor, and this can be used as the feature points. When the 3D range sensor scans the retroreflective marker, the 3D range sensor returns a high intensity value. Therefore, when the 3D range sensor and the retroreflective marker are used, the camera tracking system (1) can easily recognize and extract the feature points.

[0091] As another example, the disclosed camera tracking system (1) can additionally provide IR lighting next to the lens of the event camera (2) so that the retroreflective marker can be clearly captured by the event camera (2). The event camera (2) does not generate edge data if there is no change in light. Therefore, the camera tracking system (1) additionally including IR lighting can perform a control operation to periodically turn the IR lighting on / off, and continuously observe edge data of feature points.

[0092] A camera tracking system (1) including IR illumination may include a bandpass filter having the same wavelength as the IR illumination on the lens of the event camera (2). This embodiment allows the event camera (2) to clearly capture only the marker without being affected by external light.

[0093] The IR illumination and bandpass filter can operate in the 850 nm to 950 nm range or the 1250 nm to 1400 nm range. In this case, even when shooting outdoors, the disclosed camera tracking system (1) can accurately recognize feature points, including markers.

[0094] Meanwhile, information on the relative attitude value between the film camera (5) and the event camera (2) is essential information for replacing the attitude value of the event camera with the attitude value of the film camera. Therefore, the disclosed camera tracking system (1) performs calibration (40) to calculate the relative attitude value between the film camera (5) and the event camera (2) (hereinafter, the relative attitude value of the film camera).

[0095] As an example of calibration, a camera tracking system (1) can use motion information generated by moving a film camera (5).

[0096] In order to measure the motion of a film camera (5), the camera tracking system (1) calculates and adjusts the intrinsic parameters and extrinsic parameters of the image data acquired from the film camera (5). Here, the intrinsic parameters are parameters related to the properties of the film camera (5) itself, such as focal length and lens distortion, and the extrinsic parameters are parameters related to the position and direction of the film camera (5). The camera tracking system (1) extracts feature points from a plurality of image data (image data with adjusted parameters) captured while the film camera (5) moves or rotates. In the calibration step (40), the camera tracking system (1) measures the motion of the film camera (5) using the movement information of the extracted feature points.

[0097] When the film camera (5) moves or rotates, the event camera (2) also moves together, so the camera tracking system (1) loads the amount of movement of the event camera (2) calculated in the localization step (30).

[0098] The camera tracking system (1) calculates a relative attitude value between the film camera (5) and the event camera (2), i.e., a relative attitude value of the film camera, based on the amount of movement of the event camera (2) and the measured motion information of the film camera (5).

[0099] Meanwhile, the third embodiment using the auxiliary camera (4) can calculate the relative pose value based on the coordinates of the feature point pre-stored in the memory (9) instead of extracting the relative pose value based on the motion information. As described above in Fig. 1, since the auxiliary camera (4) is arranged to face the same direction as the film camera (5), the feature point whose 3D position is known can be captured simultaneously. The third embodiment can calculate the relative pose value through the PnP (Perspective-n-Point) algorithm using the coordinates of the feature point pre-stored.

[0100] In addition, the third embodiment can be further optimized through a nonlinear least squares method (Gauss-Newton or Levenberg-Marquardt). Specifically, the third embodiment first calculates the relative attitude value between the auxiliary camera (4) and the film camera (5), and then uses the previously stored relative attitude value between the auxiliary camera (4) and the event camera (2), thereby calculating the relative attitude value between the event camera (2) and the film camera (5).

[0101] When the relative attitude value of the film camera is calculated, the camera tracking system (1) calculates the attitude value of the film camera indicating the final position of the film camera using the relative attitude value of the film camera and the attitude value of the event camera (50).

[0102] Meanwhile, the calibration step (40) does not need to be performed every time localization is performed.

[0103] The relative attitude value of the film camera and the attitude value of the event camera can both be expressed as SE(3) matrices, and the camera tracking system (1) can calculate the final attitude value of the film camera based on the SE(3) product operation of the two attitudes.

[0104] The finally derived pose values ​​of the film camera can be used for real-time pose estimation of cameras in various fields. Specifically, the disclosed camera tracking system (1) can be used for various shooting purposes, such as real-time pose estimation of studio film cameras, real-time pose estimation of Steadicams, real-time pose estimation of crane-mounted cameras, real-time pose estimation of dollies-mounted cameras, real-time pose estimation of cameras used in outdoor shooting, pose estimation of cameras for VFX (Visual FX), and pose estimation of cameras used in chroma key shooting.

[0105] FIG. 4 is a flowchart for explaining a camera tracking method according to the disclosed first embodiment.

[0106] Referring to Fig. 4, the camera tracking system (1) executes map building (100).

[0107] Here, map building refers to 3D map data generated after taking pictures of the shooting location in advance using an event camera (2) or a 3D range sensor (e.g., a lidar sensor).

[0108] Feature points are extracted (110) from edge data generated by the event camera (2), and the feature point coordinate values ​​of the extracted feature points are generated and stored in advance (120).

[0109] The coordinate values ​​of the feature points are then used in localization and calibration.

[0110] Meanwhile, the disclosed camera tracking system (1) does not necessarily need to perform a map building step in order to calculate the attitude value of the film camera (5). That is, the coordinate values ​​of the feature points can be generated in advance through a 3D range sensor and stored in the user terminal (10), and the localization and calibration described below can be performed by loading the pre-stored feature point map.

[0111] The camera tracking system (1) compares the coordinate values ​​of feature points extracted at different times with each other (130) and calculates the pose value of the event camera through a Kalman filter or batch optimization (131).

[0112] The camera tracking system (1) measures the motion of the film camera (5) using movement information of the extracted feature points (140).

[0113] The camera tracking system (1) according to the first embodiment includes an SE (3) matrix (T) for converting a coordinate system (E) of an event camera that can be defined according to the amount of movement of the event camera (2) into a coordinate system (C) that can be defined according to the motion information of the film camera (5). EC ) is defined as in mathematical formula 5 below.

[0114]

[0115] Here R EC is a 3x3 rotation matrix, and t EC is a 3x1 position vector.

[0116] A specific time (t) i ) in the pose (pose, T) of the event camera (2) and the film camera (5) E (t i ) and T C (t i )), the relative motion (posture value) between consecutive poses (motions) can be expressed in each coordinate system (E,C). The camera tracking system (1) has a matrix (T EC ) for optimization, the problem is solved in closed form using the hand-to-eye method to obtain an initial solution, and then the LM (Levenberg-Marquardt) algorithm and the GN (Gauss-Newton) algorithm are used to obtain a more accurate relative pose value of the film camera.

[0117] The camera tracking system (1) comprehensively considers the measured motion information of the film camera (5) and the amount of movement of the event camera (2) to calculate the relative pose value between the film camera (5) and the event camera (2) (142).

[0118] The event camera's pose value (T WE ) and the relative attitude value (T EC ) is produced, the camera tracking system (1) calculates the final attitude value (T) of the film camera (5). WC ) is calculated using the mathematical formula 6 below (150), and the final position of the film camera (5) can be determined through this.

[0119]

[0120] FIG. 5 is a flowchart for explaining a camera tracking method according to the disclosed second embodiment.

[0121] The camera tracking system (1) performs map building (100). After the map building is completed, feature points are extracted (110) from edge data generated by the event camera (2), and the feature point coordinate values ​​of the extracted feature points are generated and stored in advance (120).

[0122] The camera tracking system (1) generates sensor data (200) through an IMU sensor (3).

[0123] The camera tracking system (1) compares the first feature point coordinate value and the second feature point coordinate value (130), and then, when performing localization, uses the sensor data of the IMU sensor (3) together to calculate the final event camera attitude value (131).

[0124] Specifically, the camera tracking system (1) according to the second embodiment adds constraints to the movement of the event camera (2) based on the relative rotation and position movement predicted from the sensor data of the IMU sensor (3).

[0125] The camera tracking system (1) extracts feature points based on image data captured by the film camera (5) and measures the motion of the film camera (5) by comparing the extracted feature points (140). The camera tracking system (1) calculates a relative pose value based on the measured motion information and the amount of movement of the event camera (2) (142).

[0126] When the pose value and relative pose value of the event camera are calculated, the camera tracking system (1) calculates the final pose value of the film camera (5) (150), and through this, the position of the real-time film camera (5), which was difficult to perform in the first embodiment, can be known.

[0127] FIG. 6 is a flowchart for explaining a camera tracking method according to the disclosed third embodiment.

[0128] Referring to FIG. 6, the camera tracking system (1) according to the third embodiment performs calibration based on a captured image sensed by the auxiliary camera (4).

[0129] Specifically, the camera tracking system (1) scans the shooting scene through a 3D range sensor (300).

[0130] As described above in FIG. 4, the disclosed camera tracking system (1) can also use feature points extracted from 3D map data generated in advance through a 3D range sensor in addition to edge data sensed from an event camera (2).

[0131] The camera tracking system (1) loads data on feature points extracted in advance through a 3D range sensor (310).

[0132] The camera tracking system (1) compares the coordinate values ​​of feature points extracted at different times with each other (330) and calculates the pose value of the event camera through a Kalman filter or batch optimization (331).

[0133] The camera tracking system (1) calculates the relative attitude values ​​of the auxiliary camera (4) and the film camera (5) (340).

[0134] In the third embodiment, since the auxiliary camera (4) and the film camera (5) face the same direction, feature points whose extracted feature point coordinates are known can be captured simultaneously. Therefore, the camera tracking system (1) according to the third embodiment calculates the relative pose values ​​of the auxiliary camera (4) and the film camera (5) through the PnP algorithm.

[0135] The camera tracking system (1) calculates the relative attitude values ​​of the event camera (2) and the film camera (5) (341).

[0136] Specifically, the camera tracking system (1) calculates the relative attitude value between the event camera (2) and the film camera (5) based on the relative attitude values ​​of the auxiliary camera (4) and the film camera (5) calculated in step 340 and the relative attitude values ​​between the auxiliary camera (4) and the event camera (2) stored in advance.

[0137] The camera tracking system (1) calculates the final attitude value of the film camera (5) (350).

[0138] Specifically, the camera tracking system (1) calculates the attitude value of the film camera based on the attitude value of the event camera calculated in step 331 and the relative attitude values ​​of the event camera (2) and the film camera (5) calculated in step 341. That is, the attitude value of the film camera can be calculated by the attitude value of the lidar and the relative attitude value of the film camera through a SE (3) matrix multiplication operation.

[0139] The camera tracking system (1) can transmit the calculated attitude value of the film camera to the outside through wired or wireless communication. The attitude value of the film camera calculated in the embodiment described through FIGS. 4 to 6 can be utilized in various fields such as virtual production studios, TV broadcasting, and VFX. Specifically, the virtual production studio displays an image while rendering the background in real time on a large LED wall. To this end, the virtual production image must accurately estimate the movement of the film camera (5) in real time to shoot an image with a natural background. The rendering server receives the attitude value of the film camera (5), renders the background accordingly, and then outputs it to the LED wall.

[0140] In sports, news, interviews, advertisements, internet live broadcasts, etc., AR technology is utilized by estimating the pose and lens information of a film camera in real time and transmitting this to a rendering computer. The rendering computer can produce realistic and good-looking videos by mixing 3D objects, etc. into the output image of the film camera according to the pose of the film camera. In this real-time AR technique, there is a method of directly mixing 3D objects into the image data of the film camera (5), and a method of removing the background using a chroma key cloth and then drawing 3D rendered objects on both the foreground and background. In this example of use, when the shooting camera moves, the camera tracking system (1) tracks it and transmits the pose value to the rendering computer. The rendering computer can mix 3D objects into an appropriate position based on the pose value.

[0141] In post-production of VFX or CGI, having camera movement information from the captured footage can significantly reduce the time required for post-production. Furthermore, recording point cloud data from LiDAR (2) in addition to pose values ​​can facilitate depth-related post-production in VFX.

[0142] The disclosed camera tracking method can also be applied when shooting rendered characters with a virtual camera in conjunction with a motion capture system, such as a virtual camera system (VCS). That is, the disclosed camera tracking method can also be used in estimating the pose of a virtual camera, and a virtual scouting system (VSS), which must create a virtual shooting space and find the optimal angle or view, can also perform shooting simulations based on the point cloud and pose values ​​provided by the lidar (2).

Claims

1. At least one event camera; A detachable film camera of the above event camera; and A processor for calculating the detailed value of the film camera; The above processor, Extract the coordinate values ​​of the feature points based on the edge data of the above event camera, The pose value of the event camera is calculated based on the coordinate values ​​of the extracted feature points and the localization algorithm. Calculate the relative attitude value of the film camera based on the image data captured by the above film camera, A camera tracking system that calculates the attitude value of the film camera based on the relative attitude value of the film camera and the attitude value of the event camera.

2. In paragraph 1, The above processor, Store the first feature point coordinate values ​​extracted from the edge data of the above event camera, After a certain period of time, the second feature point coordinate values ​​are extracted from the edge data collected through the scan of the above event camera, Matching the first feature point coordinate value and the second feature point coordinate value based on the distance between vectors and the feature vector, A camera tracking system that calculates the event detail value based on a pair of matched feature point coordinate values ​​and the localization algorithm.

3. In paragraph 2, The above processor, Select feature points from the point cloud using a 3D range sensor, The coordinate values ​​of the above-mentioned selected feature points are stored in advance as the coordinate values ​​of the first feature point, Extract the coordinate values ​​of the second feature point from the above event camera, A camera tracking system that matches the coordinate values ​​of the first feature point stored in advance with the coordinate values ​​of the second feature point extracted.

4. In paragraph 3, The above processor, In the above point cloud, the markers attached to the shooting scene are selected as feature points, Including the coordinate values ​​of the selected feature points as the coordinate values ​​of the first feature point, A camera tracking system that stores the coordinate values ​​of the first feature point.

5. In paragraph 1, Further comprising an IR light attached to the above event camera; The above processor, Perform on / off control of the above IR lighting, A camera tracking system that extracts the second feature point coordinate values ​​based on edge data captured by the event camera for the above marker.

6. In paragraph 5, The above event camera is, A camera tracking system further comprising a bandpass filter including a wavelength range emitted by the IR illumination.

7. In paragraph 1, Further comprising an IMU sensor provided in the above film camera; The above processor, A camera tracking system that calculates the attitude value of an event camera based on sensor data of the above IMU sensor and a localization algorithm.

8. In paragraph 2, The above processor, Generate motion information of the above film camera, A camera tracking system that calculates a relative pose value of a film camera based on the motion information of the film camera and the pose value of the event camera.

9. In paragraph 8, Further comprising an auxiliary camera provided on the film camera to photograph in the same direction as the film camera; The above processor, A camera tracking system that calculates relative pose values ​​of an auxiliary camera and a film camera based on the coordinates of the first feature point and image data captured by the auxiliary camera.

10. In paragraph 9, The above processor, A camera tracking system that calculates a relative attitude value of a film camera based on the relative attitude values ​​of the auxiliary camera and the event camera that are stored in advance and the relative attitude values ​​of the auxiliary camera and the film camera that are calculated above.

11. In paragraph 10, The above processor, A camera tracking system further comprising an output section for displaying the produced film camera attitude values.

12. In paragraph 1, The above processor, Transmit the calculated film camera detail values ​​to the outside, or A camera tracking system that performs background rendering, 3D object mixing, or VFX or CGI image processing based on the attitude values ​​of the above film camera.

Images