Camera tracking method and system for LED virtual shooting

Abstract

The present application relates to the technical field of LED virtual shooting, and particularly relates to a camera tracking method and system for LED virtual shooting. The method first acquires the corner points of a calibration plane and a shooting plane; according to the pitch angle, the direction angle and the distance corresponding to the origins of the camera coordinate system and the world coordinate system, the rotation amount and the displacement amount of the shooting plane relative to the calibration plane are determined, the corner points of the calibration plane are affinely converted to the shooting plane to obtain ideal corner points; according to the error between the corner points of the shooting plane and the ideal corner points, the distortion error amount is determined; the ideal corner points are corrected for distortion to obtain target corner points, the final rotation amount and the final displacement amount of the shooting plane relative to the calibration plane are determined, and the final rotation amount and the final displacement amount are transmitted to a calibration system to obtain the position of the camera. The present application considers the error caused by the camera imaging distortion on positioning by analyzing the transformation between the two coordinate systems, and improves the accuracy of camera positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of LED virtual shooting, in particular to a camera tracking method and system for LED virtual shooting. BACKGROUND

[0002] LED virtual shooting technology is a new type of shooting method combining LED display screen and virtual reality (VR) or augmented reality (AR) technology. It builds a large-scale LED display screen on the shooting site, which displays a virtual background image or a three-dimensional modeled virtual scene on the screen. The actors and other real objects are shot in front of this virtual background. Since the LED screen displays a high-quality dynamic background, the actors and objects can interact with the background in real time, avoiding the unnaturalness caused by the synthesis effect in traditional green screen shooting.

[0003] In LED virtual shooting technology, camera tracking is a key technology to ensure that the interaction between the virtual background and the actors or objects in the real world is consistent. Camera tracking tracks the position, angle and motion of the camera in real time to ensure that the virtual background is synchronized with the camera's perspective, thereby providing seamless and immersive shooting effects for actors and scenes.

[0004] Most existing camera tracking methods rely on tracking cameras or laser radars to determine the camera position and track the shooting camera. However, using these high-precision sensors for camera tracking requires a series of expensive tracking cameras or laser radars, greatly increasing the cost. SUMMARY

[0005] In order to solve the technical problem of high cost when using laser radar for camera tracking, the purpose of the present application is to provide a camera tracking method and system for LED virtual shooting, and the technical solution is as follows:

[0006] In the first aspect, the present application provides a camera tracking method for LED virtual shooting, which comprises:

[0007] When the camera is in the anchor position, the imaging plane where the LED screen is located is taken as the calibration plane, and the imaging plane of the camera is taken as the camera plane. The corner points of the calibration plane and the camera plane are detected to obtain the corner points of the calibration plane and the camera plane.

[0008] According to the pitch angle and the direction angle corresponding to the origins of the camera coordinate system and the world coordinate system, a rotation amount of the photographing plane relative to the calibration plane is determined; according to the distance between the origins of the camera coordinate system and the world coordinate system, a displacement amount of the photographing plane relative to the calibration plane is determined; wherein the calibration plane belongs to the world coordinate system, and the photographing plane belongs to the camera coordinate system;

[0009] Based on the rotation amount and the displacement amount, the corner points of the calibration plane are affinely converted to the photographing plane to obtain ideal corner points of the photographing plane; according to the error between the corner points of the photographing plane and the ideal corner points, a distortion error amount is determined; and the ideal corner points are corrected in distortion in combination with the distortion error amount to obtain target corner points in the camera coordinate system;

[0010] The distortion error amount is obtained by taking the distance between the corner points of the photographing plane and the ideal corner points as the distortion error amount of the ideal corner points.

[0011] When the ideal corner points are corrected in distortion to obtain the target corner points in the camera coordinate system, the calculation formula of the coordinates of the target corner points is:

[0012] ;

[0013] ;

[0014] Wherein, is the coordinate of the jth target corner point; is the weighted weight of the ith ideal corner point when the coordinate of the jth target corner point is calculated; is the coordinate of the ith ideal corner point; is the distortion error amount of the ith ideal corner point corresponding to the corner point of the photographing plane of the jth ideal corner point; β is the distance weight; and n is the number of corner points in the calibration plane.

[0015] According to the target corner points in the camera coordinate system and the corner points in the world coordinate system, final rotation amount and final displacement amount of the photographing plane relative to the calibration plane are determined, and the final rotation amount and the final displacement amount are transmitted to a calibration system to obtain the position of the camera.

[0016] Further, corner point detection is performed on the calibration plane to obtain the corner points of the calibration plane, including:

[0017] When the corner point detection is performed, the corner point response value during the corner point detection is weighted by the pixel features around the pixel points, and the corner points of the calibration plane are obtained based on the corner point response value of each corner point after weighting.

[0018] Further, the corner point response value during the corner point detection is weighted by the pixel features around the pixel points, including:

[0019] Using a 3×3 window size, a local window is constructed corresponding to each pixel. The grayscale changes in the horizontal, vertical, left diagonal, and right diagonal directions passing through the center pixel within the local window are analyzed. For the horizontal direction, the sum of the squares of the differences between the center pixel and the pixels on both sides of the horizontal direction is calculated as the directional feature value of the center pixel in the horizontal direction. The weight value of the center pixel and the directional feature value of the center pixel in the horizontal direction are multiplied to obtain the target feature value of the center pixel in the horizontal direction.

[0020] Calculate the target feature values ​​of the center pixels in the horizontal, vertical, left diagonal, and right diagonal directions of the local window respectively, and use the minimum target feature value as the weighted feature value of the center pixels of the local window;

[0021] The corner response values ​​in the corner detection algorithm are weighted using the weighted feature values ​​as weights.

[0022] Furthermore, the method for obtaining the pitch angle corresponding to the origin of the camera coordinate system and the world coordinate system is as follows:

[0023] Let the coordinates of the origin of the camera coordinate system in the world coordinate system be... Calculate the origin of the camera coordinate system The line connecting the origin of the world coordinate system and The angle between the axes serves as the pitch angle; where the camera coordinate system and the world coordinate system are... The axes are parallel.

[0024] Furthermore, the method for obtaining the orientation angles corresponding to the origins of the camera coordinate system and the world coordinate system is as follows:

[0025] Obtain the origin of the world coordinate system from the origin of the camera coordinate system. The connection is The projection line of the plane is used as the direction angle between the projection line and the x-axis.

[0026] Further, determining the rotation of the photographic plane relative to the calibration plane based on the pitch and azimuth angles corresponding to the origins of the camera coordinate system and the world coordinate system includes:

[0027] The sum of the pitch and azimuth angles is used as the rotation of the photographic plane relative to the calibration plane.

[0028] Further, determining the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system includes:

[0029] Calculate the Euclidean distance between the origins of the camera coordinate system and the world coordinate system, as the displacement of the photographic plane relative to the calibration plane.

[0030] In a second aspect, a camera tracking system for LED virtual shooting is provided, and the system comprises the following modules:

[0031] An angle point detection module is configured to, when the camera is in an anchor position, take an imaging plane where the LED screen is located as a calibration plane and take an imaging plane of the camera as a shooting plane; perform angle point detection on the calibration plane and the shooting plane to obtain angle points of the calibration plane and angle points of the shooting plane;

[0032] A parameter determination module is configured to determine a rotation amount of the shooting plane relative to the calibration plane according to a pitch angle and a direction angle corresponding to origins of a camera coordinate system and a world coordinate system; and determine a displacement amount of the shooting plane relative to the calibration plane according to a distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system, and the shooting plane belongs to the camera coordinate system.

[0033] A distortion correction module is configured to, based on the rotation amount and the displacement amount, perform affine conversion of the angle points of the calibration plane to the shooting plane to obtain ideal angle points of the shooting plane; determine a distortion error amount according to errors between the angle points of the shooting plane and the ideal angle points; and perform distortion correction on the ideal angle points in combination with the distortion error amount to obtain target angle points in the camera coordinate system.

[0034] The distortion error amount is obtained by taking a distance between the angle points of the shooting plane and the ideal angle points as a distortion error amount of the ideal angle points.

[0035] When the distortion correction is performed on the ideal angle points to obtain the target angle points in the camera coordinate system, a calculation formula of coordinates of the target angle points is:

[0036]

[0037]

[0038] wherein, is a coordinate of the jth target angle point; is a weighted weight of the ith ideal angle point when the coordinate of the jth target angle point is calculated; is a coordinate of the ith ideal angle point; is a distortion error amount of the ith ideal angle point corresponding to an angle point of the jth ideal angle point on the shooting plane; β is a distance weight; and n is a number of angle points in the calibration plane.

[0039] A position determination module is configured to determine a final rotation amount and a final displacement amount of the shooting plane relative to the calibration plane according to the target angle points in the camera coordinate system and the angle points in the world coordinate system, transmit the final rotation amount and the final displacement amount to a calibration system, and obtain a position of the camera.​​

[0040] In a third aspect, an electronic device is provided, comprising a memory and a processor, the memory storing executable code, and the processor executing the executable code to implement the embodiments of any possible implementation of the first aspect.

[0041] In a fourth aspect, a computer program product is provided, comprising computer program code which, when run on a computer, causes the computer to perform the method of the first aspect or any possible implementation of the first aspect.

[0042] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program, which, when executed on a computer, causes the computer to perform the embodiments of any possible implementation of the first aspect.

[0043] The embodiments of the present application have at least the following beneficial effects:

[0044] The present application determines the conversion parameters of the affine conversion of the photographing plane relative to the calibration plane by analyzing the distance and relative pose between the camera coordinate system and the world coordinate system, so as to convert the corner points in the two coordinate systems by the conversion parameters, and determine a basic corner point which may exist distortion error, which is also the ideal corner point of the photographing plane. However, since the camera imaging will exist distortion problem, the present application determines the distortion error caused by the camera imaging as the distortion error amount by analyzing the error between the real corner point and the ideal corner point on the calibration plane and the photographing plane. And the ideal corner point is corrected by the determined distortion error amount to obtain the target corner point in the camera coordinate system after distortion correction. The final rotation amount and the final displacement amount of the photographing plane relative to the calibration plane are determined by combining the target corner point and the corner point in the world coordinate system, and the final rotation amount and the final displacement amount are transmitted to the calibration system to obtain the position of the camera, so that the angle and position of the virtual background on the LED screen are adjusted in real time to keep synchronization with the camera, and the realistic shooting effect is realized. The present application considers the error of the camera imaging distortion on the positioning by analyzing the transformation between the two coordinate systems, and improves the accuracy of the camera positioning. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0046] Figure 1 A method flowchart of a camera tracking method for LED virtual shooting provided by an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a world coordinate system and a calibration plane provided by an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a camera coordinate system, a world coordinate system, a calibration plane and a shooting plane provided by an embodiment of the present application;

[0049] Figure 4 A schematic diagram of an image coordinate system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purposes, the following describes the camera tracking method and system for LED virtual shooting according to the present application, its specific implementation, structure, features and effects in detail in combination with the preferred embodiments and the accompanying drawings.

[0051] In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0052] In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, in addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0053] Hereinafter, the terms "first", "second" are only used for description purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs.

[0055] The embodiments of the present application are described below in combination with the accompanying drawings. Those skilled in the art can know that with the development of technology and the appearance of new scenes, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] The embodiment of the present application provides a specific implementation method of a camera tracking method and system for LED virtual shooting.

[0057] The specific scheme of the camera tracking method and system for LED virtual shooting provided by the present application will be described in detail below with reference to the drawings.

[0058] Please refer to Figure 1 which shows a step flow chart of the camera tracking method for LED virtual shooting provided by the embodiment of the present application, and the method comprises the following steps:

[0059] In step S100, when the camera is in an anchor position, the imaging plane where the LED screen is located is taken as a calibration plane, and the imaging plane of the camera is taken as a camera plane; corner point detection is performed on the calibration plane and the camera plane to obtain the corner points of the calibration plane and the corner points of the camera plane.

[0060] The core role of the camera tracking technology is to ensure that the virtual background remains consistent under different shooting angles in the LED virtual shooting process, and to ensure that the light and shadow, perspective and space sense between the objects and actors in the scene and the background can be accurately matched.

[0061] In the camera tracking method related to the embodiment of the present application, the initial relative position of the camera for virtual shooting relative to the LED screen needs to be obtained first, and the initial relative position is taken as an anchor position to track the position of the camera after movement.

[0062] Therefore, in step S100, the anchor position of the camera for virtual shooting relative to the LED screen is determined. In the embodiment of the present application, the LED screen is considered as a single flat plane, so the anchor position of the camera should be located on the extension line of the screen center point, and the distance from the LED screen is selected by the experiment personnel according to the scene and shooting requirements.

[0063] It can be known that although the camera shoots three-dimensional objects, it is a two-dimensional plane after imaging; meanwhile, the spatial position of the LED screen will not change in the process of virtual shooting, so there is a spatial mapping relationship between the camera imaging plane and the plane of the LED screen.

[0064] The imaging plane where the LED screen is located is taken as a calibration plane, a spatial coordinate is established, a distortion relationship between the corner points in the imaging plane and the corresponding corner points in the calibration plane is found, a deviation between ideal corner point coordinates and actual corner point coordinates is calculated, and the actual position and posture of the camera in space are restored through the distortion deviation.

[0065] First, corner point detection is performed on the calibration plane in the embodiment of the application, and the calibration plane is the imaging plane of the LED screen when the camera is in an anchor position. Similarly, corner point detection is also performed on the imaging plane captured by the camera. The detected corner points are taken as objects for subsequent restoration of the spatial position information of the camera.

[0066] The corner points of the calibration plane and the imaging plane are matched one by one, and the matched corner points are considered as points having the same relative position in the two planes.

[0067] Some pixels in the LED screen have strong features, specifically, the pixels in the corners of some pattern edges, the pixels on the edges of images, and the like. They are usually local maximum values of the curvature on the pattern contour.

[0068] Therefore, in the preferred embodiment of the application, the step of performing corner point detection on the calibration plane and the imaging plane is adaptively adjusted. When the corner point detection is performed, the corner point response value during the corner point detection is weighted through the pixel features around the pixel point, and the corner points of the imaging plane are obtained based on the corner point response value of each corner point after weighting. More specifically:

[0069] Each pixel point is taken as the center of a window, and a 3x3 window size is taken to form a local window corresponding to each pixel point. The gray level changes in the horizontal direction, the vertical direction, the left diagonal direction, and the right diagonal direction of the center pixel point in the local window are analyzed respectively to determine the weighted feature value corresponding to the center pixel point. Before the weighted feature value corresponding to the pixel point is determined, a weight value is assigned to each pixel point in the image. The weight value is determined according to the distribution of the Gaussian function, and different weight values are assigned to each pixel point in the image. The weight value of the center point of the image is high, and the weight values of the points around the center point are low. The reason for assigning a high weight value to the center point of the image is that the captured picture is usually located in the middle position of the image.

[0070] Taking the horizontal direction as an example, the horizontal direction passing through the center pixel point is analyzed. Specifically, the sum of squares of the differences between the center pixel point and the pixel points on both sides of the horizontal direction in the local window is calculated as the directional feature value of the center pixel point in the horizontal direction. The weight value of the center pixel point is multiplied by the directional feature value of the center pixel point in the horizontal direction to obtain the target feature value of the center pixel point in the horizontal direction.

[0071] Then, the target feature values ​​of the center pixel points in the horizontal, vertical, left diagonal, and right diagonal directions of the local window are calculated respectively.

[0072] Taking a local window size of 3×3 as an example, the formula for calculating the target feature value of pixels in the horizontal direction within the local window is as follows:

[0073] ;in, pixels in the horizontal direction The target feature value; For pixels The weight value; For pixels In the local window, horizontally excluding pixels The u-th pixel outside the local window; where U is the size of the local window after subtracting a constant 1 from the side length of the local window. When the side length of the local window is 3, the value of U is 2.

[0074] Finally, the smallest target feature value is used as the weighted feature value of the center pixel of the local window.

[0075] The weighted feature values ​​are used as weights to weight the corner response values ​​in the corner detection algorithm. It should be noted that the corner detection algorithm is a well-known technique to those skilled in the art, and the acquisition of the corner response values ​​in the corner detection algorithm is also a well-known technique to those skilled in the art, and will not be elaborated upon here.

[0076] Step S100 detects corner points in the calibration plane. Corner point detection is a crucial step in camera calibration, and its results directly affect the calibration accuracy of the camera. Subsequent steps involve obtaining corner points on the camera's imaging plane using the same process, which will not be elaborated upon here. In this embodiment of the invention, the camera's imaging plane is also referred to as the imaging plane.

[0077] Step S200: Determine the rotation of the camera plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; determine the displacement of the camera plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein, the calibration plane belongs to the world coordinate system and the camera plane belongs to the camera coordinate system.

[0078] Next, establish the world coordinate system. And the calibration plane With the world coordinate system Parallelism, in this embodiment of the invention, takes the origin of the world coordinate system as the center point of the LED screen, and sets the coordinates of any corner point as... Please see. Figure 2 ,Figure 2 The schematic diagram of the world coordinate system and the calibration plane.

[0079] The imaging plane of the LED screen is also the calibration plane , which is the coordinate of the corner point on the calibration plane in the world coordinate system. The world coordinate system can be regarded as the reference system of the objective world of the space in the studio.

[0080] It can be seen that the camera can be regarded as a point in the three-dimensional space, but the real picture will form an imaging plane at the position of the camera in the space through the mapping of the camera lens, and the imaging plane can represent the position and posture of the current camera, and the imaging plane is also the imaging plane of the LED screen.

[0081] If the positional relationship between the LED screen and the camera is directly calculated in the world coordinate system, that is, the spatial relationship between the calibration plane and the imaging plane is calculated, the relationship needs to be described by the normal vectors of the two planes and the points on the two planes, and such calculation is too complex. Therefore, another camera coordinate system can be established, wherein the imaging plane belongs to the camera coordinate system, the imaging plane is used to represent the imaging plane obtained by the camera shooting, and the camera is positioned through the positional relationship between the camera coordinate system and the world coordinate system.

[0082] Therefore, after the world coordinate system is established, the camera coordinate system is established , the imaging plane of the camera is , that is, the imaging plane is , and the imaging plane of the camera is parallel to the of the camera coordinate system. In the embodiment of the application, the origin of the camera coordinate system is the center point of the imaging image of the camera when the camera is in the anchor position, and in other embodiments, the origin of the world coordinate system and the camera coordinate system established by the implementer can also be adjusted correspondingly; the coordinates of the corner point on the imaging plane corresponding to the corner point on the calibration plane are . Please refer to Figure 3 , Figure 3 for the schematic diagram of the camera coordinate system, the world coordinate system, the calibration plane and the imaging plane; Figure 3 In the above, the camera coordinate system, the calibration plane is parallel to the plane, the axis is perpendicular to the calibration plane, the world coordinate system provides an objective reference for the camera and the calibration plane, and the calibration plane is taken as an arbitrary corner point on the LED screen for example, the corresponding corner point on the imaging plane is So the next thing to do is to find the conversion relationship between m and M, and solve the conversion relationship, which is equivalent to obtain the position and angle of the camera when shooting the picture.

[0083] First, the points in two different coordinate systems are subjected to affine transformation, and the corner point M on the calibration plane and the corner point m matched with M on the photographic plane are analyzed, and the corner point M and the corner point m are converted into homogeneous coordinates in two coordinate systems: and Then the conversion relationship between and can be expressed by the following formula: ; wherein represents the conversion parameter of the affine transformation of the corner point from the world coordinate system to the camera coordinate system.

[0084] Further analysis of the conversion parameter A, if you want to measure the spatial relationship between the two planes, you need to consider two parts: the distance between the two planes and the relative angle between the two planes. Therefore, the conversion parameter A should contain two dimensions of measurement, i.e. the displacement amount and the relative rotation amount between the two planes.

[0085] Then the conversion parameter A is expanded as:

[0086] ; wherein represents the rotation amount of the photographic plane relative to the calibration plane; represents the displacement amount of the photographic plane relative to the calibration plane.

[0087] Further, the parameters and are analyzed in detail. First, the rotation amount , since the calibration plane is parallel to the plane of the world coordinate system, and the photographic plane is parallel to the plane of the camera coordinate system, so the rotation amount can be calculated by the angle relationship between the coordinate systems. Specifically, it is known that the rotation amount should contain two parts: the azimuth angle and the pitch angle . Among them, the azimuth angle can represent the "left and right" position relationship of the camera relative to the LED screen, and the pitch angle can represent the "high and low" position relationship of the camera relative to the LED screen.

[0088] Therefore, according to the pitch angle and the azimuth angle corresponding to the origin of the camera coordinate system and the world coordinate system, the rotation amount of the photographic plane relative to the calibration plane is determined.

[0089] Taking the origin of the camera coordinate system and the origin of the world coordinate system as an example, let the coordinates of the origin of the camera coordinate system in the world coordinate system be... The corresponding rotation amount The calculation is divided into two parts:

[0090] Part 1: Pitch Angle The meaning is: due to the difference between the world coordinate system and the camera coordinate system Since the axes are parallel, the line connecting the origin of the camera coordinate system and the origin of the world coordinate system is parallel to the axis. The angle between the axes can represent the angle between the camera coordinate system and the world coordinate system. Axis perpendicular Rotation offset in the planar direction. The origin of the world coordinate system is (0,0,0).

[0091] Part Two: Azimuth The meaning is: the origin of the camera coordinate system. In addition to the above There is an offset in the planar direction, and it is still rotating. If there is a deflection in the measurement of the axis, then the azimuth angle can be expressed as a distance from the origin of the world coordinate system to the point. The connection, in The projection lines of the plane, and The angle between the axes is used to represent the spatial orientation of the camera coordinate system relative to the world coordinate system, and thus the camera's imaging plane. With calibration plane Spatial relationships between them.

[0092] Rotation The calculation method is as follows:

[0093]

[0094]

[0095]

[0096] in, The pitch angle; θ is the azimuth angle; cos is the cosine function; tan is the tangent function; The x-coordinate of the origin of the camera coordinate system relative to the origin of the world coordinate system; The ordinate of the origin of the camera coordinate system relative to the origin of the world coordinate system; The z-coordinate is the coordinate of the origin of the camera coordinate system in the world coordinate system.

[0097] The rotation amount calculated above The spatial position relationship between the camera and the LED screen is represented, and the spatial distance relationship between the camera and the LED screen is represented by a displacement amount. Therefore, the displacement amount is also represented by the Euclidean distance between the origin of the camera coordinate system and the origin of the world coordinate system . In order to determine the displacement amount of the photographing plane relative to the calibration plane according to the distance between the origins of the camera coordinate system and the world coordinate system.

[0098] In some embodiments, the displacement amount of the photographing plane relative to the calibration plane is calculated according to the following formula: .

[0099] Step S300, on the basis of the rotation amount and the displacement amount, the corner points of the calibration plane are affinely converted to the photographing plane to obtain ideal corner points of the photographing plane; according to the error between the corner points and the ideal corner points of the photographing plane, a distortion error amount is determined; and the ideal corner points are corrected for distortion in combination with the distortion error amount to obtain target corner points in the camera coordinate system.

[0100] The conversion between the photographing plane and the calibration plane can be completed through coordinate transformation. The core logic on which the above method can be established is that light propagates along a straight line. However, in reality, there are also imaging distortion errors caused by camera internal reasons such as camera lenses, and the accuracy requirement for camera tracking and positioning is very high in LED virtual shooting. Therefore, based on the above transformation, the imaging distortion also needs to be eliminated.

[0101] After completing the two-plane transformation of the above step, any corner point in the world coordinate system is transformed to the camera coordinate system through rotation and translation, becoming . Therefore, the error distance between the transformed point and the actual point can be regarded as a measure of distortion. Therefore, next, an image coordinate system is established, which is in the same plane as the calibration plane, that is, a two-dimensional coordinate system is established on the calibration plane as the image coordinate system. Please refer to Figure 4 , Figure 4 for a schematic diagram of the image coordinate system.

[0102] Taking a corner point on the calibration plane, that is, the LED screen as an example, after calculation through the above transformation model, the corner point of the calibration plane is affinely converted to the photographing plane on the basis of the rotation amount and the displacement amount to obtain the ideal corner point of the photographing plane. For example, for the corner point on the calibration plane, the corresponding point of the corner point on the photographing plane of the camera is the corner point Based on the rotation and displacement, the corner points of the calibration plane will be... Affine transformation to the photographic plane yields the ideal corner points of the photographic plane. Then calculate the ideal corner point. Points corresponding to the image on the photographic plane obtained by a camera in reality The distance between them is taken as the lens distortion error generated when the camera shoots, and the amount of distortion error is set as follows: .

[0103] Distortion error The calculation formula is: .

[0104] Distortion error This represents a single point on the LED screen. The degree of distortion appearing on the camera. This method can be used to calculate the degree of distortion at all corner points on the calibration plane. Having obtained the degree of distortion at all corner points in the imaging plane captured by the current camera, the next step is to eliminate the distortion in the camera's image.

[0105] Points on the LED screen under the calibration plane Recorded as the origin point The point corresponding to the original point on the photographic plane after calculation using the above coordinate transformation. Denote as the ideal point The corresponding points in the photographic image on the photographic plane actually captured by the camera. Recorded as actual points Taking the distortion correction of the j-th ideal corner point as an example, the method for distortion correction of the ideal corner point is as follows:

[0106] ;

[0107] ;

[0108] in, These are the coordinates of the corrected ideal corner point, which are also the coordinates of the j-th target corner point; The weighted weight of the i-th ideal corner point when calculating the coordinates of the j-th target corner point; Let i be the coordinates of the i-th ideal corner point; β represents the distortion error between the corner point of the photographic plane corresponding to the j-th ideal corner point and the i-th ideal corner point, which is also the Euclidean distance between the coordinate positions of the actual corner point corresponding to the j-th ideal corner point and the i-th ideal corner point; β is the distance weight. In this embodiment of the invention, the distance weight is set manually by the implementer according to the actual situation; n is the number of corner points in the calibration plane. It should be noted that when weighting the ideal corner points using the weighted average of the ideal corner points, the weighted average of the ideal corner points is applied to the horizontal and vertical coordinates of the ideal corner points respectively to obtain the weighted horizontal and vertical coordinates of the ideal corner points. In this embodiment of the invention, It actually contains the x and y coordinates of the j-th target corner point, which is represented here by a character. Let represent the coordinates of the j-th target corner point.

[0109] This formula indicates that for the th For an actual corner point, the ideal corner point can be obtained by calculation after coordinate transformation. To estimate it, specifically, it can be calculated using all the transformed ideal corner points, therefore... ,in This means that when calculating the coordinates of the j-th target corner point, the... The weighted values ​​of the ideal corner points are calculated by weighting all the ideal corner points to obtain the corrected actual corner points. (Regarding the weighted values...) The logic is as follows: Using the first... The distortion error is measured by the ratio of the distortion error at the j-th ideal corner point and the j-th actual corner point to the sum of the distortion errors at all points. As mentioned above, the distortion error... The actual meaning is the distance between the ideal angle point and the actual angle point, so here... This means that the greater the distance between the ideal corner point and the actual corner point, the greater the distortion error. The larger the ideal corner point, the smaller the weight should be for the correction of the actual corner point, i.e., the weighted weight. With distortion error of The power is inversely proportional. Here about... The selection can be made by the implementers based on the characteristics of the LED screen background. Specifically, The smaller, the more... The closer to the average; The larger the value, the greater the weight of the reference point that is closer to the actual point.

[0110] Step S400: Based on the target corner point in the camera coordinate system and the corner point in the world coordinate system, determine the final rotation and final displacement of the camera plane relative to the calibration plane, and transmit the final rotation and final displacement to the calibration system to obtain the position of the camera.

[0111] After the ideal corner points are corrected for distortion, the rotation amount and displacement amount of the photographing plane relative to the calibration plane are calculated according to the target corner points obtained after the distortion correction and the corner points on the calibration plane. Specifically, in step S200 in the embodiment of the present application, the rotation amount and displacement amount of the photographing plane relative to the calibration plane are determined through the origin of the camera coordinate system and the world coordinate system. After the ideal corner points are corrected for distortion to obtain the target corner points, the rotation amount and displacement amount of the photographing plane relative to the calibration plane are determined through the target corner points of the camera coordinate system and the corner points in the world coordinate system. It should be noted that each pair of corner points and target corner points can obtain a corresponding set of rotation amount and displacement amount. In the embodiment of the present application, the mode of all rotation amounts of the pairs of corner points and target corner points is taken as the final rotation amount, and the average of all displacement amounts of the pairs of corner points and target corner points is taken as the displacement amount. It should be noted that when the rotation amount and displacement amount of the photographing plane relative to the calibration plane are determined through the target corner points of the camera coordinate system and the corner points in the world coordinate system, the corner points in the world coordinate system and the corresponding corner points in the camera coordinate system can be respectively taken as the origins of the coordinate systems for calculation.

[0112] The final rotation amount and final displacement amount calculated are transmitted to an existing calibration system, and the calibration system determines the position of the camera through the currently mapped position and angle and other information, adjusts the virtual background on the LED screen in real time to keep the angle and position of the camera synchronized with the virtual background, and realizes a realistic shooting effect. Compared with the existing camera positioning method through laser radar or sensors, the method of calculating the camera position through space mapping saves a large amount of cost, and also considers the error between the imaging picture and the actual positioning caused by lens distortion, thereby improving the accuracy.

[0113] The embodiment of the present application calculates the rotation amount and displacement amount of the photographing plane relative to the calibration plane through the transformation between the two coordinate systems, and considers the error in positioning caused by the imaging distortion of the camera.

[0114] The embodiment of the present application provides a camera tracking system for LED virtual shooting, and the system comprises:

[0115] A corner point detection module is configured to, when the camera is in an anchor position, take the imaging plane where the LED screen is located as a calibration plane, and take the imaging plane of the camera as a photographing plane; and perform corner point detection on the calibration plane and the photographing plane to obtain corner points of the calibration plane and corner points of the photographing plane.

[0116] The parameter determining module is configured to determine a rotation amount of the photographing plane relative to the calibration plane according to a pitch angle and a direction angle corresponding to origins of the camera coordinate system and the world coordinate system, and determine a displacement amount of the photographing plane relative to the calibration plane according to a distance between the origins of the camera coordinate system and the world coordinate system, wherein the calibration plane belongs to the world coordinate system, and the photographing plane belongs to the camera coordinate system.

[0117] The distortion correction module is configured to convert the corner points of the calibration plane to the photographing plane to obtain ideal corner points of the photographing plane based on the rotation amount and the displacement amount, determine a distortion error amount according to errors between the corner points of the photographing plane and the ideal corner points, and correct the ideal corner points based on the distortion error amount to obtain target corner points in the camera coordinate system.

[0118] The position determining module is configured to determine a final rotation amount and a final displacement amount of the photographing plane relative to the calibration plane according to the target corner points in the camera coordinate system and the corner points in the world coordinate system, and transmit the final rotation amount and the final displacement amount to a calibration system to obtain the position of the camera.

[0119] Optionally, the transmission medium can be a wired link such as, but not limited to, a coaxial cable, an optical fiber, a digital subscriber line, and the like, or a wireless link such as, but not limited to, Wireless Fidelity (WIFI), Bluetooth, and a mobile device network, and the like.

[0120] It should be noted that the apparatus provided in the above embodiments is only used as an example for the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above described functions.

[0121] An exemplary computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to enable the computer device to perform any of the above-described camera tracking methods for LED virtual shooting.

[0122] In addition, an apparatus is also provided in the embodiments of the present application, which can include a memory and a processor, wherein the memory stores executable program code, and the processor is configured to invoke and execute the executable program code to perform the camera tracking method for LED virtual shooting provided in the embodiments of the present application.

[0123] The embodiment of the present application can divide the functions of the device according to the above method examples, for example, each function module can be divided, two or more functions can be integrated in one processing module, and the integrated module can be realized in the form of hardware. It should be noted that the division of the modules in the embodiment is illustrative, and is only a logical function division. In actual implementation, another division mode can be used.

[0124] In the case of dividing each module according to each function, the device can further include a signal uploading module, a determination module, an adjustment module and the like. It should be noted that all related contents of each step involved in the above method embodiment can be cited to the function description of the corresponding function module, and will not be described here.

[0125] It should be understood that the device provided by the embodiment of the present application is used to execute the above camera tracking method for LED virtual shooting, and thus the same effect as the above implementation method can be achieved.

[0126] In the case of using an integrated unit, the device can include a processing module and a storage module. When the device is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes and the like. The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, a combination of digital signal processing (Digital Signal Processing, DSP) and microprocessors, and the like. The storage module can be a memory.

[0127] In addition, the device provided by the embodiment of the present application can be a chip, an assembly or a module. The chip can include a connected processor and a memory. The memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the camera tracking method for LED virtual shooting provided by the above embodiment.

[0128] The embodiment of the present application also provides a computer readable storage medium, which stores computer program codes. When the computer program codes run on a computer, the computer can execute the above related method steps to realize the camera tracking method for LED virtual shooting provided by the above embodiment.

[0129] The embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer can execute the above related steps to realize the camera tracking method for LED virtual shooting provided by the above embodiment.

[0130] Among them, the device, computer readable storage medium, computer program product or chip provided by the embodiment of the application are used to execute the corresponding method provided above, so the beneficial effects that can be achieved are referable to the beneficial effects of the corresponding method provided above, which will not be repeated here. Through the description of the above implementation mode, those skilled in the art can understand that, for the convenience and brevity of description, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the application, it should be understood that the disclosed device and method can be implemented in other ways.

[0131] The device embodiment described above is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0132] It should be further understood that the terms "comprise", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or terminal devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or terminal devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0133] It should be noted that: the above-mentioned sequence of the embodiments of the application is only for description, not representing the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0134] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments.

[0135] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A camera tracking method for LED virtual filming, characterized in that, The method comprises the following steps: When the camera is in the anchor position, taking the imaging plane where the LED screen is located as a calibration plane and taking the imaging plane of the camera as a photography plane; performing corner point detection on the calibration plane and the photography plane to obtain corner points of the calibration plane and corner points of the photography plane; According to the pitch angle and the direction angle corresponding to the origins of the camera coordinate system and the world coordinate system, the rotation amount of the photography plane relative to the calibration plane is determined; according to the distance between the origins of the camera coordinate system and the world coordinate system, the displacement amount of the photography plane relative to the calibration plane is determined; wherein the calibration plane belongs to the world coordinate system and the photography plane belongs to the camera coordinate system; Based on the rotation amount and the displacement amount, the corner points of the calibration plane are affinely converted to the photography plane to obtain ideal corner points of the photography plane; according to the error between the corner points of the photography plane and the ideal corner points, the distortion error amount is determined; the ideal corner points are corrected in distortion in combination with the distortion error amount to obtain target corner points in the camera coordinate system; The method for obtaining the distortion error amount is that the distance between the corner points of the photography plane and the ideal corner points is taken as the distortion error amount of the ideal corner points; When the ideal corner points are corrected in distortion to obtain the target corner points in the camera coordinate system, the calculation formula of the coordinates of the target corner points is: ； ； wherein, is the coordinate of the jth target corner point; is the weighted weight of the ith ideal corner point when calculating the coordinate of the jth target corner point; is the coordinate of the ith ideal corner point; is the distortion error amount of the jth ideal corner point corresponding to the corner point of the photographic plane and the ith ideal corner point; β is the distance weight; n is the number of corner points in the calibration plane; According to the target corner points in the camera coordinate system and the corner points of the world coordinate system, the final rotation amount and the final displacement amount of the photography plane relative to the calibration plane are determined, and the final rotation amount and the final displacement amount are transmitted to the calibration system to obtain the position of the camera.

2. The camera tracking method for LED virtual photography of claim 1, wherein, The corner points of the calibration plane are obtained by performing corner point detection on the calibration plane, comprising: When performing the corner point detection, the corner point response value during the corner point detection is weighted by pixel features around a pixel point, and the corner points of the calibration plane are obtained based on the corner point response value of each corner point after weighting.

3. The camera tracking method for LED virtual photography of claim 2, wherein, The corner point response value during the corner point detection is weighted by pixel features around a pixel point, comprising: A local window corresponding to each pixel point is formed with a window size of 3*3, and the gray level changes of the center pixel point in the horizontal direction, the vertical direction, the left diagonal direction and the right diagonal direction in the local window are analyzed respectively; for the horizontal direction, the sum of squares of the difference values of the center pixel point and the pixel points on both sides of the horizontal direction in the local window is calculated as the direction feature value of the center pixel point in the horizontal direction; the target feature value of the center pixel point in the horizontal direction is obtained by multiplying the weight value of the center pixel point and the direction feature value of the center pixel point in the horizontal direction; The target feature values of the center pixel point in the horizontal direction, the vertical direction, the left diagonal direction and the right diagonal direction of the local window are calculated respectively, and the smallest target feature value is taken as the weighted feature value of the center pixel point of the local window; The corner point response value in the corner point detection algorithm is weighted by taking the weighted feature value as the weight.

4. The camera tracking method for LED virtual photography of claim 1, wherein, The method for obtaining the pitch angle corresponding to the origins of the camera coordinate system and the world coordinate system comprises: Let the coordinates of the origin of the camera coordinate system in the world coordinate system be... Calculate the origin of the camera coordinate system The line connecting the origin of the world coordinate system and The angle between the axes serves as the pitch angle; where the camera coordinate system and the world coordinate system are... The axes are parallel.

5. The camera tracking method for LED virtual photography of claim 1, wherein, The method for obtaining the direction angle corresponding to the origins of the camera coordinate system and the world coordinate system comprises: Obtain the origin of the world coordinate system from the origin of the camera coordinate system. The connection is in The projection line of the plane is used as the direction angle between the projection line and the x-axis.

6. The camera tracking method for LED virtual photography of claim 1, wherein, The rotation amount of the photographing plane relative to the calibration plane is determined according to the pitch angle and the direction angle corresponding to the origins of the camera coordinate system and the world coordinate system, and the rotation amount of the photographing plane relative to the calibration plane is determined according to the distance between the origins of the camera coordinate system and the world coordinate system. The sum of the pitch angle and the direction angle is taken as the rotation amount of the photographing plane relative to the calibration plane.

7. The camera tracking method for LED virtual photography of claim 1, wherein, The displacement amount of the photographing plane relative to the calibration plane is determined according to the distance between the origins of the camera coordinate system and the world coordinate system. The Euclidean distance between the origins of the camera coordinate system and the world coordinate system is calculated as the displacement amount of the photographing plane relative to the calibration plane.

8. A camera tracking system for LED virtual filming, characterized in that, The system comprises the following modules: The corner point detection module is configured to, when the camera is in the anchor position, take the imaging plane where the LED screen is located as the calibration plane and take the imaging plane of the camera as the photographing plane. The corner points of the calibration plane and the photographing plane are detected to obtain the corner points of the calibration plane and the photographing plane. The parameter determination module is configured to determine the rotation amount of the photographing plane relative to the calibration plane according to the pitch angle and the direction angle corresponding to the origins of the camera coordinate system and the world coordinate system, and determine the displacement amount of the photographing plane relative to the calibration plane according to the distance between the origins of the camera coordinate system and the world coordinate system. The distortion correction module is configured to, based on the rotation amount and the displacement amount, perform affine conversion of the corner points of the calibration plane to the photographing plane to obtain ideal corner points of the photographing plane, determine a distortion error amount according to the error between the corner points of the photographing plane and the ideal corner points, and perform distortion correction on the ideal corner points in combination with the distortion error amount to obtain target corner points in the camera coordinate system. The distance between the corner points of the photographing plane and the ideal corner points is taken as the distortion error amount of the ideal corner points. When the distortion correction is performed on the ideal corner points to obtain the target corner points in the camera coordinate system, the calculation formula of the coordinates of the target corner points is: ； ； wherein, is the coordinate of the jth target corner point; is the weighted weight of the ith ideal corner point when calculating the coordinate of the jth target corner point; is the coordinate of the ith ideal corner point; is the distortion error amount of the jth ideal corner point corresponding to the corner point of the photographic plane and the ith ideal corner point; β is the distance weight; n is the number of corner points in the calibration plane; The position determination module is configured to determine the final rotation amount and the final displacement amount of the photographing plane relative to the calibration plane according to the target corner points in the camera coordinate system and the corner points in the world coordinate system, transmit the final rotation amount and the final displacement amount to a calibration system, and obtain the position of the camera.

Images