Camera tracking method and system for LED virtual shooting

Abstract

The invention relates to the technical field of LED virtual shooting, in particular to a camera tracking method and system for LED virtual shooting. The method comprises the steps of firstly obtaining angular points of a calibration plane and a photographing plane; according to pitch angles, direction angles and distances corresponding to the original points of the camera coordinate system and the world coordinate system, determining the rotation amount and the displacement amount of the photographing plane relative to the calibration plane, and performing affine conversion on angular points of the calibration plane to the photographing plane to obtain ideal angular points; determining a distortion error amount according to an error between the angular point of the photographing plane and the ideal angular point; and performing distortion correction on the ideal angular point to obtain a target angular point, determining the final rotation amount and the final displacement amount of the photographing plane relative to the calibration plane, and transmitting the final rotation amount and the final displacement amount to a calibration system to obtain the position of the camera. According to the invention, by analyzing the transformation between the two coordinate systems, the positioning error caused by the imaging distortion of the camera is considered, and the positioning accuracy of the camera is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of LED virtual shooting, and in particular to a camera tracking method and system for LED virtual shooting. Background Art

[0002] LED virtual filming technology is a new filming method that combines LED displays with virtual reality (VR) or augmented reality (AR). It involves constructing a large-scale LED display on set, displaying a virtual background image or a three-dimensional virtual scene. Actors and other real-life objects are filmed against this virtual backdrop. Because the LED screen displays a high-quality, dynamic background, actors and objects can interact with it in real time, avoiding the unnatural effects of traditional green-screen filming due to synthetic effects.

[0003] In LED virtual filming, camera tracking is a key technology for ensuring consistent interaction between virtual backgrounds and real-world actors and objects. By tracking the camera's position, angle, and movement in real time, camera tracking ensures the virtual background is synchronized with the camera's perspective, providing a seamless and immersive experience for actors and scenes.

[0004] Most existing camera tracking methods rely on tracking cameras or LiDAR to determine the camera's position and track the camera. However, using these high-precision sensors for camera tracking requires deploying a series of expensive tracking cameras or LiDARs, significantly increasing costs. Summary of the Invention

[0005] In order to solve the technical problem of high cost when using laser radar for camera tracking, the purpose of the present invention is to provide a camera tracking method and system for LED virtual shooting. The technical solutions adopted are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a camera tracking method for LED virtual shooting, the method comprising:

[0007] When the camera is in the anchor position, the imaging plane of the LED screen is used as the calibration plane, and the imaging plane of the camera is used as the photographic plane. Corner point detection is performed on the calibration plane and the photographic plane to obtain the corner points of the calibration plane and the photographic plane.

[0008] Determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system;

[0009] Based on the rotation and displacement, the corner points of the calibration plane are affine transformed to the photographic plane to obtain the ideal corner points of the photographic plane; the distortion error is determined based on the error between the corner points of the photographic plane and the ideal corner points; based on the distortion error, the ideal corner points are subjected to distortion correction to obtain the target corner points in the camera coordinate system;

[0010] According to the target corner points in the camera coordinate system and the corner points in the world coordinate system, the final rotation and displacement of the photographic plane relative to the calibration plane are determined, and the final rotation and displacement are transmitted to the calibration system to obtain the position of the camera.

[0011] Furthermore, corner point detection is performed on the calibration plane to obtain corner points of the calibration plane, including:

[0012] When performing corner detection, the corner response values during corner detection are weighted by the pixel features around the pixel, and the corner points of the calibration plane are obtained based on the weighted corner response values of each corner point.

[0013] Furthermore, weighting the corner point response value during corner point detection by using pixel features surrounding the pixel point includes:

[0014] With a window size of 3×3, a local window corresponding to each pixel is constructed. The grayscale changes in the horizontal, vertical, left diagonal, and right diagonal directions of the central pixel in the local window are analyzed respectively. For the horizontal direction, the sum of the squares of the differences between the central pixel in the local window and the pixels on both sides of the horizontal direction is calculated as the directional eigenvalue of the central pixel in the horizontal direction. The weight value of the central pixel is multiplied by the directional eigenvalue of the central pixel in the horizontal direction to obtain the target eigenvalue of the central pixel in the horizontal direction.

[0015] Calculate the target eigenvalues of the central pixel in the horizontal, vertical, left diagonal, and right diagonal directions of the local window respectively, and take the smallest target eigenvalue as the weighted eigenvalue of the central pixel of the local window;

[0016] The weighted eigenvalues are used as weights to weight the corner point response values in the corner point detection algorithm.

[0017] Furthermore, the method for obtaining the pitch angle corresponding to the origin of the camera coordinate system and the world coordinate system is:

[0018] Assume that the origin of the camera coordinate system in the world coordinate system is (x0, y0, z0), and calculate the angle between the line connecting the origin of the camera coordinate system (x0, y0, z0) and the origin of the world coordinate system and the y-axis as the pitch angle; where the y-axes of the camera coordinate system and the world coordinate system are parallel.

[0019] Furthermore, the method for obtaining the direction angle corresponding to the origin of the camera coordinate system and the world coordinate system is:

[0020] Get the projection line of the line connecting the origin of the world coordinate system to the origin of the camera coordinate system (x0, y0, z0) on the x-z plane, and use the angle between the projection line and the x-axis as the direction angle.

[0021] Furthermore, determining the rotation amount of the photographic plane relative to the calibration plane according to the pitch angle and the azimuth angle corresponding to the origin of the camera coordinate system and the world coordinate system includes:

[0022] The sum of the pitch angle and the azimuth angle is taken as the rotation of the photographic plane relative to the calibration plane.

[0023] Furthermore, determining the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system includes:

[0024] Calculate the Euclidean distance between the origin of the camera coordinate system and the world coordinate system as the displacement of the photography plane relative to the calibration plane.

[0025] Furthermore, determining the distortion error amount based on the error between the corner point of the photographic plane and the ideal corner point includes:

[0026] The distance between the corner point of the photographic plane and the ideal corner point is taken as the distortion error of the ideal corner point.

[0027] Furthermore, the distortion correction is performed on the ideal corner point in combination with the distortion error to obtain the target corner point in the camera coordinate system, including:

[0028]

[0029] Among them, X2′(i) is the coordinate of the target corner point; λ i is the weighted weight of the i-th ideal corner point; X1(i) is the coordinate of the i-th ideal corner point; α(i) is the distortion error of the i-th ideal corner point; β is the distance weight; n is the number of corner points in the calibration plane.

[0030] In a second aspect, a camera tracking system for LED virtual filming is provided, the system comprising the following modules:

[0031] A corner point detection module is used to detect corner points of the calibration plane and the photographic plane when the camera is in the anchor position, using the imaging plane of the LED screen as the calibration plane and the imaging plane of the camera as the photographic plane, and obtaining the corner points of the calibration plane and the photographic plane;

[0032] a parameter determination module, configured to determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; and to determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system;

[0033] The distortion correction module is used to perform an affine transformation of the corner points of the calibration plane to the photographic plane based on the rotation amount and the displacement amount to obtain the ideal corner points of the photographic plane; determine the distortion error amount based on the error between the corner points of the photographic plane and the ideal corner points; and perform distortion correction on the ideal corner points based on the distortion error amount to obtain the target corner points in the camera coordinate system;

[0034] The position determination module is used to determine the final rotation and final displacement of the photographic plane relative to the calibration plane based on the target corner point in the camera coordinate system and the corner point of the world coordinate system, and transmit the final rotation and final displacement to the calibration system to obtain the position of the camera.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, various possible implementations of the first aspect are implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes: computer program code, which, when running on a computer, enables the computer to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0037] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementations of the first aspect.

[0038] The embodiments of the present invention have at least the following beneficial effects:

[0039] The present invention analyzes the distance and relative posture between the camera coordinate system and the world coordinate system to determine the transformation parameters for the affine transformation of the photographic plane relative to the calibration plane. These parameters are then used to transform the corner points in the two coordinate systems, determining an ideal base corner point that may contain distortion errors. This base corner point is also the ideal corner point of the photographic plane. However, since camera imaging can be subject to distortion, the present invention analyzes the errors between the actual corner points and the ideal corner points in the calibration and photographic planes to determine the distortion error caused by camera imaging, which serves as the distortion error amount. The ideal corner point is then corrected for distortion using the determined distortion error amount, obtaining the distortion-corrected target corner point in the camera coordinate system. Combining the target corner point with the corner point in the world coordinate system, the final rotation and displacement of the photographic plane relative to the calibration plane are determined. These final rotation and displacement values are then transmitted to a calibration system to determine the camera position. The virtual background on the LED screen is then adjusted in real time to synchronize with the camera's angle and position, achieving a realistic shooting effect. The present invention analyzes the transformation between the two coordinate systems, takes into account the error caused by camera imaging distortion in positioning, and improves the accuracy of camera positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A flowchart of a camera tracking method for LED virtual shooting provided by one embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a world coordinate system and a calibration plane provided by one embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a camera coordinate system, a world coordinate system, a calibration plane, and a photographic plane provided by one embodiment of the present invention;

[0044] Figure 4 A schematic diagram of an image coordinate system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, characteristics and effects of the camera tracking method and system for LED virtual shooting proposed in accordance with the present invention.

[0046] In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0047] In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.

[0048] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0050] The embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0051] Embodiments of the present invention provide a camera tracking method and system for LED virtual photography. This method is applicable to LED virtual photography scenarios consisting of an LED screen and a camera. The camera captures the image of the LED screen, which displays a high-quality dynamic background. The camera tracks the virtual background to ensure synchronization with the camera's perspective.

[0052] The specific scheme of the camera tracking method and system for LED virtual shooting provided by the present invention is described in detail below with reference to the accompanying drawings.

[0053] See also Figure 1 , which shows a flowchart of a camera tracking method for LED virtual shooting provided by one embodiment of the present invention, the method comprising the following steps:

[0054] Step S100: When the camera is in the anchor position, the imaging plane of the LED screen is used as the calibration plane, and the imaging plane of the camera is used as the photographic plane; corner point detection is performed on the calibration plane and the photographic plane to obtain the corner points of the calibration plane and the photographic plane.

[0055] The core role of camera tracking technology is to ensure that the virtual background remains consistent under different shooting angles during LED virtual shooting, and to ensure that the light, shadow, perspective and spatial sense between the objects and actors in the scene and the background are accurately matched.

[0056] In the camera tracking method involved in the embodiment of the present invention, it is first necessary to obtain the initial relative position of the camera used for virtual shooting relative to the LED screen, and use the initial relative position as an anchor position to perform subsequent position tracking of the camera after movement.

[0057] Therefore, in step S100, the anchor position of the camera used for virtual filming relative to the LED screen is determined. In this embodiment of the present invention, the LED screen is considered a single, flat surface, so the camera's anchor position should be located on the extension line of the screen's center point. The distance from the LED screen can be selected by the experimenter based on the scene and filming requirements.

[0058] It can be seen that although the camera shoots a three-dimensional object, the image formed by the camera is a two-dimensional plane; at the same time, during the virtual shooting process, the spatial position of the LED screen will not change, so there is a spatial mapping relationship between the camera imaging plane and the plane of the LED screen.

[0059] The imaging plane of the LED screen is used as the calibration plane. By establishing spatial coordinates, the distortion relationship between the corner points in the imaging plane and the corresponding corner points in the calibration plane is found, and the deviation between the ideal corner point coordinates and the actual corner point coordinates is calculated. The actual position and posture of the camera in space are restored through the distortion deviation.

[0060] First, in this embodiment of the present invention, corner point detection is performed on the calibration plane, which is the imaging plane of the LED screen when the camera is in the anchor position. Similarly, corner point detection is performed on the photographic plane captured by the camera. The detected corner points are used as the basis for subsequent restoration of the camera's spatial position information.

[0061] The corner points of the calibration plane and the photographic plane are matched one by one, and the matched corner points are considered to be the points with the same relative positions in the two planes.

[0062] Because some pixels in the LED screen have strong characteristics, specifically pixels at the corners of certain patterns, pixels on the edge of the image, etc. They are usually the local maximum value of the curvature on the pattern outline.

[0063] Therefore, as a preferred embodiment of the present invention, the steps of performing corner point detection on the calibration plane and the photographic plane are adaptively adjusted. When performing corner point detection, the corner point response values during corner point detection are weighted by the pixel features surrounding the pixel point. Based on the weighted corner point response values of each corner point, the corner points of the imaging plane are obtained. More specifically:

[0064] Each pixel is used as the center of a window with a size of 3×3, forming a local window corresponding to each pixel. The grayscale changes in the local window through the center pixel in the horizontal, vertical, left diagonal, and right diagonal directions are analyzed to determine the weighted eigenvalue corresponding to the center pixel. Before determining the weighted eigenvalue corresponding to the pixel, each pixel in the image is assigned a weight value. The weight value is determined based on the distribution of the Gaussian function, and each pixel in the image is assigned a different weight value. The center of the image has a higher weight value than the surrounding pixels. The reason for assigning a high weight value to the center of the image is that the captured scene is usually located in the middle of the image.

[0065] Taking the horizontal direction as an example, the analysis is performed along the horizontal direction passing through the center pixel. Specifically, the sum of the squares of the differences between the center pixel and the pixels on both sides of the horizontal direction in the local window is calculated as the directional eigenvalue of the center pixel in the horizontal direction. The weight value of the center pixel is multiplied by the directional eigenvalue of the center pixel in the horizontal direction to obtain the target eigenvalue of the center pixel in the horizontal direction.

[0066] Then, the target feature values of the central pixel in the horizontal, vertical, left diagonal and right diagonal directions of the local window are calculated respectively;

[0067] Among them, taking the size of the local window as 3×3 and the horizontal direction within the local window as an example, the calculation formula for the target feature value of the pixel point in the horizontal direction is:

[0068] Among them, S(x,y) is the target feature value of the pixel point (x,y) in the horizontal direction; ω(x,y) is the weight value of the pixel point (x,y); I(x,y) u It is the u-th pixel point in the horizontal direction of the local window of the pixel point (x, y), excluding the pixel point (x, y). The value of U is the size of the side length of the local window minus the constant 1. When the side length of the local window is 3, the value of U is 2.

[0069] Finally, the minimum target eigenvalue is used as the weighted eigenvalue of the central pixel of the local window.

[0070] The weighted eigenvalues are used as weights to weight the corner response values in the corner detection algorithm. It should be noted that the corner detection algorithm is well known to those skilled in the art, and obtaining the corner response values in the corner detection algorithm is also well known to those skilled in the art and will not be described in detail here.

[0071] Step S100 detects corner points in the calibration plane. Corner point detection is a key step in camera calibration, and its results directly impact the calibration accuracy. In subsequent steps, corner points of the camera's imaging plane are acquired using the same process described above, and will not be further elaborated here. In this embodiment of the present invention, the camera's imaging plane is also referred to as the photographic plane.

[0072] Step S200: Determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system.

[0073] Next, establish the world coordinate system X w ―Y w ―Z w , and calibrate the plane Ω p X of the world coordinate system w ―O―Y w In the embodiment of the present invention, the origin of the world coordinate system is taken as the center point of the LED screen, and the coordinates of any corner point are set as M=(x w ,y w ,z w ). See Figure 2 , Figure 2 Schematic diagram of the world coordinate system and calibration plane.

[0074] The imaging plane where the LED screen is located is also the calibration plane Ω p , in the world coordinate system, it is the X w ―O―Y w The world coordinate system can be used as a reference system for the objective world in the studio space.

[0075] It can be seen that the camera can be regarded as a point in three-dimensional space, but the real picture will form an imaging plane at the position of the camera in space through the mapping of the camera lens. This imaging plane can represent the current position and posture of the camera. This imaging plane is also the imaging plane where the LED screen is located.

[0076] If the orientation relationship between the LED screen and the camera is calculated directly in the world coordinate system, that is, the spatial relationship between the calibration plane and the photographic plane is calculated, the relationship needs to be described using the normal vectors of the two planes and the points on the two planes. This calculation is too complicated. Therefore, another camera coordinate system can be established, in which the photographic plane belongs to the camera coordinate system. The photographic plane is used to represent the imaging plane captured by the camera, and the camera is positioned based on the orientation relationship between the camera coordinate system and the world coordinate system.

[0077] So after establishing the world coordinate system, establish the camera coordinate system X c ―Y c ―Z c , the imaging plane of the camera is Ω c , that is, the photographic plane is Ω c , and the imaging plane of the camera is aligned with the X coordinate system of the camera c ―O―Y c In the embodiment of the present invention, the origin of the camera coordinate system is the center point of the image formed by the camera when the camera is in the anchor position. In other embodiments, the origins of the world coordinate system and the camera coordinate system can be adjusted accordingly by the implementer; the photographic plane Ω c The corner point M on the calibration plane is (x w ,y w ,z w ) corresponds to the coordinates of the corner point m=(x c ,y c ,z c ). See Figure 3 , Figure 3 Schematic diagram of the camera coordinate system, world coordinate system, calibration plane and photographic plane; Figure 3 In, X c ―Y c ―Z c is the camera coordinate system, and the calibration plane Ω c With X c ―O―Y c Plane parallel, Z c The axis is perpendicular to the calibration plane, X w ―Y w ―Z w The world coordinate system provides an objective reference for the camera and the calibration plane, and the calibration plane is an arbitrary corner point M=(x w ,y w ,z w ) as an example, the corresponding corner point on the photographic plane is m=(x c ,y c ,z c), so the next thing we need to do is to find the conversion relationship between m and M. Solving the conversion relationship is equivalent to obtaining the position and angle of the camera when shooting the picture.

[0078] First, to perform affine transformation on the points in two different coordinate systems, it is necessary to analyze the corner point M on the calibration plane and the corner point m that matches M on the photographic plane, and convert the corner point M and the corner point m into homogeneous coordinates in the two coordinate systems: (x w ,y w ,z w ,1) and (x c ,y c ,z c ,1). Then (x w ,y w ,z w ,1) and (x c ,y c ,z c ,1) can be expressed by the following formula: Among them, A represents the conversion parameter when the corner point is transformed from the world coordinate system to the camera coordinate system.

[0079] Analyzing the transformation parameter A further, if we want to measure the spatial relationship between two planes, we need to consider two components: the distance between the two planes and the relative angle between them. Therefore, the transformation parameter A should include a two-dimensional measurement: the displacement between the two planes and the relative rotation.

[0080] Then the conversion parameter A is expanded to:

[0081] Where R represents the rotation of the photographic plane relative to the calibration plane; T represents the displacement of the photographic plane relative to the calibration plane.

[0082] Furthermore, we analyze the parameters R and T in detail. First, the rotation R, due to the calibration plane Ω p X of the world coordinate system w ―O―Y w Surface parallel, photographic plane Ω c X of the camera coordinate system c ―O―Y c The planes are parallel, so the rotation R can be calculated by the angular relationship between the coordinate systems. Specifically, it can be seen that the rotation should contain two parts: azimuth And the pitch angle θ. The azimuth angle can represent the "left and right" position relationship of the camera relative to the LED screen, while the pitch angle can represent the "high and low" position relationship of the camera relative to the LED screen.

[0083] Therefore, the rotation of the photographic plane relative to the calibration plane is determined according to the pitch angle and direction angle corresponding to the origin of the camera coordinate system and the world coordinate system.

[0084] Taking the origin of the camera coordinate system and the origin of the world coordinate system as an example, assuming that the origin coordinates of the camera coordinate system in the world coordinate system are (x0, y0, z0), the calculation of the corresponding rotation amount R is divided into two parts:

[0085] The first part, the pitch angle θ, means that since the world coordinate system is parallel to the y-axis of the camera coordinate system, the angle between the line connecting the origin of the camera coordinate system and the origin of the world coordinate system and the y-axis can be calculated to represent the rotation offset of the camera coordinate system relative to the world coordinate system in the x-z plane perpendicular to the y-axis. The origin of the world coordinate system is (0,0,0).

[0086] The second part is the azimuth The meaning is: the origin of the camera coordinate system (x0, y0, z0) not only has an offset in the direction of the x-z plane mentioned above, but also has a deflection in the measurement around the y-axis. Then the azimuth angle can be expressed by the angle between the projection line of the line connecting the origin of the world coordinate system to the point (x0, y0, z0) on the x-z plane and the x-axis. In this way, these two angles can represent the spatial orientation relationship of the camera coordinate system with respect to the world coordinate system, and can also represent the camera's photographic plane Ω. c and calibration plane Ω p The spatial relationship between them.

[0087] The calculation method of the rotation amount R is:

[0088]

[0089] Where θ is the pitch angle; is the azimuth angle; cos is the cosine function; tan is the tangent function; x0 is the horizontal coordinate of the origin of the camera coordinate system in the world coordinate system; y0 is the vertical coordinate of the origin of the camera coordinate system in the world coordinate system; z0 is the z coordinate of the origin of the camera coordinate system in the world coordinate system.

[0090] The rotation R calculated above represents the spatial positional relationship between the camera and the LED screen. The spatial distance between the camera and the LED screen is represented by the displacement. Therefore, the displacement T is also represented by the Euclidean distance between the origin of the camera coordinate system (x0, y0, z0) and the origin of the world coordinate system. This allows the displacement of the camera plane relative to the calibration plane to be determined based on the distance between the origins of the camera and world coordinate systems.

[0091] In some embodiments, the calculation formula for the displacement T of the photographic plane relative to the calibration plane is:

[0092] In step S300, based on the rotation amount and the displacement amount, the corner points of the calibration plane are affine transformed to the photographic plane to obtain the ideal corner points of the photographic plane; the distortion error amount is determined based on the error between the corner points of the photographic plane and the ideal corner points; and the distortion error amount is combined with the distortion error amount to correct the ideal corner points to obtain the target corner points in the camera coordinate system.

[0093] The conversion between the photographic plane and the calibration plane can be achieved through coordinate transformation. The core logic behind the above method is that light propagates in a straight line. However, in reality, there are imaging distortion errors caused by internal factors such as the camera lens. In LED virtual shooting, the accuracy of camera tracking and positioning is very high, so the above transformation also needs to eliminate imaging distortion.

[0094] After completing the two-plane transformation in the previous step, any corner point (x w ,y w ,z w ,1), and transform it into the camera coordinate system through rotation and translation, becoming (x c ,y c ,z c ,1), then the error distance between the corresponding transformed point and the actual point can be regarded as a measure of distortion. So next, we establish the image coordinate system u―v, which belongs to the same plane as the calibration plane, that is, to establish a two-dimensional coordinate system on the calibration plane as the image coordinate system, see Figure 4 , Figure 4 A schematic diagram of the image coordinate system.

[0095] Taking a corner point (u0, v0) on the calibration plane (i.e., the LED screen) as an example, the transformation model described above calculates the ideal corner point of the calibration plane by affine transformation, based on the rotation and displacement. For example, for corner point (u0, v0) on the calibration plane, the corresponding point on the camera's photographic plane is corner point (u2, v2). Based on the rotation and displacement, the corner point (u0, v0) on the calibration plane is affine transformed to the photographic plane, resulting in the ideal corner point (u1, v1). The distance between the ideal corner point (u1, v1) and the corresponding point (u2, v2) on the photographic plane image captured by the camera is then calculated as the lens distortion error generated by the camera during the capture. Let α be the distortion error.

[0096] The calculation formula of the distortion error α is:

[0097] The distortion error α represents the degree of distortion of a point (u0, v0) on the LED screen as seen by the camera. This method can be used to calculate the degree of distortion for all corner points on the calibration plane. Once the degree of distortion for all corner points on the current camera plane is determined, the next step is to eliminate the distortion in the camera's image.

[0098] The point (u0, v0) on the LED screen under the calibration plane is recorded as the original point X0, the corresponding point (u1, v1) on the photographic plane obtained by the above coordinate transformation is recorded as the ideal point X1, and the corresponding point (u2, v2) in the photographic image actually captured by the camera on the photographic plane is recorded as the actual point X2. Then the method for distortion correction of the ideal corner point is:

[0099]

[0100] Among them, X2′(i) is the coordinate of the corrected ideal corner point, which is also the coordinate of the target corner point; λ i is the weighted weight of the i-th ideal corner point; X1(i) is the coordinate of the i-th ideal corner point; α(i) is the distortion error of the i-th ideal corner point; β is the distance weight. In the embodiment of the present invention, the distance weight is manually set by the implementer according to the actual situation; n is the number of corner points in the calibration plane. It should be noted that when the ideal corner point is weighted by the weighted weight of the ideal corner point, the weighted weight of the ideal corner point is weighted on the horizontal and vertical coordinates of the ideal corner point respectively to obtain the horizontal and vertical coordinates of the weighted ideal coordinates. In the embodiment of the present invention, X2′(i) actually contains the horizontal and vertical coordinates of the ideal corner point, and the coordinates of the ideal corner point are represented by a character X2′(i).

[0101] This formula indicates that for the ith actual corner point, it can be estimated based on the ideal corner point X1 obtained after coordinate transformation. Specifically, it can be calculated through all the transformed ideal corner points, so where λ i It represents the weighted weight of the i-th ideal corner point. By performing weighted calculation on all the ideal corner points, the corrected actual corner point is obtained. i , its logic is as follows: the ratio of the distortion error between the i-th ideal corner point and the actual corner point to the sum of the distortion errors of all points is used to measure. From the above, it can be seen that the actual meaning of the distortion error α is the distance between the ideal corner point and the actual corner point, so here The meaning is that the farther the ideal corner point is from the actual corner point, that is, the larger the distortion error α is, the smaller the weight of the correction for the actual corner point should be, that is, the weighted weight λ i It is inversely proportional to the power of β of the distortion error α. The choice of β here can be made by the implementer according to the characteristics of the LED screen background. Specifically, the smaller β is, the larger the overall λ is. i The closer to the mean value, the larger β is, and the greater the weight of the reference point that is closer to the actual point.

[0102] In step S400, the final rotation and displacement of the photographic plane relative to the calibration plane are determined based on the target corner point in the camera coordinate system and the corner point in the world coordinate system. The final rotation and displacement are transmitted to the calibration system to obtain the position of the camera.

[0103] After the distortion correction is performed on the ideal corner point, the rotation and displacement of the photographic plane relative to the calibration plane are calculated based on the target corner point obtained after the distortion correction and the corner point on the calibration plane. Specifically: in step S200 in the embodiment of the present invention, the rotation and displacement of the photographic plane relative to the calibration plane are determined by the origin of the camera coordinate system and the world coordinate system. After the distortion correction is performed on the ideal corner point to obtain the target corner point, the rotation and displacement of the photographic plane relative to the calibration plane are determined by the target corner point in the camera coordinate system and the corner point in the world coordinate system. It should be noted that each diagonal point and target corner point can obtain a corresponding set of rotation and displacement. In the embodiment of the present invention, the mode of the rotation of all diagonal points and target corner points is taken as the final rotation; the average of the displacement of all diagonal points and target corner points is taken as the displacement. It should be noted that when determining the rotation and displacement of the photographic plane relative to the calibration plane through the target corner point of the camera coordinate system and the corner point in the world coordinate system, the corner point in the world coordinate system and the corresponding corner point in the camera coordinate system can be used as the origin of the coordinate system for calculation.

[0104] The calculated final rotation and displacement are transmitted to the existing calibration system. Using information such as the current mapped position and angle, the calibration system determines the camera's position and adjusts the virtual background on the LED screen in real time to sync with the camera's angle and position, achieving realistic filming. This method of inferring camera position through spatial mapping significantly reduces costs compared to existing camera positioning methods using lidar or sensors. It also accounts for errors between the image and actual positioning caused by lens distortion, improving accuracy.

[0105] The embodiment of the present invention obtains the rotation and displacement of the photographic plane relative to the calibration plane through transformation calculation between the two coordinate systems, and takes into account the positioning error caused by camera imaging distortion.

[0106] An embodiment of the present invention provides a camera tracking system for LED virtual shooting, the system comprising:

[0107] A corner point detection module is used to detect corner points of the calibration plane and the photographic plane when the camera is in the anchor position, using the imaging plane of the LED screen as the calibration plane and the imaging plane of the camera as the photographic plane, and obtaining the corner points of the calibration plane and the photographic plane;

[0108] a parameter determination module, configured to determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; and to determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system;

[0109] The distortion correction module is used to perform an affine transformation of the corner points of the calibration plane to the photographic plane based on the rotation amount and the displacement amount to obtain the ideal corner points of the photographic plane; determine the distortion error amount based on the error between the corner points of the photographic plane and the ideal corner points; and perform distortion correction on the ideal corner points based on the distortion error amount to obtain the target corner points in the camera coordinate system;

[0110] The position determination module is used to determine the final rotation and final displacement of the photographic plane relative to the calibration plane based on the target corner point in the camera coordinate system and the corner point of the world coordinate system, and transmit the final rotation and final displacement to the calibration system to obtain the position of the camera.

[0111] Optionally, the transmission medium can be a wired link, such as but not limited to coaxial cable, optical fiber and digital subscriber line, or a wireless link, such as but not limited to Wireless Fidelity (WIFI), Bluetooth and mobile device network.

[0112] It should be noted that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above.

[0113] An embodiment of the present invention provides a computer device. Exemplarily, the computer device includes: a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the computer device can perform any of the aforementioned camera tracking methods for LED virtual photography.

[0114] In addition, an embodiment of the present invention also protects a device, which may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform the camera tracking method for LED virtual shooting provided by an embodiment of the present invention.

[0115] In embodiments of the present invention, the device may be divided into functional modules based on the above-described method examples. For example, these modules may correspond to individual functional modules, or two or more functions may be integrated into a single processing module. The integrated modules may be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and represents only a logical functional division. In actual implementation, other division methods may be employed.

[0116] In the case of dividing each module into modules corresponding to each function, the device may further include a signal uploading module, a determination module, an adjustment module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0117] It should be understood that the apparatus provided in the embodiment of the present invention is used to execute the camera tracking method for LED virtual shooting, and thus can achieve the same effect as the above-mentioned implementation method.

[0118] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a device, the processing module may be used to control and manage the operation of the device. The storage module may be used to support the device in executing mutual program codes, etc. The processing module may be a processor or a controller that may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0119] In addition, the device provided in the embodiment of the present invention can specifically be a chip, component or module, and the chip may include a connected processor and memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the camera tracking method for LED virtual shooting provided in the above embodiment.

[0120] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer program code. When the computer program code is run on a computer, the computer executes the above-mentioned related method steps to implement the camera tracking method for LED virtual shooting provided in the above embodiment.

[0121] An embodiment of the present invention further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the camera tracking method for LED virtual shooting provided in the above embodiment.

[0122] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways.

[0123] The device embodiments described above are merely illustrative. For example, the division into modules or units represents only one logical functional division. Actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another device, or omitting or disabling certain features. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through an interface, or indirect coupling or communication connection between devices or units may be electrical, mechanical, or otherwise.

[0124] It should also be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal device comprising the element.

[0125] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0127] The above content is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A camera tracking method for LED virtual shooting, characterized in that: The method comprises the following steps: When the camera is in the anchor position, the imaging plane of the LED screen is used as the calibration plane, and the imaging plane of the camera is used as the photographic plane. Corner point detection is performed on the calibration plane and the photographic plane to obtain the corner points of the calibration plane and the photographic plane. Determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system; Based on the rotation and displacement, the corner points of the calibration plane are affine transformed to the photographic plane to obtain the ideal corner points of the photographic plane; the distortion error is determined based on the error between the corner points of the photographic plane and the ideal corner points; based on the distortion error, the ideal corner points are subjected to distortion correction to obtain the target corner points in the camera coordinate system; According to the target corner points in the camera coordinate system and the corner points in the world coordinate system, the final rotation and displacement of the photographic plane relative to the calibration plane are determined, and the final rotation and displacement are transmitted to the calibration system to obtain the position of the camera.

2. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: Perform corner detection on the calibration plane to obtain the corner points of the calibration plane, including: When performing corner detection, the corner response values during corner detection are weighted by the pixel features around the pixel, and the corner points of the calibration plane are obtained based on the weighted corner response values of each corner point.

3. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The weighting of the corner point response value during corner point detection by using pixel features surrounding the pixel point includes: With a window size of 3×3, a local window corresponding to each pixel is constructed. The grayscale changes in the horizontal, vertical, left diagonal, and right diagonal directions of the central pixel in the local window are analyzed respectively. For the horizontal direction, the sum of the squares of the differences between the central pixel in the local window and the pixels on both sides of the horizontal direction is calculated as the directional eigenvalue of the central pixel in the horizontal direction. The weight value of the central pixel is multiplied by the directional eigenvalue of the central pixel in the horizontal direction to obtain the target eigenvalue of the central pixel in the horizontal direction. Calculate the target eigenvalues of the central pixel in the horizontal, vertical, left diagonal, and right diagonal directions of the local window respectively, and take the smallest target eigenvalue as the weighted eigenvalue of the central pixel of the local window; The weighted eigenvalues are used as weights to weight the corner point response values in the corner point detection algorithm.

4. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The method for obtaining the pitch angle corresponding to the origin of the camera coordinate system and the world coordinate system is: Assume that the origin of the camera coordinate system in the world coordinate system is (x0, y0, z0), and calculate the angle between the line connecting the origin of the camera coordinate system (x0, y0, z0) and the origin of the world coordinate system and the y-axis as the pitch angle; where the y-axes of the camera coordinate system and the world coordinate system are parallel.

5. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The method for obtaining the direction angle corresponding to the origin of the camera coordinate system and the world coordinate system is: Get the projection line of the line connecting the origin of the world coordinate system to the origin of the camera coordinate system (x0, y0, z0) on the x-z plane, and use the angle between the projection line and the x-axis as the direction angle.

6. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The step of determining the rotation amount of the photographic plane relative to the calibration plane according to the pitch angle and the azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system includes: The sum of the pitch angle and the azimuth angle is taken as the rotation of the photographic plane relative to the calibration plane.

7. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: Determining the displacement of the photographic plane relative to the calibration plane based on the distance between the origin of the camera coordinate system and the world coordinate system includes: Calculate the Euclidean distance between the origin of the camera coordinate system and the world coordinate system as the displacement of the photography plane relative to the calibration plane.

8. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The step of determining the distortion error amount based on the error between the corner point of the photographic plane and the ideal corner point includes: The distance between the corner point of the photographic plane and the ideal corner point is taken as the distortion error of the ideal corner point.

9. The camera tracking method for LED virtual shooting according to claim 1, characterized in that: The step of performing distortion correction on the ideal corner point based on the distortion error to obtain the target corner point in the camera coordinate system includes: Among them, X2′(i) is the coordinate of the target corner point; λ i is the weighted weight of the i-th ideal corner point; X1(i) is the coordinate of the i-th ideal corner point; α(i) is the distortion error of the i-th ideal corner point; β is the distance weight; n is the number of corner points in the calibration plane.

10. A camera tracking system for LED virtual shooting, characterized in that: The system includes the following modules: A corner point detection module is used to use the imaging plane of the LED screen as the calibration plane and the imaging plane of the camera as the photographic plane when the camera is in the anchor position; Perform corner point detection on the calibration plane and the photographic plane to obtain the corner points of the calibration plane and the photographic plane; a parameter determination module, configured to determine the rotation of the photographic plane relative to the calibration plane based on the pitch angle and azimuth angle corresponding to the origins of the camera coordinate system and the world coordinate system; and to determine the displacement of the photographic plane relative to the calibration plane based on the distance between the origins of the camera coordinate system and the world coordinate system; wherein the calibration plane belongs to the world coordinate system and the photographic plane belongs to the camera coordinate system; The distortion correction module is used to perform an affine transformation of the corner points of the calibration plane to the photographic plane based on the rotation amount and the displacement amount to obtain the ideal corner points of the photographic plane; determine the distortion error amount based on the error between the corner points of the photographic plane and the ideal corner points; and perform distortion correction on the ideal corner points based on the distortion error amount to obtain the target corner points in the camera coordinate system; The position determination module is used to determine the final rotation and final displacement of the photographic plane relative to the calibration plane based on the target corner point in the camera coordinate system and the corner point of the world coordinate system, and transmit the final rotation and final displacement to the calibration system to obtain the position of the camera.

Images