Method for visually inspecting camera matrix of monitoring camera based on live-action three-dimensional model

Abstract

The invention discloses a method for visually inspecting a camera matrix of a monitoring camera based on a live-action three-dimensional model, and the method comprises the steps: decomposing a camera matrix M to obtain the coordinate of a camera focus F, calculating the photographing light LP of a homonymy point through an image space point P and the camera matrix, obtaining a calculated object space point W'through calculating the intersection point of a straight line LP and a three-dimensional live-action model S, and obtaining the camera matrix of the monitoring camera through the calculated object space point W '. Obtaining a difference vector D according to the real object space point W and the calculated object space point W ', namely the error of the camera matrix; and finally, drawing a vector D in the three-dimensional real scene model S by taking the real object space point W as a starting point, and marking a specific numerical value of the D. The method solves the problems that an existing precision test method of a camera matrix is not fused with a real scene, is not visual enough, and is difficult to obtain an object variance value of errors and distribution characteristics of a three-dimensional space.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a method for visually inspecting a camera matrix of a monitoring camera based on a real-scene three-dimensional model. Background Art

[0002] When fusing camera images or videos with 3D real-world models, the camera matrix is ​​needed to associate the camera image coordinate system with the 3D geographic model coordinate system. The camera matrix is ​​the product of the camera intrinsic parameter matrix and the camera extrinsic parameter matrix. The correctness of the camera matrix directly affects the effect of fusing camera images or videos with 3D real-world models based on the camera matrix, and also affects the accuracy of the 3D spatial coordinates corresponding to any image point in the camera image calculated based on the camera matrix.

[0003] Traditional methods for testing the accuracy of a single camera matrix include the image-space test method, which is to convert the object-space coordinates of known homonymous points into image-space coordinates through the camera matrix, and then compare the image-space difference between the object-space coordinates and the image-space coordinates of the homonymous points as the test accuracy of the camera matrix; however, since the object-space error is positively correlated with the image-space error, it is also positively correlated with the viewing distance divided by the focal length, resulting in that the image-space error result cannot be directly equivalent to the object-space error result. The object-space error has unique application value. Therefore, this method cannot integrate the tested image-space error result with the three-dimensional real-scene model, and is not intuitive enough to display the distribution characteristics of the object-space difference value and the three-dimensional space. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] Based on the above problems, the present invention provides a method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model, so as to solve the problem that the existing accuracy inspection method of the camera matrix is ​​not integrated with the real scene, is not intuitive enough, and is difficult to obtain the object square difference value of its error and the distribution characteristics of the three-dimensional space.

[0006] (II) Technical solution

[0007] Based on the above technical problems, the present invention provides a method for visually inspecting a camera matrix of a surveillance camera based on a real-scene three-dimensional model, comprising the following steps:

[0008] S1. Decompose the camera matrix M to obtain the coordinates of the camera focus F;

[0009] S2, based on the collinearity of the image point, object point and the origin of the focal coordinate system, the photographic light of the same-name point is calculated through the image point P and the camera focus F, which is recorded as the straight line L P ;

[0010] S3. Calculate the straight line L P The intersection point with the 3D real scene model S is obtained to obtain the calculated object point W';

[0011] S4. Obtain the difference vector D based on the real object point W and the calculated object point W', that is, the error of the camera matrix:

[0012]

[0013] S5. Draw a difference vector D in the three-dimensional real scene model S with the real object point W as the starting point, and mark the specific value of the difference vector D.

[0014] Further, the S1 includes:

[0015] S11. Decompose the camera matrix M based on the transformation formula from the world coordinate system to the image pixel coordinate system, write the camera matrix M into an augmented matrix form, and decompose the camera matrix M into R and t:

[0016]

[0017] Here, R is the matrix consisting of the first 3*3 items of M, and t is the matrix consisting of the last 3*1 items of M;

[0018] S12, according to t is the coordinate of the origin of the camera coordinate system in the world coordinate system, obtain the coordinate of the camera focus F in the world coordinate system, expressed as F(X t ,Y t ,Z t ).

[0019] Further, the S2 includes:

[0020] S21, based on the fact that the image point, the object point and the origin of the focal coordinate system are collinear, let the coordinates of the image point P in the focal coordinate system be P C , the coordinates of the object point W in the focal coordinate system are W C , then there is a vector starting at the origin of the focal coordinate system: λ is a non-zero constant;

[0021] S22, translate the object point W from the world coordinate system to the focus coordinate system to obtain W C ,Right now:

[0022]

[0023] X c =X w -X t , Y c =Y w -Y t , Z c =Z w -Z t

[0024] S23, derive P according to the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix c The unit vector P c1 and W c The unit vector W c1 Relationship:

[0025] S24, two rays ±R passing through the camera focus F and collinear in the world coordinate system P Expressed as (F, ±P C1 ), the two rays with opposite directions at the same point form the straight line L corresponding to the photographic light passing through the focus at the same point P .

[0026] Further, the S23 includes:

[0027] The transformation formula from the focus coordinate system to the image pixel coordinate system is derived from the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix:

[0028]

[0029] Z c is an unknown constant, and Reversible, so

[0030]

[0031] make:

[0032]

[0033] Thus P c The unit vector P c1 and W c The unit vector W c1 The relationship is:

[0034]

[0035] Furthermore, the S5 further includes: connecting the focus F and the calculated object point W' to obtain a line segment L FW' , add a solid line segment L in the 3D real scene model S FW' .

[0036] Furthermore, the straight line L P The intersection point with the three-dimensional real scene model S is extracted by the three-dimensional model software.

[0037] The present invention also discloses a system for visually inspecting a camera matrix of a monitoring camera based on a real-scene three-dimensional model, comprising:

[0038] at least one processor; and at least one memory in communication with the processor, wherein:

[0039] The memory stores program instructions that can be executed by the processor, and the processor can execute the method by calling the program instructions.

[0040] The present invention also discloses a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method.

[0041] (III) Beneficial effects

[0042] The above technical solution of the present invention has the following advantages:

[0043] (1) The present invention calculates the photographic light rays of the same-name point passing through the focus according to the principle of three points in a line, thereby calculating the position of the object point, thereby obtaining the actual error of the camera matrix in the real-scene three-dimensional model, that is, the error of the object point. Compared with the image error, it can better represent the error result of the camera matrix.

[0044] (2) The present invention marks the error of the object point in the real-scene three-dimensional model, thereby integrating the object error result with the three-dimensional real-scene model, and can more intuitively display the distribution characteristics of the object difference value and the three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0046] Figure 1 A schematic flow chart of a method for visually inspecting a camera matrix of a monitoring camera based on a real-scene three-dimensional model according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of three points on a collinear line according to an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of photographic light calculated based on a camera matrix and real photographic light in a real-scene three-dimensional model according to an embodiment of the present invention;

[0049] Figure 4 It is an enlarged diagram of the schematic diagram of the photographic light calculated based on the camera matrix and the real photographic light in the real-scene three-dimensional model according to an embodiment of the present invention, and a visualized schematic diagram of the actual error of the camera matrix in the real-scene three-dimensional model. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] The embodiment of the present invention is a method for visually inspecting the camera matrix of a monitoring camera based on a real-scene three-dimensional model. Figure 1 As shown, the following steps are included:

[0052] S1. Decompose the camera matrix M to obtain the coordinates of the camera focus F.

[0053] Specific methods:

[0054] S11. Decompose the camera matrix M based on the transformation formula from the world coordinate system to the image pixel coordinate system, write M into an augmented matrix form, and decompose M into R and t:

[0055]

[0056] Here R is only the matrix composed of the first 3*3 items of M, not the rotation matrix or intrinsic matrix of the camera extrinsic parameters, and t is the matrix composed of the last 3*1 items of M.

[0057] S12, according to t is the coordinate of the origin of the camera coordinate system in the world coordinate system, obtain the coordinate of the camera focus F in the world coordinate system, expressed as F(X t ,Y t ,Z t );

[0058] According to the geometric meaning of the matrix operation, t can be set as the coordinate of the origin of the camera coordinate system in the world coordinate system, that is, t is the coordinate of the camera focus F in the W coordinate system, which can be expressed as F(X t ,Y t ,Z t ).

[0059] S2, based on the collinearity of the image point, object point and the origin of the focal coordinate system, the photographic light of the same-name point is calculated through the image point P and the camera focus F, which is recorded as the straight line L P .

[0060] S21, based on the fact that the image point, the object point and the origin of the focal coordinate system are collinear, let the coordinates of the image point P in the focal coordinate system be P C , the coordinates of the object point W in the focal coordinate system are W C , then there is a vector starting at the origin of the focal coordinate system: λ is a non-zero constant;

[0061] Since the camera matrix is ​​calculated based on the collinearity of the camera focus, the image point and the object point of the same name. Based on formula ①, it can be seen that translating the origin of the object coordinates to t is equivalent to setting the focal coordinates as the origin, and defining the coordinate system at this time as the focal coordinate system. Therefore, the three points are collinear, that is, the image point, the object point and the origin of the focal coordinate system are collinear. Let the coordinates of the image point P in the focal coordinate system be P C , the coordinates of the object point W in the focal coordinate system are W C , then there is a vector starting at the origin of the focal coordinate system: λ is a non-zero constant.

[0062] S22, translate the object point W from the world coordinate system to the focus coordinate system to obtain W C ,Right now:

[0063]

[0064] X c =X w -X t , Y c =Y w -Y t , Z c =Z w -Z t

[0065] S23, derive P according to the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix c The unit vector P c1 and W c The unit vector W c1 relationship;

[0066] According to the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix, the transformation formula from the focus coordinate system to the image pixel coordinate system can be derived:

[0067]

[0068] The above formula can be understood as only one ray can be determined through (u, v), Z c Represents the distance between the object plane and the focal point F. The object plane refers to Figure 2 The plane passing through W' and parallel to the pixel plane, Z c is an unknown constant, and Reversible, so

[0069]

[0070] make:

[0071] For W C The unit vector W C1You don't need to consider Z C The absolute value of , only its positive or negative value needs to be considered. -1 The unit vector of the calculated vector is P C1 ,but:

[0072]

[0073] S24, two rays ±R passing through the camera focus F and collinear in the world coordinate system P It can be expressed as (F, ±P C1 ), the two rays with opposite directions at the same point form the straight line L corresponding to the photographic light passing through the focus at the same point P .

[0074] S3. Calculate the straight line L P The intersection point with the three-dimensional real scene model S is obtained to obtain the calculated object point W'.

[0075] Straight Line L P is the photographic ray passing through the focus F at the same point, then the straight line L P Also consider the image point and the object point, then the straight line L P The intersection point W' with the real-scene 3D model S is the calculated object point.

[0076] Calculate the straight line L P There are many ways to find the intersection with the three-dimensional real-scene model S, including but not limited to extracting it through the software of the three-dimensional model. For example, on Cesium, pick up the first intersecting object through rays and return the object and the intersection position.

[0077] S4. Obtain a difference vector D, ie, the error of the camera matrix, according to the real object point W and the calculated object point W'.

[0078] After the object point W' is calculated, the error size and direction are calculated and displayed based on the true value W. W is the real position of the object point with the same name corresponding to the image point P in the real 3D model, which needs to be manually marked in the real 3D model. W' is the object position automatically calculated by the camera matrix and the real 3D model, and D represents the actual error of the camera matrix in the real 3D model.

[0079]

[0080] S5. Visualization: Connect the focus F and the calculated object point W' to obtain the line segment L FW' , add a solid line segment L in the 3D real scene model S FW' , draw the vector D with the real object point W as the starting point, and mark the specific value of D.

[0081] Line segment L FW'It is used to visualize the photographic light corresponding to the image point P, that is, from which object point P passes through which straight line (i.e. light path) to reach the camera focus. Figure 3 As shown, the line segments in the figure represent the photographic light L calculated based on the camera matrix at the four corner points of the camera FW' and the light L from the real point to the camera focus FW . Enlarge the image as Figure 4 As shown in the figure, W is the real object point position marked manually, and W' is the object point calculated by the camera matrix and the real scene 3D model. FW Represents the light L from the real point to the camera focus FW , L FW The intersection with the model represents the real position W of the manually marked image point P in the real scene 3D model; the light L in the figure FW' Represents the photographic line segment L calculated by the camera matrix FW' , L FW' The intersection with the model represents the position W' with error calculated by this method; the error of the camera matrix in the object space is reflected in the difference between W and W'. The blue arrow represents the error vector

[0082] You can also not display line segment L FW' , we only need to display the error vector D. The core of visualization is to display the error vector D.

[0083] Finally, it should be noted that the above method can be converted into software program instructions, which can be implemented by using a system including a processor and a memory, or by computer instructions stored in a non-transitory computer-readable storage medium. The above integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above software functional unit is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0084] In summary, the above method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model has the following advantages:

[0085] (1) The present invention calculates the photographic light rays of the same-name point passing through the focus according to the principle of three points in a line, thereby calculating the position of the object point, thereby obtaining the actual error of the camera matrix in the real-scene three-dimensional model, that is, the error of the object point. Compared with the image error, it can better represent the error result of the camera matrix.

[0086] (2) The present invention marks the error of the object point in the real-scene three-dimensional model, thereby integrating the object error result with the three-dimensional real-scene model, and can more intuitively display the distribution characteristics of the object difference value and the three-dimensional space.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model, characterized in that: The following steps are involved: S1. Decompose the camera matrix M to obtain the coordinates of the camera focus F; S2. Based on the fact that the image point, object point and the origin of the focal coordinate system are collinear, the photographic light of the same-name point is calculated through the image point P and the camera focus F, which is recorded as the straight line L P ; S3. Calculate the straight line L P The intersection point with the 3D real scene model S is obtained to obtain the calculated object point W'; S4. Obtain the difference vector D based on the real object point W and the calculated object point W', that is, the error of the camera matrix: S5. Draw a difference vector D in the three-dimensional real scene model S with the real object point W as the starting point, and mark the specific value of the difference vector D.

2. The method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model according to claim 1, characterized in that: The S1 includes: S11. Decompose the camera matrix M based on the transformation formula from the world coordinate system to the image pixel coordinate system, write the camera matrix M into an augmented matrix form, and decompose the camera matrix M into R and t: Here, R is the matrix consisting of the first 3*3 items of M, and t is the matrix consisting of the last 3*1 items of M; S12, according to t is the coordinate of the origin of the camera coordinate system in the world coordinate system, obtain the coordinate of the camera focus F in the world coordinate system, expressed as F(X t ,Y t ,Z t ).

3. The method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model according to claim 1, characterized in that: The S2 includes: S21, based on the fact that the image point, the object point and the origin of the focal coordinate system are collinear, let the coordinates of the image point P in the focal coordinate system be P C , the coordinates of the object point W in the focal coordinate system are W C , then there is a vector starting at the origin of the focal coordinate system: λ is a non-zero constant; S22, translate the object point W from the world coordinate system to the focus coordinate system to obtain W C ,Right now: X c =X w -X t ,Y c =Y w -Y t ,Z c =Z w -Z t S23, derive P according to the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix c The unit vector P c1 and W c The unit vector W c1 Relationship: S24, two rays ±R passing through the camera focus F and collinear in the world coordinate system P Expressed as (F, ±P C1 ), the two rays with opposite directions at the same point form the straight line L corresponding to the photographic light passing through the focus at the same point P .

4. The method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model according to claim 3, characterized in that: The S23 includes: The transformation formula from the focus coordinate system to the image pixel coordinate system is derived from the transformation formula from the world coordinate system to the image pixel coordinate system via the camera matrix: Z c is an unknown constant, and Reversible, so make: Thus P c The unit vector P c1 and W c The unit vector W c1 The relationship is:

5. The method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model according to claim 1, characterized in that: The step S5 further includes: connecting the focus F and the calculated object point W' to obtain a line segment L FW' , add a solid line segment L in the 3D real scene model S FW' .

6. The method for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model according to claim 1, characterized in that: The straight line L P The intersection point with the three-dimensional real scene model S is extracted by the three-dimensional model software.

7. A system for visually inspecting the camera matrix of a surveillance camera based on a real-scene three-dimensional model, characterized in that: include: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 6 by calling the program instructions.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method according to any one of claims 1 to 6.

Images