A 3D positioning method for a spherical camera based on binocular vision fusion

Abstract

The application provides a kind of 3D positioning method of ball camera based on binocular vision fusion, specifically comprising: S1, camera calibration;S2, establish world coordinate system;S3, binocular-ball camera joint external parameter calibration;S4, calculate the relationship between the coordinates of two cameras;S5, according to the camera coordinates of the ball camera obtained, rotate the binocular camera to the position of the camera coordinates of the ball camera by polar coordinate mode, so that it is in the center of the ball camera picture;And the like steps.The application can be operated on a binocular camera with a larger field of view, the range of 3D positioning can be increased, and point-to-point coordinate conversion is carried out, so that 3D positioning is very accurate, and flexibility is increased, and viewing operation can be performed at any time in the binocular camera.

Description

Technical Field

[0001] This invention relates to the field of camera positioning technology, and specifically to a 3D positioning method for PTZ cameras based on binocular vision fusion. Background Technology

[0002] 3D positioning of a PTZ camera refers to setting the target's coordinates in the image and rotating the PTZ camera to center the target within the camera's field of view. The principle of 3D positioning is explained in [link to 3D positioning principle]. Figure 1 (C is the lens center, O is the image center point, P is the target point's position on the image, and A is the image boundary point), using the formula The horizontal and vertical angular differences between the target point P on the image and the center of the image's field of view can be obtained. Using these horizontal and vertical angular differences, the pan-tilt head can be rotated by a specified angle.

[0003] However, existing 3D positioning technologies have the following main drawbacks:

[0004] (1) The field of view is small, and the range that can be located is small. The field of view of a single lens of a PTZ camera is usually ≤60°, which makes it difficult to cover a large range of scenes.

[0005] (2) The algorithm for converting coordinates to angles has certain errors.

[0006] (3) There is the effect of image distortion.

[0007] (4) Low operating efficiency: Users need to repeatedly adjust the field of view in the magnified image of the PTZ camera, which significantly increases the time required for multi-target operation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a 3D positioning method for PTZ cameras based on binocular vision fusion. This method can be operated on binocular cameras with a large field of view, increasing the range of 3D positioning and enabling point-to-point coordinate transformation, resulting in highly accurate 3D positioning. It also increases flexibility, allowing for real-time viewing and operation on the binocular camera.

[0009] To achieve the above technical solution, the present invention provides a 3D positioning method for a PTZ camera based on binocular vision fusion, specifically including the following steps:

[0010] S1. Camera Calibration: The PTZ camera and the stereo camera are calibrated using the nine-square checkerboard calibration method, and the intrinsic parameters dome_cameraMatrix and the distortion coefficients dome_distCoeffs of the PTZ camera are obtained, as well as the intrinsic parameters Binocular_cameraMatrix and the distortion coefficients Binocular_distCoeffs of the stereo camera.

[0011] S2, establish a world coordinate system: take the center of the spherical camera as the origin of the coordinate system, the direction in which the front of the spherical camera is located is the z-axis, and the direction in which the top is located is the y-axis;

[0012] S3, binocular-spherical camera joint external parameter calibration: find the coordinate relationship between the spherical camera and the binocular camera for binding;

[0013] S4, calculate the relationship between the coordinates of the two cameras: take the pixel coordinates of a point P on the binocular camera to calculate the world coordinates Pw corresponding to the point P and the camera coordinates Pc of the spherical camera;

[0014] S5, according to the calculated camera coordinates of the spherical camera, rotate the binocular camera to the position of the camera coordinates of the spherical camera through the polar coordinate mode, so that it is in the center of the spherical camera picture.

[0015] Preferably, before camera calibration, the positions of the two cameras are fixed, the relative positions are kept unchanged, and after the positions are fixed, they are not moved. If there is movement, it needs to be fixed again.

[0016] Preferably, the specific way of binding the coordinate relationship between the spherical camera and the binocular camera in step S3 is as follows:

[0017] S31, based on the current position, the binocular camera takes a picture panorama and saves it, and the spherical camera takes nine pictures according to the picture panorama, which is equivalent to dividing the picture panorama into nine parts. It is required that the nine pictures must include all parts of the picture panorama, and the centers of the nine pictures must be in the picture panorama. When the spherical camera takes nine pictures, the horizontal angle and the vertical angle of the spherical camera are recorded;

[0018] S32, the ten pictures are processed for distortion removal;

[0019] S33, the nine pictures after distortion removal are respectively matched with the picture panorama after distortion removal, and the homography matrix is estimated from the screened matching points. The homography matrix describes the geometric transformation relationship between the two images;

[0020] S34, the center point of the complete image is transformed to the original image space by using the homography matrix, and the transformed center point coordinates are obtained, and finally nine points in the original image are obtained, which correspond one by one to the centers of the nine spherical camera images;

[0021] S35, according to the horizontal angle and the vertical angle of the spherical camera when taking pictures, the world coordinates corresponding to the center points of the nine pictures are calculated;

[0022] S36, using the function of opencv to obtain the rotation R (rvec) and translation vector T (tvec) of the binocular camera, and calculating the relationship between the coordinates of two cameras through R and T.

[0023] Preferably, in the step S32, the function cv: undistort is used for the distortion removal processing.

[0024] Preferably, in the step S36, the formula of the opencv function is: bool cv: solvePnP (InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess = false, int flags = SOLVEPNP_ITERATIVE).

[0025] Preferably, in the step S4, the pixel coordinates of a point P on the binocular camera are obtained through formula 1 to obtain the world coordinates Pw corresponding to the point P, and then the camera coordinates Pc of the spherical camera are obtained through formula 2.

[0026]

[0027]

[0028] Wherein, dx and dy represent how many mm each column and each row represents, that is, 1 pixel = dx mm, (u0, v0) is the image coordinate, and (u, v) is the pixel coordinate.

[0029]

[0030] Preferably, in the step S5, the operation mode of the polar coordinates is: the proportion of the frame to the entire binocular camera picture -> the proportion of the frame to the spherical camera picture -> the proportion of the focal length -> the proportion of the zoom value, wherein the proportion of the frame to the spherical camera picture is proportional to the focal length, and the focal length is proportional to the zoom value.

[0031] The ball camera 3D positioning method based on binocular vision fusion provided by the application has the beneficial effects that:

[0032] (1) The ball camera 3D positioning method based on binocular vision fusion has a unique world coordinate system construction strategy: taking the rotation center of the ball camera holder as the origin, combining the PTZ angle to calculate the space coordinates in real time, realizing point-to-point coordinate conversion, and making the 3D positioning very accurate.

[0033] (2) The ball camera 3D positioning method based on binocular vision fusion adopts a double-machine calibration system, uses a nine-square chessboard sampling external parameter calibration process, and a robust external parameter solving algorithm based on RANSAC-PnP.

[0034] (3) The ball camera 3D positioning method based on binocular vision fusion can be operated on a binocular camera with a larger field of view, can make the range of 3D positioning larger, and can be viewed at any time in the binocular camera, increasing flexibility.

[0035] (4) The ball camera 3D positioning method based on binocular vision fusion binds the binocular camera and the ball camera in depth through an advanced and complex algorithm. When a user selects a target in the vast field of view covered by the binocular camera, the ball camera system will quickly respond. If the user selects a point, at this time, the intelligent algorithm inside the system will quickly calculate the angle and direction parameters that the ball camera needs to rotate according to the pre-set coordinate conversion model, until the selected point is stably located at the center of the ball camera picture. When the user selects a region, the ball camera will still rotate to the specified region according to the established algorithm, and then enlarge the selected region, so that the user can clearly observe the fine scene in the region. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a prior art 3D positioning principle diagram.

[0037] Figure 2 is a flowchart of the present application.

[0038] Figure 3 is a world coordinate system diagram in the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. All other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0040] Embodiment: A ball camera 3D positioning method based on binocular vision fusion.

[0041] Referring to Figures 1 to 3 The present application also provides a ball camera 3D positioning method based on binocular vision fusion, specifically comprising the following steps:

[0042] (1) First, the positions of the dome camera and the binocular camera need to be fixed, and the relative positions should not be changed. After fixing the positions, do not move them again. If there is any movement, you need to start over. This is to ensure the stability of the positions between the dome camera and the binocular camera.

[0043] (2) Camera calibration: the dome camera and the binocular camera are calibrated by themselves. The calibration method is the nine-square chessboard calibration method. The internal parameters of the dome camera are obtained: dome_cameraMatrix, the distortion coefficients of the dome camera: dome_distCoeffs; the internal parameters of the binocular camera: Binocular_cameraMatrix, the distortion coefficients of the binocular camera: Binocular_distCoeffs. The nine-square chessboard sampling extrinsic calibration process is used, which is convenient for subsequent calculation of the robust extrinsic parameter solving algorithm based on RANSAC-PnP, and improves the accuracy. Moreover, the distortion coefficients of the dome camera and the binocular camera are selected, which can be combined with the subsequent composite distortion compensation algorithm to realize distortion correction optimization.

[0044] (3) Establish the world coordinate system, with the center of the dome camera as the origin of the coordinate system, the direction in front of the dome camera as the z-axis, and the direction above as the y-axis, as shown in Figure 3 .

[0045] The reason for taking the center of the dome camera as the origin of the coordinate system: the dome camera can rotate and the current angle of rotation of the dome camera can be known, which is convenient for obtaining the world coordinates corresponding to the points on the dome camera. If other things are selected as the origin, such as a calibration board, there will not be a calibration board in the actual application scenario (such as outdoor high places).

[0046] Through the unique world coordinate system construction strategy: taking the center of the dome camera gimbal as the origin, the spatial coordinates are calculated in real time combined with the PTZ angle, realizing point-to-point coordinate conversion, and making the 3D positioning very accurate.

[0047] (4) Binocular-dome camera joint extrinsic calibration: the calibration between the two cameras, finding the coordinate relationship between the dome camera and the binocular camera for binding. The specific way is:

[0048] ① Based on the current position, the binocular camera takes a picture panorama and saves it. The dome camera takes nine pictures according to the picture panorama (the image resolution of the two cameras is the same), which is equivalent to dividing the picture panorama into nine parts. The nine pictures must include all parts of the picture panorama, and the centers of the nine pictures must be in the picture panorama. When the dome camera takes the nine pictures, record the horizontal angle and the vertical angle pt of the dome camera.

[0049] 2. Undistort the ten images: use the function cv::undistort to undistort the images.

[0050] 3. Match the nine undistorted images with the undistorted panorama image to estimate the homography matrix, which describes the geometric transformation relationship between the two images.

[0051] 4. Transform the center point of the complete image to the original image space using the homography matrix to obtain the transformed center point coordinates. Finally, nine points in the original image are obtained, corresponding to the centers of the nine spherical camera images.

[0052] 5. According to the pt rotated when the spherical camera was shooting, the world coordinates of the center points of the nine images can be calculated (z is normalized)

[0053] 6. Use the function of opencv:

[0054] bool cv::solvePnP(InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess = false, int flags = SOLVEPNP_ITERATIVE)

[0055] The solvePnP function estimates the pose (i.e., rotation and translation vectors) of an object by given 3D points and their corresponding points in 2D images. These vectors can be used to transform 3D points in the object's coordinate system to the camera's coordinate system, enabling object localization and tracking in the scene.

[0056] Through this function, the rotation and translation vectors R (rvec) and T (tvec) of the binocular camera can be obtained, and through R and T, the relationship between the coordinates of the two cameras can be obtained.

[0057] Through the above operations, the spatial coordinate conversion chain between binocular and spherical cameras is constructed, achieving sub-pixel level positioning (error ≤0.1°), establishing a high-precision mapping model, and realizing point-to-point coordinate conversion between binocular and spherical cameras, making 3D positioning very accurate.

[0058] (5) Calculate the relationship between the two cameras: Since the two machines are in the same world coordinate system, the relationship between the two cameras can be established according to this benchmark. The spherical camera and the binocular camera have their own pixel coordinate system, image coordinate system, and camera coordinate system, and they share the world coordinate system.

[0059] The specific calculation method is: take the pixel coordinates of a point P on the binocular camera (if it is a drawn box, take the center point of the box), through formula 1 below, the world coordinates Pw corresponding to P can be solved, and through formula 2, the camera coordinates Pc of the spherical camera can be solved.

[0060]

[0061]

[0062] Where dx and dy represent how many mm each column and each row represents, i.e. 1 pixel = dx mm. (u0, v0) is the image coordinate, and (u, v) is the pixel coordinate.

[0063]

[0064] Through the above algorithm, a nonlinear mapping relationship between the target area and the zoom value is realized, and the binocular camera and the spherical camera are deeply bound. When the user selects a target in the vast field of view covered by the binocular camera, the spherical system will quickly respond.

[0065] (6) According to the camera coordinates of the spherical camera obtained, rotate to the position by polar coordinates, so that it is in the center of the spherical camera picture (if the box is taken on the binocular camera, it needs to be enlarged according to the size of the box).

[0066] Method: The proportion of the box to the entire binocular camera picture -> the proportion of the box in the spherical camera picture -> the proportion of the focal length -> the proportion of the zoom value

[0067] Explanation: The proportion of the box in the spherical camera picture is proportional to the focal length, and the focal length is proportional to the zoom value.

[0068] This 3D positioning method for PTZ cameras based on binocular vision fusion employs an advanced and complex algorithm to deeply bind the binocular and PTZ cameras. When a user selects a target within the vast field of view covered by the binocular camera, the PTZ camera system responds rapidly. If the user selects a point, the system's intelligent algorithm quickly calculates the required rotation angle and direction parameters of the PTZ camera based on a pre-set coordinate transformation model, until the selected point is stably centered in the PTZ camera's view. When the user selects an area, the PTZ camera will still rotate to the designated area according to the predetermined algorithm, then magnify the selected area, allowing the user to clearly observe the fine details within that area.

[0069] This 3D positioning method for PTZ cameras based on binocular vision fusion features a unique world coordinate system construction strategy: using the PTZ camera's pan-tilt-zoom (PTZ) rotation center as the origin, it calculates spatial coordinates in real time by combining the PTZ angle, achieving point-to-point coordinate transformation and resulting in highly accurate 3D positioning. This method employs a dual-camera calibration system, using a nine-square checkerboard sampling extrinsic parameter calibration process, and a robust extrinsic parameter solution algorithm based on RANSAC-PnP. This 3D positioning method can be operated on binocular cameras with a large field of view, increasing the range of 3D positioning and allowing for real-time viewing and operation within the binocular camera, thus increasing flexibility.

[0070] This 3D positioning method for PTZ cameras based on binocular vision fusion has the following significant advantages over existing technologies:

[0071] (1) Expanded field of view: A 180° ultra-wide-angle coverage is achieved through binocular cameras, solving the problem of limited field of view of PTZ cameras.

[0072] (2) Establish a high-precision mapping model: Construct a spatial coordinate transformation chain between the binocular and PTZ cameras to achieve sub-pixel-level positioning (error ≤ 0.1°).

[0073] (3) Distortion correction optimization: A composite distortion compensation algorithm is designed by integrating dual-target calibration and PTZ camera calibration data.

[0074] (4) Adaptive zoom strategy: The optimal zoom value is dynamically calculated based on the size of the target area to achieve intelligent operation of "selection and focus".

[0075] The above description is only a preferred embodiment of the present invention, but the present invention should not be limited to the content disclosed in the embodiments and drawings. Therefore, any equivalent or modified embodiments made without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A 3D positioning method for a spherical camera based on binocular vision fusion, characterized in that Specifically comprising the following steps: S1, camera calibration: using nine palace chessboard calibration method to calibrate the dome camera and binocular camera, and obtaining the intrinsic parameters of the dome camera dome_cameraMatrix, the distortion coefficient of the dome camera dome_distCoeffs, the intrinsic parameters of the binocular camera Binocular_cameraMatrix, and the distortion coefficient of the binocular camera Binocular_distCoeffs; S2, establishing a world coordinate system: taking the center of the dome camera as the origin of the coordinate system, and taking the direction in front of the dome camera as the z axis and the direction above as the y axis; S3, binocular-dome camera joint parameter calibration: finding the coordinate relationship between the dome camera and the binocular camera for binding; S4, calculating the relationship between the coordinates of the two cameras: taking the pixel coordinates of a point P on the binocular camera to calculate the world coordinates Pw corresponding to P and the camera coordinates Pc of the dome camera; S5, according to the calculated camera coordinates of the dome camera, rotating the binocular camera to the position of the camera coordinates of the dome camera by the polar coordinate method, so that it is in the center of the dome camera picture; The specific way of binding the coordinate relationship between the dome camera and the binocular camera in step S3 is as follows: S31, based on the current position, the binocular camera takes a picture panorama and saves it, and the dome camera takes nine pictures according to the picture panorama, which is equivalent to dividing the picture panorama into nine parts, and requires that the nine pictures must include all parts of the picture panorama, and the centers of the nine pictures must be in the picture panorama. When the dome camera takes nine pictures, record the horizontal angle and vertical angle of the dome camera; S32, de-distortion processing is performed on the ten pictures; S33, feature point matching is performed on the nine de-distorted pictures and the de-distorted picture panorama respectively, and a homography matrix is estimated from the screened matching points, which describes the geometric transformation relationship between the two images; S34, the center point of the complete image is transformed to the original image space using the homography matrix, and the transformed center point coordinates are obtained, and finally nine points in the original image are obtained, which correspond one-to-one to the centers of the nine dome camera images; S35, according to the horizontal angle and vertical angle of the dome camera when taking pictures, the world coordinates corresponding to the center points of the nine pictures are calculated; S36, using the function of opencv to obtain the rotation R(rvec) and translation vector T(tvec) of the binocular camera, and calculating the relationship between the coordinates of the two cameras through R and T.

2. The binocular vision fusion based 3D positioning method of a spherical camera according to claim 1, wherein, Before camera calibration, fix the positions of the two cameras, keep the relative positions unchanged, and after fixing the positions, do not move, if there is movement, it is necessary to start fixing again.

3. The binocular vision fusion based 3D positioning method of a spherical camera according to claim 1, wherein, In step S32, the function cv:undistort is used for de-distortion processing.

4. The binocular vision fusion based 3D positioning method of a spherical camera according to claim 1, wherein, The formula of the opencv function in the step S36 is: bool cv::solvePnP (InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess = false, int flags = SOLVEPNP_ITERATIVE).

5. The binocular vision fusion based spherical camera 3D positioning method of claim 1, wherein, The pixel coordinates of a point P on the binocular camera are obtained by formula 1 to obtain the world coordinates Pw corresponding to the point P, and then the camera coordinates Pc of the spherical camera are obtained by formula 2 in the step S4. Wherein, dx and dy represent how many mm each column and each row respectively represents, that is, 1 pixel = dx mm, (u0, v0) is the image coordinates, and (u, v) is the pixel coordinates; 6. The binocular vision fusion based spherical camera 3D positioning method of claim 1, wherein, In the step S5, the operation mode of the polar coordinates is: the proportion of the frame to the entire binocular camera picture -> the proportion of the frame to the spherical camera picture -> the proportion of the focal length -> the proportion of the zoom value, wherein the proportion of the frame to the spherical camera picture is proportional to the focal length, and the focal length is proportional to the zoom value.

Images