External parameter calibration method and device for binocular monitoring camera and binocular monitoring camera

Abstract

The present disclosure provides an external parameter calibration method, device, binocular monitoring camera, and computer-readable storage medium for a binocular monitoring camera. The external parameter calibration method for a binocular monitoring camera includes: obtaining a first scene image collected by a left-eye camera and a second scene image collected by a right-eye camera at the same moment; obtaining a scene depth image according to the first scene image and the second scene image; extracting a background depth image defined by a reference region in the scene depth image, where the reference region is determined according to a still-life region in the scene initialization depth image; determining whether the external parameters of the binocular monitoring camera are damaged according to the background depth image and a reference depth image defined by the reference region in the scene initialization depth image; when it is determined that the external parameters of the binocular monitoring camera are damaged, determining an external parameter compensation coefficient for the binocular monitoring camera according to the reference depth image; and calibrating the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

Description

Technical Field

[0001] The present disclosure relates to the technical field of binocular monitoring, and particularly relates to an external parameter calibration method, device, binocular monitoring camera, and computer-readable storage medium for a binocular monitoring camera. Background Art

[0002] Security monitoring systems are applied in various industries in our country, such as shopping malls, factories, and squares. Common security monitoring systems generally use monocular monitoring cameras.

[0003] Currently, some security monitoring systems have adopted binocular monitoring cameras. A binocular monitoring camera includes a left-eye camera and a right-eye camera that are fixedly connected. Among them, the left-eye camera serves as an auxiliary camera and the right-eye camera serves as a main camera, or the left-eye camera serves as the main camera and the right-eye camera serves as the auxiliary camera. Usually, there is a certain positional deviation between the coordinate systems of the left-eye camera and the right-eye camera. The chip of the binocular monitoring camera calculates a depth image of the monitored object in space based on the images captured by the left-eye camera and the right-eye camera respectively, that is, binocular stereo matching is performed. Depth refers to the distance from a point in the scene to the longitudinal plane (XY plane, perpendicular to the ground) where the main camera is located. The depth image is consistent with the coordinate system of the main camera, and each pixel in the depth image represents the distance from the corresponding area in the scene to the longitudinal plane where the main camera is located. The security monitoring system using a binocular monitoring camera can not only achieve monitoring but also obtain parameter information such as the size and volume of the monitored object. Therefore, it has broad application value.

[0004] In the related art, the binocular baseline of a binocular monitoring camera (that is, the distance between the optical axes of the left-eye camera and the right-eye camera) is short, generally less than 20 centimeters, and the ranging range generally does not exceed 10 meters. Therefore, it is usually only used in small scenes. If the binocular baseline is made long in order to pursue an application scenario of a large scene (the scene depth is about 200 meters), the structural stability of the binocular monitoring camera will become poor, and further the external parameters of the binocular monitoring camera will be damaged, resulting in a sharp increase in the ranging error. Summary of the Invention

[0005] Embodiments of the present disclosure provide an external parameter calibration method, device, binocular monitoring camera, and computer-readable storage medium for a binocular monitoring camera, so that the binocular monitoring camera can meet the accuracy requirements of large-scene applications and improve the applicable scene range.

[0006] According to one aspect of the embodiments of the present disclosure, an external parameter calibration method for a binocular monitoring camera is provided. The binocular monitoring camera includes a left-eye camera and a right-eye camera. The external parameter calibration method includes:

[0007] Obtain a first scene image collected by the left-eye camera and a second scene image collected by the right-eye camera at the same moment;

[0008] Obtain a scene depth image based on the first scene image and the second scene image;

[0009] Extract the background depth image defined by the reference region in the scene depth image, where the reference region is determined according to the still object region in the initial scene depth image captured after the binocular monitoring camera is installed on site and its external parameters are initially calibrated;

[0010] Judge whether the external parameters of the binocular monitoring camera are damaged according to the background depth image and the reference depth image defined by the reference region in the initial scene depth image;

[0011] When it is determined that the external parameters of the binocular monitoring camera are damaged, determine the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image;

[0012] Calibrate the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

[0013] In some embodiments, judging whether the external parameters of the binocular monitoring camera are damaged according to the background depth image and the reference depth image includes:

[0014] When the depth distribution density of the background depth image is less than the density threshold, and / or, the average depth error of the background depth image compared to the reference depth image is greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are damaged;

[0015] When the depth distribution density of the background depth image is not less than the density threshold, and the average depth error of the background depth image compared to the reference depth image is not greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are not damaged.

[0016] In some embodiments, in the left-eye camera and the right-eye camera, one is the main camera and the other is the auxiliary camera; determining the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image includes:

[0017] Determine the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the auxiliary camera to rotate the coordinate system with reference to the main camera according to the reference depth image.

[0018] In some embodiments, determining the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the auxiliary camera to rotate the coordinate system with reference to the main camera according to the reference depth image includes:

[0019] Loop through multiple roll angles roll within the set roll angle value range and multiple pitch angles pitch within the set pitch angle value range, and in each access loop, according to the roll angle roll, pitch angle pitch, and initial yaw angle yaw 00 obtain the first depth distribution density of the depth image defined by the reference region corresponding thereto;

[0020] Determine the roll angle roll0 and pitch angle pitch0 corresponding to the maximum value in the first depth distribution density for each access cycle;

[0021] Traverse multiple yaw angles yaw within the set yaw angle value range, and when accessing each yaw angle yaw, obtain the first mean depth error of the depth image defined by the reference area compared to the reference depth image according to the yaw angle yaw, roll angle roll0, and pitch angle pitch0;

[0022] Determine the yaw angle yaw0 corresponding to the minimum value in the first mean depth error obtained when accessing each yaw angle yaw;

[0023] According to the yaw angle yaw0, roll angle roll0, and pitch angle pitch0, obtain the second depth distribution density of the depth image defined by the reference area;

[0024] When the second depth distribution density is not less than the density threshold, determine the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the auxiliary camera to rotate the coordinate system with the main camera as the reference according to the yaw angle yaw0, roll angle roll0, and pitch angle pitch0; otherwise, adjust and increase the set roll angle value range and set pitch angle value range, and return to the step of cyclic traversal.

[0025] In some embodiments, the external parameter calibration method further includes:

[0026] When it is determined that the external parameters of the binocular monitoring camera are damaged, output a prompt message for damaged external parameters; and / or

[0027] When it is determined that the external parameters of the binocular monitoring camera are not damaged, output the scene depth image.

[0028] In some embodiments, obtaining the scene depth image according to the first scene image and the second scene image includes:

[0029] According to the first scene image, obtain multiple first to-be-processed scene images with different resolutions;

[0030] According to the second scene image, obtain multiple second to-be-processed scene images with the same resolutions corresponding one by one to the multiple first to-be-processed scene images;

[0031] For each resolution, obtain the to-be-processed scene depth image according to the corresponding first to-be-processed scene image and second to-be-processed scene image;

[0032] Perform fusion processing on the to-be-processed scene depth images with multiple resolutions to obtain the scene depth image.

[0033] In some embodiments, the external parameter calibration method further includes: after acquiring the first scene image and the second scene image and before obtaining the scene depth image, performing binocular parallel correction processing on the first scene image and the second scene image.

[0034] In some embodiments, the distance between the optical axes of the left camera and the right camera is between 20 cm and 300 cm.

[0035] According to another aspect of the embodiments of the present disclosure, there is provided an external parameter calibration device for a binocular monitoring camera. The binocular monitoring camera includes a left camera and a right camera. The external parameter calibration device includes:

[0036] An acquisition unit, configured to acquire a first scene image acquired by the left camera and a second scene image acquired by the right camera at the same moment;

[0037] A processing unit, configured to obtain a scene depth image according to the first scene image and the second scene image;

[0038] An extraction unit, configured to extract a background depth image defined by a reference area in the scene depth image, where the reference area is determined according to a still object area in an initial scene depth image captured after the binocular monitoring camera is installed on site and externally parameter initially calibrated;

[0039] A judgment unit, configured to judge whether the external parameters of the binocular monitoring camera are damaged according to the background depth image and a reference depth image defined by the reference area in the initial scene depth image;

[0040] A determination unit, configured to, when it is determined that the external parameters of the binocular monitoring camera are damaged, determine an external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image;

[0041] A calibration unit, configured to calibrate the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

[0042] According to still another aspect of the embodiments of the present disclosure, there is provided a binocular monitoring camera, including: a memory and a processor coupled to the memory. The processor is configured to execute the external parameter calibration method of the binocular monitoring camera according to any one of the foregoing technical solutions based on instructions stored in the memory.

[0043] According to yet another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the external parameter calibration method of the binocular monitoring camera according to any one of the foregoing technical solutions is implemented.

[0044] By adopting the technical solutions of the above embodiments of the present disclosure, the depth error caused by the damage of the external parameters can be overcome, thereby improving the accuracy of the depth image of the captured scene. Since the external parameters of the binocular monitoring camera can be adaptively calibrated according to the actual damage situation and no human intervention is required, the binocular monitoring camera can be used in large scenes.

[0045] Of course, when implementing the products or methods of any embodiment of the present disclosure, it is not necessarily required to achieve all the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required to be used in the description of the embodiments of the present disclosure or related technologies. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] Figure 1a Schematic diagram of a binocular monitoring camera designed based on a large baseline in some embodiments of the present disclosure;

[0048] Figure 1b Monitoring scene images and depth images captured by a binocular monitoring camera in some embodiments of the present disclosure;

[0049] Figure 2a Monitoring scene images captured by a binocular monitoring camera after being installed on site and calibrated for external parameters in some embodiments of the present disclosure;

[0050] Figure 2b Depth images captured by a binocular monitoring camera after being installed on site and calibrated for external parameters in some embodiments of the present disclosure;

[0051] Figure 2c Depth images captured by a binocular monitoring camera after the external parameters are damaged in some embodiments of the present disclosure;

[0052] Figure 2d Depth images captured by a binocular monitoring camera after self-calibration of the external parameters in some embodiments of the present disclosure;

[0053] Figure 3 Flow schematic diagram of the external parameter calibration method of a binocular monitoring camera in some embodiments of the present disclosure;

[0054] Figure 4 Flow schematic diagram of determining the external parameter compensation coefficient in some embodiments of the present disclosure;

[0055] Figure 5 Schematic diagram of the external parameter calibration device of a binocular monitoring camera in some embodiments of the present disclosure;

[0056] Figure 6 Schematic diagram of a binocular monitoring camera in some embodiments of the present disclosure. Specific embodiments

[0057] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0058] In the related art, the main reason why binocular monitoring cameras are rarely used in large scenes is that large scenes require a longer binocular baseline. However, when the binocular baseline is made longer, the structural stability will deteriorate, which may lead to damage to the external parameters of the binocular monitoring camera, that is, the external parameters change beyond the allowable error range, and the system cannot accurately obtain the depth image of the monitoring scene based on the captured images of the left-eye camera and the right-eye camera. Among them, the external parameters of the binocular monitoring camera refer to the rotation and translation parameters between the coordinate systems of the left-eye camera and the right-eye camera, that is, the rotation and translation parameters for transforming the points in the coordinate system of the auxiliary camera to the coordinate system of the main camera. The internal parameters of the binocular monitoring camera refer to the parameters for transforming points from the world coordinate system to the respective coordinate systems of the left-eye camera and the right-eye camera. The internal parameters are independent of the length of the binocular baseline and are inherent parameters. Therefore, making the binocular baseline longer has basically no impact on the internal parameters.

[0059] The embodiments of the present disclosure provide an external parameter calibration method, device, binocular monitoring camera, and computer-readable storage medium for a binocular monitoring camera, so that the binocular monitoring camera can meet the accuracy requirements of large-scene applications and improve the applicable scene range.

[0060] As Figure 1a shown, the binocular monitoring camera 100 provided by the embodiments of the present disclosure is applied to a large monitoring scene, and its binocular baseline L can take values in the range of 20 cm to 300 cm. The binocular monitoring camera 100 includes a left-eye camera 1L and a right-eye camera 1R connected by a connecting member 2. Among them, the left-eye camera 1L is the main camera and the right-eye camera 1R is the auxiliary camera, or the left-eye camera 1L is the auxiliary camera and the right-eye camera 1R is the main camera. In the binocular monitoring camera, based on the coordinate system of the main camera, there is a certain positional deviation between the coordinate systems of the auxiliary camera and the main camera. The depth image of the monitored object 001 collected by the binocular monitoring camera uses the same coordinate system as the main camera.

[0061] As Figure 1bAs shown in the figure above, the upper figure is an image of a monitoring scene captured by a binocular monitoring camera at a set resolution, and the lower figure is the depth image corresponding to the monitoring scene in the upper figure. In the depth image, the color and grayscale of each pixel are used to represent the depth of the scene area corresponding to the pixel, and pixels with different colors and / or grayscales correspond to different depths. The security monitoring system uses a binocular monitoring camera, which can not only achieve video monitoring, but also obtain some geometric parameter information of the monitored object, such as size, volume, etc., through depth calculation.

[0062] In the embodiments of the present disclosure, the specific type of the binocular monitoring camera is not limited. For example, it can be an RGB binocular monitoring camera, and the left-eye camera and the right-eye camera collect color images. The binocular monitoring camera can also be an infrared binocular monitoring camera, which includes a group of infrared light sources. The left-eye camera and the right-eye camera collect color images in the daytime scene and infrared images in the night vision scene.

[0063] The following technical solutions of each embodiment provided by the embodiments of the present disclosure are particularly applicable to large monitoring scenes, such as Figure 1a As shown in the figure, it is required that the binocular baseline L takes values in the range of 20 cm to 300 cm. In addition, according to requirements, it can also be used in some small monitoring scenes (the binocular baseline is generally less than 20 cm), and the present disclosure does not make specific limitations on this.

[0064] Taking the binocular monitoring camera applied to a large monitoring scene as an example, during production, the left-eye camera and the right-eye camera will be internally calibrated. As mentioned above, the internal parameters of the binocular monitoring camera are the inherent parameters of the left-eye camera and the right-eye camera respectively. When the binocular monitoring camera is installed on site, the operator will use a black and white checkerboard calibration board to calibrate the external parameters of the binocular monitoring camera (that is, the rotation and translation parameters for transforming points in the auxiliary camera coordinate system to the main camera coordinate system), and the calibrated external parameters are stored in the chip or memory of the binocular monitoring camera. After on-site installation and completion of the external parameter calibration, the binocular monitoring camera can be officially started to work.

[0065] In some embodiments of the present disclosure, after the left-eye camera and the right-eye camera collect images and before calculating the depth image based on the two images, the system will perform binocular parallel correction on the two images according to the external parameters of the binocular monitoring camera, so that the same object has the same size in the two images and the same ordinate (Y coordinate) in the longitudinal plane (XY plane). Through binocular parallel correction, the non-parallel left-eye camera and right-eye camera can be converted into a parallel binocular system that is strictly aligned in the longitudinal plane.

[0066] Since a large monitoring scene requires a longer binocular baseline for the binocular monitoring camera, and a longer binocular baseline will lead to poor structural stability, which may in turn cause damage to the external parameters of the binocular monitoring camera. At this time, if the external parameters are not compensated, the system will not be able to obtain an accurate depth image. As Figures 2a to 2d shown in the figureFigure 2a It is a monitoring scene image captured by a binocular monitoring camera after being installed on-site and externally calibrated. Figure 2b It is a depth image (i.e., the scene initialization depth image in the following text) captured by a binocular monitoring camera after being installed on-site and externally calibrated. Figure 2c It is a depth image captured by a binocular monitoring camera after the external parameters are damaged. Figure 2d It is a depth image captured by a binocular monitoring camera after self-calibrating the external parameters. Comparing Figure 2b and Figure 2c It can be seen that after the external parameters of the binocular monitoring camera are damaged, in the same reference area 00S, Figure 2c the depth distribution density (characterizing the density of depth value distribution) is significantly sparse, and the depth also has a large deviation (reflected in the change of pixel color and grayscale).

[0067] Based on this, as Figure 3 shown, the embodiments of the present disclosure provide a method for externally calibrating a binocular monitoring camera, which can be used for adaptively calibrating the external parameters of the binocular monitoring camera. The external parameter calibration method includes the following steps S1 to step S6.

[0068] In step S1, obtain the first scene image collected by the left-eye camera and the second scene image collected by the right-eye camera at the same moment.

[0069] After the binocular monitoring camera is started, the left-eye camera and the right-eye camera work simultaneously. As mentioned above, the first scene image and the second scene image can be RGB color images or infrared images.

[0070] As Figure 2a shown, by synthesizing the first scene image and the second scene image, the monitoring scene image 80 can be obtained.

[0071] In some embodiments of the present disclosure, the external parameter calibration method further includes: after step S1 and before the following step S2, perform binocular parallel correction processing on the first scene image and the second scene image according to the external parameters of the binocular monitoring camera.

[0072] In step S2, obtain the scene depth image according to the first scene image and the second scene image. As Figure 2c shown is the scene depth image 30 captured by the binocular monitoring camera after being used for a period of time.

[0073] In this step, the depth calculation method used is not limited. For example, the multi-view stereo method, photometric stereo vision method, chromaticity shaping method, defocus inference method, and depth calculation method based on machine learning can be used, etc.

[0074] In some embodiments, the above step S2 includes the following sub-steps one to sub-step four.

[0075] In sub-step one, based on the first scene image, multiple first to-be-processed scene images with different resolutions are obtained.

[0076] For example, the user sets the resolutions of the images captured by the left-eye camera and the right-eye camera to 1920×1080 (Full High Definition, FHD). Based on the first scene image with a resolution of 1920×1080 (Full High Definition, FHD) captured by the left-eye camera, three first to-be-processed scene images with resolutions of 1920×1080 (Full High Definition, FHD), 1280×720 (High Definition, HD), and 480×640 (Standard Definition, SD) are obtained.

[0077] In sub-step two, based on the second scene image, multiple second to-be-processed scene images with the same resolutions as the multiple first to-be-processed scene images one by one are obtained.

[0078] For example, based on the second scene image with a resolution of 1920×1080 (Full High Definition, FHD) captured by the right-eye camera, three second to-be-processed scene images with resolutions of 1920×1080 (Full High Definition, FHD), 1280×720 (High Definition, HD), and 480×640 (Standard Definition, SD) are obtained.

[0079] In sub-step three, for each resolution, based on the corresponding first to-be-processed scene image and second to-be-processed scene image, a to-be-processed scene depth image is obtained. For example, three to-be-processed scene depth images are obtained according to the above three resolutions.

[0080] In sub-step three, the to-be-processed scene depth images with multiple resolutions are fused to obtain a scene depth image.

[0081] By fusing the to-be-processed scene depth images at multiple resolutions, the void areas in the obtained scene depth image can be reduced, making the judgment result of the following step S4 more accurate.

[0082] Return to Figure 3 , in step S3, the background depth image defined by the reference area in the scene depth image is extracted, where the reference area is determined according to the still-life area in the scene initialization depth image captured after the binocular monitoring camera is installed on-site and the external parameters are initially calibrated.

[0083] As Figure 2b shown, the scene initialization depth image 50 is the depth image captured after the binocular monitoring camera is installed on-site and the external parameters are calibrated. The reference area 00S is determined according to the still-life area in the scene initialization depth image 50. The reference area is, for example, an area with good daylighting and not easily blocked in the building. The reference area can be delimited by the operator input on-site or remotely through the terminal device.

[0084] If the external parameters of the binocular monitoring camera are not damaged, the background depth image 31 should be substantially the same as the reference depth image 51 in the scene initialization depth image 50, and the depth value difference within the image is small. However, by comparing Figure 2b and Figure 2c it can be clearly seen that the depth value difference between the two images is large, indicating that the external parameters of the binocular monitoring camera have been damaged.

[0085] Return to Figure 3 , in step S4, based on the background depth image and the reference depth image defined by the reference area in the scene initialization depth image, determine whether the external parameters of the binocular monitoring camera are damaged.

[0086] In some embodiments, the above step S4 includes:

[0087] When the depth distribution density of the background depth image is less than the density threshold, and / or the average depth error of the background depth image compared to the reference depth image is greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are damaged;

[0088] When the depth distribution density of the background depth image is not less than the density threshold, and the average depth error of the background depth image compared to the reference depth image is not greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are not damaged.

[0089] When the external parameters of the binocular monitoring camera are damaged, the depth distribution density and / or the average depth error of the captured scene depth image will change significantly. As Figure 2c shown, after the external parameters are damaged, the depth values of the background depth image 31 defined by the reference area 00S are no longer continuously distributed, the distribution density decreases significantly, and in addition, the average depth error also increases significantly, and the pixel colors and grayscales of the background depth image 31 compared to the reference depth image 51 change greatly.

[0090] Therefore, it is possible to determine whether the external parameters of the binocular monitoring camera are damaged based on the depth distribution density and the average depth error of the scene depth image.

[0091] Return to Figure 3 , in step S5, when it is determined that the external parameters of the binocular monitoring camera are damaged, determine the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image.

[0092] In some embodiments, step S5 includes: determining the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image by using a control loop optimization algorithm.

[0093] In the process of implementing the embodiments of the present disclosure, the inventor found that during the use of the binocular monitoring camera, the external parameters are damaged, mainly because the rotation parameters of the transformation from the coordinate system of the auxiliary camera to the coordinate system of the main camera change, while the translation parameters of the two coordinate systems change less.

[0094] In some embodiments of the present disclosure, to improve the system calculation and processing speed, the translation parameters of the transformation from the coordinate system of the auxiliary camera to the coordinate system of the main camera are ignored. The determined external parameter compensation coefficients include: the yaw angle compensation coefficient, the roll angle compensation coefficient, and the pitch angle compensation coefficient for the rotation of the coordinate system of the auxiliary camera with reference to the main camera.

[0095] In some other embodiments of the present disclosure, to further improve the accuracy of the depth image captured by the binocular monitoring camera, the translation parameters and rotation parameters of the transformation from the coordinate system of the auxiliary camera to the coordinate system of the main camera can also be considered simultaneously.

[0096] As Figure 4 shown, in an embodiment of the present disclosure, the translation parameters of the transformation from the coordinate system of the auxiliary camera to the coordinate system of the main camera are ignored, and the control loop optimization algorithm is used to determine the external parameter compensation coefficients of the binocular monitoring camera, which specifically includes the following steps S501 to S508.

[0097] In step S501, a plurality of roll angles roll within the set roll angle value range and a plurality of pitch angles pitch within the set pitch angle value range are traversed in a loop, and in each access loop, according to the roll angle roll, the pitch angle pitch, and the initial yaw angle yaw 00 , the first depth distribution density of the depth image defined by the reference area corresponding to the parameter is obtained.

[0098] The set roll angle value range and the set pitch angle value range can be determined according to experience or the accuracy requirements that the device needs to achieve. In an embodiment, the set roll angle value range (i.e., the value range of the roll angle roll) is -2° ≤ roll ≤ 2°, and a plurality of roll angles roll within the set roll angle value range, such as -2°, -1.999°, -1.998°... respectively. The set pitch angle value range (i.e., the value range of the pitch angle pitch) is -2° ≤ pitch ≤ 2°, and a plurality of pitch angles pitch within the set pitch angle value range, such as -2°, -1.999°, -1.998°... respectively.

[0099] The initial yaw angle yaw 00 is the external parameter obtained after the on-site installation and calibration of the binocular monitoring camera, and in the control loop optimization algorithm, it is first assumed that it has not changed.

[0100] Taking the above angle value range as an example, the process of loop traversal is as follows:

[0101] In the first loop, access roll = -2°, pitch = -2°, and based on this roll angle, pitch angle, and the initial yaw angle yaw 00 , obtain the first depth distribution density D1 of the depth image defined by the reference region corresponding to these parameters;

[0102] In the second loop, access roll = -1.999°, pitch = -1.999°, and based on this roll angle, pitch angle, and the initial yaw angle yaw 00 , obtain the first depth distribution density D2 of the depth image defined by the reference region corresponding to these parameters;

[0103] In the third loop, access roll = -1.998°, pitch = -1.998°, and based on this roll angle, pitch angle, and the initial yaw angle yaw 00 , obtain the first depth distribution density D3 of the depth image defined by the reference region corresponding to these parameters;

[0104] ……, and so on in sequence, to obtain n first depth distribution densities D1 to Dn.

[0105] In step S502, determine the roll angle roll0 and pitch angle pitch0 corresponding to the maximum value among the first depth distribution densities corresponding to each access loop.

[0106] That is, determine the roll angle roll0 and pitch angle pitch0 corresponding to the maximum value among D1 to Dn.

[0107] In step S503, traverse multiple yaw angles yaw within the set yaw angle value range, and when accessing each yaw angle yaw, based on the yaw angle yaw, roll angle roll0, and pitch angle pitch0, obtain the first depth error mean value of the depth image defined by the reference region corresponding to these parameters compared to the reference depth image.

[0108] The set yaw angle value range can be determined according to experience or the accuracy requirements that the device needs to achieve. In one embodiment, the set yaw angle value range (i.e., the value range of the yaw angle yaw) is -1° ≤ yaw ≤ 1°, and multiple yaw angles yaw within the set yaw angle value range, such as -1°, -0.999°, -0.998°... respectively.

[0109] In this step, the traversal process is as follows:

[0110] Access yaw = -1°, and based on this yaw angle, roll angle roll0, and pitch angle pitch0 obtained in the previous step S502, obtain the first mean depth error δ1 of the depth image defined by the reference region corresponding to this parameter compared to the reference depth image;

[0111] Access yaw = -0.999°, and based on this yaw angle, roll angle roll0, and pitch angle pitch0, obtain the first mean depth error δ2 of the depth image defined by the reference region corresponding to this parameter compared to the reference depth image;

[0112] Access yaw = -0.998°, and based on this yaw angle, roll angle roll0, and pitch angle pitch0, obtain the first mean depth error δ3 of the depth image defined by the reference region corresponding to this parameter compared to the reference depth image;

[0113] ……, and so on in sequence, to obtain m first mean depth errors δ1 to δm.

[0114] In step S504, determine the yaw angle yaw0 corresponding to the minimum value among the first mean depth errors obtained each time the yaw angle is accessed.

[0115] That is, determine the yaw angle yaw0 corresponding to the minimum value among δ1 to δm.

[0116] In step S505, based on the yaw angle yaw0, roll angle roll0, and pitch angle pitch0, obtain the second depth distribution density of the depth image defined by the reference region corresponding to this parameter.

[0117] In step S506, determine whether the second depth distribution density is not less than the density threshold. If so, it is considered that the calibration accuracy requirement is met, and the process proceeds to step S507; otherwise, it is considered that the calibration accuracy requirement is not met, and the process proceeds to step S508.

[0118] In step S507, based on the yaw angle yaw0, roll angle roll0, and pitch angle pitch0, determine the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the auxiliary camera to rotate the coordinate system with reference to the main camera.

[0119] For example, if the roll angle roll0 = -0.108°, pitch angle pitch0 = 0.96°, and yaw angle yaw0 = -0.048° obtained in the previous step, then obtain the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient based on this parameter.

[0120] In step S508, adjust and increase the set roll angle value range and / or the set pitch angle value range, and return to the step of cyclic traversal. For example, expand the set roll angle value range to -2.2° ≤ roll ≤ 2.2°, and expand the set pitch angle value range to -2.2° ≤ pitch ≤ 2.2°, and find the compensation coefficient that meets the accuracy requirements within a larger value range.

[0121] In step S6, calibrate the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

[0122] As Figure 2d shown, after the self-calibration correction of the external parameters of the binocular monitoring camera in this step, calculate the scene depth image again according to the external parameters calibrated in this step, and a more accurate scene depth image 30' can be obtained. Comparing Figure 2c with the background depth image 31 in Figure 2d and the background depth image 31' in Figure 2b , it can be seen that the density of the depth value distribution increases, which is basically the same as the density of the background depth image 51 in Figure 2b . In addition, the difference in the mean depth error also decreases significantly.

[0123] In some embodiments of the present disclosure, when it is determined that the external parameters of the binocular monitoring camera are not damaged, the scene depth image can continue to be output. For example, the scene depth image is output to a cloud device or a terminal device through a wireless transceiver module inside the binocular monitoring camera.

[0124] When it is determined that the external parameters of the binocular monitoring camera are damaged, an external parameter damage prompt message can also be output simultaneously to prompt relevant personnel to perform intervention processing according to the actual situation. For example, after overhauling the installation and structure of the binocular monitoring camera, perform on-site external parameter calibration again and re-determine the scene initialization depth image.

[0125] In summary, by using the external parameter calibration method provided in the above embodiments of the present disclosure, the increase in depth error caused by damaged external parameters can be overcome, thereby improving the accuracy of the captured scene depth image. Since the external parameters of the binocular monitoring camera can be adaptively calibrated according to the actual damage situation and no manual intervention is required, the binocular monitoring camera can be used in large scenes.

[0126] In the embodiments of the present disclosure, the distance between the optical axes of the left-eye camera and the right-eye camera can be within the range of 20 cm to 300 cm. Compared with the binocular monitoring camera used in small scenes in the related art, the binocular baseline length can be increased to 10 times or more, and the accuracy of adaptive calibration can meet the equivalent pixel error within 1.5 pixels, so that the ranging accuracy can be increased to 10 times or more.

[0127] As Figure 5As shown in the figure, an external parameter calibration device 600 for a binocular monitoring camera is further provided in an embodiment of the present disclosure. The binocular monitoring camera includes a left-eye camera and a right-eye camera. The external parameter calibration device includes:

[0128] An acquisition unit 61, configured to acquire a first scene image acquired by the left-eye camera and a second scene image acquired by the right-eye camera at the same moment;

[0129] A processing unit 62, configured to obtain a scene depth image according to the first scene image and the second scene image;

[0130] An extraction unit 63, configured to extract a background depth image defined by a reference area in the scene depth image, where the reference area is determined according to a still-life area in an initial scene depth image captured after the binocular monitoring camera is installed on-site and externally parameter initially calibrated;

[0131] A judgment unit 64, configured to judge whether the external parameters of the binocular monitoring camera are damaged according to the background depth image and a reference depth image defined by the reference area in the initial scene depth image;

[0132] A determination unit 65, configured to determine an external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image when it is determined that the external parameters of the binocular monitoring camera are damaged;

[0133] A calibration unit 66, configured to calibrate the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

[0134] Similar to the beneficial effects of the foregoing embodiments, by using the external parameter calibration device provided in the above embodiments of the present disclosure, the increase in depth error caused by damaged external parameters can be overcome, thereby improving the accuracy of the captured scene depth image, especially applicable to large-scale scene monitoring scenarios.

[0135] As Figure 6 shown, some embodiments of the present disclosure further provide a binocular monitoring camera 700, including: a memory 71 and a processor 72 coupled to the memory 71. The processor 72 is configured to execute the external parameter calibration method of the binocular monitoring camera according to any one of the foregoing embodiments based on instructions stored in the memory 71.

[0136] The memory 71 may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. The memory 71 may also be at least one storage device located far from the foregoing processor 72.

[0137] The above-mentioned processor 72 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0138] The product types of the binocular monitoring cameras include, but are not limited to, RGB binocular monitoring cameras and infrared binocular monitoring cameras. In one embodiment, the left-eye camera and the right-eye camera of the binocular monitoring camera are integrally connected to a connector designed based on a large baseline. In another embodiment, the left-eye camera and the right-eye camera of the binocular monitoring camera are detachably connected to a connector designed based on a large baseline, and are configured not to be installed and fixed at the factory, and are installed and fixed by the installer at the installation site.

[0139] The chip types inside the binocular monitoring camera include, but are not limited to, ARM (Advanced RISC Machine), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processer).

[0140] Some embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the external parameter calibration method of the binocular monitoring camera in any of the foregoing technical solutions.

[0141] In addition, in some other embodiments of the present disclosure, there is also provided a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the external parameter calibration method of the binocular monitoring camera in any of the above embodiments.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0143] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising such element.

[0144] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the device, electronic device, and computer-readable storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0145] The above are only the preferred embodiments of the present disclosure and are not intended to limit the protection scope of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure are all included in the protection scope of the present disclosure.

Claims

1. A method for calibrating the external parameters of a binocular monitoring camera, the binocular monitoring camera comprising a left-eye camera and a right-eye camera, characterized in that, The external parameter calibration method includes: Obtaining a first scene image collected by a left-eye camera and a second scene image collected by a right-eye camera at the same moment; Obtaining a scene depth image based on the first scene image and the second scene image; Extracting a background depth image defined by a reference area in the scene depth image, where the reference area is determined according to the still object area in the initialized depth image of the scene taken after the binocular monitoring camera is installed on-site and externally parameter initially calibrated; When the depth distribution density of the background depth image is less than the density threshold, and / or, when the average depth error between the background depth image and the reference depth image defined by the reference area in the initialized depth image of the scene is greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are damaged; when the depth distribution density of the background depth image is not less than the density threshold, and the average depth error between the background depth image and the reference depth image defined by the reference area in the initialized depth image of the scene is not greater than the error threshold, it is determined that the external parameters of the binocular monitoring camera are not damaged; When it is determined that the external parameters of the binocular monitoring camera are damaged, determining the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image; Calibrating the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

2. The external parameter calibration method according to claim 1, wherein Among the left-eye camera and the right-eye camera, one is the main camera and the other is the auxiliary camera; determining the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image includes: Determining the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the coordinate system rotation of the auxiliary camera with the main camera as the reference according to the reference depth image.

3. The external parameter calibration method according to claim 1, wherein Determining the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the coordinate system rotation of the auxiliary camera with the main camera as the reference according to the reference depth image includes: Iterate through multiple roll angles roll within the set roll angle value range and multiple pitch angles pitch within the set pitch angle value range, and in each access loop, based on the roll angle roll, pitch angle pitch, and initial yaw angle yaw 00 , obtain the first depth distribution density of the depth image defined by the reference area corresponding thereto; Determining the roll angle roll0 and pitch angle pitch0 corresponding to the maximum value in the first depth distribution density corresponding to each access cycle; Traversing multiple yaw angles yaw within the set yaw angle value range, and when accessing each yaw angle yaw, obtaining the first average depth error between the depth image defined by the reference area and the reference depth image according to the yaw angle yaw, roll angle roll0, and pitch angle pitch0; Determining the yaw angle yaw0 corresponding to the minimum value among the first average depth errors obtained when accessing each yaw angle yaw; Obtaining the second depth distribution density of the depth image defined by the reference area according to the yaw angle yaw0, roll angle roll0, and pitch angle pitch0; When the second depth distribution density is not less than the density threshold, determining the yaw angle compensation coefficient, roll angle compensation coefficient, and pitch angle compensation coefficient for the coordinate system rotation of the auxiliary camera with the main camera as the reference according to the yaw angle yaw0, roll angle roll0, and pitch angle pitch0; otherwise, adjusting and increasing the set roll angle value range and set pitch angle value range, and returning to the step of cyclic traversal.

4. The external parameter calibration method according to claim 1, characterized in that It further includes: When it is determined that the external parameters of the binocular monitoring camera are damaged, outputting an external parameter damage prompt message; and / or When it is determined that the external parameters of the binocular monitoring camera are not damaged, outputting the scene depth image.

5. The external parameter calibration method according to claim 1, wherein Based on the first scene image and the second scene image, a scene depth image is obtained, including: Based on the first scene image, multiple first to-be-processed scene images with different resolutions are obtained; Based on the second scene image, multiple second to-be-processed scene images with the same resolutions as the multiple first to-be-processed scene images are obtained one by one; For each resolution, based on the corresponding first to-be-processed scene image and second to-be-processed scene image, a to-be-processed scene depth image is obtained; The to-be-processed scene depth images of multiple resolutions are subjected to fusion processing to obtain the scene depth image.

6. The external parameter calibration method according to claim 1 or 5, characterized in that It further includes: After obtaining the first scene image and the second scene image and before obtaining the scene depth image, binocular parallel correction processing is performed on the first scene image and the second scene image.

7. The external parameter calibration method according to claim 1, characterized in that The distance between the optical axes of the left-eye camera and the right-eye camera is between 20 cm and 300 cm.

8. An external parameter calibration device for a binocular monitoring camera, the binocular monitoring camera comprising a left-eye camera and a right-eye camera, characterized in that, The external parameter calibration device includes: An acquisition unit for acquiring the first scene image acquired by the left-eye camera and the second scene image acquired by the right-eye camera at the same moment; A processing unit for obtaining a scene depth image based on the first scene image and the second scene image; An extraction unit for extracting a background depth image defined by a reference area in the scene depth image, where the reference area is determined according to the still-life area in the scene initialization depth image captured after the binocular monitoring camera is installed on-site and the external parameter initial calibration is performed; A judgment unit for determining that the external parameters of the binocular monitoring camera are damaged when the depth distribution density of the background depth image is less than the density threshold, and / or, the average depth error of the background depth image compared with the reference depth image defined by the reference area in the scene initialization depth image is greater than the error threshold; determining that the external parameters of the binocular monitoring camera are not damaged when the depth distribution density of the background depth image is not less than the density threshold and the average depth error of the background depth image compared with the reference depth image defined by the reference area in the scene initialization depth image is not greater than the error threshold; A determination unit for determining the external parameter compensation coefficient of the binocular monitoring camera according to the reference depth image when it is determined that the external parameters of the binocular monitoring camera are damaged; A calibration unit for calibrating the external parameters of the binocular monitoring camera according to the external parameter compensation coefficient.

9. A binocular monitoring camera, characterized in that, It includes: A memory and a processor coupled to the memory, and the processor is configured to execute the external parameter calibration method of the binocular monitoring camera according to any one of claims 1-7 based on the instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by the processor, the external parameter calibration method of the binocular monitoring camera according to any one of claims 1-7 is implemented.

Images