An internal reference calibration method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a kind of internal reference calibration method, device, electronic equipment and storage medium, involve computer technology field, including obtaining the reference size of target measurement area in the image acquisition area of binocular camera;Obtain the depth image of image acquisition area;Utilize depth image to determine the target size of target measurement area, calculate the size error of target size relative to reference size;The ratio of the number of valid pixel points in depth image accounts for the total number of all pixel points;Utilize size error and ratio, determine the cost function of current internal reference of binocular camera;Adjust the current internal reference of binocular camera, return the step of obtaining the depth image of image acquisition area, obtain the cost function of multiple internal reference;The internal reference that cost function meets preset cost condition is determined as the internal reference of binocular camera after calibration.The scheme provided in the present application embodiment can reduce the complexity of internal reference calibration, and improve the accuracy of internal reference calibration.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an internal reference calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Binocular cameras are widely used in scenarios such as passenger flow statistics, volume measurement, and driver assistance. During application, vibration, impact, thermal expansion, and other factors may cause deformation of the binocular camera, which will change the pose relationship between the left and right cameras and thus change the intrinsic parameters of the binocular camera. In this case, the intrinsic parameters of the binocular camera need to be calibrated.

[0003] In related technologies, to calibrate the intrinsic parameters of a stereo camera, it is necessary to obtain images of the left and right cameras respectively under an image acquisition scenario, extract feature points from the left and right camera images respectively, match the feature points between the left and right camera images to obtain feature point pairs, and solve the pose relationship between the left and right cameras based on the positions of the feature point pairs in the left and right camera images and the epipolar geometry relationship between the left and right cameras. Then, change the image acquisition scenario of the stereo camera to obtain images of the left and right cameras under the new scenario, and recalculate the pose relationship between the left and right cameras based on the above images and the previously obtained pose relationship. Repeat the above steps to achieve multiple iterations of calculating the pose relationship, and finally obtain a stable pose relationship. Finally, use the stable pose relationship to calibrate the intrinsic parameters of the stereo camera.

[0004] While the above-mentioned approach can calibrate the intrinsic parameters of a stereo camera, it has several drawbacks. First, the calibration process requires multiple camera movements to change the image acquisition scene and capture images of different scenarios, making it unsuitable for static applications where cameras are fixed and cannot be moved, such as passenger flow statistics or volume measurement. Second, the above approach relies on feature point matching, which is highly dependent on the image acquisition scene and has low robustness. When there are few or concentrated feature points in the image acquisition scene, the obtained feature point pairs are difficult to accurately reflect the pose relationship between the left and right cameras, thus making it difficult to accurately calibrate the intrinsic parameters of the stereo camera.

[0005] This shows that the related technologies suffer from high complexity and low accuracy in internal parameter calibration. Summary of the Invention

[0006] The purpose of this application is to provide an internal reference calibration method, apparatus, electronic device, and storage medium to reduce the complexity of internal reference calibration and improve its accuracy. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide an internal reference calibration method, the method comprising:

[0008] Obtain the reference dimensions of the target measurement area within the image acquisition area of ​​the binocular camera;

[0009] Obtain a depth image of the image acquisition area, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the stereo camera using the current intrinsic parameters of the stereo camera;

[0010] The target size of the target measurement area is determined using the depth image, and the size error of the target size relative to the reference size is calculated.

[0011] Calculate the ratio of the number of valid pixels in the depth image to the total number of all pixels, wherein the valid pixels are: pixels in the depth image whose pixel values ​​can reflect actual depth information;

[0012] Using the size error and ratio, a cost function for the current intrinsic parameters of the binocular camera is determined, wherein the cost function is used to reflect the magnitude of the error of the current intrinsic parameters of the binocular camera;

[0013] Adjust the current intrinsic parameters of the binocular camera, return to the step of obtaining the depth image of the image acquisition area, and obtain a cost function for multiple intrinsic parameters;

[0014] The intrinsic parameters that satisfy the preset cost conditions of the cost function are determined as the intrinsic parameters after the binocular camera is calibrated.

[0015] In one embodiment of this application, the cost function for determining the current intrinsic parameters of the binocular camera using the size error and ratio includes:

[0016] The cost function c of the current intrinsic parameters of the stereo camera is calculated according to the following formula:

[0017] c = α*(1-S1) + β*S2

[0018] Wherein, α is a preset first weight, β is a preset second weight, S1 represents the ratio, and S2 represents the size error.

[0019] In one embodiment of this application, obtaining the depth image of the image acquisition area includes:

[0020] Obtain the left and right camera images captured by the left and right cameras of the stereo camera;

[0021] The images obtained from the left and right cameras are corrected using the current intrinsic parameters of the binocular cameras respectively;

[0022] Image processing is performed on the corrected left and right camera images to obtain a depth image reflecting the depth information of the image acquisition area, and the confidence level of each pixel in the depth image is obtained. The confidence level of each pixel represents the probability that the depth information reflected by the pixel value of that pixel is the actual depth information.

[0023] The calculation of the ratio of the number of valid pixels in the depth image to the total number of pixels includes:

[0024] The number of valid pixels in the depth image with a confidence level higher than a preset confidence threshold is counted.

[0025] Calculate the ratio of the number of valid pixels to the total number of all pixels.

[0026] In one embodiment of this application, adjusting the current intrinsic parameters of the binocular camera includes:

[0027] The descent gradient of the cost function is calculated using a nonlinear optimization algorithm, and the adjustment step size of the binocular camera intrinsic parameters is determined based on the descent gradient.

[0028] Adjust the current intrinsic parameters of the binocular camera according to the adjustment step size.

[0029] Secondly, embodiments of this application provide an internal reference calibration device, the device comprising:

[0030] The reference size acquisition module is used to obtain the reference size of the target measurement area within the image acquisition area of ​​the binocular camera;

[0031] A depth image acquisition module is used to acquire a depth image of the image acquisition area, wherein the depth image is an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the stereo camera using the current intrinsic parameters of the stereo camera;

[0032] The size error calculation module is used to determine the target size of the target measurement area using the depth image, and to calculate the size error of the target size relative to the reference size.

[0033] The ratio calculation module is used to calculate the ratio of the number of effective pixels in the depth image to the total number of all pixels, wherein the effective pixels are: pixels in the depth image whose pixel values ​​can reflect actual depth information;

[0034] The cost function determination module is used to determine the cost function of the current intrinsic parameters of the binocular camera using the size error and ratio;

[0035] The intrinsic parameter adjustment module is used to adjust the current intrinsic parameters of the binocular camera, trigger the depth image acquisition module, and obtain a cost function of multiple intrinsic parameters.

[0036] The intrinsic parameter calibration module is used to determine the intrinsic parameters of the cost function that satisfy the preset cost conditions as the calibrated intrinsic parameters of the binocular camera.

[0037] In one embodiment of this application, the cost function determination module is specifically used for:

[0038] The cost function c of the current intrinsic parameters of the stereo camera is calculated according to the following formula:

[0039] c = α*(1-S1) + β*S2

[0040] Wherein, α is a preset first weight, β is a preset second weight, S1 represents the ratio, and S2 represents the size error.

[0041] In one embodiment of this application, the depth image acquisition module is specifically used for:

[0042] Obtain the left and right camera images captured by the left and right cameras of the stereo camera;

[0043] The images obtained from the left and right cameras are corrected using the current intrinsic parameters of the binocular cameras respectively;

[0044] Image processing is performed on the corrected left and right camera images to obtain a depth image reflecting the depth information of the image acquisition area, and the confidence level of each pixel in the depth image is obtained. The confidence level of each pixel represents the probability that the depth information reflected by the pixel value of that pixel is the actual depth information.

[0045] The ratio calculation module is specifically used for:

[0046] The number of valid pixels in the depth image with a confidence level higher than a preset confidence threshold is counted.

[0047] Calculate the ratio of the number of valid pixels to the total number of all pixels.

[0048] In one embodiment of this application, the internal parameter adjustment module is specifically used for:

[0049] The descent gradient of the cost function is calculated using a nonlinear optimization algorithm, and the adjustment step size of the binocular camera intrinsic parameters is determined based on the descent gradient.

[0050] Adjust the current intrinsic parameters of the binocular camera according to the adjustment step size, trigger the depth image acquisition module, and obtain a cost function for multiple intrinsic parameters.

[0051] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] Memory, used to store computer programs;

[0053] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0055] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the internal parameter calibration methods described above.

[0056] Beneficial effects of the embodiments in this application:

[0057] The intrinsic parameter calibration scheme provided in this application embodiment can obtain the reference size of the target measurement area within the image acquisition area of ​​the binocular camera; obtain a depth image of the image acquisition area, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the binocular camera using the current intrinsic parameters of the binocular camera; determine the target size of the target measurement area using the depth image, and calculate the size error of the target size relative to the reference size; calculate the ratio of the number of effective pixels in the depth image to the total number of all pixels, wherein effective pixels are: pixels in the depth image whose pixel values ​​can reflect the actual depth information; determine the cost function of the current intrinsic parameters of the binocular camera using the size error and the ratio, wherein the cost function is used to reflect the error magnitude of the current intrinsic parameters of the binocular camera; adjust the current intrinsic parameters of the binocular camera, return to the step of obtaining the depth image of the image acquisition area, and obtain the cost functions of multiple intrinsic parameters; determine the intrinsic parameters whose cost functions satisfy the preset cost conditions as the calibrated intrinsic parameters of the binocular camera.

[0058] On the one hand, this allows for the calibration of the intrinsic parameters of the binocular camera based on a fixed target measurement area. During the calibration process, there is no need to move the binocular camera to change the image acquisition scene. This is suitable for calibrating the intrinsic parameters of the binocular camera in static scenes where the binocular camera is fixed and cannot be moved.

[0059] On the other hand, in the solution provided by the embodiments of this application, the cost function of the intrinsic parameters of the binocular camera is calculated by using the size error and ratio, and the calibrated intrinsic parameters are determined based on the cost function. The intrinsic parameters are determined without the need to rely on feature point pairs, which reduces the dependence on the image acquisition scene and improves the robustness of the intrinsic parameter calibration. When there are few or relatively concentrated feature points in the image acquisition scene, the intrinsic parameter calibration of the binocular camera can also be achieved.

[0060] Therefore, the solution provided in the embodiments of this application can reduce the complexity of internal parameter calibration and improve the accuracy of internal parameter calibration. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0062] Figure 1 A flowchart illustrating an internal parameter calibration method provided in this application embodiment;

[0063] Figure 2 A schematic flowchart of another internal parameter calibration method provided in this application embodiment;

[0064] Figure 3 This is a schematic diagram of the structure of an internal reference calibration device provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0067] To reduce the complexity of internal reference calibration and improve its accuracy, embodiments of this application provide an internal reference calibration method, apparatus, electronic device, and storage medium, which will be described in detail below.

[0068] See Figure 1 , Figure 1This application provides a flowchart illustrating an intrinsic parameter calibration method. This method can be applied to electronic devices such as computers, servers, NVRs (Network Video Recorders), DVRs (Digital Video Recorders), and stereo cameras. The aforementioned intrinsic parameter calibration method for stereo cameras can be used in applications such as automotive driver assistance, robot navigation, passenger flow statistics, and volumetric measurement to address the problem of intrinsic parameter failure in stereo cameras deployed in these scenarios.

[0069] like Figure 1 As shown, the above internal reference calibration method includes the following steps S101-S107:

[0070] S101, obtain the reference size of the target measurement area within the image acquisition area of ​​the binocular camera.

[0071] The target measurement area can be the area where a graphic drawn within the image acquisition area of ​​the binocular camera is located. The graphic can be a square frame, a triangle frame, a pentagram, etc. Alternatively, the target measurement area can also be the area where an object is located within the image acquisition area of ​​the binocular camera. The object can be a ball, a water cup, a calibration plate, a box, a sign, etc.

[0072] Specifically, the actual size of the target measurement area within the image acquisition area of ​​the stereo camera can be obtained as a reference size. This size can include the length, width, height, and radius of the target measurement area, as well as the distance of the target measurement area relative to the stereo camera.

[0073] In one embodiment of this application, the size of the target measurement area, obtained manually, can be used as a reference size;

[0074] In addition, parameter information of the target measurement area can be obtained, and the size of the target measurement area can be determined from the parameter information as a reference size. For example, assuming that the target measurement area is the area where a preset sign is located, the parameter information of the sign can be obtained, and the length, width, height and other information of the sign can be obtained from the parameter information as a reference size of the target measurement area.

[0075] S102, Obtain the depth image of the image acquisition area.

[0076] Among them, the depth image is: an image that reflects the depth information of the image acquisition area, obtained by processing the images acquired by the left and right cameras of the stereo camera using the current intrinsic parameters of the stereo camera.

[0077] The aforementioned internal parameters may include: the focal length of the binocular camera, pixel size, the position of the left and right cameras, and the pose relationship between the left and right cameras.

[0078] Specifically, the binocular camera includes left and right cameras, namely the left eye camera and the right eye camera. The left and right cameras are used to acquire images of the image acquisition area, and the left eye camera image and the right eye camera image are acquired by the right eye camera. These are used as the left and right camera images. Then, the current intrinsic parameters of the binocular camera are used to process the left and right camera images to obtain a depth image that reflects the depth information of the image acquisition area.

[0079] In one embodiment of this application, the depth image sent by the binocular camera can be directly obtained. Specifically, after the binocular camera acquires images from the left and right cameras in the image acquisition area, it can use the current intrinsic parameters of the binocular camera to perform image processing on the left and right camera images to obtain a depth image reflecting the depth information of the image acquisition area. Then, the depth image is sent to the electronic device, which then obtains the depth image of the image acquisition area.

[0080] In addition, after the binocular camera acquires images from the left and right cameras in the image acquisition area, it can send the images to an electronic device. The electronic device obtains the current intrinsic parameters of the binocular camera and uses these intrinsic parameters to process the received images from the left and right cameras to obtain a depth image that reflects the depth information of the image acquisition area.

[0081] S103, use the depth image to determine the target size of the target measurement area, and calculate the size error of the target size relative to the reference size.

[0082] Specifically, since the target measurement area is located within the image acquisition area of ​​the stereo camera, the aforementioned depth image contains the target measurement area. The pixel values ​​of the pixels in the depth image reflect depth information. Using the depth information reflected by the pixel values ​​in the obtained depth image, the size of the target measurement area in the depth image can be determined as the target size. After obtaining the target size, the error between the target size and the reference size can be calculated as the size error.

[0083] Since the target size mentioned above is obtained based on the depth image, and the depth image is obtained based on the current intrinsic parameters of the stereo camera, the size error mentioned above can reflect the error of the current intrinsic parameters of the stereo camera. The larger the size error, the larger the error of the current intrinsic parameters, and the smaller the size error, the smaller the error of the current intrinsic parameters.

[0084] In one embodiment of this application, when determining the target size of a target measurement area using a depth image, the pixel value of the pixel point reflecting the target measurement area can be determined from the depth image. Since the pixel value of each pixel point in the depth image can reflect depth information, the distance of the target measurement area relative to the binocular camera can be obtained based on the above pixel value, and the above distance can be used as the target size of the target measurement area.

[0085] In another embodiment of this application, point cloud data corresponding to a pixel can be obtained based on the pixel value of the pixel, and then the target size of the target measurement area can be obtained using the point cloud data.

[0086] In one embodiment of this application, when calculating the dimensional error of the target size relative to the reference size, the difference between the target size and the reference size, the absolute value of the difference, etc., can be calculated as the dimensional error. Alternatively, the above difference, the absolute value of the difference, etc., can be further weighted to obtain the dimensional error.

[0087] S104, calculate the ratio of the number of valid pixels in the depth image to the total number of all pixels.

[0088] Among them, effective pixels are: pixels in the depth image whose pixel values ​​can reflect actual depth information.

[0089] Specifically, pixels whose pixel values ​​can reflect actual depth information can be identified from the depth image as valid pixels. Then, the number of valid pixels is counted, and the ratio of the number of valid pixels to the total number of pixels in the depth image is calculated.

[0090] The above ratio reflects the density of the depth image; the higher the density, the higher the accuracy of the depth image. Since the depth image is obtained based on the current intrinsic parameters of the stereo camera, the density also reflects the accuracy of the current intrinsic parameters. The higher the density, the smaller the error of the current intrinsic parameters and the higher the accuracy; the lower the density, the larger the error of the current intrinsic parameters and the lower the accuracy.

[0091] S105, a cost function for determining the current intrinsic parameters of the binocular camera using size error and ratio.

[0092] The cost function is used to reflect the error magnitude of the current intrinsic parameters of the stereo camera.

[0093] Specifically, the aforementioned size error and ratio both reflect the error of the stereo camera's current intrinsic parameters. A smaller size error and higher density indicate a smaller error in the current intrinsic parameters; conversely, a larger size error and lower density indicate a larger error in the current intrinsic parameters. Based on this, the cost function for the stereo camera's current intrinsic parameters can be determined using the aforementioned size error and ratio. This cost function also reflects the magnitude of the error in the stereo camera's current intrinsic parameters. A smaller cost function indicates a smaller size error and a higher ratio obtained based on the current intrinsic parameters, thus indicating a smaller error in the current intrinsic parameters; conversely, a larger cost function indicates a larger size error and a smaller ratio obtained based on the current intrinsic parameters, thus indicating a larger error in the current intrinsic parameters.

[0094] S106, adjust the current intrinsic parameters of the binocular camera, return to step S102, and obtain the cost function of multiple intrinsic parameters.

[0095] Specifically, the intrinsic parameters of the binocular camera can be adjusted, and then the process can return to step S102 above. Using the adjusted intrinsic parameters, the depth image of the image acquisition area can be obtained again. Based on the newly obtained depth image, the new target size of the target measurement area can be obtained, and the ratio of the number of effective pixels in the new depth image to the total number of all pixels can be obtained. Then, the cost function of the adjusted intrinsic parameters can be determined using the new target size and the new ratio. In this way, after each return to step S102 above, a new adjusted intrinsic parameter and the cost function of the adjusted intrinsic parameter can be obtained. Finally, multiple intrinsic parameters and their cost functions can be obtained.

[0096] In one embodiment of this application, a preset number of intrinsic parameters can be pre-set. When adjusting the intrinsic parameters of the binocular camera, the preset number of intrinsic parameters can be iterated through sequentially. After the iteration is complete, the cost function of the preset number of intrinsic parameters is obtained. The preset number can be 5, 10, 30, etc.

[0097] In addition, in one embodiment of this application, the current intrinsic parameters of the binocular camera can be adjusted according to a preset adjustment step size. In this case, the loop can be terminated when the number of iterations reaches a preset threshold, and the cost function of the aforementioned threshold plus one intrinsic parameter can be obtained.

[0098] In one embodiment of this application, when adjusting the intrinsic parameters of a stereo camera, all parameters of the stereo camera can be adjusted, or only some parameters of the stereo camera's intrinsic parameters can be adjusted. For example, assuming that the intrinsic parameters of the stereo camera include parameters such as focal length, vertical field of view, horizontal field of view, pose, and distortion coefficient, when adjusting the intrinsic parameters of the stereo camera, since the pose of the stereo camera can be obtained through a pose sensor, only parameters such as the focal length, vertical field of view, horizontal field of view, and distortion coefficient can be adjusted.

[0099] S107, the intrinsic parameters of the cost function that satisfy the preset cost conditions are determined as the intrinsic parameters after the binocular camera calibration.

[0100] The aforementioned cost condition is used to determine whether the corresponding intrinsic parameters meet the requirements based on the cost function. It can be understood that when the cost function of the intrinsic parameters meets the cost condition, it means that the size error calculated using the intrinsic parameters is small and the ratio of the number of effective pixels in the obtained depth image to the total number of all pixels is high, thus indicating that the error of the intrinsic parameters is small relative to the true intrinsic parameters of the stereo camera.

[0101] Specifically, after obtaining the cost functions of multiple intrinsic parameters, the intrinsic parameters that satisfy the preset cost conditions can be determined from the above multiple intrinsic parameters and used as the intrinsic parameters after the binocular camera is calibrated.

[0102] In one embodiment of this application, the aforementioned cost condition can be the minimum cost function. In this case, the intrinsic parameter with the minimum cost function can be selected from multiple intrinsic parameters as the intrinsic parameter after binocular camera calibration.

[0103] In addition, the aforementioned cost condition can also be that the cost function is less than a preset evaluation threshold. In this case, an intrinsic parameter whose cost function is less than the aforementioned evaluation threshold can be selected from multiple intrinsic parameters, and then one of these intrinsic parameters can be selected as the intrinsic parameter after binocular camera calibration.

[0104] In the intrinsic parameter calibration scheme provided in the above embodiments, the reference size of the target measurement area within the image acquisition area of ​​the binocular camera can be obtained; a depth image of the image acquisition area can be obtained, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the binocular camera using the current intrinsic parameters of the binocular camera; the target size of the target measurement area can be determined using the depth image, and the size error of the target size relative to the reference size can be calculated; the ratio of the number of effective pixels in the depth image to the total number of all pixels can be calculated, wherein effective pixels are: pixels in the depth image whose pixel values ​​can reflect the actual depth information; the cost function of the current intrinsic parameters of the binocular camera can be determined using the size error and the ratio, wherein the cost function is used to reflect the error magnitude of the current intrinsic parameters of the binocular camera; the current intrinsic parameters of the binocular camera can be adjusted, and the step of obtaining the depth image of the image acquisition area can be returned to obtain the cost functions of multiple intrinsic parameters; the intrinsic parameters whose cost functions satisfy the preset cost conditions are determined as the intrinsic parameters after binocular camera calibration.

[0105] On the one hand, this allows for the calibration of the intrinsic parameters of the binocular camera based on a fixed target measurement area. During the calibration process, there is no need to move the binocular camera to change the image acquisition scene. This is suitable for calibrating the intrinsic parameters of the binocular camera in static scenes where the binocular camera is fixed and cannot be moved.

[0106] On the other hand, in the solution provided by the above embodiments, the cost function of the intrinsic parameters of the binocular camera is calculated by using the size error and ratio, and the calibrated intrinsic parameters are determined based on the cost function. The intrinsic parameters are determined without the need to rely on feature point pairs, which reduces the dependence on the image acquisition scene and improves the robustness of the intrinsic parameter calibration. When there are few or relatively concentrated feature points in the image acquisition scene, the intrinsic parameter calibration of the binocular camera can also be achieved.

[0107] Therefore, the solution provided in the above embodiments can reduce the complexity of internal parameter calibration and improve the accuracy of internal parameter calibration.

[0108] In one embodiment of this application, when determining the cost function in step S105 above, the cost function c of the current intrinsic parameters of the stereo camera can be calculated according to the following formula:

[0109] c = α*(1-S1) + β*S2

[0110] Wherein, α is a preset first weight, β is a preset second weight, S1 represents the ratio, and S2 represents the dimensional error. The first weight can be 0.5, 0.6, 0.3, 1.2, 4, etc., and the second weight can be 0.4, 0.7, 2.5, 6, etc. The sum of the first weight and the second weight can be 1 or not 1, and the embodiments of this application do not limit this.

[0111] Alternatively, the proportion of the aforementioned dimensional error to the aforementioned ratio can be directly calculated to obtain the cost function, but this application does not limit this approach.

[0112] In one embodiment of this application, when obtaining the depth image in step S102 above, the left and right camera images captured by the left and right cameras of the stereo camera can be obtained; the obtained left and right camera images are corrected using the current intrinsic parameters of the stereo camera; the corrected left and right camera images are processed to obtain a depth image reflecting the depth information of the image acquisition area, and the confidence level of each pixel in the depth image is obtained.

[0113] The confidence level of each pixel represents the probability that the depth information reflected by the pixel value is the actual depth information.

[0114] Specifically, the images from the left and right cameras can be corrected using the current intrinsic parameters of the binocular cameras. Then, stereo matching is performed on the corrected images to calculate the disparity map between them. Finally, the disparity map is converted into a depth map, thus obtaining a depth image reflecting the depth information of the image acquisition area. During the process of obtaining the depth image, the confidence level of each pixel in the depth image can also be obtained. This confidence level represents the probability that the depth information reflected by the pixel value is the actual depth information.

[0115] In the above scheme, when calculating the ratio of the number of valid pixels in the depth image to the total number of all pixels, the number of valid pixels in the depth image with a confidence level higher than a preset confidence level threshold can be counted; the ratio of the number of valid pixels to the total number of all pixels can be calculated.

[0116] Specifically, pixels with a confidence level higher than a preset confidence threshold can be identified from the depth image as valid pixels. Then, the number of these valid pixels is counted, and the ratio of the number of valid pixels to the total number of all pixels is calculated.

[0117] In one embodiment of this application, when adjusting the current intrinsic parameters of the stereo camera in step S106, a nonlinear optimization algorithm can be used to calculate the descent gradient of the cost function, and the adjustment step size of the stereo camera intrinsic parameters can be determined based on the descent gradient; the current intrinsic parameters of the stereo camera are adjusted according to the adjustment step size.

[0118] Specifically, after obtaining the cost function, a nonlinear optimization algorithm can be used to calculate the descent gradient of the cost function, and the adjustment step size of the intrinsic parameters can be determined based on the gradient. Then, the current intrinsic parameters are adjusted according to the determined adjustment step size. This can reduce the cost function of the adjusted intrinsic parameters, which helps to obtain better intrinsic parameters.

[0119] In the above scheme, it can be determined whether the calculated descent gradient has reached the preset gradient threshold. If not, the adjustment step size of the intrinsic parameter is determined based on the gradient. Then, the current intrinsic parameter is adjusted according to the determined adjustment step size, and the process returns to step S102 to continue to obtain the cost function of the adjusted intrinsic parameter until the calculated descent gradient reaches the preset gradient threshold. Then, the intrinsic parameter is no longer adjusted.

[0120] In this case, for step S105, the intrinsic parameters obtained after the last adjustment can be directly used as the intrinsic parameters after the binocular camera is calibrated.

[0121] Specifically, when the descent gradient reaches the preset gradient threshold, it indicates that the calculated cost function is gradually stabilizing and no longer decreasing significantly. In this case, the intrinsic parameters can be adjusted without further adjustment, and the intrinsic parameters obtained after the last adjustment can be directly used as the intrinsic parameters after the binocular camera calibration.

[0122] In the above scheme, the intrinsic parameters of the stereo camera can be continuously optimized, so that the corresponding cost function continues to decrease. Finally, the intrinsic parameters corresponding to the minimum cost function are determined as the intrinsic parameters after stereo camera calibration. In this case, the smaller the size error and the larger the ratio obtained based on the calibrated intrinsic parameters, the smaller the error of the obtained intrinsic parameters, thereby realizing the calibration of the stereo camera intrinsic parameters.

[0123] See Figure 2, Figure 2 This is a flowchart illustrating another internal parameter calibration method provided in an embodiment of this application. The method includes the following steps S201-S207:

[0124] S201, obtain the reference size of the target measurement area within the image acquisition area of ​​the binocular camera.

[0125] S202, obtain the depth image of the image acquisition area, and obtain the confidence level of each pixel in the depth image.

[0126] Among them, the depth image is: an image that reflects the depth information of the image acquisition area, obtained by processing the images acquired by the left and right cameras of the stereo camera using the current intrinsic parameters of the stereo camera.

[0127] The confidence level of each pixel represents the probability that the depth information reflected by the pixel value is the actual depth information.

[0128] S203, count the number of valid pixels in the depth image with a confidence level higher than a preset confidence threshold, and calculate the ratio of the number of valid pixels to the total number of all pixels as the density of the depth image.

[0129] S204, use the depth image to determine the target size of the target measurement area, and calculate the size error of the target size relative to the reference size.

[0130] S205 uses size error and density to determine the cost function of the current intrinsic parameters of the stereo camera.

[0131] The cost function is used to reflect the error magnitude of the current intrinsic parameters of the stereo camera.

[0132] Specifically, the cost function c of the current intrinsic parameters of the stereo camera can be calculated using the following formula:

[0133] c = α*(1-S1) + β*S2

[0134] Where α is the preset first weight, β is the preset second weight, S1 represents the density, and S2 represents the size error.

[0135] S206, use a nonlinear optimization algorithm to calculate the descent gradient of the cost function, and determine the adjustment step size of the binocular camera intrinsic parameters based on the descent gradient; adjust the current intrinsic parameters of the binocular camera according to the adjustment step size, return to step S202, and obtain the cost function of multiple intrinsic parameters until the calculated descent gradient reaches the preset gradient threshold.

[0136] S207, the internal parameters obtained after the last adjustment are determined as the internal parameters after the binocular camera calibration.

[0137] In the intrinsic parameter calibration scheme provided in the above embodiments, a region can first be selected in the image acquisition area of ​​the binocular camera as the target measurement region. Then, the size information of the region is precisely measured manually to obtain the reference size. The size information includes, but is not limited to, the vertical distance from the region to the camera, the length and width of the region, etc. Next, the depth map is calculated using the current intrinsic parameters of the binocular camera to obtain a depth image. The size information of the target measurement region is obtained using the depth image as the target size. The error of the target size relative to the reference size is calculated, and the density of the depth image is determined. The cost function corresponding to the current intrinsic parameters is calculated using the density and the size error.

[0138] At this point, the error of the current intrinsic parameters may be large, resulting in a low density and a large size error, thus leading to a high cost function result. Therefore, the intrinsic parameters of the stereo camera can be fine-tuned based on the current intrinsic parameters to reduce the cost function. The adjusted intrinsic parameters can then be used as the current intrinsic parameters and substituted into the next depth map calculation and intrinsic parameter adjustment step until the iteration ends.

[0139] On the one hand, this allows for the calibration of the intrinsic parameters of the binocular camera based on a fixed target measurement area. During the calibration process, there is no need to move the binocular camera to change the image acquisition scene. This is suitable for calibrating the intrinsic parameters of the binocular camera in static scenes where the binocular camera is fixed and cannot be moved.

[0140] On the other hand, in the solution provided by the above embodiments, the cost function of the intrinsic parameters of the binocular camera is calculated by using the size error and ratio, and the calibrated intrinsic parameters are determined based on the cost function. The intrinsic parameters are determined without the need to rely on feature point pairs, which reduces the dependence on the image acquisition scene and improves the robustness of the intrinsic parameter calibration. When there are few or relatively concentrated feature points in the image acquisition scene, the intrinsic parameter calibration of the binocular camera can also be achieved.

[0141] Therefore, the solution provided in the above embodiments can reduce the complexity of internal parameter calibration and improve the accuracy of internal parameter calibration.

[0142] Corresponding to the above-described internal reference calibration method, this application also provides an internal reference calibration device, which will be described in detail below.

[0143] See Figure 3 , Figure 3 This is a schematic diagram of an internal reference calibration device provided in an embodiment of this application. The device includes:

[0144] Reference size acquisition module 301 is used to obtain the reference size of the target measurement area within the image acquisition area of ​​the binocular camera;

[0145] The depth image acquisition module 302 is used to acquire a depth image of the image acquisition area, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the stereo camera using the current intrinsic parameters of the stereo camera;

[0146] The size error calculation module 303 is used to determine the target size of the target measurement area using the depth image, and to calculate the size error of the target size relative to the reference size.

[0147] The ratio calculation module 304 is used to calculate the ratio of the number of effective pixels in the depth image to the total number of all pixels, wherein the effective pixels are: pixels in the depth image whose pixel values ​​can reflect actual depth information;

[0148] The cost function determination module 305 is used to determine the cost function of the current intrinsic parameters of the binocular camera using the size error and ratio;

[0149] The intrinsic parameter adjustment module 306 is used to adjust the current intrinsic parameters of the binocular camera, trigger the depth image acquisition module, and obtain a cost function of multiple intrinsic parameters.

[0150] The intrinsic parameter calibration module 307 is used to determine the intrinsic parameters of the cost function that satisfy the preset cost conditions as the calibrated intrinsic parameters of the binocular camera.

[0151] In one embodiment of this application, the cost function determination module 305 is specifically used for:

[0152] The cost function c of the current intrinsic parameters of the stereo camera is calculated according to the following formula:

[0153] c = α*(1-S1) + β*S2

[0154] Wherein, α is a preset first weight, β is a preset second weight, S1 represents the ratio, and S2 represents the size error.

[0155] In one embodiment of this application, the depth image acquisition module 302 is specifically used for:

[0156] Obtain the left and right camera images captured by the left and right cameras of the stereo camera;

[0157] The images obtained from the left and right cameras are corrected using the current intrinsic parameters of the binocular cameras respectively;

[0158] Image processing is performed on the corrected left and right camera images to obtain a depth image reflecting the depth information of the image acquisition area, and the confidence level of each pixel in the depth image is obtained. The confidence level of each pixel represents the probability that the depth information reflected by the pixel value of that pixel is the actual depth information.

[0159] The ratio calculation module 304 is specifically used for:

[0160] The number of valid pixels in the depth image with a confidence level higher than a preset confidence threshold is counted.

[0161] Calculate the ratio of the number of valid pixels to the total number of all pixels.

[0162] In one embodiment of this application, the internal parameter adjustment module 306 is specifically used for:

[0163] The descent gradient of the cost function is calculated using a nonlinear optimization algorithm, and the adjustment step size of the binocular camera intrinsic parameters is determined based on the descent gradient.

[0164] Adjust the current intrinsic parameters of the binocular camera according to the adjustment step size, trigger the depth image acquisition module, and obtain a cost function for multiple intrinsic parameters.

[0165] In the intrinsic parameter calibration scheme provided in the above embodiments, the reference size of the target measurement area within the image acquisition area of ​​the binocular camera can be obtained; a depth image of the image acquisition area can be obtained, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the binocular camera using the current intrinsic parameters of the binocular camera; the target size of the target measurement area can be determined using the depth image, and the size error of the target size relative to the reference size can be calculated; the ratio of the number of effective pixels in the depth image to the total number of all pixels can be calculated, wherein effective pixels are: pixels in the depth image whose pixel values ​​can reflect the actual depth information; the cost function of the current intrinsic parameters of the binocular camera can be determined using the size error and the ratio, wherein the cost function is used to reflect the error magnitude of the current intrinsic parameters of the binocular camera; the current intrinsic parameters of the binocular camera can be adjusted, and the step of obtaining the depth image of the image acquisition area can be returned to obtain the cost functions of multiple intrinsic parameters; the intrinsic parameters whose cost functions satisfy the preset cost conditions are determined as the intrinsic parameters after binocular camera calibration.

[0166] On the one hand, this allows for the calibration of the intrinsic parameters of the binocular camera based on a fixed target measurement area. During the calibration process, there is no need to move the binocular camera to change the image acquisition scene. This is suitable for calibrating the intrinsic parameters of the binocular camera in static scenes where the binocular camera is fixed and cannot be moved.

[0167] On the other hand, in the solution provided by the above embodiments, the cost function of the intrinsic parameters of the binocular camera is calculated by using the size error and ratio, and the calibrated intrinsic parameters are determined based on the cost function. The intrinsic parameters are determined without the need to rely on feature point pairs, which reduces the dependence on the image acquisition scene and improves the robustness of the intrinsic parameter calibration. When there are few or relatively concentrated feature points in the image acquisition scene, the intrinsic parameter calibration of the binocular camera can also be achieved.

[0168] Therefore, the solution provided in the above embodiments can reduce the complexity of internal parameter calibration and improve the accuracy of internal parameter calibration.

[0169] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0170] Memory 403 is used to store computer programs;

[0171] The processor 401 is used to implement the steps of the intrinsic parameter calibration method when executing the program stored in the memory 403.

[0172] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0173] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0174] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0175] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0176] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described intrinsic parameter calibration methods.

[0177] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the intrinsic parameter calibration methods described in the above embodiments.

[0178] In the intrinsic parameter calibration scheme provided in the above embodiments, the reference size of the target measurement area within the image acquisition area of ​​the binocular camera can be obtained; a depth image of the image acquisition area can be obtained, wherein the depth image is: an image reflecting the depth information of the image acquisition area obtained by image processing of the images acquired by the left and right cameras of the binocular camera using the current intrinsic parameters of the binocular camera; the target size of the target measurement area can be determined using the depth image, and the size error of the target size relative to the reference size can be calculated; the ratio of the number of effective pixels in the depth image to the total number of all pixels can be calculated, wherein effective pixels are: pixels in the depth image whose pixel values ​​can reflect the actual depth information; the cost function of the current intrinsic parameters of the binocular camera can be determined using the size error and the ratio, wherein the cost function is used to reflect the error magnitude of the current intrinsic parameters of the binocular camera; the current intrinsic parameters of the binocular camera can be adjusted, and the step of obtaining the depth image of the image acquisition area can be returned to obtain the cost functions of multiple intrinsic parameters; the intrinsic parameters whose cost functions satisfy the preset cost conditions are determined as the intrinsic parameters after binocular camera calibration.

[0179] On the one hand, this allows for the calibration of the intrinsic parameters of the binocular camera based on a fixed target measurement area. During the calibration process, there is no need to move the binocular camera to change the image acquisition scene. This is suitable for calibrating the intrinsic parameters of the binocular camera in static scenes where the binocular camera is fixed and cannot be moved.

[0180] On the other hand, in the solution provided by the above embodiments, the cost function of the intrinsic parameters of the binocular camera is calculated by using the size error and ratio, and the calibrated intrinsic parameters are determined based on the cost function. The intrinsic parameters are determined without the need to rely on feature point pairs, which reduces the dependence on the image acquisition scene and improves the robustness of the intrinsic parameter calibration. When there are few or relatively concentrated feature points in the image acquisition scene, the intrinsic parameter calibration of the binocular camera can also be achieved.

[0181] Therefore, the solution provided in the above embodiments can reduce the complexity of internal parameter calibration and improve the accuracy of internal parameter calibration.

[0182] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0183] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0184] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0185] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An internal reference calibration method, characterized by, The method comprises: obtaining a reference size of a target measurement region in an image acquisition region of a binocular camera; obtaining a depth image of the image acquisition region, wherein the depth image is an image reflecting depth information of the image acquisition region, which is obtained by image processing on images captured by left and right cameras of the binocular camera using current intrinsic parameters of the binocular camera; determining a target size of the target measurement region using the depth image, and calculating a size error of the target size relative to the reference size; counting a number of valid pixel points in the depth image whose confidence is higher than a preset confidence threshold, and calculating a ratio of the number of valid pixel points to a total number of all pixel points; wherein the valid pixel points are pixel points in the depth image whose pixel values can reflect actual depth information; determining a cost function of the current intrinsic parameters of the binocular camera using the size error and the ratio, wherein the cost function is used to reflect an error size of the current intrinsic parameters of the binocular camera; adjusting the current intrinsic parameters of the binocular camera, returning to the step of obtaining the depth image of the image acquisition region, and obtaining cost functions of multiple intrinsic parameters; determining intrinsic parameters whose cost functions satisfy a preset cost condition as calibrated intrinsic parameters of the binocular camera.

2. The method of claim 1, wherein, The step of determining the cost function of the current intrinsic parameters of the binocular camera using the size error and the ratio comprises: calculating the cost function c of the current intrinsic parameters of the binocular camera according to the following formula: ； wherein the α is a preset first weight, the β is a preset second weight, the S1 represents the ratio, and the S2 represents the size error.

3. The method of claim 1, wherein, The step of obtaining the depth image of the image acquisition region comprises: obtaining left and right camera images captured by left and right cameras of the binocular camera; correcting the obtained left and right camera images respectively using the current intrinsic parameters of the binocular camera; performing image processing on the corrected left and right camera images to obtain a depth image reflecting depth information of the image acquisition region, and obtaining a confidence of each pixel point in the depth image, wherein the confidence of each pixel point represents a probability that depth information reflected by a pixel value of the pixel point is actual depth information.

4. The method according to any one of claims 1 to 3, characterized in that, The step of adjusting the current intrinsic parameters of the binocular camera comprises: calculating a descending gradient of the cost function using a nonlinear optimization algorithm, determining an adjustment step length of the intrinsic parameters of the binocular camera based on the descending gradient, and adjusting the current intrinsic parameters of the binocular camera according to the adjustment step length. The device comprises:

5. An internal reference calibration device, characterized in that a reference size obtaining module configured to obtain a reference size of a target measurement region in an image acquisition region of a binocular camera; a depth image obtaining module configured to obtain a depth image of the image acquisition region, wherein the depth image is an image reflecting depth information of the image acquisition region, which is obtained by image processing on images captured by left and right cameras of the binocular camera using current intrinsic parameters of the binocular camera; a size error calculating module configured to determine a target size of the target measurement region using the depth image, and calculate a size error of the target size relative to the reference size; ​ The ratio calculation module is configured to calculate a ratio of a number of valid pixels in the depth image to a total number of all pixels, wherein the valid pixels are pixels in the depth image whose pixel values can reflect actual depth information. The cost function determination module is configured to determine a cost function of the current intrinsic parameter of the binocular camera by using the size error and the ratio. The intrinsic parameter adjustment module is configured to adjust the current intrinsic parameter of the binocular camera, trigger the depth image obtaining module, and obtain cost functions of a plurality of intrinsic parameters. The intrinsic parameter calibration module is configured to determine, as the calibrated intrinsic parameter of the binocular camera, an intrinsic parameter whose cost function satisfies a preset cost condition. The ratio calculation module is specifically configured to: count a number of valid pixels in the depth image whose confidence is higher than a preset confidence threshold; and calculate a ratio of the number of valid pixels to a total number of all pixels.

6. The apparatus of claim 5, wherein, The cost function determination module is specifically configured to: calculate the cost function c of the current intrinsic parameter of the binocular camera according to the following formula: ； wherein the α is a preset first weight, the β is a preset second weight, the S1 represents the ratio, and the S2 represents the size error.

7. The apparatus of claim 5, wherein, The depth image obtaining module is specifically configured to: obtain left and right camera images captured by left and right cameras of the binocular camera; correct the obtained left and right camera images respectively by using the current intrinsic parameter of the binocular camera; perform image processing on the corrected left and right camera images to obtain a depth image reflecting depth information of the image capturing region, and obtain a confidence of each pixel in the depth image, wherein the confidence of each pixel represents a probability that depth information reflected by a pixel value of the pixel is actual depth information.

8. The apparatus of any one of claims 5-7, wherein, The intrinsic parameter adjustment module is specifically configured to: calculate a descending gradient of the cost function by using a nonlinear optimization algorithm, determine an adjustment step of the intrinsic parameter of the binocular camera based on the descending gradient; and adjust the current intrinsic parameter of the binocular camera according to the adjustment step, trigger the depth image obtaining module, and obtain cost functions of a plurality of intrinsic parameters. The device comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus.

9. An electronic device, comprising: The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method steps in any one of claims 1-4. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-4.

Images