A method, apparatus, and electronic device for determining calibration parameters

Abstract

An embodiment of the present invention provides a calibration parameter determination method, device, and electronic device, which relate to the field of calibration. The method includes: obtaining multiple images collected by a vehicle-mounted camera to be calibrated; determining an image to be processed in the images collected for a zebra crossing scene, and performing undistortion processing on the image to be processed to obtain a target image; then, using a preset deep learning algorithm, determining a vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and extracting a specified edge line in each segmentation map using an edge extraction algorithm; establishing a constraint equation regarding the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated; and solving the constraint equation based on a preset solution target determined using the geometric characteristics of the zebra crossing to obtain the calibration parameters. Applying the solution provided by the embodiment of the present invention can improve the display accuracy of the stitched image for the road conditions around the vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of calibration, and particularly to a method and device for determining calibration parameters and an electronic device. Background Art

[0002] Currently, with the continuous development of automotive technology and road traffic, the number of vehicles traveling on the road is gradually increasing. Therefore, how to improve driving safety has attracted more and more attention from users.

[0003] Generally, multiple on-vehicle cameras can be installed on a vehicle so that each on-vehicle camera can capture the road conditions within its viewing angle range. After that, the road condition images collected by each on-vehicle camera are stitched together and presented to the user. In this way, it can help the user better understand the road condition environment where the vehicle is located, and adjust the driving strategy based on the road condition environment, thereby improving driving safety.

[0004] For example, an on-vehicle camera is installed in front of, behind, on the left side, and on the right side of the vehicle body respectively. Thus, as Figure 1(a) - Figure 1(e) shown, the images collected by the above-mentioned on-vehicle cameras can be obtained. Among them, Fig. 1(a) is the front view collected by the on-vehicle camera installed in front of the vehicle body, Fig. 1(b) is the left view collected by the on-vehicle camera installed on the left side of the vehicle body, Fig. 1(c) is the right view collected by the on-vehicle camera installed on the right side of the vehicle body, and Fig. 1(d) is the rear view collected by the on-vehicle camera installed behind the vehicle body. Furthermore, by performing surround stitching on the above-mentioned front view, left view, right view, and rear view, the surround stitching image shown in Fig. 1(e) can be obtained. In this way, the user can more clearly understand the road conditions around the vehicle by viewing the above-mentioned surround stitching image.

[0005] However, when the vehicle passes through a relatively bumpy road surface, the on-vehicle camera carried by it may shift in position along with the bumping of the vehicle body. As a result, when stitching the images collected by the on-vehicle camera, stitching errors will occur in the obtained stitching image, reducing the display accuracy of the above-mentioned stitching image for the road conditions around the vehicle.

[0006] Based on this, there is an urgent need for a method for determining calibration parameters, which can calibrate the images collected by the on-vehicle camera when stitching the images collected by the on-vehicle camera, reduce the stitching errors of the obtained stitching image, and improve the display accuracy of the above-mentioned stitching image for the road conditions around the vehicle. Summary of the Invention

[0007] The purpose of the embodiments of the present invention is to provide a method and device for determining calibration parameters and an electronic device to improve the display accuracy of the stitching image of the on-vehicle camera for the road conditions around the vehicle. The specific technical solutions are as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for determining calibration parameters, the method comprising:

[0009] Obtain multiple images collected by a vehicle-mounted camera to be calibrated;

[0010] Determine a to-be-processed image from the images collected for a zebra crossing scene, and perform a distortion removal process on the to-be-processed image to obtain a target image;

[0011] Using a preset deep learning algorithm, determine a vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract a specified edge line in each segmentation map; wherein, the specified edge line has a specified geometric relationship with the traffic direction of the zebra crossing;

[0012] Establish a constraint equation regarding the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated; wherein, the internal parameters include: a principal point and a focal length; the calibration parameters include: a pitch angle, a yaw angle, and a roll angle;

[0013] Based on a preset solution target determined using the geometric characteristics of the zebra crossing, solve the constraint equation to obtain the calibration parameters; wherein, the solution target includes: the specified edge lines being parallel.

[0014] Optionally, in a specific implementation manner, at least one vehicle-mounted camera is respectively installed in at least one of the front, rear, left, and right sides of the vehicle body where the vehicle-mounted camera to be calibrated is located; the vehicle-mounted camera to be calibrated includes one or more of the vehicle-mounted cameras installed on the vehicle.

[0015] Optionally, in a specific implementation manner, the vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body of the vehicle;

[0016] The using a preset deep learning algorithm to determine a vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and using an edge extraction algorithm to extract a specified edge line in each segmentation map includes:

[0017] Using a preset deep learning algorithm, determine a first vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract a first specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the first edge lines in the segmentation maps of each line intersect at the first vanishing point.

[0018] Optionally, in a specific implementation, the vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed on the left side and / or the right side of the vehicle body;

[0019] Using the preset deep learning algorithm, determining the vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and using the edge extraction algorithm to extract the specified edge line in each segmentation map, includes:

[0020] Using the preset deep learning algorithm, determining the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and using the edge extraction algorithm to extract the second specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the traffic direction of the zebra crossing in the segmentation map of each line intersect at the second vanishing point.

[0021] Optionally, in a specific implementation, if the target image includes multiple lines in the zebra crossing, the solution target further includes: each line in the zebra crossing has the same width, and / or, each line in the zebra crossing has the same height;

[0022] If the number of lines in the zebra crossing included in the target image is greater than 2, the solution target further includes: at least one of each line in the zebra crossing having the same width, each line in the zebra crossing having the same height, and the same spacing between each group of adjacent lines in the zebra crossing.

[0023] Optionally, in a specific implementation, the constraint equation is an equation about the coordinates of the vanishing point in the image coordinate system corresponding to the target image, the coordinates of the principal point in the image coordinate system, the horizontal focal length of the vehicle-mounted camera to be calibrated, the vertical focal length of the vehicle-mounted camera to be calibrated, the pitch angle, the yaw angle, and the roll angle.

[0024] Optionally, in a specific implementation, the number of vehicle-mounted cameras to be calibrated is multiple, and the method further includes:

[0025] Calibrating the images collected by the vehicle-mounted camera to be calibrated using the calibration parameters;

[0026] Stitching the images collected by the calibrated vehicle-mounted camera to be calibrated to obtain a stitched image.

[0027] In a second aspect, an embodiment of the present invention provides a calibration parameter determination device, the device includes:

[0028] An image acquisition module, configured to acquire multiple images collected by a vehicle-mounted camera to be calibrated;

[0029] A processing module, configured to determine a to-be-processed image in an image collected for a zebra crossing scene, and perform a distortion removal process on the to-be-processed image to obtain a target image;

[0030] A determination module, configured to use a preset deep learning algorithm to determine a vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract a specified edge line in each segmentation map; wherein, the specified edge line has a specified geometric relationship with the traffic direction of the zebra crossing;

[0031] An equation establishment module, configured to establish a constraint equation regarding the vanishing point, the internal parameters of the to-be-calibrated vehicle-mounted camera, and the calibration parameters of the to-be-calibrated vehicle-mounted camera; wherein, the internal parameters include: a principal point and a focal length; the calibration parameters include: a pitch angle, a yaw angle, and a roll angle;

[0032] A solution module, configured to solve the constraint equation based on a preset solution target determined by using the geometric characteristics of the zebra crossing to obtain the calibration parameters; wherein, the solution target includes: the specified edge lines are parallel.

[0033] Optionally, in a specific implementation manner, at least one vehicle-mounted camera is respectively installed in at least one of the front, rear, left, and right sides of the vehicle body where the to-be-calibrated vehicle-mounted camera is located; the to-be-calibrated vehicle-mounted camera includes one or more of the vehicle-mounted cameras installed on the vehicle.

[0034] Optionally, in a specific implementation manner, the to-be-calibrated vehicle-mounted camera includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body of the vehicle;

[0035] The determination module is specifically configured to:

[0036] Use a preset deep learning algorithm to determine a first vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract a first specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the first edge lines in the segmentation maps of the respective lines intersect at the first vanishing point.

[0037] Optionally, in a specific implementation manner, the to-be-calibrated vehicle-mounted camera includes at least one vehicle-mounted camera installed on the left and / or right sides of the vehicle body of the vehicle;

[0038] The determination module is specifically configured to:

[0039] Using a preset deep learning algorithm, determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the second specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the traffic direction of the zebra crossing in the segmentation map of each line intersect at the second vanishing point.

[0040] Optionally, in a specific implementation, if the target image includes multiple lines in the zebra crossing; then the solution objective further includes: each line in the zebra crossing has the same width, and / or, each line in the zebra crossing has the same height;

[0041] If the number of lines in the zebra crossing included in the target image is greater than 2; then the solution objective further includes at least one of: each line in the zebra crossing has the same width, each line in the zebra crossing has the same height, and the spacing between each group of adjacent lines in the zebra crossing is the same.

[0042] Optionally, in a specific implementation, the constraint equation is an equation about the coordinates of the vanishing point in the image coordinate system corresponding to the target image, the coordinates of the principal point in the image coordinate system, the horizontal focal length of the vehicle-mounted camera to be calibrated, the vertical focal length of the vehicle-mounted camera to be calibrated, the pitch angle, the yaw angle, and the roll angle.

[0043] Optionally, in a specific implementation, the number of vehicle-mounted cameras to be calibrated is multiple, and the device further includes:

[0044] A calibration module, configured to calibrate the images collected by the vehicle-mounted cameras to be calibrated by using the calibration parameters;

[0045] A stitching module, configured to stitch the images collected by the vehicle-mounted cameras to be calibrated after calibration to obtain a stitched image.

[0046] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0047] The memory is used for storing a computer program;

[0048] The processor is configured to implement the steps of any of the above method embodiments when executing the program stored in the memory.

[0049] Fourthly, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented. Fifthly, an embodiment of the present invention further provides a computer program product containing instructions, which when running on a computer, enables the computer to execute the steps of any of the above method embodiments.

[0050] Advantages of the embodiments of the present invention:

[0051] As can be seen above, when applying the method provided by the embodiments of the present invention to determine the calibration parameters of a to-be-calibrated vehicle-mounted camera, first, multiple images collected by the to-be-calibrated vehicle-mounted camera are obtained; then, in the images collected for the zebra crossing scene, a to-be-processed image is determined, and the to-be-processed image is de-distorted to obtain a target image; then, using a preset deep learning algorithm, the vanishing point of the zebra crossing included in the target image and the segmentation maps of each line of the zebra crossing are determined, and an edge extraction algorithm is used to extract the specified edge lines in each segmentation map that have a specified geometric relationship with the traffic direction of the zebra crossing; thus, a constraint equation regarding the vanishing point, the internal parameters of the to-be-calibrated vehicle-mounted camera, and the calibration parameters of the to-be-calibrated vehicle-mounted camera can be established; furthermore, based on a preset solution target determined using the geometric characteristics of the zebra crossing, the above constraint equation can be solved to obtain the above calibration parameters.

[0052] Based on this, when applying the solution provided by the embodiments of the present invention to determine the calibration parameters of a to-be-calibrated vehicle-mounted camera, images including zebra crossings collected by the vehicle-mounted camera can be used, combined with the geometric characteristics of the zebra crossing, and by solving the constraint equation regarding the vanishing point of the zebra crossing, the internal parameters of the to-be-calibrated vehicle-mounted camera, and the calibration parameters, the calibration parameters of the to-be-verified vehicle-mounted camera can be obtained. Furthermore, when there are multiple to-be-verified vehicle-mounted cameras, during the process of stitching the images collected by the above multiple to-be-verified vehicle-mounted cameras, the calibration parameters of each to-be-verified vehicle-mounted camera obtained above can be used to calibrate the above images, so that the stitched image can more accurately display the road conditions around the vehicle. Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0054] Figure 1(a) - Figure 1(d) Respectively, images collected by a vehicle-mounted camera provided by an embodiment of the present invention;

[0055] Figure 1(e) is for Figure 1(a) - Figure 1(d)The stitched image obtained by stitching;

[0056] Figure 2 It is a schematic flowchart of a calibration parameter determination method provided by an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of the installation position of a vehicle-mounted camera provided by an embodiment of the present invention;

[0058] Figure 4(a) - Figure 4(b) They are respectively schematic diagrams of the scenes of the images collected by the vehicle-mounted camera provided by an embodiment of the present invention;

[0059] Figure 5(a) - Figure 5(b) They are respectively schematic diagrams of the images including zebra crossings collected by the vehicle-mounted camera provided by an embodiment of the present invention;

[0060] Figure 6(a) - Figure 6(b) They are respectively for Figure 5(a) - Figure 5(b) The schematic diagrams after the distortion correction process;

[0061] Figure 7 It is a specific example diagram of a zebra crossing provided by an embodiment of the present invention;

[0062] Figure 8(a) - Figure 8(b) They are respectively specific example diagrams for determining the vanishing point of the zebra crossing and extracting the specified edge line of the area where the zebra crossing is located in FIG. 6(a);

[0063] Figure 9(a) - Figure 9(b) They are respectively specific example diagrams for determining the vanishing point of the zebra crossing and extracting the specified edge line of the area where the zebra crossing is located in FIG. 6(a);

[0064] Figure 10 It is a schematic diagram of the mapping relationship between the world coordinate system and the camera coordinate system provided by an embodiment of the present invention;

[0065] Figure 11(a) - Figure 11(b) They are respectively schematic diagrams of a stitched image provided by an embodiment of the present invention;

[0066] Figure 12 It is a schematic structural diagram of a calibration parameter determination device provided by an embodiment of the present invention;

[0067] Figure 13 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0068] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the scope of protection of the present invention.

[0069] Generally, multiple on-vehicle cameras can be installed on a vehicle so that each on-vehicle camera can capture the road conditions within its viewing angle. After that, the road condition images collected by each on-vehicle camera are stitched together and the stitched road condition image is presented to the user. In this way, it can help the user better understand the road condition environment where the vehicle is located and adjust the driving strategy based on the road condition environment, thereby improving driving safety.

[0070] However, when the vehicle passes through a bumpy road surface, the on-vehicle camera carried by it may shift in position along with the bumping of the vehicle body. As a result, when stitching the images collected by the on-vehicle camera, the resulting stitched image has stitching errors, reducing the display accuracy of the above stitched image for the road conditions around the vehicle.

[0071] To solve the above technical problems, an embodiment of the present invention provides a method for determining calibration parameters.

[0072] Among them, this method can be applied to an on-vehicle camera added with a functional module for executing this method, so that the on-vehicle camera executes this method to determine its own calibration parameters; it can also be applied to an electronic device that provides a service for determining calibration parameters for the on-vehicle camera. The above electronic device can be a mobile phone in communication with the on-vehicle camera, a monitoring console for monitoring the on-vehicle camera, etc. The above electronic device can receive the images collected by the on-vehicle camera and execute the method provided by the present invention to determine the calibration parameters of the on-vehicle camera. Based on this, the embodiment of the present invention does not specifically limit the execution subject of this method.

[0073] A method for determining calibration parameters provided by an embodiment of the present invention may include the following steps:

[0074] Obtain multiple images collected by the on-vehicle camera to be calibrated;

[0075] Determine the image to be processed in the images collected for the zebra crossing scene, and perform de-distortion processing on the image to be processed to obtain a target image;

[0076] Using a preset deep learning algorithm, determine the vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the specified edge line in each segmentation map; wherein, the specified edge line has a specified geometric relationship with the passing direction of the zebra crossing;

[0077] Establish a constraint equation for the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated; wherein, the internal parameters include: the principal point and the focal length; the calibration parameters include: the pitch angle, the yaw angle, and the roll angle;

[0078] Based on a preset solution target determined by using the geometric characteristics of the zebra crossing, solve the constraint equation to obtain the calibration parameters; wherein, the solution target includes: the specified edge line is parallel.

[0079] As can be seen above, when applying the method provided by the embodiment of the present invention to determine the calibration parameters of the vehicle-mounted camera to be calibrated, first, obtain multiple images collected by the vehicle-mounted camera to be calibrated; then, determine the image to be processed in the images collected for the zebra crossing scene, and perform distortion removal processing on the image to be processed to obtain a target image; then, use a preset deep learning algorithm to determine the vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the specified edge line in each segmentation map that has a specified geometric relationship with the traffic direction of the zebra crossing; thus, a constraint equation for the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated can be established; furthermore, based on a preset solution target determined by using the geometric characteristics of the zebra crossing, the above constraint equation can be solved to obtain the above calibration parameters.

[0080] Based on this, when applying the solution provided by the embodiment of the present invention to determine the calibration parameters of the vehicle-mounted camera to be calibrated, an image including a zebra crossing collected by the vehicle-mounted camera can be used, combined with the geometric characteristics of the zebra crossing, and by solving the constraint equation for the vanishing point of the zebra crossing, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters, the calibration parameters of the vehicle-mounted camera to be verified can be obtained. Furthermore, when there are multiple vehicle-mounted cameras to be verified, during the process of stitching the images collected by the above multiple vehicle-mounted cameras to be verified, the calibration parameters of each vehicle-mounted camera to be verified obtained above can be used to calibrate the above images, so that the stitched image can more accurately display the road conditions around the vehicle.

[0081] Next, with reference to the accompanying drawings, a method for determining calibration parameters provided by the embodiment of the present invention will be specifically described.

[0082] Figure 2 It is a flowchart of a method for determining calibration parameters provided by an embodiment of the present invention. As Figure 2 shown, the method may include the following steps S201-S205.

[0083] S201: Obtain multiple images collected by the vehicle-mounted camera to be calibrated;

[0084] During the driving of the vehicle, the in-vehicle camera can collect images in real time. Thus, when the vehicle drives through different road scenes, the in-vehicle camera can collect images for the above-mentioned different road scenes. For example, images collected for the zebra crossing scene, images collected for the lane line scene, and so on. In this way, when determining the calibration parameters of the in-vehicle camera to be calibrated, multiple images collected by the in-vehicle camera to be calibrated can be obtained first.

[0085] Among them, the in-vehicle camera to be calibrated can be various types of cameras such as a fish-eye camera, a dome camera, and a bullet camera. This is reasonable, and the type of the in-vehicle camera to be calibrated is not specifically limited in the embodiments of the present invention.

[0086] Moreover, the in-vehicle camera to be calibrated can be one or multiple. When the in-vehicle camera to be calibrated is one, it can be installed in any direction of the vehicle body; when the in-vehicle camera to be calibrated is multiple, the multiple in-vehicle cameras to be calibrated can be installed in the same direction of the vehicle body or in different directions of the vehicle body. For example, when the number of in-vehicle cameras to be calibrated is 3, two of the in-vehicle cameras to be calibrated are installed in front of the vehicle body, and the other in-vehicle camera to be calibrated is installed on the left side of the vehicle body. This is reasonable, and the specific number and installation position of the in-vehicle camera to be calibrated are not specifically limited in the embodiments of the present invention.

[0087] Optionally, in a specific implementation manner, at least one in-vehicle camera is installed in at least one of the front, rear, left, and right directions of the vehicle body where the in-vehicle camera to be calibrated is located; the in-vehicle camera to be calibrated includes one or more of the in-vehicle cameras installed on the vehicle.

[0088] For example, as Figure 3 shown, in-vehicle cameras A, B, C, and D are installed in the front, rear, left, and right directions of the vehicle body of vehicle E respectively. Furthermore, the in-vehicle camera to be calibrated can include at least one of the above-mentioned in-vehicle cameras A - D. Exemplarily, all of the above-mentioned in-vehicle cameras A - D are in-vehicle cameras to be calibrated.

[0089] When the in-vehicle camera to be calibrated is multiple, for each in-vehicle camera to be calibrated, the images collected by the in-vehicle camera to be calibrated can be obtained. Furthermore, the calibration parameters of the in-vehicle camera to be calibrated can be determined by using the images collected by the in-vehicle camera to be calibrated. That is to say, in the embodiments of the present invention, the calibration parameters of each in-vehicle camera to be calibrated are independently determined by using the images independently collected by each in-vehicle camera to be calibrated.

[0090] During the driving process of the vehicle, in order to facilitate the user to view the road conditions around the vehicle in real time, the in-vehicle camera usually performs image acquisition in real time. Therefore, when obtaining multiple images collected by the to-be-calibrated in-vehicle camera, multiple images collected by the to-be-calibrated in-vehicle camera when the vehicle passes through a zebra crossing can be obtained.

[0091] Optionally, during the process of the vehicle passing through the zebra crossing, the in-vehicle camera can collect multiple images of the zebra crossing scene in real time. Thus, for the to-be-calibrated in-vehicle camera, multiple images collected by the to-be-calibrated in-vehicle camera when its vehicle passes through the zebra crossing can be obtained. Furthermore, among the above multiple images, the to-be-processed images for determining the calibration parameters of the to-be-calibrated in-vehicle camera can be determined.

[0092] During the driving process of the vehicle, when the vehicle jolts, the installation position of the in-vehicle camera installed on the vehicle may change with the jolting of the vehicle. As a result, the external parameters of the in-vehicle camera change. For example, the pitch angle, yaw angle, and roll angle of the in-vehicle camera change. Based on this, in order to ensure the accuracy of the road condition images around the vehicle displayed to the user based on the images collected by the in-vehicle camera during subsequent driving, when the vehicle jolts, multiple images collected by the to-be-calibrated in-vehicle camera can be obtained, so as to determine the calibration parameters of the to-be-calibrated in-vehicle camera.

[0093] Based on this, optionally, after detecting that the vehicle where the in-vehicle camera is located generates a preset driving state, obtain multiple images of the zebra crossing scene collected by the to-be-calibrated in-vehicle camera when its vehicle returns to the normal driving state and first passes through the zebra crossing;

[0094] Wherein, the driving state is a state that can cause the installation position of the to-be-calibrated in-vehicle camera to change.

[0095] In this specific implementation manner, a driving state that can cause the installation position of the to-be-calibrated in-vehicle camera to change can be preset. For example, vehicle jolting, vehicle sudden stop, etc. These are all reasonable. In the embodiments of the present invention, the above driving states are not specifically limited as long as they are driving states that can cause the installation position of the to-be-calibrated in-vehicle camera to change. And when the vehicle generates the above driving state, the installation position of the in-vehicle camera installed on the vehicle may or may not change. These are all reasonable.

[0096] Furthermore, during the driving process of the vehicle where the to-be-calibrated in-vehicle camera is located, the driving state of the vehicle where the to-be-calibrated camera is located can be detected in real time. In this way, after detecting that the above vehicle generates a preset driving state, multiple images of the zebra crossing scene collected by the to-be-calibrated in-vehicle camera when its vehicle returns to the normal driving state and first passes through the zebra crossing can be obtained.

[0097] For example, at a certain moment, the vehicle where the vehicle-mounted camera to be calibrated is located jolts because it runs over a stone on the road. After passing the stone, the driving state of the vehicle returns to normal. During the continuous driving of the vehicle, multiple images of the zebra crossing scene collected by the vehicle-mounted camera to be calibrated when the vehicle first passes through the zebra crossing can be obtained, and a to-be-processed image can be determined from the above multiple images.

[0098] S202: Determine a to-be-processed image from the images collected for the zebra crossing scene, and perform distortion removal processing on the to-be-processed image to obtain a target image;

[0099] Generally, zebra crossings have geometric characteristics. For example, the specified edge lines of each line in the zebra crossing are parallel to each other, each edge line of each line in the zebra crossing is a straight line, the width and height of each line in the zebra crossing are the same respectively, and the spacing between each adjacent two lines in the zebra crossing is the same. Therefore, the geometric characteristics of the above zebra crossing can be used as the calibration basis for the vehicle-mounted camera to be calibrated. Thus, when determining the calibration parameters of the vehicle-mounted camera to be calibrated, the calibration parameters of the vehicle-mounted camera to be calibrated can be determined by using the images of the zebra crossing scene collected by the vehicle-mounted camera to be calibrated.

[0100] In this way, after obtaining multiple images collected by the vehicle-mounted camera to be calibrated, the scene classification algorithm can be used to identify and classify the scenes photographed by the above multiple images to obtain the images collected for the zebra crossing scene.

[0101] As Figure 4(a) - 4(b) shown, Fig. 4(a) is an image collected by the vehicle-mounted camera to be calibrated for the zebra crossing scene, and Fig. 4(b) is an image collected by the vehicle-mounted camera to be calibrated for the lane line scene.

[0102] Among them, the above scene classification algorithm can be a spatial pyramid matching algorithm or a deep learning algorithm, such as a convolution algorithm; this is all reasonable and is not specifically limited in the embodiments of the present invention.

[0103] Since the so-called zebra crossing refers to a set of lines composed of multiple mutually parallel white solid lines, therefore, the above target image may include one line in the zebra crossing or multiple lines in the zebra crossing.

[0104] Furthermore, a to-be-processed image can be determined from the obtained images collected for the zebra crossing scene.

[0105] Among them, when the on-vehicle camera to be calibrated is installed at different positions on the vehicle body, the road conditions around the vehicle collected by it are different. Furthermore, the images of the zebra crossing scene collected by it are also different. For example, the on-vehicle camera installed in the front of the vehicle body can collect the image of the zebra crossing scene as shown in Fig. 5(a), and the on-vehicle camera installed on the left side of the vehicle body can collect the image of the zebra crossing scene as shown in Fig. 5(b).

[0106] When determining the image to be processed from the images collected for the zebra crossing scene, it is reasonable to use the candidate image with the largest number of lines in the zebra crossing contained in the above-mentioned multiple images as the image to be processed, or use the image with the earliest shooting time among the above-mentioned multiple images as the image to be processed, or use the image with the best image quality among the above-mentioned multiple images as the image to be processed. The embodiments of the present invention do not specifically limit the method of determining the image to be processed from multiple images of the zebra crossing scene above.

[0107] Moreover, since the images collected by the above-mentioned on-vehicle camera to be calibrated may have image distortion caused by the camera type. For example, when the above-mentioned on-vehicle camera to be calibrated is a fish-eye camera or a dome camera, the images collected by the to-be-calibrated camera may have edge distortion, etc. Therefore, after determining the image to be processed from the images of the zebra crossing scene collected by the on-vehicle camera to be calibrated, considering that the image to be processed may have distortion, in order to improve the accuracy of the finally determined calibration parameters, the to-be-processed image can be first de-distorted to obtain a target image for determining the calibration parameters.

[0108] Optionally, in a specific implementation manner, when the above-mentioned on-vehicle camera to be calibrated is a fish-eye camera, the internal parameters of the on-vehicle camera to be calibrated can be used to de-distort the above-mentioned to-be-processed image to obtain a target image. Furthermore, the above-mentioned to-be-processed image can be de-distorted, and the de-distorted to-be-processed image can be used as the target image for determining the calibration parameters.

[0109] In this specific implementation manner, when the above-mentioned on-vehicle camera to be calibrated is a fish-eye camera, after determining the to-be-processed image collected by the on-vehicle camera to be calibrated, the internal parameters of the on-vehicle camera to be calibrated can be used to de-distort the above-mentioned to-be-processed image to obtain a target image for determining the calibration parameters of the on-vehicle camera to be calibrated.

[0110] Among them, the image distortion can include radial distortion and tangential distortion. When de-distorting the target image using the internal parameters of the on-vehicle camera, the coordinates (a distorted , b distorted ) of the point G(a, b) after de-distortion can be calculated as follows:[[]]END]]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] Among them, a is the abscissa of point G in the image coordinate system, and b is the ordinate of point G in the image coordinate system; a distorted is the abscissa of point G in the image coordinate system after distortion removal, and b distorted is the ordinate of point G in the image coordinate system after distortion removal; cx is the abscissa of the principal point in the image coordinate system; cy is the ordinate of the principal point in the image coordinate system; fx is the horizontal focal length of the vehicle-mounted camera to be calibrated; fy is the vertical focal length of the vehicle-mounted camera to be calibrated; x is the abscissa of the corresponding point of point G in the world coordinate system; y is the ordinate of the corresponding point of point G in the world coordinate system; r is the vertical axis coordinate of the corresponding point of point G in the world coordinate system;

[0117] For the sake of convenience of description, the corresponding point of point G in the world coordinate system can be called a calibration point. Then, the above x can be understood as: the abscissa of the calibration point in the world coordinate system; the above y can be understood as the ordinate of the calibration point in the world coordinate system; the above ry can be understood as the vertical axis coordinate of the calibration point in the world coordinate system;

[0118] Furthermore, the above x distorted is the abscissa of the calibration point in the world coordinate system after distortion removal; y distorted is the ordinate of the calibration point in the world coordinate system after distortion removal; k1, k2, k3, p1, and p2 are all preset parameters, and each of them is a constant.

[0119] When the calculated above a distorted and b distorted are not integers, the pixel values of the surrounding integer coordinates can be used. Through bilinear interpolation or the nearest neighbor method, the above a distorted and b distorted are rounded to obtain the coordinates of point G in the image coordinate system after distortion removal.

[0120] Based on the same concept, when the vehicle-mounted camera to be calibrated is a spherical camera or other cameras that may cause distortion of the captured images, the to-be-processed images collected by the vehicle-mounted camera to be calibrated can be first subjected to distortion removal processing according to the above method to obtain a target image for determining the calibration parameters of the vehicle-mounted camera to be calibrated.

[0121] Exemplarily, when the in-vehicle camera to be calibrated is a fish-eye camera and is installed in front of the vehicle body, the to-be-processed image captured by the in-vehicle camera to be calibrated is the image shown in Fig. 5(a). By performing a distortion removal process on the to-be-processed image, a target image after distortion removal as shown in Fig. 6(a) can be obtained; when the in-vehicle camera to be calibrated is a fish-eye camera and is installed on the left side of the vehicle body, the to-be-processed image captured by the in-vehicle camera to be calibrated is the image shown in Fig. 5(b). By performing a distortion removal process on the to-be-processed image, a target image after distortion removal as shown in Fig. 6(b) can be obtained.

[0122] S203: Using a preset deep learning algorithm, determine the vanishing point of the zebra crossing included in the target image and the segmentation maps of each line of the zebra crossing, and use an edge extraction algorithm to extract the specified edge lines in each segmentation map;

[0123] Among them, the specified edge lines have a specified geometric relationship with the traffic direction of the zebra crossing;

[0124] After obtaining the above target image, for this target image, a preset deep learning algorithm can be used to determine the vanishing point of the zebra crossing included in the target image and the segmentation maps of each line of the zebra crossing. Then, an edge extraction algorithm can be used to extract the specified edge lines in each segmentation map that have a specified geometric relationship with the traffic direction of the zebra crossing.

[0125] Among them, pedestrians cross the road in the traffic direction indicated by the zebra crossing. Thus, the walking direction of pedestrians can be understood as the traffic direction of the zebra crossing. For example, as Figure 7 shown, the white line segments 1-5 in the figure are part of the line segments that make up the zebra crossing; the direction indicated by the arrow is the traffic direction of the zebra crossing.

[0126] As Figure 7 shown, for each line segment in the zebra crossing, each edge line in this line segment is a straight line, and the longer edge line in this line segment is perpendicular to the traffic direction of the zebra crossing, and the shorter edge line is parallel to the traffic direction of the zebra crossing. Thus, based on the geometric relationship between the above edge lines and the traffic direction of the zebra crossing, a specified geometric relationship can be determined in the above geometric relationship. Furthermore, the specified edge lines that have the above-specified geometric relationship with the traffic direction of the zebra crossing can be extracted in the target image.

[0127] Among them, the above-specified geometric relationship can be perpendicular or parallel.

[0128] Moreover, the above deep learning algorithm can be an unsupervised pre-training algorithm or a convolutional neural algorithm; the above edge extraction algorithm can be a Sobel algorithm or a Prewitt algorithm, which are all reasonable. In the embodiments of the present invention, the above deep learning algorithm and the above edge extraction algorithm are not specifically limited.

[0129] For example, for the segmentation map of each line of the zebra crossing, the Sobel algorithm can be used to extract each edge point in the segmentation map. Then, through line fitting, each edge line in the segmentation map can be obtained. Thus, a specified edge line that has a specified geometric relationship with the traffic direction of the zebra crossing in each segmentation map can be extracted.

[0130] In addition, the installation positions of the vehicle-mounted cameras on the vehicle are different, and the images collected for the zebra crossing scene can be different. Based on this, for the vehicle-mounted cameras to be calibrated at different installation positions, the determined target images can be different. Furthermore, for each target image, the vanishing point and the specified edge line of the zebra crossing determined can also be different.

[0131] Optionally, in a specific implementation manner, the vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body; the above step S203 may include the following step 21:

[0132] Step 21: Use a preset deep learning algorithm to determine the first vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the first specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map;

[0133] Among them, the extension lines of the first edge lines in the segmentation maps of each line intersect at the first vanishing point.

[0134] In this specific implementation manner, when the vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body, for the determined target image, a deep learning algorithm can be used to determine the first vanishing point of the zebra crossing included in the target image and the segmentation map of each line in the zebra crossing. Then, an edge extraction algorithm can be used to extract each first specified edge line perpendicular to the traffic direction of the zebra crossing in the segmentation maps of each line in the above zebra crossing. And the extension lines of the above first edge lines intersect at the above first vanishing point.

[0135] Exemplarily, when the in-vehicle camera to be calibrated is an in-vehicle camera installed in front of the vehicle body, the target image for determining the calibration parameters of the in-vehicle camera to be calibrated is Figure 6(a). Subsequently, a deep learning algorithm can be used to determine the segmentation map of each line of the zebra crossing included in the target image and the vanishing point of the zebra crossing. As shown in Figure 8(a), the black long strip areas in the figure are the segmentation maps of each line of the zebra crossing, and the vanishing point G in the figure is the first vanishing point of the zebra crossing. Subsequently, an edge extraction algorithm can be used to extract the first edge line perpendicular to the traffic direction of the zebra crossing in the segmentation map of each line. As shown in Figure 8(b), the edge line of the long side of each black long strip area is the first edge line in the segmentation map of the line. Moreover, the extension lines of multiple first edge lines intersect at the vanishing point G.

[0136] Optionally, in a specific implementation, the in-vehicle camera to be calibrated includes at least one in-vehicle camera installed on the left side and / or the right side of the vehicle body; the above step S203 may include the following step 31:

[0137] Step 31: Use a preset deep learning algorithm to determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the second specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the traffic direction of the zebra crossing in the segmentation maps of each line intersect at the second vanishing point.

[0138] In this specific implementation, when the in-vehicle camera to be calibrated includes at least one in-vehicle camera installed on the left side and / or the right side of the vehicle body, for the determined target image, a deep learning algorithm can be used to determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line in the zebra crossing. Then, an edge extraction algorithm can be used to extract each second specified edge line perpendicular to the traffic direction of the zebra crossing in the segmentation maps of each line in the above zebra crossing. The extension lines of the edge lines parallel to the communication direction of the zebra crossing in the segmentation maps of each line intersect at the above second vanishing point.

[0139] Exemplarily, when the in-vehicle camera to be calibrated is an in-vehicle camera installed on the left side of the vehicle body, the target image for determining the calibration parameters of the in-vehicle camera to be calibrated is Figure 6(b). Subsequently, a deep learning algorithm can be used to determine the segmentation map of each line of the zebra crossing included in the target image and the vanishing point of the zebra crossing. As shown in Figure 9(a), the black long strip areas in the figure are the segmentation maps of each line of the zebra crossing, and the vanishing point H in the figure is the second vanishing point of the zebra crossing. Subsequently, an edge extraction algorithm can be used to extract the second edge line perpendicular to the traffic direction of the zebra crossing in the segmentation map of each line. As shown in Figure 9(b), the edge line of the long side of each black long strip area is the second edge line in the segmentation map of the line. In addition, the edge line of the short side of each black long strip area is parallel to the traffic direction of the zebra crossing, and the extension lines of the edge lines of the short sides of each black long strip area intersect at the vanishing point H.

[0140] S204: Establish a constraint equation regarding the vanishing point, the internal parameters of the in-vehicle camera to be calibrated, and the calibration parameters of the in-vehicle camera to be calibrated;

[0141] Among them, the internal parameters include: the principal point and the focal length; the calibration parameters include: the pitch angle, the yaw angle, and the roll angle;

[0142] For an in-vehicle camera, the coordinates of the above-mentioned vanishing point of the zebra crossing in the image coordinate system of the target image, the internal parameters of the in-vehicle camera to be calibrated, and the calibration parameters of the in-vehicle camera to be calibrated are related and interact with each other. Since the coordinates of the above-mentioned vanishing point and the above-mentioned internal parameters are known, therefore, the calibration parameters of the in-vehicle camera to be calibrated can be calculated by using the relationship between the above-mentioned vanishing point, internal parameters, and calibration parameters and the known vanishing point and internal parameters.

[0143] Based on this, first, a constraint equation regarding the above-mentioned vanishing point, the above-mentioned internal parameters, and the above-mentioned calibration parameters can be established according to the coordinates of the above-mentioned vanishing point of the zebra crossing in the image coordinate system of the target image, the internal parameters of the in-vehicle camera to be calibrated, and the calibration parameters of the in-vehicle camera to be calibrated. Among them, the above-mentioned internal parameters can include the principal point and the focal length, and the above-mentioned calibration parameters can include: the pitch angle, the yaw angle, and the roll angle. The so-called principal point is the image center of the image captured by the in-vehicle camera.

[0144] That is to say, a constraint equation regarding the vanishing point of the zebra crossing, the principal point, the focal length, the pitch angle, the yaw angle, and the roll angle can be established.

[0145] Optionally, in a specific implementation manner, the above-mentioned constraint equation is an equation regarding the coordinates of the vanishing point in the image coordinate system corresponding to the target image, the coordinates of the principal point in the image coordinate system, the horizontal focal length of the in-vehicle camera to be calibrated, the vertical focal length of the in-vehicle camera to be calibrated, the pitch angle, the yaw angle, and the roll angle.

[0146] In this specific implementation, after obtaining the vanishing point, principal point, horizontal focal length of the vehicle-mounted camera to be calibrated, and vertical focal length of the vehicle-mounted camera to be calibrated in the target image, constraint equations regarding the coordinates of the vanishing point in the image coordinate system corresponding to the target image, the coordinates of the principal point in the image coordinate system, the horizontal focal length of the vehicle-mounted camera to be calibrated, the vertical focal length of the vehicle-mounted camera to be calibrated, pitch angle, yaw angle, and roll angle can be established.

[0147] Optionally, as Figure 10 shown, the mapping relationship between the camera coordinate system \(0_c - X_cY_cZ_c\) and the world coordinate system \(0_{cw} - X_{cw}Y_{cw}Z_{cw}\) is as follows:

[0148]

[0149] Wherein, \(X_c\) is the horizontal axis coordinate in the camera coordinate system; \(Y_c\) is the vertical axis coordinate in the camera coordinate system; \(Z_c\) is the vertical axis coordinate in the camera coordinate system; \(X_{cw}\) is the horizontal axis coordinate in the world coordinate system; \(Y_{cw}\) is the horizontal axis coordinate in the world coordinate system; \(Z_{cw}\) is the vertical axis coordinate in the world coordinate system; pitch is the pitch angle of the camera, and roll is the roll angle of the camera.

[0150] The so-called yaw angle of the camera refers to: the angle between the lane line and the \(Y_{cw}\) axis in the world coordinate system \(0_{cw} - X_{cw}Y_{cw}Z_{cw}\), denoted as yaw. Thus, the coordinates of a point on the lane line in the world coordinate system can be expressed as \((Y_{cw}*\tan(yaw), Y_{cw}, 0)\).

[0151] In addition, the mapping relationship between the coordinates \((u, v)\) of the vanishing point of the lane line in the image in the image coordinate system and the coordinates \((X_d, Y_d, Z_d)\) of this vanishing point in the camera coordinate system \(0_c - X_cY_cZ_c\) can be expressed as:

[0152]

[0153] Wherein, \(X_d\) is the horizontal axis coordinate of the above-mentioned vanishing point in the camera coordinate system; \(Y_d\) is the vertical axis coordinate of the above-mentioned vanishing point in the camera coordinate system; \(Z_d\) is the vertical axis coordinate of the above-mentioned vanishing point in the camera coordinate system; \(c_x\) is the abscissa of the principal point in the image coordinate system; \(c_y\) is the ordinate of the principal point in the image coordinate system; \(f_x\) is the horizontal focal length of the vehicle-mounted camera to be calibrated; \(f_y\) is the vertical focal length of the vehicle-mounted camera to be calibrated, and \(\alpha\) is a preset parameter and is a constant.

[0154] In this way, in the above \((Y_{cw}*\tan(yaw), Y_{cw}, 0)\), when \(Y_{cw} ightarrow \infty\), by linking the above two mapping relationships, the obtained \((u, v)\) is the coordinates of the vanishing point in the image coordinate system in the target image.

[0155] Therefore, by solving the above two mapping relationships and taking the limit of the equation after solving, the constraint equations for the vanishing point, principal point, focal length, pitch angle, yaw angle, and roll angle can be obtained:

[0156]

[0157]

[0158] Among them, u is the abscissa of the vanishing point in the image coordinate system corresponding to the target image; v is the ordinate of the vanishing point in the image coordinate system; cx is the abscissa of the principal point in the image coordinate system; cy is the ordinate of the principal point in the image coordinate system; fx is the horizontal focal length of the vehicle-mounted camera to be calibrated; fy is the vertical focal length of the vehicle-mounted camera to be calibrated; pitch is the pitch angle; yaw is the yaw angle; roll is the roll angle.

[0159] S205: Based on the solution target determined by using the geometric characteristics of the zebra crossing preset, solve the constraint equation to obtain the calibration parameters;

[0160] Among them, the solution target includes: the specified edge lines are parallel.

[0161] Usually, the zebra crossing has geometric characteristics. For example, the specified edge lines of each line in the zebra crossing are parallel to each other, each edge line of each line in the zebra crossing is a straight line, the width and height of each line in the zebra crossing are the same respectively, and the distance between each adjacent two lines in the zebra crossing is the same. Therefore, the geometric characteristics of the above zebra crossing can be used as the calibration basis for the vehicle-mounted camera to be calibrated. Thus, based on the geometric characteristics of the zebra crossing, the solution target can be determined, and based on the above solution target, the above constraint equation can be solved to obtain the calibration parameters of the vehicle-mounted camera to be calibrated.

[0162] Among them, the above solution target determined by using the geometric characteristics of the zebra crossing preset can include: the specified edge lines extracted above are parallel.

[0163] Optionally, the above solution target determined by using the geometric characteristics of the zebra crossing preset can also include: the specified edge lines extracted above are all straight lines.

[0164] In this way, by solving the above constraint equation, the pitch angle, yaw angle, and roll angle of the vehicle-mounted camera to be calibrated can be obtained simultaneously. Furthermore, it can be avoided that after calculating one or several of the calibration parameters separately, the calibrated parameters are affected when calculating other calibration parameters subsequently, resulting in poor accuracy of the finally obtained calibration parameters.

[0165] Optionally, based on the above-mentioned solution objective, the LM (Levenberg-Marquardt) algorithm can be used to solve the above-mentioned constraint equations to obtain the calibration parameters of the in-vehicle camera to be calibrated.

[0166] Among them, the above LM algorithm is an optimization algorithm. Furthermore, the above solution objective is the optimization objective of this optimization algorithm.

[0167] Since for different in-vehicle cameras to be calibrated, the target images obtained for the in-vehicle camera to be calibrated can be different. Moreover, due to different shooting angles and shooting distances, the number of lines in the zebra crossing included in each target image is different. When at least one line in the zebra crossing is included in the target image, the geometric characteristics of the edge lines of each line can be used to determine the above solution objective. When multiple lines in the zebra crossing are included in the target image, the geometric relationship between each line can also be used to determine the above solution objective.

[0168] Optionally, in a specific implementation, if the target image includes multiple lines in the zebra crossing, the solution objective further includes: the width of each line in the zebra crossing is the same, and / or the height of each line in the zebra crossing is the same.

[0169] In this specific implementation, when multiple zebra crossings are included in the above target image, the solution objective can further include that the width of each line in the zebra crossing is the same, and / or the height of each line in the zebra crossing is the same.

[0170] That is, the above solution objective includes that the specified edge lines of the lines in the zebra crossing are parallel, and at least one of the widths of the individual lines in the zebra crossing being the same and the heights of the individual images in the zebra crossing being the same.

[0171] Similarly, when multiple lines in the zebra crossing are included in the target image and the number of lines is greater than 2, the above lines can also be grouped, and the geometric relationship between each group of lines can be used to determine the above solution objective.

[0172] Optionally, in a specific implementation, if the number of lines in the zebra crossing included in the target image is greater than 2, the solution objective further includes at least one of the width of each line in the zebra crossing being the same, the height of each line in the zebra crossing being the same, and the spacing between adjacent groups of lines in the zebra crossing being the same.

[0173] In this specific implementation, when the number of lines in the zebra crossing included in the above target image is greater than 2, every two adjacent lines in the zebra crossing can be taken as a group. Thus, the solution objective can further include at least one of the width of each line in the zebra crossing being the same, the height of each line in the zebra crossing being the same, and the spacing between adjacent groups of lines in the zebra crossing being the same.

[0174] After determining the above calibration parameters, the photos captured by the to-be-calibrated camera can be calibrated using the above calibration parameters, so that the images output by the to-be-calibrated vehicle-mounted camera can more accurately restore the road conditions at the vehicle's location.

[0175] Optionally, in a specific implementation, for a calibration parameter determination method provided by an embodiment of the present invention, when the number of to-be-calibrated vehicle-mounted cameras is multiple, the following steps 41-42 may further be included:

[0176] Step 41: Calibrate the images captured by the to-be-calibrated vehicle-mounted camera using the calibration parameters;

[0177] Step 42: Stitch the images captured by the calibrated to-be-calibrated vehicle-mounted camera to obtain a stitched image.

[0178] In this specific implementation, after obtaining the calibration parameters of the to-be-calibrated vehicle-mounted camera, the images captured by the to-be-calibrated vehicle-mounted camera can be calibrated using the above calibration parameters.

[0179] Among them, the internal parameters of the to-be-calibrated vehicle-mounted camera and the above calibration parameters can be used to map each point on the image captured by the to-be-calibrated vehicle-mounted camera to the world coordinate system. Furthermore, the points obtained through mapping in the world coordinate system are mapped back to the image coordinate system again to obtain a new image. The new image obtained above is the image after calibrating the image captured by the to-be-calibrated vehicle-mounted camera.

[0180] After calibrating the images captured by the to-be-calibrated vehicle-mounted camera using the above calibration parameters, the calibrated multiple images can be stitched to obtain a stitched image.

[0181] For example, when a vehicle-mounted camera is installed in front of the vehicle body, behind the vehicle body, on the left side of the vehicle body, and on the right side of the vehicle body where the to-be-calibrated vehicle-mounted camera is located, the to-be-calibrated vehicle-mounted camera can be the above four vehicle-mounted cameras. After determining the calibration parameters of each to-be-calibrated vehicle-mounted camera, the images captured by the to-be-calibrated vehicle-mounted camera can be calibrated based on the above calibration parameters, and then the calibrated images can be stitched.

[0182] Exemplarily, such as Figure 11(a) - 11(b)As shown in the figure, FIG. 11(a) is a stitched image obtained by directly stitching the images collected by each vehicle-mounted camera to be calibrated. Due to the change in the installation positions of the vehicle-mounted cameras to be calibrated, there are stitching errors in the obtained stitched image, and the road conditions around the vehicle cannot be accurately displayed; FIG. 11(b) is a stitched image obtained by calibrating the images collected by each vehicle-mounted camera to be calibrated according to the above calibration parameters and then stitching the calibrated images. The obtained stitched image can accurately display the road conditions around the vehicle.

[0183] In this way, stitching the calibrated images can improve the accuracy of the stitched image in displaying the road conditions around the vehicle. Then, the above stitched image can be displayed to the user so that the user can accurately understand the road conditions around the vehicle based on the above stitched image.

[0184] Optionally, after completing the calibration of the images collected by the vehicle-mounted camera to be calibrated using the above calibration parameters, the calibrated images can be output as the image acquisition results of the vehicle-mounted camera to be calibrated.

[0185] For example, when there is only one vehicle-mounted camera to be calibrated, after completing the calibration of the images collected by the vehicle-mounted camera to be calibrated using the above calibration parameters, the calibrated images can be output as the image acquisition results of the vehicle-mounted camera to be calibrated.

[0186] As can be seen from the above, when applying the solution provided in the embodiment of the present invention to determine the calibration parameters of the vehicle-mounted camera to be calibrated, images including zebra crossings collected by the vehicle-mounted camera can be used, combined with the geometric characteristics of the zebra crossings, and by solving the constraint equations regarding the vanishing point of the zebra crossing, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters, the calibration parameters of the vehicle-mounted camera to be calibrated can be obtained. Furthermore, when there are multiple vehicle-mounted cameras to be calibrated, during the process of stitching the images collected by the above multiple vehicle-mounted cameras to be calibrated, the calibration parameters of each vehicle-mounted camera to be calibrated obtained above can be used to calibrate the above images so that the stitched image can more accurately display the road conditions around the vehicle.

[0187] Based on the same inventive concept, corresponding to the Figure 2 shown calibration parameter determination method provided in the embodiment of the present invention, the embodiment of the present invention also provides a calibration parameter determination device.

[0188] Figure 12 As shown in the structural schematic diagram of a calibration parameter determination method provided in the embodiment of the present invention, Figure 12 the device may include the following modules:

[0189] An image acquisition module 1210, configured to acquire multiple images collected by a vehicle-mounted camera to be calibrated;

[0190] A processing module 1220 is configured to determine a to-be-processed image in an image collected for a zebra crossing scene, and perform distortion removal processing on the to-be-processed image to obtain a target image;

[0191] A determination module 1230 is configured to use a preset deep learning algorithm to determine a vanishing point of the zebra crossing included in the target image and a segmentation map of each line of the zebra crossing, and extract a specified edge line in each segmentation map by using an edge extraction algorithm; wherein, the specified edge line has a specified geometric relationship with the traffic direction of the zebra crossing;

[0192] An equation establishment module 1240 is configured to establish a constraint equation regarding the vanishing point, the internal parameters of the to-be-calibrated vehicle-mounted camera, and the calibration parameters of the to-be-calibrated vehicle-mounted camera; wherein, the internal parameters include: a principal point and a focal length; the calibration parameters include: a pitch angle, a yaw angle, and a roll angle;

[0193] A solution module 1250 is configured to solve the constraint equation based on a preset solution target determined by using the geometric characteristics of the zebra crossing to obtain the calibration parameters; wherein, the solution target includes: the specified edge lines are parallel.

[0194] As can be seen above, by applying the solution provided in the embodiment of the present invention, when determining the calibration parameters of the to-be-calibrated vehicle-mounted camera, an image including a zebra crossing collected by the vehicle-mounted camera can be used, combined with the geometric characteristics of the zebra crossing, and the calibration parameters of the to-be-calibrated vehicle-mounted camera can be obtained by solving a constraint equation regarding the vanishing point of the zebra crossing, the internal parameters of the to-be-calibrated vehicle-mounted camera, and the calibration parameters. Furthermore, when there are multiple to-be-calibrated vehicle-mounted cameras, during the process of stitching the images collected by the above multiple to-be-calibrated vehicle-mounted cameras, the calibration parameters of each of the above to-be-calibrated vehicle-mounted cameras obtained above can be used to calibrate the above images, so that the stitched image can more accurately display the road conditions around the vehicle.

[0195] Optionally, in a specific implementation manner, at least one vehicle-mounted camera is respectively installed in at least one of the front, rear, left, and right sides of the vehicle body where the to-be-calibrated vehicle-mounted camera is located; the to-be-calibrated vehicle-mounted camera includes one or more of the vehicle-mounted cameras installed on the vehicle.

[0196] Optionally, in a specific implementation manner, the to-be-calibrated vehicle-mounted camera includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body of the vehicle;

[0197] The determination module 1230 is specifically configured to:

[0198] Using a preset deep learning algorithm, determine the first vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the first specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the first edge lines in the segmentation maps of the respective lines intersect at the first vanishing point.

[0199] Optionally, in a specific implementation manner, the vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed on the left side and / or the right side of the vehicle body of the vehicle;

[0200] The determining module 1230 is specifically configured to:

[0201] Using a preset deep learning algorithm, determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the second specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the traffic direction of the zebra crossing in the segmentation maps of the respective lines intersect at the second vanishing point.

[0202] Optionally, in a specific implementation manner, if the target image includes multiple lines in the zebra crossing; then the solution target further includes: each line in the zebra crossing has the same width, and / or, each line in the zebra crossing has the same height;

[0203] If the number of lines in the zebra crossing included in the target image is greater than 2; then the solution target further includes: at least one of each line in the zebra crossing having the same width, each line in the zebra crossing having the same height, and the same spacing between each group of adjacent lines in the zebra crossing.

[0204] Optionally, in a specific implementation manner, the constraint equation is an equation regarding the coordinates of the vanishing point in the image coordinate system corresponding to the target image, the coordinates of the principal point in the image coordinate system, the horizontal focal length of the vehicle-mounted camera to be calibrated, the vertical focal length of the vehicle-mounted camera to be calibrated, the pitch angle, the yaw angle, and the roll angle.

[0205] Optionally, in a specific implementation manner, the number of vehicle-mounted cameras to be calibrated is multiple, and the device further includes:

[0206] A calibration module, configured to calibrate the images collected by the vehicle-mounted camera to be calibrated using the calibration parameters;

[0207] A stitching module, configured to stitch the images collected by the vehicle-mounted camera to be calibrated after calibration to obtain a stitched image.

[0208] An embodiment of the present invention also provides an electronic device, such as Figure 13 shown, which includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. Among them, the processor 1301, the communication interface 1302, and the memory 1303 complete communication with each other through the communication bus 1304.

[0209] The memory 1303 is used to store computer programs.

[0210] When the processor 1301 executes the programs stored on the memory 1303, it implements the steps of any of the calibration parameter determination methods provided in the above embodiments of the present invention.

[0211] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0212] The communication interface is used for the communication between the above electronic device and other devices.

[0213] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0214] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0215] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of any of the above-described calibration parameter determination methods are implemented.

[0216] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the calibration parameter determination methods in the above embodiments.

[0217] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0218] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0219] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the relevant parts of the method embodiments for the related content.

[0220] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for determining calibration parameters, characterized in that, The method includes: Obtaining multiple images collected by the vehicle-mounted camera to be calibrated; Determining a to-be-processed image from the images collected for the zebra crossing scene, and performing distortion removal processing on the to-be-processed image to obtain a target image; Using a preset deep learning algorithm to determine the vanishing point of the zebra crossing included in the target image and the segmentation maps of the respective lines of the zebra crossing, and using an edge extraction algorithm to extract the specified edge lines in each segmentation map; wherein, the extension lines of the multiple specified edge lines intersect at the vanishing point, and the specified edge line has a specified geometric relationship with the traffic direction of the zebra crossing; Establishing a constraint equation regarding the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated; wherein, the internal parameters include: the principal point and the focal length; the calibration parameters include: the pitch angle, the yaw angle, and the roll angle; Based on a preset solution objective determined using the geometric characteristics of the zebra crossing, solving the constraint equation to obtain the calibration parameters; wherein, the solution objective includes: the specified edge lines being parallel; Wherein, the constraint equation is: Wherein, u is the abscissa of the vanishing point in the image coordinate system corresponding to the target image; v is the ordinate of the vanishing point in the image coordinate system; cx is the abscissa of the principal point in the image coordinate system; cy is the ordinate of the principal point in the image coordinate system; fx is the horizontal focal length of the vehicle-mounted camera to be calibrated; fy is the vertical focal length of the vehicle-mounted camera to be calibrated; pitch is the pitch angle; yaw is the yaw angle; roll is the roll angle.

2. The method according to claim 1, wherein At least one vehicle-mounted camera is respectively installed in at least one of the front, rear, left, and right sides of the vehicle body where the vehicle-mounted camera to be calibrated is located; the vehicle-mounted camera to be calibrated includes one or more of the vehicle-mounted cameras installed on the vehicle.

3. The method according to claim 2, wherein The vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body of the vehicle; The using a preset deep learning algorithm to determine the vanishing point of the zebra crossing included in the target image and the segmentation maps of the respective lines of the zebra crossing, and using an edge extraction algorithm to extract the specified edge lines in each segmentation map includes: Using a preset deep learning algorithm to determine the first vanishing point of the zebra crossing included in the target image and the segmentation maps of the respective lines of the zebra crossing, and using an edge extraction algorithm to extract the first specified edge lines perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the first edge lines in the respective lines' segmentation maps intersect at the first vanishing point.

4. The method according to claim 2, characterized in that, The vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed on the left and / or right sides of the vehicle body of the vehicle; The using a preset deep learning algorithm to determine the vanishing point of the zebra crossing included in the target image and the segmentation maps of the respective lines of the zebra crossing, and using an edge extraction algorithm to extract the specified edge lines in each segmentation map includes: Using a preset deep learning algorithm, determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the second specified edge line perpendicular to the traffic direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the traffic direction of the zebra crossing in the segmentation maps of the respective lines intersect at the second vanishing point.

5. The method according to claim 1, wherein if the target image includes multiple lines in the zebra crossing, the solution objective further includes: each line in the zebra crossing has the same width, and / or, each line in the zebra crossing has the same height; or if the number of lines in the zebra crossing included in the target image is greater than 2, the solution objective further includes at least one of: each line in the zebra crossing has the same width, each line in the zebra crossing has the same height, and the distance between each group of adjacent lines in the zebra crossing is the same.

6. The method according to claim 1, wherein The number of vehicle-mounted cameras to be calibrated is multiple, and the method further includes: calibrating the images collected by the vehicle-mounted cameras to be calibrated using the calibration parameters; stitching the images collected by the calibrated vehicle-mounted cameras to be calibrated to obtain a stitched image.

7. A calibration parameter determination device, characterized in that, The device includes: an image acquisition module, configured to acquire multiple images collected by a vehicle-mounted camera to be calibrated; a processing module, configured to determine a to-be-processed image in the images collected for the zebra crossing scenario and perform a distortion removal process on the to-be-processed image to obtain a target image; a determination module, configured to use a preset deep learning algorithm to determine the vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the specified edge line in each segmentation map; wherein, the extension lines of the multiple specified edge lines intersect at the vanishing point, and the specified edge line has a specified geometric relationship with the traffic direction of the zebra crossing; an equation establishment module, configured to establish a constraint equation regarding the vanishing point, the internal parameters of the vehicle-mounted camera to be calibrated, and the calibration parameters of the vehicle-mounted camera to be calibrated; wherein, the internal parameters include: the principal point and the focal length; the calibration parameters include: the pitch angle, the yaw angle, and the roll angle; a solution module, configured to solve the constraint equation based on a preset solution objective determined using the geometric characteristics of the zebra crossing to obtain the calibration parameters; wherein, the solution objective includes: the specified edge lines are parallel; wherein, the constraint equation is: wherein, u is the abscissa of the vanishing point in the image coordinate system corresponding to the target image; v is the ordinate of the vanishing point in the image coordinate system; cx is the abscissa of the principal point in the image coordinate system; cy is the ordinate of the principal point in the image coordinate system; fx is the horizontal focal length of the vehicle-mounted camera to be calibrated; fy is the vertical focal length of the vehicle-mounted camera to be calibrated; pitch is the pitch angle; yaw is the yaw angle; roll is the roll angle.

8. The device according to claim 7, wherein At least one vehicle-mounted camera is installed in at least one of the directions in front of the vehicle body, behind the vehicle body, on the left side of the vehicle body, and on the right side of the vehicle body where the vehicle-mounted camera to be calibrated is located; the vehicle-mounted camera to be calibrated includes one or more of the vehicle-mounted cameras installed on the vehicle. And / or The vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed in front of and / or behind the vehicle body of the vehicle. The determining module is specifically configured to: Using a preset deep learning algorithm, determine the first vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the first specified edge line perpendicular to the passing direction of the zebra crossing in each segmentation map; wherein, the extension lines of the first edge lines in the segmentation maps of each line intersect at the first vanishing point. And / or The vehicle-mounted camera to be calibrated includes at least one vehicle-mounted camera installed on the left side and / or the right side of the vehicle body of the vehicle. The determining module is specifically configured to: Using a preset deep learning algorithm, determine the second vanishing point of the zebra crossing included in the target image and the segmentation map of each line of the zebra crossing, and use an edge extraction algorithm to extract the second specified edge line perpendicular to the passing direction of the zebra crossing in each segmentation map; wherein, the extension lines of the edge lines parallel to the passing direction of the zebra crossing in the segmentation maps of each line intersect at the second vanishing point. And / or If the target image includes multiple lines in the zebra crossing; then the solution target further includes: each line in the zebra crossing has the same width, and / or, each line in the zebra crossing has the same height; or, if the number of lines in the zebra crossing included in the target image is greater than 2; then the solution target further includes: at least one of each line in the zebra crossing having the same width, each line in the zebra crossing having the same height, and the same spacing between each group of adjacent lines in the zebra crossing. And / or The number of the vehicle-mounted cameras to be calibrated is multiple, and the device further includes: A calibration module, configured to calibrate the images collected by the vehicle-mounted cameras to be calibrated using the calibration parameters. A stitching module, configured to stitch the images collected by the calibrated vehicle-mounted cameras to be calibrated to obtain a stitched image.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. The memory is used to store computer programs. The processor, when executing the programs stored on the memory, implements the method steps described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-6.

Images