External parameter calibration method of image acquisition device, program product and equipment

Abstract

The invention discloses an external parameter calibration method of an image acquisition device, a program product and equipment. The method comprises the following steps: acquiring an initial image acquired by the image acquisition device; acquiring position coordinates of the calibration reference point based on an initial image displayed by the user interface; obtaining calibrated internal parameters of the image acquisition device; calculating an initial external parameter based on the internal parameter and the position coordinate of the calibration reference point; acquiring a current scene image acquired by the image acquisition device, and identifying a vehicle target from the current scene image to obtain current vehicle target data; updating the position of the calibration reference point based on the current vehicle target data; and based on the updated position of the calibration reference point, the internal parameter and the initial external parameter, calculating the calibrated external parameter.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an external parameter calibration method for an image acquisition device, a computer program product and vehicle equipment. Background Art

[0002] The calibration of the image acquisition device includes obtaining accurate intrinsic and extrinsic parameters of the image acquisition device. The intrinsic parameters of the image acquisition device are the parameters of the image acquisition device itself, such as focal length, optical center position, etc. The extrinsic parameters are related to the installation position of the image acquisition device, and include rotation matrices and translation vectors. For vehicle-mounted image acquisition devices, accurate extrinsic and intrinsic parameters of the image acquisition device are crucial for subsequent assisted driving tasks, positioning tasks, multi-sensor fusion tasks, etc. The intrinsic parameters can often be obtained by the manufacturer through a calibration plate with a standard pattern at the time of shipment.

[0003] The external parameter calibration of image acquisition devices is generally divided into two categories: calibration using a calibration plate and self-calibration of the image acquisition device. The method using a calibration plate can only obtain the parameters of the image acquisition device in a static state, and cannot solve the changes in the camera's external parameters caused by the inevitable bumps and shakes of the vehicle during its movement on the road. Existing self-calibration methods for image acquisition devices mostly use lane line detection information for calibration, but in infrared cameras, lane lines cannot always be clearly imaged. This method is more suitable for visible light cameras, but not for infrared cameras. Summary of the invention

[0004] In order to solve the existing technical problems, the present invention provides an extrinsic parameter calibration method, a computer program product and a vehicle equipment for an image acquisition device, which can perform an online real-time self-calibration process and continuously correct the extrinsic parameter, thereby improving the accuracy of the extrinsic parameter calibration.

[0005] In a first aspect, a method for calibrating external parameters of an image acquisition device is provided, comprising: acquiring an initial image acquired by the image acquisition device; acquiring position coordinates of a calibration reference point based on the initial image displayed on a user interface; acquiring internal parameter parameters calibrated for the image acquisition device; calculating initial external parameter parameters based on the internal parameter parameters and the position coordinates of the calibration reference point; acquiring a current scene image acquired by the image acquisition device, identifying a vehicle target from the current scene image, and obtaining current vehicle target data; updating the position of the calibration reference point based on the current vehicle target data; and calculating the calibrated external parameter parameters based on the updated position of the calibration reference point, the internal parameter parameters, and the initial external parameter parameters.

[0006] In a second aspect, a computer program product is provided, including a computer program, which, when executed by a controller, implements the external parameter calibration method of the image acquisition device as described in any embodiment of the present application.

[0007] According to a third aspect, a vehicle device is provided, comprising a memory and a controller, wherein the memory stores a computer program, and when the computer program is executed by the controller, the controller executes the external parameter calibration method of the image acquisition device as described in any embodiment of the present application.

[0008] The embodiment of the present application first acquires an initial image and displays the initial image on a user interface, acquires the position coordinates of a calibration reference point based on the displayed initial image, and acquires the calibrated intrinsic parameter, and obtains the initial extrinsic parameter based on the selected calibration reference point and the intrinsic parameter; then, during vehicle driving, the current vehicle target data is updated by continuously acquiring multiple frames of image data, and the position of the calibration reference point is continuously updated based on the vehicle target data in the multiple frames, and the external parameter parameters of the image acquisition device are continuously updated by using the updated position of the calibration reference point, the intrinsic parameter and the initial external parameter, to complete the online real-time self-calibration process, and continuously correct the external parameter, thereby improving the accuracy of the external parameter correction, and the system can also be applied to the calibration of infrared cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A diagram showing an application environment of an external parameter calibration method for an image acquisition device in an embodiment;

[0010] Figure 2 is a flow chart of an external parameter calibration method for an image acquisition device in one embodiment;

[0011] Figure 3 A schematic diagram of a user interface displaying a calibration reference point in an embodiment;

[0012] Figure 4 A schematic diagram of external and internal parameters in one embodiment;

[0013] Figure 5 is a schematic diagram of an imaging model based on calibration reference points in an embodiment;

[0014] Figure 6 Schematic diagram of a flow chart of calculating calibrated extrinsic parameters in an extrinsic calibration method for an image acquisition device in one embodiment;

[0015] Figure 7 is a schematic diagram of the actual widths of various vehicle types in one embodiment;

[0016] Figure 8 is a schematic diagram of an external parameter calibration device of an image acquisition device in one embodiment;

[0017] Fig. 9 Schematic diagram of vehicle equipment in one embodiment. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the technical field of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0020] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it should be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] See also Figure 1 , is an application environment diagram of an external parameter calibration method for an image acquisition device in an embodiment. The external parameter calibration method for an image acquisition device is applied to a vehicle device 10, and the vehicle device 10 includes a display terminal 11, an image acquisition device 12, and a controller 13. The image acquisition device 12 is used to acquire an image in front of the vehicle device 10, and the controller 13 assists in the calibration of the external parameter of the image acquisition device 12 according to the image acquired by the image acquisition device 12. The display terminal 11 is used to display images and various user interfaces for interaction with the user, so as to facilitate accurate calibration of the external parameter.

[0022] The image acquisition device 12 may be a combination of one or more sensors. The image acquisition device 12 may be a monocular vision sensor or a multi-vision sensor. For example, it may be a combination of one or more sensors including a thermal imaging sensor, a visible light image sensor, a millimeter wave sensor, a laser radar sensor, an infrared thermal imaging sensor, and a depth sensor. The image acquisition device 12 is disposed in front of the vehicle device 10, for example, installed in the front bumper of the vehicle device 10.

[0023] The controller 13 may be one or more. When there are multiple controllers 13, the multiple controllers may be integrated on one chip or independently arranged on each chip. The vehicle device 10 is a device installed on any type of mobile body, such as a vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, or a robot, etc.

[0024] The vehicle device 10 may also include other sensor modules, including but not limited to environmental perception sensors and motion posture sensors. Environmental perception sensors include but are not limited to one or more combinations of the following sensors: brightness sensors, temperature sensors, haze sensors and other environmental sensors. Motion posture sensors include but are not limited to one or more combinations of the following: inertial sensors (Inertial Measurement Unit, IMU), speed sensors, acceleration sensors, gyroscope sensors, geomagnetic sensors, rotation vector sensors, steering wheel angle sensors, level sensors, tilt sensors, vibration sensors, displacement sensors and gravity sensors, etc.

[0025] The screen is the surface on which the display terminal 11 displays images. The screen is arranged in front of the driver so that the driver can view the images. The display terminal 11 includes but is not limited to displays such as liquid crystal displays, projectors with projector screens, etc. The display terminal 11 can be a display located on the central control of the vehicle device 10, or a projector arranged at the rear of the vehicle device 10 to project images onto the projector screen.

[0026] The controller 13 may be one or more. When there are multiple controllers 13, the multiple controllers may be integrated on one chip or independently arranged on each chip. The vehicle device 10 is a device installed on any type of mobile body, such as a vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, or a robot, etc.

[0027] See also Figure 2 , is a flow chart of an external parameter calibration method for an image acquisition device provided in an embodiment of the present application. The external parameter calibration method for an image acquisition device is applied to vehicle equipment, and the external parameter calibration method for an image acquisition device includes the following steps:

[0028] S11, acquiring an initial image captured by an image acquisition device.

[0029] In this embodiment, when the image acquisition device is installed in front of the vehicle equipment, the intrinsic parameters of the image acquisition device have been calibrated, and the intrinsic parameters mainly include focal length, principal point coordinates, pixel size and distortion parameters, etc. When a certain external parameter is calibrated later, it can be obtained by calibration using one or more intrinsic parameters.

[0030] The initial image refers to the image captured after the image acquisition device is installed horizontally. When taking the initial image, the vehicle equipment can be placed at the edge of the road and parallel to the road; or the vehicle equipment can be controlled to drive in a straight line to control the image acquisition device to capture the initial image.

[0031] S12. Based on the initial image displayed on the user interface, obtain the position coordinates of the calibration reference point.

[0032] In this embodiment, the calibration reference point indicates a reference point for calibrating the extrinsic parameter. The initial extrinsic parameter can be calibrated according to the position of the calibration reference point to obtain more accurate extrinsic parameters. The calibration personnel can select a calibration reference point from the initial image based on the displayed initial image. For example, a point can be selected by touch, or multiple line segments intersecting at a point can be drawn, and the intersection point can be used as the calibration reference point.

[0033] like Figure 3 As shown, Figure 3 This is a schematic diagram of a user interface displaying a calibration reference point in an embodiment. The calibration personnel mark two intersecting line segments on the original image on the user interface, and the intersection is the calibration reference point. Of course, a point can also be directly specified as a calibration reference point on the user interface, such as by touching a point with a finger, or marking a point on the user interface.

[0034] S13: Acquire calibrated intrinsic parameters of the image acquisition device.

[0035] In this embodiment, the acquired intrinsic parameters may include but are not limited to the following or a combination of multiple ones: focal length, principal point coordinates, optical center position, pixel size, distortion parameters, etc.

[0036] S14. Calculate initial extrinsic parameters based on the intrinsic parameters and the position coordinates of the calibration reference point.

[0037] In this embodiment, the internal parameter is a parameter that has been calibrated before leaving the factory and is a relatively accurate parameter. Therefore, the initial external parameter can be obtained by combining the internal parameter and the position coordinates of the calibration reference point. In this embodiment, the initial external parameter includes one or more of the following combinations: initial pitch angle and initial azimuth angle.

[0038] like Figure 4 As shown, Figure 4The figure is a schematic diagram of external parameters in one embodiment. The Z-axis direction is along the center line of the vehicle and points to the forward direction of the vehicle. The Y-axis represents the vertical direction. The X-axis is the horizontal direction, pointing from one side of the vehicle to the other side and perpendicular to the YZ plane. The vehicle rotation motion corresponding to these three coordinate axes includes: roll around the Z-axis, that is, the roll angle; pitch around the X-axis, that is, the pitch angle, the pitch angle is the installation angle and installation direction of the image acquisition device relative to the horizontal plane; and yaw around the Y-axis, that is, the azimuth angle. After the image acquisition device is installed relatively horizontally, the roll angle does not change much. Generally, after the initial roll angle is obtained using the imaging model of the image acquisition device, subsequent calibration correction is not required. However, when the vehicle equipment is driving, the uneven road in the driving environment will affect the pitch angle of the image acquisition device. Therefore, after obtaining the initial pitch angle, subsequent calibration is required to obtain a more accurate pitch angle.

[0039] S15, acquiring the current scene image acquired by the image acquisition device, identifying the vehicle target from the current scene image, and obtaining current vehicle target data.

[0040] In this embodiment, the vehicle equipment can be controlled to travel normally on the road, and the environmental image in front of the vehicle can be collected. The collected environmental image exists in the form of a frame video stream. The current scene image indicates the image collected in the current frame. The vehicle target includes various types of vehicles, including but not limited to cars, buses, trucks (trunks), tank trucks, muck trucks, other vehicles, two-wheeled vehicles, three-wheeled vehicles, etc. For each type of vehicle, the actual width size when leaving the factory can be obtained.

[0041] In this embodiment, current input data can be formed based on the current scene image, and the current input data can be input into a pre-trained target recognition network model, and the target can be output based on the pre-trained target recognition network model, wherein the target recognition network model is trained based on the training data set.

[0042] In some embodiments, the method further includes: obtaining a training data set, wherein each training sample in the training data set includes a training image and a category label corresponding to the training image, and the category label indicates the type of vehicle; during the training process, obtaining a training input sample from the training data set, forming training input data based on the training input sample, and outputting the output data in this iteration through the target recognition network model in iterative training; calculating the loss value of this iteration based on the output data in this iteration and the category label corresponding to the training input data, judging whether the current iteration meets the iteration termination condition based on the loss value of this iteration, and then using the target recognition network model after stopping the iteration as the pre-trained target recognition network model. The training images may include various types of vehicles, and the richer the types of vehicles, the more corresponding training images, so that after the training is completed, the target recognition network model can accurately identify various types of vehicle targets.

[0043] The current vehicle target data includes the target frame data of the current vehicle target, and the target frame data includes but is not limited to the size and position of the target frame, etc. The target frame is output by the target recognition network model and is used to frame the corresponding vehicle target. One vehicle target corresponds to one target frame.

[0044] S16. Based on the current vehicle target data, update the position of the calibration reference point.

[0045] In this embodiment, since the vehicle equipment can capture many target frames of vehicle targets in each frame during driving, the position coordinates of the bottom edge of each target frame can be obtained in each frame. According to the imaging model of the image acquisition device, the position of the calibration reference point is continuously updated to obtain a more accurate calibration reference point.

[0046] S17 calculates the calibrated extrinsic parameters based on the updated position of the calibration reference point, the intrinsic parameters and the initial extrinsic parameters.

[0047] In this embodiment, since the position of the calibration reference point is continuously updated in real time based on the captured frame image, the calibrated extrinsic parameter is continuously updated and calculated based on the updated position of the calibration reference point, the intrinsic parameter and the initial extrinsic parameter.

[0048] like Figure 5 As shown, Figure 5 Schematic diagram of an imaging model based on calibration reference points in an embodiment, Hc represents the height of the camera from the ground, yh represents the ordinate of the reference calibration point, and yb represents the ordinate corresponding to the bottom edge of the target frame of the photographed vehicle, wherein the ordinate is the coordinate in the vertical direction. The calibrated extrinsic parameters are calculated according to the ordinate corresponding to the bottom edge of the target frame of the photographed vehicle, the ordinate of the reference calibration point, the intrinsic parameter and the initial extrinsic parameter.

[0049] In the above embodiment, an initial image is first acquired and displayed on a user interface. Based on the displayed initial image, the position coordinates of the calibration reference point are acquired, and the calibrated internal parameter parameters are acquired. Based on the selected calibration reference point and the internal parameter parameters, the initial external parameter parameters can be obtained to complete the initial calibration process. Then, during the driving process of the vehicle, the current vehicle target data is updated by continuously acquiring multiple frames of image data, and the position of the calibration reference point is continuously updated based on the vehicle target data in the multiple frames. The updated position of the calibration reference point, the internal parameter parameters and the initial external parameter parameters are used to continuously update the external parameter parameters of the image acquisition device to complete the online real-time self-calibration process, and the external parameter parameters are continuously corrected, thereby improving the accuracy of the external parameter correction. At the same time, it can also be applied to the calibration of infrared cameras.

[0050] In some embodiments, the calibration reference point indicates a vanishing point in the initial image.

[0051] In this embodiment, the vanishing point indicates the point where a group of parallel lines converge in the distance in the initial image. Therefore, when shooting the initial image, the vehicle device is placed at the edge of the road, parallel to the road; or the vehicle device is controlled to drive along a straight line, and the initial image is captured, so that the initial image can include at least one group of parallel road lines, and the vanishing point can be found in the initial image by using this group of parallel road lines. According to the imaging model, a geometric relationship is established by using the property that the line connecting the vanishing point and the optical center of the image acquisition device is parallel to the parallel lines forming the vanishing point plane, thereby obtaining various parameters, such as the line connecting the vanishing point and the optical center is parallel to the parallel lines forming the vanishing point plane. Therefore, if there are several groups of parallel lines perpendicular to each other in the scene, the lines connecting their corresponding vanishing points and the optical center of the camera are also perpendicular to each other. This can be used to determine the rotation matrix of the image acquisition device relative to the world coordinate system. Moreover, by adopting the vanishing point online calibration, the external parameter parameters can be continuously updated, and the accuracy of the subsequent calibration external parameters can be improved.

[0052] In the above embodiment, the vanishing point is set as the calibration reference point. In combination with the imaging model, a geometric relationship can be established and can be used for online calibration to continuously update the extrinsic parameters, thereby improving the accuracy of subsequent calibration extrinsic parameters.

[0053] In some embodiments, before acquiring the initial image captured by the image acquisition device, the method further includes:

[0054] Determining whether the image acquisition device is level relative to a horizontal plane;

[0055] When the image acquisition device is not level with respect to the horizontal plane, first prompt information is output to prompt that the image acquisition device is not level with respect to the horizontal plane.

[0056] In this embodiment, if the image acquisition device is not installed horizontally, it will affect the calibration accuracy of the subsequent external parameters. Therefore, a level meter can be used to detect whether the image acquisition device is installed horizontally. It can also be determined whether the image acquisition device is horizontal by analyzing the vanishing points in the image. If the scene captured by the image acquisition device contains straight lines parallel to the ground (such as lines on the road or the floor), these straight lines should remain parallel in the image and do not intersect at any vanishing points. If these lines intersect, it means that the image acquisition device is not horizontal. The edges and lines in the image can also be analyzed. If all horizontal lines in the image are parallel to the upper or lower edge of the image, the image acquisition device may be basically horizontal. If the lines are tilted, it may indicate that the image acquisition device needs to be adjusted.

[0057] When the image acquisition device is not level with respect to the horizontal plane, a first prompt message is output to prompt that the image acquisition device is not level with respect to the horizontal plane, so as to prompt the calibration personnel to promptly check whether the image acquisition device is installed level.

[0058] In the above embodiment, whether the image acquisition device is level relative to the horizontal plane is automatically detected, so that no human judgment is required, thereby improving the calibration accuracy.

[0059] In some embodiments, acquiring the position coordinates of the calibration reference point based on the initial image displayed on the user interface includes:

[0060] Based on the displayed initial image, a designation operation for designating the calibration reference point is acquired, and based on the designation operation, the position coordinates of the calibration reference point are acquired.

[0061] Optionally, acquiring a designation operation for designating the calibration reference point based on the displayed initial image, and acquiring the position coordinates of the calibration reference point based on the designation operation, includes:

[0062] Based on the displayed initial image, obtaining a point designation operation, and based on the point designation operation, obtaining the position coordinates of the calibration reference point; or

[0063] Based on the displayed initial image, a cross-line marking operation is acquired, and based on the cross-line marking operation, the position coordinates of the calibration reference point are acquired.

[0064] In this embodiment, the point designation operation includes but is not limited to touch operations and point marking operations in the user interface. Touch operations include but are not limited to single-point touch or multi-point touch operations. After touching a certain position through a touch operation, the position point corresponding to the touch operation is obtained, and then the obtained position point is used as a calibration reference point. Or the position point on the user interface is marked, such as drawing a circle, etc., and the marked position of the marking operation is obtained, and the marked position is used as a calibration reference point. In an optional implementation, an image analysis method can be used to extract multiple candidate vanishing points from the initial image and display them on the user interface, and a selection operation based on the multiple candidate vanishing points displayed on the user interface is obtained, and the vanishing point corresponding to the selected operation is used as a calibration reference point. It is also possible to specify a cross line in the user interface, obtain a cross line marking operation based on the user interface, obtain a corresponding cross line based on the cross line marking operation, obtain the intersection of the cross lines, and determine the intersection of the cross lines as a calibration reference point. For example, Figure 3 As shown in the figure, draw a cross line on the user interface and use the intersection of the cross line as the calibration reference point.

[0065] In the above embodiment, a specified operation can be performed on the initial image displayed on the user interface, so that the user can intuitively select the calibration reference point; and the calibration reference point can be obtained based on the point specification operation, and the calibration reference point can also be specified through the cross line marking operation, which facilitates multi-style interaction and improves the efficiency of calibration.

[0066] In some embodiments, the initial external parameter includes an initial pitch angle and / or an initial azimuth angle, the internal parameter includes an optical center position coordinate and a focal length parameter, and the initial external parameter is calculated based on the internal parameter and the position coordinate of the calibration reference point, including:

[0067] Based on the internal reference parameters and the position coordinates of the calibration reference point, and using the initial pitch angle calculation formula, the initial pitch angle is calculated; wherein the initial pitch angle calculation formula is as follows:

[0068] pitch angle_ori = atan((cy―vp.y),fy), and / or

[0069] Based on the internal parameter and the position coordinates of the calibration reference point, and using the initial azimuth calculation formula, the initial azimuth is calculated; wherein the calculation formula of the initial azimuth is as follows:

[0070] yaw angle_ori = atan((cx―vp.x),fx),

[0071] Among them, pitch angle_ori is the initial pitch angle, yaw angle_oriis the initial azimuth, cx represents the horizontal coordinate of the optical center position in the image coordinate system, cy represents the vertical coordinate of the optical center position in the image coordinate system, fx represents the pixel length of the focal length in the x-axis direction in the image coordinate system, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, vp represents the calibration reference point, vp.x represents the horizontal coordinate of the calibration reference point in the image coordinate system, and vp.y represents the vertical coordinate of the calibration reference point in the image coordinate system.

[0072] In this embodiment, after the initial azimuth angle is obtained, the azimuth angle is generally not greatly affected during the vehicle's driving process, so the azimuth angle does not need to be corrected subsequently.

[0073] In the above embodiment, the initial pitch angle and / or the initial azimuth angle can be calculated using the intrinsic parameters and the position coordinates of the calibration reference point, thereby facilitating the subsequent correction of some extrinsic parameters and improving the accuracy of the extrinsic calibration.

[0074] In some embodiments, the step of S16 includes:

[0075] S61. Calculate the current vertical coordinate of the calibration reference point based on the current vehicle target data.

[0076] In some optional embodiments, the calculating, based on the current vehicle target data, the current vertical coordinate of the calibration reference point includes:

[0077] Acquire the height data of the image acquisition device from the ground, and acquire the actual width of each current vehicle target;

[0078] Calculate the vehicle image width of each current vehicle target according to the target frame of each current vehicle target;

[0079] Calculating an average actual vehicle width according to the actual width of each current vehicle target, and calculating an average vehicle image width according to the vehicle image width of each current vehicle target;

[0080] Based on the height data, the average actual vehicle width and the average vehicle image width, the average ordinate corresponding to the calibration reference point is calculated, and based on the average ordinate corresponding to the calibration reference point, the current ordinate is obtained.

[0081] like Figure 7 As shown, the actual vehicle widths of six different vehicle types are displayed, and different symbols are used to represent the width values ​​of different vehicle types.

[0082] S62: Obtain the vertical coordinate of the calibration reference point in a preset frame before the current frame.

[0083] In this embodiment, the previous preset frame is the N frames before the current frame. If N=1, the previous preset frame is the previous frame.

[0084] S63: Weighting the current vertical coordinate and the vertical coordinate in the previous preset frame to obtain a weighted vertical coordinate.

[0085] In an optional implementation, weighting the current vertical coordinate and the vertical coordinate in the previous preset frame to obtain a weighted vertical coordinate includes:

[0086] The current ordinate and the ordinate in the previous preset frame are weighted by using a weighted formula to obtain a weighted ordinate, wherein the weighted formula is as follows:

[0087]

[0088] where y h (t) represents the weighted ordinate corresponding to the calibration reference point in the current frame t, represents the current ordinate corresponding to the calibration reference point in the current frame collected at the current time t, y h (t-1) represents the ordinate corresponding to the calibration reference point in the t-1 frame, and γ represents a parameter.

[0089] S64. Based on the weighted ordinate, obtain the updated position of the calibration reference point.

[0090] In this embodiment, since the image acquisition device is already horizontal relative to the horizontal plane, the horizontal coordinate of the calibration reference point does not change much, so it does not need to be updated. However, since the vehicle will bump up and down during the formation process, the vertical coordinate of the calibration reference point needs to be corrected, and the weighted vertical coordinate is used as the vertical coordinate of the updated calibration reference point.

[0091] In the above embodiment, the current vertical coordinate is calculated by continuously acquiring updated vehicle target data, and the current vertical coordinate is corrected based on the vertical coordinate of the calibration reference point calculated in the historical frame. After the accurate vertical coordinate of the calibration reference point is obtained, it is convenient to obtain accurate external parameter parameters in the subsequent process, thereby improving the real-time performance of the external parameter correction and improving the accuracy of the external parameter correction.

[0092] In an optional implementation, the initial extrinsic parameter includes an initial pitch angle, the intrinsic parameter includes an optical position coordinate, and a focal length parameter, and the step of calculating the calibrated extrinsic parameter based on the updated position of the calibration reference point, the intrinsic parameter, and the initial extrinsic parameter includes:

[0093] Based on the updated position of the calibration reference point, the memory parameters and the initial external parameter parameters, and using the pitch angle calculation formula, the calibrated external parameter parameters are calculated, wherein the external parameter calculation formula is as follows:

[0094] pitch angle =σ1*atan((cy―y h (t)),fy)+(1―σ1)*pitch angle_ori ,

[0095] Where pitch angle represents the pitch angle after calibration, σ1 is the coefficient, cy represents the ordinate of the optical center position in the image coordinate system, y h (t) The updated ordinate corresponding to the calibration reference point in the current frame t, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, pitch angle_ori Indicates the initial pitch angle.

[0096] In this embodiment, due to the up and down bumps of the vehicle during driving, the longitudinal coordinate of the calibration reference point may not be very accurate. The image in front of the vehicle equipment during driving can be continuously acquired, and the current longitudinal coordinate can be calculated based on the vehicle target data in the current frame. The current longitudinal coordinate is corrected based on the longitudinal coordinate of the calibration reference point calculated in the historical frame. After obtaining the accurate longitudinal coordinate of the calibration reference point, it is convenient to obtain accurate external parameter parameters later.

[0097] In the above embodiment, the current vertical coordinate is calculated by continuously acquiring updated vehicle target data, and the current vertical coordinate is corrected based on the vertical coordinate of the calibration reference point calculated in the historical frame. After the accurate vertical coordinate of the calibration reference point is obtained, it is convenient to obtain accurate external parameter parameters in the subsequent process, thereby improving the real-time performance of the external parameter correction and improving the accuracy of the external parameter correction.

[0098] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a controller, implements the external parameter calibration method of the image acquisition device described in any embodiment of the present application.

[0099] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program that implements each step of the target recognition method can be an external parameter calibration device for the image acquisition device.

[0100] See also Figure 8An embodiment of the present application provides an external parameter calibration device for an image acquisition device, comprising: an acquisition module 81, used to acquire an initial image acquired by the image acquisition device; the acquisition module 81 is also used to acquire the position coordinates of a calibration reference point based on the initial image displayed on a user interface; the acquisition module 81 is also used to acquire the internal parameter parameters that have been calibrated for the image acquisition device; a calculation module 82 is used to calculate initial external parameter parameters based on the internal parameter parameters and the position coordinates of the calibration reference point; the acquisition module 81 is also used to acquire a current scene image acquired by the image acquisition device, identify a vehicle target from the current scene image, and obtain current vehicle target data; the calculation module 82 is also used to update the position of the calibration reference point based on the current vehicle target data; the calculation module 82 is also used to calculate the calibrated external parameter parameters based on the updated position of the calibration reference point, the internal parameter parameters and the initial external parameter parameters.

[0101] Optionally, the calibration reference point indicates a vanishing point in the initial image.

[0102] Optionally, the output module 83 is used to:

[0103] Before acquiring an initial image captured by an image acquisition device, determining whether the image acquisition device is horizontal relative to a horizontal plane;

[0104] When the image acquisition device is not level with respect to the horizontal plane, first prompt information is output to prompt that the image acquisition device is not level with respect to the horizontal plane.

[0105] Optionally, the acquisition module 81 is further used for:

[0106] Based on the displayed initial image, a designation operation for designating the calibration reference point is acquired, and based on the designation operation, the position coordinates of the calibration reference point are acquired.

[0107] Optionally, the acquisition module 81 is further used for:

[0108] Based on the displayed initial image, obtaining a point designation operation, and based on the point designation operation, obtaining the position coordinates of the calibration reference point; or

[0109] Based on the displayed initial image, a cross-line marking operation is acquired, and based on the cross-line marking operation, the position coordinates of the calibration reference point are acquired.

[0110] Optionally, the calculation module 82 is further used for:

[0111] Based on the internal reference parameters and the position coordinates of the calibration reference point, and using the initial pitch angle calculation formula, the initial pitch angle is calculated; wherein the initial pitch angle calculation formula is as follows:

[0112] pitchangle_ori = atan((cy―vp.y),fy), and / or

[0113] Based on the internal parameter and the position coordinates of the calibration reference point, and using the initial azimuth calculation formula, the initial azimuth is calculated; wherein the calculation formula of the initial azimuth is as follows:

[0114] yaw angle_ori = atan((cx―vp.x),fx),

[0115] Among them, pitch angle_ori is the initial pitch angle, yaw angle_ori is the initial azimuth, cx represents the horizontal coordinate of the optical center position in the image coordinate system, cy represents the vertical coordinate of the optical center position in the image coordinate system, fx represents the pixel length of the focal length in the x-axis direction in the image coordinate system, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, vp represents the calibration reference point, vp.x represents the horizontal coordinate of the calibration reference point in the image coordinate system, and vp.y represents the vertical coordinate of the calibration reference point in the image coordinate system.

[0116] Optionally, the calculation module 82 is further used for:

[0117] Calculating the current vertical coordinate of the calibration reference point based on the current vehicle target data;

[0118] Obtaining the ordinate of the calibration reference point in a preset frame before the current frame;

[0119] Weighting the current vertical coordinate and the vertical coordinate in the previous preset frame to obtain a weighted vertical coordinate;

[0120] Based on the weighted ordinate, the position of the updated calibration reference point is obtained.

[0121] Optionally, the calculation module 82 is further used for:

[0122] Acquire the height data of the image acquisition device from the ground, and acquire the actual width of each current vehicle target;

[0123] Calculate the vehicle image width of each current vehicle target according to the target frame of each current vehicle target;

[0124] Calculating an average actual vehicle width according to the actual width of each current vehicle target, and calculating an average vehicle image width according to the vehicle image width of each current vehicle target;

[0125] Based on the height data, the average actual vehicle width and the average vehicle image width, the average ordinate corresponding to the calibration reference point is calculated, and based on the average ordinate corresponding to the calibration reference point, the current ordinate is obtained.

[0126] Optionally, the calculation module 82 is further used for:

[0127] The current ordinate and the ordinate in the previous preset frame are weighted by using a weighted formula to obtain a weighted ordinate, wherein the weighted formula is as follows:

[0128]

[0129] where y h (t) represents the weighted ordinate corresponding to the calibration reference point in the current frame t, represents the current ordinate corresponding to the calibration reference point in the current frame collected at the current time t, y h (t-1) represents the ordinate corresponding to the calibration reference point in the t-1 frame, and γ represents a parameter.

[0130] Optionally, the calculation module 82 is further used for:

[0131] Based on the updated position of the calibration reference point, the memory parameters and the initial external parameter parameters, and using the pitch angle calculation formula, the calibrated external parameter parameters are calculated, wherein the external parameter calculation formula is as follows:

[0132] pitch angle =σ1*atan((cy―y h (t)),fy)+(1―σ1)*pitch angle_ori ,

[0133] Where pitch angle represents the pitch angle after calibration, σ1 is the coefficient, cy represents the ordinate of the optical center position in the image coordinate system, y h (t) The updated ordinate corresponding to the calibration reference point in the current frame t, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, pitch angle_ori Indicates the initial pitch angle.

[0134] It can be understood by those skilled in the art that Figure 8The structure of the external parameter calibration device of the image acquisition device does not constitute a limitation on the external parameter calibration device of the image acquisition device, and the various modules can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of the controller in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the controller can call and execute the operations corresponding to the above modules. In other embodiments, the external parameter calibration device of the image acquisition device may include more or fewer modules than those shown in the figure.

[0135] See also Fig. 9 On the other hand, a vehicle device 10 is provided, including a controller 13 and a memory 14. The memory 14 stores a computer program. When the computer program is executed by the controller, the controller 13 executes the steps of the external parameter calibration method of the image acquisition device provided in any of the above embodiments of the present application.

[0136] The controller 13 is a control center, which uses various interfaces and lines to connect various parts of the entire computer device, and executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 14, and calling data stored in the memory 14. Optionally, the controller 13 may include one or more processing cores; preferably, the controller 13 may integrate an application controller and a modem controller, wherein the application controller mainly processes the operating system, user pages and application programs, etc., and the modem controller mainly processes wireless communications. It is understandable that the above-mentioned modem controller may not be integrated into the controller 13.

[0137] The memory 14 can be used to store software programs and modules. The controller 13 executes various functional applications and data processing by running the software programs and modules stored in the memory 14. The memory 14 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 14 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 14 can also include a memory controller to provide the controller 13 with access to the memory 14.

[0138] On the other hand, an embodiment of the present application further provides a storage medium storing a computer program. When the computer program is executed by a controller, the controller executes the steps of the external parameter calibration method of the image acquisition device provided in any of the above embodiments of the present application.

[0139] Those skilled in the art can understand that all or part of the processes in the methods provided in the above embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0140] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for calibrating external parameters of an image acquisition device, characterized in that: include: Acquire an initial image acquired by an image acquisition device; Based on the initial image displayed on the user interface, the position coordinates of the calibration reference point are obtained; Acquiring calibrated intrinsic parameters of the image acquisition device; Calculating initial extrinsic parameters based on the intrinsic parameters and the position coordinates of the calibration reference point; Acquire the current scene image acquired by the image acquisition device, identify the vehicle target from the current scene image, and obtain current vehicle target data; Based on the current vehicle target data, updating the position of the calibration reference point; Based on the updated position of the calibration reference point, the intrinsic parameter and the initial extrinsic parameter, the calibrated extrinsic parameter is calculated.

2. The extrinsic parameter calibration method of the image acquisition device according to claim 1, characterized in that: The calibration reference point indicates a vanishing point in the initial image.

3. The extrinsic parameter calibration method of the image acquisition device according to claim 1, characterized in that: Before acquiring the initial image captured by the image acquisition device, the method further includes: Determining whether the image acquisition device is level relative to a horizontal plane; When the image acquisition device is not level relative to the horizontal plane, first prompt information is output to prompt that the image acquisition device is not level relative to the horizontal plane.

4. The external parameter calibration method of the image acquisition device according to claim 1, characterized in that: The obtaining of the position coordinates of the calibration reference point based on the initial image displayed on the user interface comprises: Based on the displayed initial image, a designation operation for designating the calibration reference point is acquired, and based on the designation operation, the position coordinates of the calibration reference point are acquired.

5. The external parameter calibration method of the image acquisition device according to claim 4, characterized in that: The acquiring, based on the displayed initial image, a designation operation for designating the calibration reference point, and acquiring, based on the designation operation, position coordinates of the calibration reference point, comprises: Based on the displayed initial image, obtaining a point designation operation, and based on the point designation operation, obtaining the position coordinates of the calibration reference point; or Based on the displayed initial image, a cross-line marking operation is acquired, and based on the cross-line marking operation, the position coordinates of the calibration reference point are acquired.

6. The external parameter calibration method of the image acquisition device according to claim 1, characterized in that: The initial external parameter includes an initial pitch angle and / or an initial azimuth angle, the internal parameter includes an optical center position coordinate and a focal length parameter, and the initial external parameter is calculated based on the internal parameter and the position coordinate of the calibration reference point, including: Based on the internal reference parameters and the position coordinates of the calibration reference point, and using the initial pitch angle calculation formula, the initial pitch angle is calculated; wherein the initial pitch angle calculation formula is as follows: pitch angle_ori = atan((cy-vp.y),fy), and / or Based on the internal parameter and the position coordinates of the calibration reference point, and using the initial azimuth calculation formula, the initial azimuth is calculated; wherein the calculation formula of the initial azimuth is as follows: yaw angle_ori =atan((cx-vp.x),fx), Among them, pitch anle_ori is the initial pitch angle, yaw angle_ori is the initial azimuth, cx represents the horizontal coordinate of the optical center position in the image coordinate system, cy represents the vertical coordinate of the optical center position in the image coordinate system, fx represents the pixel length of the focal length in the x-axis direction in the image coordinate system, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, vp represents the calibration reference point, vp.x represents the horizontal coordinate of the calibration reference point in the image coordinate system, and vp.y represents the vertical coordinate of the calibration reference point in the image coordinate system.

7. The external parameter calibration method of the image acquisition device according to claim 1, characterized in that: The updating of the position of the calibration reference point based on the current vehicle target data comprises: Calculating the current vertical coordinate of the calibration reference point based on the current vehicle target data; Obtaining the ordinate of the calibration reference point in a preset frame before the current frame; Weighting the current vertical coordinate and the vertical coordinate in the previous preset frame to obtain a weighted vertical coordinate; Based on the weighted ordinate, the position of the updated calibration reference point is obtained.

8. The external parameter calibration method of the image acquisition device according to claim 7, characterized in that: The step of calculating the current vertical coordinate of the calibration reference point based on the current vehicle target data includes: Acquire the height data of the image acquisition device from the ground, and acquire the actual width of each current vehicle target; Calculate the vehicle image width of each current vehicle target according to the target frame of each current vehicle target; Calculating an average actual vehicle width according to the actual width of each current vehicle target, and calculating an average vehicle image width according to the vehicle image width of each current vehicle target; Based on the height data, the average actual vehicle width and the average vehicle image width, the average ordinate corresponding to the calibration reference point is calculated, and based on the average ordinate corresponding to the calibration reference point, the current ordinate is obtained.

9. The external parameter calibration method of the image acquisition device according to claim 7, characterized in that: The step of weighting the current vertical coordinate and the vertical coordinate in the previous preset frame to obtain a weighted vertical coordinate includes: The current ordinate and the ordinate in the previous preset frame are weighted by using a weighted formula to obtain a weighted ordinate, wherein the weighted formula is as follows: where y h (t) represents the weighted ordinate corresponding to the calibration reference point in the current frame t, represents the current ordinate corresponding to the calibration reference point in the current frame collected at the current time t, y h (t-1) represents the ordinate corresponding to the calibration reference point in the t-1 frame, and γ represents a parameter.

10. The external parameter calibration method of the image acquisition device according to claim 1, characterized in that: The initial external parameter includes an initial pitch angle, the internal parameter includes an optical position coordinate, a focal length parameter, and the calibrated external parameter is calculated based on the updated position of the calibration reference point, the internal parameter and the initial external parameter, including: Based on the updated position of the calibration reference point, the memory parameters and the initial external parameter parameters, and using the pitch angle calculation formula, the calibrated external parameter parameters are calculated, wherein the external parameter calculation formula is as follows: pitch angle =σ1*atan((cy-y h (t)),fy)+(1-σ1)*pitch angle_ori , Pitch angle represents the pitch angle after calibration, σ1 is the coefficient, cy represents the ordinate of the optical center position in the image coordinate system, y h (t) The updated ordinate corresponding to the calibration reference point in the current frame t, fy represents the pixel length of the focal length in the y-axis direction in the image coordinate system, pitch angle_ori Indicates the initial pitch angle.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the controller, the external parameter calibration method of the image acquisition device as claimed in any one of claims 1 to 10 is implemented.

12. A vehicle device, characterized in that: It comprises a memory and a controller, wherein the memory stores a computer program, and when the computer program is executed by the controller, the controller executes the external parameter calibration method of the image acquisition device according to any one of claims 1 to 10.

Images