Vehicle-mounted camera dynamic external parameter estimation method, system, equipment and medium

Abstract

The invention provides a vehicle-mounted camera dynamic external parameter estimation method, system and device and a medium, and the method comprises the steps: extracting angular points of a previous-frame downsampling image to obtain an initial angular point set, and filtering the initial angular point set according to a previous-frame target detection result to obtain a previous-frame angular point set; performing angular point tracking on the angular point set of the previous frame by adopting an angular point tracking algorithm through various of the vanishing point of the previous frame, the vanishing point of the lane line of the previous frame, the vanishing point of the lane line of the current frame, the target detection result of the previous frame and the target detection result of the current frame to obtain an angular point set of the current frame; and acquiring a four-degree-of-freedom affine transformation matrix of the current frame, a vanishing point of the current frame and a yaw angle rate through the angular point set of the previous frame and the angular point set of the current frame. According to the invention, external parameter estimation is carried out by combining the vanishing point of the lane line and the target detection result, so that the stability of pixel tracking is enhanced, the efficiency of a tracking algorithm is improved, and real-time, continuous and accurate external parameters of the camera are realized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent assisted driving and relates to a method, system, device and medium for estimating dynamic extrinsic parameters of a vehicle-mounted camera. Background Art

[0002] In the field of intelligent assisted driving, the external parameters of the vehicle camera are the basic and core functions for achieving accurate distance and speed measurement. However, due to the influence of various factors such as road bumps, vehicle steering, acceleration and deceleration, the external parameters of the vehicle camera will change in real time. The constantly changing external parameters pose a challenge to the stability of distance measurement, speed measurement and target tracking.

[0003] At present, the existing vehicle-mounted camera extrinsic parameters include those estimated through lane lines or visual technology. For example, Chinese patent CN115731302A discloses a distance measurement method, system, device and medium for estimating camera extrinsic parameters based on vanishing points. This method determines the vanishing point through data annotation and automatic annotation of the Heatmap network model, thereby realizing dynamic estimation of camera extrinsic parameters. The core of this method is to determine the vanishing point through the intersection of lane lines, but in scenes where lane lines are blocked, blurred or non-existent, this method has the risk of failure. For another example, Chinese invention patent CN113554711A discloses a camera online calibration method, device, computer equipment and storage medium. This method is based on visual odometer and estimates the camera pose of the current frame by bundling constraints on local and global camera poses. This method mainly relies on the matching of image features and the solution and optimization of multi-view geometry. Although it has high accuracy, it consumes a lot of computing resources and is difficult to implement on low-computing power platforms. Summary of the invention

[0004] In order to solve the technical problems that the method of estimating camera extrinsic parameters through the vanishing point of the lane line has the risk of failure in the scene where the lane line is blocked, unclear or non-existent, and the method of estimating camera extrinsic parameters through the odometer consumes too much computing resources and is difficult to deploy and apply on a low computing power platform, the present invention discloses a method for estimating dynamic extrinsic parameters of a vehicle-mounted camera, the method comprising the following steps:

[0005] S101, obtaining a previous frame downsampled image, a previous frame target detection result, and a previous frame lane line vanishing point of a previous frame image, and obtaining a current frame downsampled image, a current frame target detection result, and a current frame lane line vanishing point of a current frame image;

[0006] S102, extracting corner points of the previous frame downsampled image to obtain an initial corner point set, and filtering the initial corner point set according to the previous frame target detection result to obtain a previous frame corner point set;

[0007] S103, using a corner point tracking algorithm to track the corner point set of the previous frame to obtain a corner point set of the current frame by using multiple of the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame, and the target detection result of the current frame;

[0008] S104, performing affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain a 4-DOF affine transformation matrix of the current frame, and obtaining a current frame vanishing point and a yaw angle rate from the previous frame to the current frame according to the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame.

[0009] An embodiment of the present invention further provides a vehicle-mounted camera dynamic extrinsic parameter estimation system, comprising a downsampling module, a target detection module, a lane line acquisition module, a corner point acquisition module, a corner point tracking module and an extrinsic parameter acquisition module.

[0010] Wherein, the downsampling module is used to obtain a previous frame downsampled image of a previous frame image and a current frame downsampled image of a current frame image;

[0011] The target detection module is used to obtain the previous frame target detection result of the previous frame image and the current frame target detection result of the current frame image;

[0012] The lane line acquisition module is used to acquire the vanishing point of the lane line of the current frame of the current frame image;

[0013] The corner point acquisition module is used to extract the corner points of the previous frame downsampled image to obtain an initial corner point set, and filter the initial corner point set according to the previous frame target detection result to obtain the previous frame corner point set;

[0014] The corner point tracking module is used to track the corner point set of the previous frame using a corner point tracking algorithm to obtain the corner point set of the current frame through multiple of the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame and the target detection result of the current frame;

[0015] The external parameter acquisition module is used to perform affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain the 4-DOF affine transformation matrix of the current frame, and obtain the vanishing point of the current frame and the yaw angle rate from the previous frame to the current frame according to the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame.

[0016] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods for estimating dynamic extrinsic parameters of a vehicle-mounted camera when executing the computer program, so as to solve the technical problems that the method for estimating camera extrinsic parameters through the vanishing point of the lane line may fail in a scene where the lane line is blocked, unclear or non-existent, and that estimating camera extrinsic parameters through the odometer consumes too much computing resources and is difficult to deploy and apply on a low computing power platform.

[0017] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned methods for estimating dynamic extrinsic parameters of a vehicle-mounted camera, so as to solve the technical problems that the method for estimating camera extrinsic parameters through the vanishing point of the lane line may fail in scenarios where the lane line is blocked, unclear or non-existent, and that estimating camera extrinsic parameters through the odometer consumes too much computing resources and is difficult to deploy and apply on a low-computing power platform.

[0018] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the camera extrinsic parameter dynamic estimation method of the present invention, first, obtains the downsampled image, target detection result and lane line vanishing point of the image; secondly, extracts the corner points of the downsampled image of the previous frame to obtain the initial corner point set, and filters the initial corner point set according to the target detection result of the previous frame to obtain the corner point set of the previous frame; then, through the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame and the target detection result of the current frame, the corner point set of the previous frame is tracked to obtain the corner point set of the current frame; finally, the corner point set of the previous frame and the corner point set of the current frame are affine transformed to obtain the 4-degree-of-freedom affine transformation matrix of the current frame, and the confidence of the lane line vanishing point of the current frame and the 4-degree-of-freedom affine transformation matrix of the current frame are combined to obtain the vanishing point of the current frame and the yaw angular rate from the previous frame to the current frame.

[0019] The dynamic extrinsic parameter estimation method for a vehicle-mounted camera of the present invention takes image pixel tracking technology as its core and combines lane line detection information with target frame detection information. On the one hand, it can achieve real-time, continuous and accurate estimation of camera extrinsic parameters with lower computing resource consumption; on the other hand, it enhances the stability of pixel tracking and improves the efficiency of the tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of the method for estimating dynamic extrinsic parameters of a vehicle-mounted camera of the present invention;

[0022] Figure 2 It is a principle block diagram of a method for estimating dynamic extrinsic parameters of a vehicle-mounted camera disclosed in an embodiment of the present invention;

[0023] Figure 3 It is an architecture diagram of a dynamic extrinsic parameter estimation system for a vehicle-mounted camera disclosed in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0025] Among them, 301, downsampling module; 302, target detection module; 303, lane line acquisition module; 304, corner point acquisition module; 305, corner point tracking module; 306, external parameter acquisition module; 401, memory; 402, processor. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0027] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0028] The embodiment of the present invention discloses a method for estimating dynamic external parameters of a vehicle-mounted camera, see Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0029] S101, obtaining a previous frame downsampled image, a previous frame target detection result, and a previous frame lane line vanishing point of a previous frame image, and obtaining a current frame downsampled image, a current frame target detection result, and a current frame lane line vanishing point of a current frame image;

[0030] S102, extracting corner points of the previous frame downsampled image to obtain an initial corner point set, and filtering the initial corner point set according to the previous frame target detection result to obtain a previous frame corner point set;

[0031] S103, using a corner point tracking algorithm to track the corner point set of the previous frame to obtain a corner point set of the current frame by using multiple of the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame, and the target detection result of the current frame;

[0032] S104, performing affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain a 4-DOF affine transformation matrix of the current frame, and obtaining a current frame vanishing point and a yaw angle rate from the previous frame to the current frame according to the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame.

[0033] When the above step S101 is implemented, the previous frame image and the current frame image can be collected by a vehicle-mounted camera on the detection vehicle, and the previous frame image and the current frame image are respectively subjected to downsampling processing, lane line detection and target detection to obtain a downsampled image, a lane line detection result and a target detection result. Specifically, the above step S101 includes the following steps:

[0034] S1011 . Downsampling the previous frame image and the current frame image to obtain a previous frame downsampled image and a current frame downsampled image.

[0035] Although the present invention will lose a certain degree of accuracy when downsampling the original image of the collected image, it can significantly reduce the computing power requirements of subsequent modules. Since different models of airborne cameras may have different resolutions and the performance of the computing platform is also different, the downsampling ratio and the resolution of the final downsampled image can be adjusted according to demand. For example, when the original image resolution is 1280×720, a 5-fold downsampling ratio can be selected for downsampling processing to obtain a downsampled image resolution of 256×144.

[0036] S1012, obtaining a lane line vanishing point of a previous frame image and a lane line vanishing point of a current frame image.

[0037] The lane line vanishing point of the current frame and the lane line vanishing point of the previous frame can both be detected using traditional lane line detection methods. For example, the previous frame image and the current frame image are first converted into grayscale images and denoised and smoothed using a Gaussian filter. Then, algorithms such as Canny are used to perform edge detection and lane line extraction to obtain at least two lane line equations. Finally, the coordinates of the vanishing point and its confidence level are determined using the obtained lane line equations.

[0038] S1013, obtaining the previous frame target detection result and the current frame target detection result of the previous frame image and the current frame image.

[0039] In the present invention, the image target detection result can be achieved by a target detection algorithm, a semantic segmentation algorithm, an instance segmentation algorithm, etc. For example, a target detection algorithm such as YOLO, Faster R-CNN, etc. can be used to identify the target in the image to obtain a detection frame of each target. The image can also be pixel-classified by a semantic segmentation network such as U-Net, DeepLab, etc., and the region of interest can be extracted from the classification result as the target to obtain each target detection result. The target detection result can also be obtained by obtaining the precise outline of each target through an instance segmentation model such as Mask R-CNN.

[0040] When the above step S102 is specifically implemented, first, the corner points are extracted from the previous frame downsampled image by a corner point extraction algorithm to obtain an initial corner point set, and then the initial corner point set is filtered to obtain the final previous frame corner point set. In the present invention, the Harris corner point extraction algorithm, the FAST corner point extraction algorithm, the uniform point extraction algorithm, or the corner point extraction algorithm obtained by deep learning training, as well as the corner point extractor developed in the future, can be used to extract corner points. In order to meet the needs of low computing power platforms, the present invention preferably uses the uniform point extraction method to extract corner points. On the downsampled image with a resolution of 256×144, the points are uniformly scattered with a step size of N=16. The initial corner point set obtained has a three-dimensional width and height of 16 and 9 respectively, and a total of 144 corner points are obtained.

[0041] Dynamic targets may often appear in driving scenes. If these dynamic targets block the camera's field of view, the dynamic points (corner points) extracted in the blocked area will affect the subsequent corner point tracking results and affine transformation results, thereby reducing the accuracy of the external parameters. Therefore, in the present invention, the initial corner point set is filtered in combination with the target detection result of the previous frame to remove the corner points in the initial corner point set that fall into the target frame.

[0042] In an improved embodiment, the extracted corner points have their own gradients, but the corner points with unclear gradients may fail in corner point tracking, which in turn affects the estimation of subsequent external parameters. Therefore, the corner points are further filtered by their gradients in the present invention. The present invention performs filtering on the corner points based on the gradients, and the filtering can be placed before or after filtering the initial corner point set by the previous frame target detection result, which can be achieved in the following two ways:

[0043] S1021. Calculate the gradient of each corner point in the initial corner point set, filter the initial corner point set according to a gradient threshold, and extract the corner points in the initial corner point set whose gradients are greater than the gradient threshold to obtain a final initial corner point set.

[0044] Alternatively, S1022, calculate the gradient of each corner point in the previous frame corner point set, filter the previous frame corner point set according to a gradient threshold, extract the corner points in the previous frame corner point set whose gradient is greater than the gradient threshold to obtain a final previous frame corner point set.

[0045] When the above steps S1021 and S1022 are implemented, the gradient calculation operators commonly used in the visual field, such as Roberts, Sobel, Scharr, etc., can be used to calculate the gradient of each corner point in eight directions, namely horizontal, vertical, and diagonal. When the gradients are all less than a certain gradient threshold g (such as g=7), it means that the gradient feature of the corner point is not obvious and needs to be filtered out.

[0046] When the above step S103 is specifically implemented, the corner points in the current frame downsampled image corresponding to the previous frame downsampled image can be realized by existing descriptor feature matching or deep learning. In order to meet the requirements of the low computing power platform in the present invention, the general LK (Lucas-Kanade) optical flow algorithm is preferably selected to perform corner point tracking. The LK optical flow tracking has the characteristics of fast speed. It is more suitable for the case where the camera motion fluctuation between adjacent frames in the vehicle-mounted scene is small. For the camera motion fluctuation between adjacent frames, that is, in the bumpy scene, due to the drastic change of the camera posture, the simple LK optical flow tracking algorithm may be difficult to cope with. Therefore, the traditional LK optical flow tracking is improved in the present invention by introducing the lane line vanishing point and target detection results of the previous and next frames to estimate the motion amount of the corner point, thereby providing an initial value close to the actual value for the corner point estimation of the current frame downsampled image, and then constructing a LK pyramid optical flow tracking algorithm.

[0047] Specifically, the vanishing point of the previous frame is defined as represents the horizontal and vertical coordinates of the vanishing point P of the previous frame, which can be obtained through the external parameters output by the previous frame; the vanishing point of the lane line in the current frame is It can be obtained by performing lane line detection on the current frame image. Indicates the horizontal and vertical coordinates of the lane line vanishing point c in the current frame, c c represents the confidence of the lane line vanishing point c in the current frame; the target detection result of the previous frame is represented by the set O as N represents the total number of targets in the previous frame image; the target detection result of the current frame is represented by the set O: M represents the total number of targets in the current frame image; the corner point set of the previous frame is I represents the total number of corners after filtering in the previous frame, a set. The above set O is a set of target detection frames on a single frame image. Each detection frame o is a structure represented by the normalized coordinates of the upper left corner and the lower right corner of the rectangular frame. Among them, the process of obtaining the corner point set of the current frame by corner point tracking is as follows:

[0048] S1031, selecting tracking parameters from a previous frame vanishing point, a previous frame lane line vanishing point, a current frame lane line vanishing point, a previous frame target detection result, and a current frame target detection result according to a confidence threshold and the confidence of the lane line vanishing point of the current frame;

[0049] S1032, calculating the horizontal coordinate offset and the vertical coordinate offset of the vanishing points of the previous and next frames according to the tracking parameters;

[0050] S1033. Calculate the initial corner point set of the current frame according to the horizontal coordinate offset and the vertical coordinate offset, and use the corner point tracking algorithm to obtain the tracking position of each corner point in the previous frame corner point set in the downsampled image of the current frame through the initial corner points of the current frame to obtain the corner point set of the current frame.

[0051] In the specific implementation of step S1031, the tracking parameter is selected from the vanishing point of the previous frame, the vanishing point of the lane line of the previous frame, the vanishing point of the lane line of the current frame, the target detection result of the previous frame, and the target detection result of the current frame according to the confidence threshold and the confidence of the lane line vanishing point of the current frame, including:

[0052] S10311. When the confidence level is ≥ the confidence level threshold, the lane line vanishing point of the previous frame and the lane line vanishing point of the current frame are selected as the tracking parameters. When the confidence level of the vanishing point of the current frame obtained by lane line detection is ≥ the confidence level threshold, it indicates that the camera floating in the previous and next frames is relatively small, and the offset of the vanishing points of the previous and next frames calculated by the traditional LK optical flow tracking algorithm is also relatively accurate.

[0053] S10312, when the confidence is less than the confidence threshold, select the target detection result of the previous frame, the target detection result of the current frame and the vanishing point of the previous frame as the tracking parameters. When the confidence of the lane line vanishing point of the current frame obtained by lane line detection is less than the confidence threshold, it means that the camera floating in the front and back frames is relatively large, and it may be in a bumpy scene, the lane line is blocked, there is no lane line, the lane line is curved, etc. At this time, it is necessary to use the improved traditional LK pyramid optical flow tracking algorithm of the present invention to calculate the offset of the vanishing points of the front and back frames.

[0054] In the specific implementation of the above step S1032, the calculation of the horizontal coordinate offset and the vertical coordinate offset of the vanishing points of the previous and next frames according to the tracking parameters includes:

[0055] S10321. When the confidence level is greater than or equal to the confidence level threshold, calculate the horizontal coordinate difference and the vertical coordinate difference between the lane line vanishing point of the previous frame and the lane line vanishing point of the current frame to obtain the horizontal coordinate offset and the vertical coordinate offset, wherein the offset Δv can be expressed as:

[0056] S10322: When the confidence level is less than the confidence level threshold, calculate the horizontal coordinate difference and the vertical coordinate difference between the vanishing point of the previous frame and the vanishing point of the current frame through the target detection result of the previous frame, the target detection result of the current frame and the vanishing point of the previous frame to obtain the horizontal coordinate offset and the vertical coordinate offset.

[0057] In specific implementation, the upper and lower ordinates of the detection box of a target in the previous frame can be expressed as The vertical coordinates of the upper and lower edges of the detection box of the target in the current frame are expressed as The vertical coordinate of the vanishing point in the previous frame is Then the vertical coordinate of the vanishing point of the current frame is Satisfy the objective function By solving the objective function, we can calculate Since it is impossible to accurately obtain the tracking information of the detection frame of the current frame before dynamic external parameter estimation, it can be considered that for all the detection frames of the current frame At this time, the offset Δv can be expressed as

[0058] After calculating the offset of the vanishing points of the previous and next frames, the initial corner point set of the current frame can be obtained as The previous frame corner point set P p The initial corner point set of the current frame is used as the input of the corner point tracking algorithm (such as the LK optical flow tracking algorithm), and the final corner point set P of the current frame can be output. c .

[0059] After obtaining the corner point set of the current frame, we can use the corner point set P of the previous frame to obtain the corner point set P of the current frame. p and the current frame corner point set P c Estimation of external parameters is performed, and the external parameters include the 4-DOF affine transformation matrix of the current frame, the vanishing point of the current frame, and the yaw angular rate from the previous frame to the current frame. The above step S104 includes the following steps:

[0060] S1041, through the previous frame corner point set P p and the current frame corner point set P c Perform an affine transformation to obtain the 4-DOF affine transformation matrix of the current frame. The 4-DOF affine transformation matrix of the current frame can be expressed as: Among them, θ represents the rotation angle of the corner point set, s represents the scale factor of the corner point set, and t x ,t y They are the translation components on the x and y axes in the pixel coordinate system respectively.

[0061] S1042, obtaining the current frame vanishing point and the yaw angular rate according to the current frame vanishing point and the current frame 4-DOF affine transformation matrix, specifically includes:

[0062] S10421. When the confidence of the lane line vanishing point of the current frame is greater than or equal to the confidence threshold, use the lane line vanishing point of the current frame as the vanishing point of the current frame.

[0063] S10422: When the confidence level is less than the confidence level threshold, calculate the vanishing point of the current frame by using the 4-DOF affine transformation matrix of the current frame, the vanishing point of the previous frame and a given static vanishing point.

[0064] Specifically, when the confidence level is less than the confidence level threshold, the current frame vanishing point v c You can use the vanishing point v of the previous frame p And the current frame 4 degrees of freedom affine transformation matrix A to estimate the affine transformation projection Since there is an error in the estimation of the 4-DOF affine transformation matrix A of the current frame, in order to correct the error accumulation, the present invention combines the static vanishing point v s , and define a high-pass filter to simulate the affine transformation projection Av p and the static vanishing point v s The weight of the static vanishing point v s It is calibrated when the vehicle-mounted camera is installed and is a given value. The vanishing point v of the current frame c It can be calculated by the following formula:

[0065] v c =αAv p +(1-α)v s ;

[0066] Among them, the weight α is defined as:

[0067]

[0068] in, yes and v p The difference of the vertical axis; the weight α is a high-pass filter, which means: when |Δv y When | is small, it means that the vehicle has a smaller bump amplitude. y | has a low signal-to-noise ratio, so a smaller pass weight is set; when |Δv y When | is larger, it means that the vehicle has a larger bump amplitude. y |The signal-to-noise ratio is high, so a larger pass weight is set.

[0069] S10423, calculating the yaw rate by using the vanishing point of the current frame and the vanishing point of the previous frame. Specifically, the yaw rate is calculated by calculating the difference between the horizontal coordinates of the vanishing point of the current frame and the vanishing point of the previous frame. Combined with the pixel focal length f of the onboard camera and the timestamps t of the previous and next frames p and t c , through the formula Calculated.

[0070] The principle of the camera extrinsic parameter dynamic estimation method of the present invention is: first, a downsampled image, a target detection result and a lane line vanishing point of an image are obtained; second, the corner points of the downsampled image of the previous frame are extracted to obtain an initial corner point set, and the initial corner point set is filtered according to the target detection result of the previous frame to obtain the corner point set of the previous frame; then, the corner point set of the previous frame is tracked through the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame and the target detection result of the current frame to obtain the corner point set of the current frame; finally, the corner point set of the previous frame and the corner point set of the current frame are affine transformed to obtain a 4-degree-of-freedom affine transformation matrix of the current frame, and the vanishing point and yaw angular rate of the current frame are obtained in combination with the confidence of the lane line vanishing point of the current frame and the 4-degree-of-freedom affine transformation matrix of the current frame.

[0071] The dynamic extrinsic parameter estimation method for a vehicle-mounted camera of the present invention takes image pixel tracking technology as its core and combines lane line detection information with target frame detection information. On the one hand, it can achieve real-time, continuous and accurate estimation of camera extrinsic parameters with lower computing resource consumption; on the other hand, it enhances the stability of pixel tracking and improves the efficiency of the tracking algorithm.

[0072] Based on the same inventive concept, an embodiment of the present invention also provides a vehicle-mounted camera dynamic extrinsic parameter estimation system, as described in the following embodiments. Since the principle of solving the problem by the vehicle-mounted camera dynamic extrinsic parameter estimation system is similar to the vehicle-mounted camera dynamic extrinsic parameter estimation method disclosed in the above embodiments, the implementation of the vehicle-mounted camera dynamic extrinsic parameter estimation system can refer to the implementation of the above method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0073] Figure 3 is a structural block diagram of a vehicle-mounted camera dynamic external parameter estimation system disclosed in an embodiment of the present invention, such as Figure 3 As shown, the system includes a downsampling module 301, a target detection module 302, a lane line acquisition module 303, a corner point acquisition module 304, a corner point tracking module 305 and an external parameter acquisition module 306. The structure is described below.

[0074] The downsampling module 301 is used to obtain a previous frame downsampled image of a previous frame image and a current frame downsampled image of a current frame image;

[0075] The target detection module 302 is used to obtain the previous frame target detection result of the previous frame image and the current frame target detection result of the current frame image;

[0076] The lane line acquisition module 303 is used to acquire the current frame lane line vanishing point of the current frame image;

[0077] The corner point acquisition module 304 is used to extract the corner points of the previous frame downsampled image to obtain an initial corner point set, and filter the initial corner point set according to the previous frame target detection result to obtain the previous frame corner point set;

[0078] The corner point tracking module 305 is used to track the corner point set of the previous frame using a corner point tracking algorithm to obtain the corner point set of the current frame through multiple of the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame and the target detection result of the current frame;

[0079] The external parameter acquisition module 306 is used to perform affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain the 4-DOF affine transformation matrix of the current frame, and obtain the vanishing point of the current frame and the yaw angle rate from the previous frame to the current frame based on the lane line vanishing point of the current frame and the 4-DOF affine transformation matrix of the current frame.

[0080] In an optional embodiment, the external parameter acquisition module 306 includes an affine transformation matrix calculation module, a vanishing point acquisition module and a yaw angle rate calculation module.

[0081] The affine transformation matrix calculation module is used to perform affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain a 4-DOF affine transformation matrix of the current frame.

[0082] The vanishing point acquisition module is used to use the lane line vanishing point of the current frame as the vanishing point of the current frame when the confidence of the lane line vanishing point of the current frame is greater than or equal to the confidence threshold; when the confidence is less than the confidence threshold, calculate the vanishing point of the current frame through the 4-DOF affine transformation matrix of the current frame, the vanishing point of the previous frame and the given static vanishing point.

[0083] The yaw angle rate calculation module is used to calculate the yaw angle rate from the previous frame to the current frame through the current frame vanishing point and the previous frame vanishing point.

[0084] In this embodiment, a computer device is provided, such as Figure 4 As shown, it includes a memory 401, a processor 402 and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any of the above-mentioned methods for estimating dynamic external parameters of a vehicle-mounted camera is implemented.

[0085] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0086] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned methods for estimating dynamic extrinsic parameters of a vehicle-mounted camera.

[0087] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0088] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for estimating dynamic external parameters of a vehicle-mounted camera, characterized in that: include: Obtain the previous frame downsampled image, the previous frame target detection result and the previous frame lane line vanishing point of the previous frame image, and obtain the current frame downsampled image, the current frame target detection result and the current frame lane line vanishing point of the current frame image; Extracting corner points of the previous frame downsampled image to obtain an initial corner point set, and filtering the initial corner point set according to the previous frame target detection result to obtain a previous frame corner point set; Using a plurality of the vanishing points of the previous frame, the lane line vanishing points of the previous frame, the lane line vanishing points of the current frame, the target detection results of the previous frame, and the target detection results of the current frame, a corner point tracking algorithm is used to track the corner point set of the previous frame to obtain a corner point set of the current frame; Affine transformation is performed on the corner point set of the previous frame and the corner point set of the current frame to obtain a 4-DOF affine transformation matrix of the current frame, and the vanishing point of the current frame and the yaw angle rate from the previous frame to the current frame are obtained according to the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame.

2. The method for estimating dynamic external parameters of a vehicle-mounted camera according to claim 1, characterized in that: Also includes: The gradient of each corner point in the initial corner point set is calculated, the initial corner point set is filtered according to a gradient threshold, and the corner points in the initial corner point set whose gradient is greater than the gradient threshold are extracted to obtain a final initial corner point set.

3. The method for estimating dynamic external parameters of a vehicle-mounted camera according to claim 1, characterized in that: Also includes: The gradient of each corner point in the previous frame corner point set is calculated, the previous frame corner point set is filtered according to a gradient threshold, and the corner points in the previous frame corner point set whose gradient is greater than the gradient threshold are extracted to obtain a final previous frame corner point set.

4. The method for estimating dynamic external parameters of a vehicle-mounted camera according to any one of claims 1 to 3, characterized in that: The method of obtaining a current frame corner point set by performing corner point tracking on the previous frame corner point set using a corner point tracking algorithm through multiple of a previous frame vanishing point, a previous frame lane line vanishing point, a current frame lane line vanishing point, a previous frame target detection result, and a current frame target detection result includes: Selecting tracking parameters from a previous frame vanishing point, a previous frame lane line vanishing point, a current frame lane line vanishing point, a previous frame target detection result, and a current frame target detection result according to a confidence threshold and a confidence of the lane line vanishing point of the current frame; Calculating the horizontal coordinate offset and the vertical coordinate offset of the vanishing points of the previous and next frames according to the tracking parameters; The initial corner point set of the current frame is calculated according to the horizontal coordinate offset and the vertical coordinate offset, and the tracking position of each corner point in the previous frame corner point set in the downsampled image of the current frame is obtained by using the corner point tracking algorithm through the initial corner points of the current frame to obtain the corner point set of the current frame.

5. The method for estimating dynamic external parameters of a vehicle-mounted camera according to claim 4, characterized in that: The selecting of tracking parameters from the vanishing point of the previous frame, the vanishing point of the lane line of the previous frame, the vanishing point of the lane line of the current frame, the target detection result of the previous frame, and the target detection result of the current frame according to the confidence threshold and the confidence of the lane line vanishing point of the current frame includes: When the confidence level is greater than or equal to the confidence level threshold, selecting the lane line vanishing point of the previous frame and the lane line vanishing point of the current frame as the tracking parameters; When the confidence level is less than the confidence level threshold, the target detection result of the previous frame, the target detection result of the current frame and the vanishing point of the previous frame are selected as the tracking parameters.

6. The method for estimating dynamic external parameters of a vehicle-mounted camera according to claim 5, characterized in that: The step of calculating the horizontal coordinate offset and the vertical coordinate offset of the vanishing points of the previous and next frames according to the tracking parameters includes: When the confidence level is greater than or equal to the confidence level threshold, a horizontal coordinate difference and a vertical coordinate difference between the lane line vanishing point of the previous frame and the lane line vanishing point of the current frame are calculated to obtain the horizontal coordinate offset and the vertical coordinate offset; When the confidence level is less than the confidence level threshold, the horizontal coordinate difference and the vertical coordinate difference between the vanishing point of the previous frame and the vanishing point of the current frame are calculated by using the target detection result of the previous frame, the target detection result of the current frame and the vanishing point of the previous frame to obtain the horizontal coordinate offset and the vertical coordinate offset.

7. The method for estimating dynamic external parameters of a vehicle-mounted camera according to claim 1, characterized in that: The obtaining the vanishing point of the current frame and the yaw angle rate from the previous frame to the current frame according to the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame includes: When the confidence of the lane line vanishing point of the current frame is greater than or equal to the confidence threshold, the lane line vanishing point of the current frame is used as the vanishing point of the current frame; When the confidence level is less than the confidence level threshold, calculating the vanishing point of the current frame by using the 4-DOF affine transformation matrix of the current frame, the vanishing point of the previous frame and the given static vanishing point; The yaw angle rate from the previous frame to the current frame is calculated by using the current frame vanishing point and the previous frame vanishing point.

8. A vehicle-mounted camera dynamic external parameter estimation system, characterized in that: include: A downsampling module, the downsampling module is used to obtain a previous frame downsampled image of a previous frame image and a current frame downsampled image of a current frame image; A target detection module, the target detection module is used to obtain a previous frame target detection result of the previous frame image and a current frame target detection result of the current frame image; A lane line acquisition module, the lane line acquisition module is used to obtain a current frame lane line vanishing point of the current frame image; A corner point acquisition module, the corner point acquisition module is used to extract the corner points of the previous frame downsampled image to obtain an initial corner point set, and filter the initial corner point set according to the previous frame target detection result to obtain the previous frame corner point set; A corner point tracking module, the corner point tracking module is used to track the corner point set of the previous frame using a corner point tracking algorithm to obtain a corner point set of the current frame through multiple of the vanishing point of the previous frame, the lane line vanishing point of the previous frame, the lane line vanishing point of the current frame, the target detection result of the previous frame, and the target detection result of the current frame; An external parameter acquisition module is used to perform affine transformation on the corner point set of the previous frame and the corner point set of the current frame to obtain a 4-DOF affine transformation matrix of the current frame, and obtain a current frame vanishing point and a yaw angle rate from the previous frame to the current frame based on the lane line vanishing point of the current frame, the vanishing point of the previous frame and the 4-DOF affine transformation matrix of the current frame.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for estimating dynamic extrinsic parameters of a vehicle-mounted camera according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method for estimating dynamic extrinsic parameters of a vehicle-mounted camera according to any one of claims 1 to 7.

Images