Track-based automatic external parameter calibration method and related equipment

Abstract

The invention provides a track-based automatic external parameter calibration method and related equipment. The method comprises the following steps: acquiring a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor; respectively carrying out feature extraction on the three-dimensional point cloud image and the two-dimensional image, and determining a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image; performing similarity matching by using a first distance between the first feature and the first sensor, a frame number of the first feature in the three-dimensional point cloud image, a second distance between the second feature and the second sensor, and a frame number of the second feature in the two-dimensional image to obtain a matching result; and projecting the three-dimensional point cloud image to the two-dimensional image, optimizing the projection error by using the matching result and the internal reference matrix of the second sensor until the projection error is minimum, and obtaining an external reference matrix between the first sensor and the second sensor. And external parameter calibration of the first sensor and the second sensor is realized without a reference object and depending on a large number of computing resources.

Description

Technical Field

[0001] The present application relates to the technical field of sensor parameter calibration, and in particular to a trajectory-based automatic heterodyne calibration method and related equipment. Background Art

[0002] In intelligent transportation systems, roadside perception technology is one of the key technologies to improve traffic safety and efficiency. By fusing data from multiple sensors, roadside perception can achieve wider range and higher precision environmental perception, providing strong support for applications such as traffic control, infrastructure-assisted autonomous driving, and vehicle-road collaborative perception. Among various sensor fusion solutions, the combination of 4D imaging radar and camera has attracted widespread attention due to its cost-effectiveness and performance advantages. However, to achieve effective fusion of radar and camera, the problem of external parameter calibration must be solved first, that is, accurately determining the spatial position and posture relationship between the two sensors.

[0003] After the external parameter calibration is completed, the spatial relationship between the radar and camera sensors may drift due to factors such as weather, which may also cause corresponding errors in the external parameter calibration. Manual inspection and recalibration are required, resulting in a waste of labor resources. Summary of the invention

[0004] In view of this, the purpose of the present application is to propose a trajectory-based automatic heterodyne calibration method and related equipment that overcome the above problems or at least partially solve the above problems.

[0005] Based on the above objectives, the first aspect of the present application provides 1. a trajectory-based automatic extrinsic parameter calibration method, characterized in that it includes:

[0006] Acquire a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor;

[0007] Performing feature extraction on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image;

[0008] Perform similarity matching using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and a second distance between the second feature and the second sensor and the number of frames in which the second feature is located in the two-dimensional image to obtain a matching result;

[0009] The three-dimensional point cloud image is projected onto the two-dimensional image, and the projection error is optimized using the matching result and the intrinsic parameter matrix of the second sensor until the projection error is minimized, thereby obtaining an extrinsic parameter matrix between the first sensor and the second sensor.

[0010] In some possible implementations, performing feature extraction on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image includes:

[0011] Extracting lane line segments in the two-dimensional image to determine lane line trajectories of the two-dimensional image;

[0012] Determining a motion trajectory of a moving object between lane line trajectories of the two-dimensional image as a second feature of the two-dimensional image;

[0013] A straight line fitting is performed on the three-dimensional point cloud in the three-dimensional point cloud image, and a motion trajectory of the moving object in the three-dimensional point cloud image is determined as a first feature of the three-dimensional point cloud image.

[0014] In some possible implementations, determining a second distance between the second feature and the second sensor includes:

[0015] Determine the intersection point of the extended lines of the lane line trajectory of the two-dimensional image;

[0016] determining a center point of the second feature;

[0017] Determining a near point of a lane line trajectory of a lane in which the second feature is located;

[0018] Determining an endpoint of a lane segment closest to the second feature;

[0019] A second distance of the moving object in the second feature from the second sensor is determined using the intersection point, the center point, the near point and the end point.

[0020] In some possible implementations, determining the distance of the second feature from the second sensor by using the intersection point, the center point, the near point, and the endpoint includes:

[0021] Predetermining the actual length of the lane segment;

[0022] Determining the pixel length of the lane segment in the two-dimensional image;

[0023] Determine a mapping relationship between the two-dimensional image and the actual according to the actual length and the pixel length;

[0024] Determine a first pixel distance of a lane segment in the two-dimensional image by using the mapping relationship;

[0025] Determine a second pixel distance between the endpoint and the near point in the two-dimensional image using the endpoint and the near point;

[0026] Using the center point and the intersection point, determining a third pixel distance between the center point and the intersection point along the lane line trajectory direction in the two-dimensional image;

[0027] Determine a fourth pixel distance between the near point and the intersection point in the two-dimensional image using the near point and the intersection point;

[0028] Determine the distance of the second sensor of the moving object in the second characteristic distance according to the actual length of the lane segment, the first pixel distance, the second pixel distance, the third pixel distance, the fourth pixel distance and the mapping relationship;

[0029] The distance of the second feature from the second sensor is expressed as:

[0030] In some possible implementations, the similarity is expressed as:

[0031]

[0032] Among them, d r represents the distance between the first feature and the first sensor, d i represents the distance between the second feature and the second sensor, f r Indicates the frame number of the first feature in the 3D point cloud image, f i represents the number of frames representing the second feature in the two-dimensional image, and λ represents the scaling factor.

[0033] In some possible implementations, the extrinsic parameter matrix between the first sensor and the second sensor is expressed as:

[0034]

[0035] in, represents the optimization result, K represents the internal parameter matrix of the second sensor, T represents the external parameter matrix between the first sensor and the second sensor, and u i , x i Respectively represent the first feature and the second feature associated in the matching result.

[0036] In some possible implementations, the three-dimensional point cloud image is projected onto the two-dimensional image, and the matching result and the intrinsic parameter matrix of the first sensor are optimized until the projection error is minimized to obtain an extrinsic parameter matrix between the first sensor and the second sensor, and then the following further includes:

[0037] Determining a maximum projection error, and monitoring the maximum projection error in real time;

[0038] In response to the maximum projection error being greater than a preset error threshold, determining that drift occurs between the first sensor and the second sensor, and re-determining an extrinsic parameter matrix between the first sensor and the second sensor;

[0039] The maximum projection error is expressed as:

[0040]

[0041] Where K represents the internal parameter matrix of the second sensor, represents the optimization result, u j , x j Respectively represent the first and second features associated in the matching results after re-matching, z d Indicates a first distance of a new first feature from the first sensor.

[0042] In a second aspect of the present application, a trajectory-based automatic extrinsic parameter calibration device is provided, comprising:

[0043] An acquisition module, used to acquire a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor at the same time;

[0044] A feature extraction module, used to extract features from the three-dimensional point cloud image and the two-dimensional image respectively, and determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image;

[0045] a feature matching module, configured to perform similarity matching using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and a second distance between the second feature and the second sensor and a second distance between the second feature and the two-dimensional image, to obtain a matching result;

[0046] An external parameter matrix module is used to project the three-dimensional point cloud image to the two-dimensional image, optimize the projection error using the matching result and the internal parameter matrix of the first sensor until the projection error is minimized, and obtain the external parameter matrix between the first sensor and the second sensor.

[0047] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0048] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0049] From the above, it can be seen that the present application provides a trajectory-based automatic heterodyne calibration method and related equipment, which uses the motion trajectory of the three-dimensional point cloud image for straight line fitting as the second feature, and uses the trajectory feature for calibration, which is more stable and has higher calibration accuracy than the point cloud feature in the three-dimensional point cloud image. At the same time, the distance of the second feature from the second sensor is estimated by the vanishing point projection invariant, which does not rely on complex depth sensors, significantly reducing the computational complexity, and using the first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and the second distance between the second feature and the second sensor and the number of frames in which the second feature is located in the two-dimensional image for similarity matching, and frame-distance domain association is performed to achieve matching of the first feature and the second feature, and at the same time, the external parameter matrix of the first sensor and the second sensor is obtained by optimization to achieve calibration of the position and posture of the first sensor and the second sensor.

[0050] In addition, by utilizing maximum projection error monitoring to determine a recalibration mechanism, after spatial drift of the first sensor and / or the second sensor causes a change in position or posture of the first sensor and / or the second sensor, recalibration is performed to reduce monitoring costs and recalibration costs.

[0051] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 This is a flow chart of the trajectory-based automatic heterodyne calibration method according to an embodiment of the present application;

[0054] Figure 2 A schematic diagram of determining the minimum distance for lane change behavior of a moving object in a three-dimensional point cloud image according to an embodiment of the present application;

[0055] Figure 3 A schematic diagram of determining a second distance in a two-dimensional image according to an embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of frame-range domain association according to an embodiment of the present application;

[0057] Figure 5Schematic diagram of a trajectory-based automatic heterodyne calibration device according to an embodiment of the present application;

[0058] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0060] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0061] In the related art, the external parameter calibration method faces multiple challenges when applied on the road side. First, radars and cameras as sensors are usually installed in high places that are difficult to reach or on remote infrastructure. The target-based calibration method in the related art requires specific reference objects, so it becomes impractical. Secondly, although the use of data-driven methods can achieve automatic calibration to a certain extent, these computationally intensive algorithms are difficult to apply in practice due to the limited computing resources of road-side equipment. In addition, the radars and cameras on the road side as sensors are easily affected by climatic factors, causing the spatial relationship between sensors to drift, which may also cause response errors in the external parameter calibration, requiring manual inspection and recalibration, resulting in a waste of human resources.

[0062] Based on this, an embodiment of the present application provides a trajectory-based automatic extrinsic parameter calibration method, including: acquiring a three-dimensional point cloud image collected by a first sensor and a two-dimensional image collected by a second sensor in the same time period; performing feature extraction on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image; using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, performing similarity matching with a second distance between the second feature and the second sensor and a second distance between the second feature and the two-dimensional image to obtain a matching result; projecting the three-dimensional point cloud image to the two-dimensional image, optimizing the projection error using the matching result and the intrinsic parameter matrix of the second sensor until the projection error is minimized, and obtaining an extrinsic parameter matrix between the first sensor and the second sensor.

[0063] Figure 1 The flowchart of the method 100 for dynamically deploying network stealth according to the embodiment of the present application is shown. The purpose of the method 100 is to first extract features from the three-dimensional point cloud image and the two-dimensional image, match the first feature with the second feature by estimating the distance of the second distance of the second feature from the second sensor and associating the frame distance of the first feature in the three-dimensional point cloud image and the second feature in the two-dimensional image, and finally project the three-dimensional point cloud image to the two-dimensional image, optimize the projection error using the matching result and the intrinsic parameter matrix of the second sensor to obtain the extrinsic parameter matrix between the first sensor and the second sensor, and in the present application, use the trajectory feature for calibration, which is more stable than the point cloud feature in the three-dimensional point cloud image, and finally use the projection error to determine the spatial drift of the first sensor and / or the second sensor, and automatically recalibrate. In this process, no specific reference object is required for calibration, and no complex algorithm is required, thus avoiding dependence on computing resources.

[0064] The method 100 begins with step 101 . In step 101 , a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor are acquired.

[0065] In this embodiment, the first sensor may be a radar, and the second sensor may be a camera, such as a CCD camera.

[0066] The trajectory-based automatic extrinsic parameter calibration method provided in this embodiment can be applied to a processor of a road perception system. The application scenario can be in a roadside perception system or device.

[0067] It should be noted that, for the three-dimensional point cloud images and two-dimensional images respectively collected by the first sensor and the second sensor, in order to facilitate the subsequent feature extraction and matching for the same target, as well as the application in road perception, the first sensor and the second sensor collect the images synchronously in time series.

[0068] Then, in step 102, feature extraction is performed on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image.

[0069] In this embodiment, for the two-dimensional image, it can be an image corresponding to a certain frame at a certain moment, or it can be a video composed of a two-dimensional image sequence. Since videos are composed of still pictures, and these still pictures are called frames, in any frame, the two-dimensional image can fully reflect the second feature existing therein.

[0070] As for the three-dimensional point cloud image, which is collected by radar, it is composed of sparse three-dimensional point clouds. Unlike two-dimensional images, there are gaps between three-dimensional point clouds. Therefore, at any moment, the current three-dimensional point cloud may not be able to feedback the features corresponding to the first feature. The stability and accuracy of the three-dimensional point cloud are poor. Directly using the features in the two-dimensional image at a certain moment to match the features in the three-dimensional image point to point, the matching difficulty is high and the success rate is low. Therefore, in the embodiment of the present application, by linear fitting the three-dimensional point cloud image, the trajectory feature of the moving object is used as the second feature. In the subsequent matching process, within the time interval, the first feature of any frame in the two-dimensional point cloud image is matched and associated with the second feature in the three-dimensional point cloud image, and the external parameter matrix between the first sensor and the second sensor is obtained to determine the spatial position and posture relationship between the two sensors.

[0071] In some embodiments, performing feature extraction on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image includes:

[0072] Lane line segments in the two-dimensional image are extracted to determine the lane line trajectory of the two-dimensional image; the motion trajectory of the moving object between the lane line trajectories of the two-dimensional image is determined as the second feature of the two-dimensional image; and the three-dimensional point cloud in the three-dimensional point cloud image is fitted with a straight line to determine the motion trajectory of the moving object in the image as the first feature of the three-dimensional point cloud image.

[0073] In this embodiment, the camera captures the two-dimensional image, and the lane lines and lane areas in the road are identified by lane line detection technology. The vehicle detection and tracking algorithm (such as ByteTrack) can be used to capture the motion trajectory of the moving object. If the moving object changes lanes, key trajectory points (such as the points where the moving object just starts to cross the lane line and the points where it completely crosses the lane line) are extracted, and the direction of the lane change and the lane where the moving object is located are recorded to form the motion trajectory of the moving object in the two-dimensional image as the second feature output.

[0074] In the 3D point cloud image captured by the second sensor, the goal is to use the collected 3D point cloud to generate the motion trajectory of the moving object, and identify the lane change behavior by analyzing the deviation of the moving object from the center line of the lane. Due to the sparsity of the 3D point cloud, the 3D point cloud is fitted with a straight line through the Hough transform in the 3D space to generate the motion trajectory of the moving object as the first feature of the 3D point cloud image.

[0075] In some embodiments, performing straight line fitting on a three-dimensional point cloud in a three-dimensional point cloud image to determine a motion trajectory of a moving object in the three-dimensional point cloud image as a first feature in the three-dimensional point cloud image includes:

[0076] The three-dimensional point cloud is clustered by a clustering algorithm to determine the center line of the lane. For example, the OPTOCS clustering algorithm can be used to cluster the three-dimensional point cloud.

[0077] The distance between the moving object and the center line of the lane is used as the minimum deviation distance.

[0078] In response to the moving object deviating from the lane centerline by more than the minimum deviation distance, it is determined that the moving object has a lane change behavior, and the change trajectory is recorded.

[0079] Linear fitting is performed on the three-dimensional point cloud and the transformation trajectory, and the motion trajectory of the moving object in the three-dimensional point cloud image is determined as the first feature of the three-dimensional point cloud image.

[0080] refer to Figure 2 As shown, the minimum deviation distance is expressed as:

[0081]

[0082] Where L is the width of the lane and w is the width of the moving object.

[0083] In some exemplary embodiments, the moving object may be a vehicle, a pedestrian, or the like.

[0084] At this point, the motion trajectory of the moving object in the two-dimensional image and the motion trajectory of the moving object in the three-dimensional point cloud image have been extracted, providing a basis for subsequent feature association matching.

[0085] In step 103, a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image are used to perform similarity matching with a second distance between the second feature and the second sensor and the number of frames in which the second feature is located in the two-dimensional image to obtain a matching result.

[0086] Specifically, when the first sensor is a radar, the three-dimensional point cloud image can include three-dimensional distance information. Therefore, after the straight line is fitted into the motion trajectory of the moving object in the three-dimensional point cloud image as the first feature of the three-dimensional point cloud image, the first distance between the first sensor and the first feature can be directly obtained by the first sensor. For the second sensor as a camera, the distance between the first sensor and the second feature cannot be directly obtained, no matter which frame the second feature is in the two-dimensional image where the moving object is located. In order to realize the correlation matching of the trajectory features collected by two different sensors to ensure the accuracy of the external parameter matrix, distance matching is first required. Before distance matching, the second distance between the second feature and the second sensor is estimated.

[0087] Specifically, a distance of the second feature from the second sensor is estimated using a vanishing point projection invariant in the two-dimensional image.

[0088] In some embodiments, reference Figure 3 As shown, determining a second distance between the second feature and the second sensor includes:

[0089] Determine the intersection point of the extended lines of the lane line trajectory of the two-dimensional image, and use the intersection point as the vanishing point V of the lane line.

[0090] Determine the center point of the second feature. In this embodiment, the second feature as a whole is the motion trajectory of the moving object. The moving object exists in each frame image, so there will be multiple center points of the moving object, and the center point of the moving object cannot be effectively determined. Therefore, the center point A of the second feature is determined by determining any frame image in the two-dimensional image. j , the second feature center point is the center point of the moving object.

[0091] Determine the near point L of the lane line trajectory of the lane where the second feature is located i .

[0092] Determine the endpoint B of the lane segment closest to the second feature j .

[0093] A second distance of the second feature from the second sensor is determined using the intersection point, the center point, the near point, and the endpoint.

[0094] In this embodiment, by finding the vanishing point of the lane line, that is, the intersection of the extended lines of the lane line trajectory, the second distance of the second feature from the second sensor can be determined according to the relative position of the object in the two-dimensional image.

[0095] Specifically, determining a second distance of the second feature from the second sensor by using the intersection point, the center point, the near point, and the endpoint includes:

[0096] Predetermine the actual length of the lane segment. The actual length of the lane segment is pre-set based on the actual length of each lane line in the current road.

[0097] Determine the pixel length of the lane line segment in the two-dimensional image; determine the mapping relationship between the two-dimensional image and the actual length based on the actual length and pixel length. For example, if the actual length of the lane line segment is 1m, and the pixel length of the lane line segment in the two-dimensional image is 50PPI, then the actual length corresponding to each 1PPI pixel length is 0.02m. In this way, by determining the pixel distance of different objects in the two-dimensional image, its relative distance in reality can be determined.

[0098] Determine the first pixel distance of the lane segment in the two-dimensional image by using the mapping relationship;

[0099] Using the endpoint and the near point, determining a second pixel distance between the endpoint and the near point in the two-dimensional image;

[0100] Using the center point and the end point, determine the third pixel distance between the center point and the near point along the lane line trajectory in the two-dimensional image;

[0101] Using the near point and the intersection point, determining a fourth pixel distance between the near point and the intersection point in the two-dimensional image;

[0102] The distance of the second feature from the second sensor is determined according to the actual length of the lane segment, the first pixel distance, the second pixel distance, the third pixel distance, and the mapping relationship.

[0103] According to the intersection-over-union ratio, the distance between the second feature and the second sensor is expressed as:

[0104]

[0105] Right now

[0106] in, Indicates the actual distance corresponding to the second pixel distance, represents the second pixel distance, represents the third pixel distance, represents the first pixel distance, Indicates the fourth pixel distance.

[0107] Here, the second distance of the second feature from the second sensor in the two-dimensional image is determined. In the process of matching the second feature of the camera with the first feature of the radar, in addition to the distance information, time consistency is also a key factor. Therefore, the frame distance and association method are introduced to project the first feature and the second feature into a time-space domain, wherein the space domain uses the first distance and the second distance for matching, while in the time domain, time alignment is required. Specifically, if the number of frames of the first feature in the three-dimensional image matches the number of frames of the second feature in the two-dimensional image, time alignment can be achieved.

[0108] refer to Figure 4 As shown, the first feature and the second feature are projected into a unified domain, where the x-axis represents the frame number and the y-axis represents the distance between the feature and the sensor. In this domain, the first feature is a line segment and the second feature is a point. By comparing the similarity of the trajectories of the first feature and the second feature in the frame domain, the similarity is also used to determine the correlation between the first feature and the second feature. A threshold is preset in the weak similarity region, and the first feature is determined to match the second feature to obtain a matching result.

[0109] In some embodiments, similarity is expressed as:

[0110]

[0111] Among them, d r represents the distance between the first feature and the first sensor, d i represents the distance between the second feature and the second sensor, f r Indicates the frame number of the first feature in the 3D point cloud image, f i represents the frame number of the second feature in the two-dimensional image, and λ represents the scale factor, whose value is related to the average speed of all current moving objects and can be obtained by the millimeter wave radar.

[0112] pass Figure 4 It can be seen that there are two sets of first features and second features that overlap in frame number and distance, and the matching is completed.

[0113] Then in step 104, the three-dimensional point cloud image is projected onto a two-dimensional image, and the projection error is optimized using the matching result and the intrinsic parameter matrix of the second sensor until the projection error is minimized, thereby obtaining an extrinsic parameter matrix between the first sensor and the second sensor.

[0114] It can be seen from the above embodiments that after matching, only two groups of first features and second features are matched in the frame-range domain, and the matched trajectory features need to be optimized to obtain the final extrinsic parameter matrix.

[0115] Projection error refers to the difference between the corresponding electronic components captured by the actual second sensor and the three-dimensional point cloud after being projected onto the two-dimensional image plane. This error measures the accuracy of the three-dimensional point cloud in the two-dimensional image coordinates. The smaller the error, the more accurate the calibration of the first sensor and the second sensor.

[0116] In some embodiments, the extrinsic parameter matrix between the first sensor and the second sensor is expressed as:

[0117]

[0118] in, represents the optimization result, K represents the internal parameter matrix of the second sensor, T represents the external parameter matrix between the first sensor and the second sensor, and u i , x i Respectively represent the first feature and the second feature associated in the matching result.

[0119] At this point, the optimized external parameter matrix T is output, completing the first calibration of the position and attitude of the first sensor and the second sensor.

[0120] Since the first sensor and the second sensor are installed at a high place on the road side, climate and environmental factors (such as strong wind, vibration, etc.) may cause the position or posture of the sensor to drift, affecting the alignment accuracy between the first sensor and the second sensor, resulting in a decrease in the matching correlation rate of the first feature and the second feature after the first calibration. The present application designs an automatic recalibration mechanism to monitor the spatial drift of the first sensor and / or the second sensor, and automatically start the recalibration process when the drift exceeds a preset threshold.

[0121] In some embodiments, the recalibration process includes:

[0122] Determine the maximum projection error and monitor the maximum projection error in real time;

[0123] In response to the maximum projection error being greater than a preset error threshold, it is determined that drift occurs between the first sensor and the second sensor, and an extrinsic parameter matrix between the first sensor and the second sensor is re-determined.

[0124] The maximum projection error is expressed as:

[0125]

[0126] Where K represents the internal parameter matrix of the second sensor, represents the optimization result, u j , x j Respectively represent the first and second features associated in the matching results after re-matching, z d Indicates a first distance of a new first feature from the first sensor.

[0127] Output the optimized extrinsic parameter matrix, complete the recalibration, store the projection error and drift detection data in the database, provide real-time feedback on the extrinsic parameter status, and provide maintenance suggestions.

[0128] The trajectory-based automatic external parameter calibration method provided in the embodiment of the present application uses the motion trajectory of the three-dimensional point cloud image for straight line fitting as the second feature, and uses the trajectory feature for calibration. Compared with the point cloud feature in the three-dimensional point cloud image, it is more stable and has higher calibration accuracy. At the same time, the distance of the second feature from the second sensor is estimated by the vanishing point projection invariant, which does not rely on complex depth sensors and significantly reduces the computational complexity. The first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image are used to perform similarity matching with the second distance between the second feature and the second sensor and the number of frames in which the second feature is located in the two-dimensional image, and frame-distance domain association is performed to achieve matching of the first feature and the second feature. At the same time, the external parameter matrix of the first sensor and the second sensor is obtained by optimization to achieve calibration of the position and posture of the first sensor and the second sensor.

[0129] In addition, by utilizing maximum projection error monitoring to determine a recalibration mechanism, after spatial drift of the first sensor and / or the second sensor causes a change in position or posture of the first sensor and / or the second sensor, recalibration is performed to reduce monitoring costs and recalibration costs.

[0130] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0131] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a trajectory-based automatic extrinsic parameter calibration device.

[0133] refer to Figure 5, the trajectory-based automatic extrinsic parameter calibration device comprises:

[0134] An acquisition module 601 is used to acquire a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor at the same time;

[0135] A feature extraction module 602 is used to extract features from the three-dimensional point cloud image and the two-dimensional image respectively, and determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image;

[0136] A feature matching module 603 is used to perform similarity matching using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and a second distance between the second feature and the second sensor and a second distance between the second feature and the two-dimensional image to obtain a matching result;

[0137] The external parameter matrix module 604 is used to project the three-dimensional point cloud image to the two-dimensional image, optimize the projection error using the matching result and the internal parameter matrix of the first sensor until the projection error is minimized, and obtain the external parameter matrix between the first sensor and the second sensor.

[0138] In some embodiments, a recalibration module 605 is further included, which is used to determine the maximum projection error and monitor the maximum projection error in real time;

[0139] When the maximum projection error is greater than a preset error threshold, it is determined that drift occurs between the first sensor and the second sensor, and an external parameter matrix between the first sensor and the second sensor is re-determined;

[0140] The maximum projection error is expressed as:

[0141]

[0142] Where K represents the internal parameter matrix of the second sensor, represents the optimization result, u j , x j Respectively represent the first and second features associated in the matching results after re-matching, z d Indicates a first distance of a new first feature from the first sensor.

[0143] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0144] The device of the above embodiment is used to implement the corresponding trajectory-based automatic extrinsic parameter calibration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0145] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the trajectory-based automatic extrinsic parameter calibration method described in any of the above embodiments is implemented.

[0146] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0147] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0148] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0149] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0150] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0151] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0152] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0153] The electronic device of the above embodiment is used to implement the corresponding trajectory-based automatic extrinsic parameter calibration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0154] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the trajectory-based automatic external parameter calibration method as described in any of the above embodiments.

[0155] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0156] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the trajectory-based automatic extrinsic parameter calibration method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0157] Based on the same inventive concept, corresponding to the trajectory-based automatic external parameter calibration method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the trajectory-based automatic external parameter calibration method. Corresponding to the execution subject corresponding to each step in each embodiment of the trajectory-based automatic external parameter calibration method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0158] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the trajectory-based automatic extrinsic parameter calibration method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0159] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0160] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0161] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0162] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A trajectory-based automatic extrinsic parameter calibration method, characterized in that: include: Acquire a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor; Performing feature extraction on the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image; Perform similarity matching using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and a second distance between the second feature and the second sensor and the number of frames in which the second feature is located in the two-dimensional image to obtain a matching result; The three-dimensional point cloud image is projected onto the two-dimensional image, and the projection error is optimized using the matching result and the intrinsic parameter matrix of the second sensor until the projection error is minimized, thereby obtaining an extrinsic parameter matrix between the first sensor and the second sensor.

2. The method according to claim 1, characterized in that Extracting features from the three-dimensional point cloud image and the two-dimensional image respectively to determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image includes: Extracting lane line segments in the two-dimensional image to determine lane line trajectories of the two-dimensional image; Determining a motion trajectory of a moving object between lane line trajectories of the two-dimensional image as a second feature of the two-dimensional image; A straight line fitting is performed on the three-dimensional point cloud in the three-dimensional point cloud image, and a motion trajectory of the moving object in the three-dimensional point cloud image is determined as a first feature of the three-dimensional point cloud image.

3. The method according to claim 2, characterized in that Determining a second distance between the second feature and the second sensor includes: Determine the intersection point of the extended lines of the lane line trajectory of the two-dimensional image; determining a center point of the second feature; Determining a near point of a lane line trajectory of a lane in which the second feature is located; Determining an endpoint of a lane segment closest to the second feature; A second distance of the moving object in the second feature from the second sensor is determined using the intersection point, the center point, the near point and the end point.

4. The method according to claim 3, characterized in that The determining the distance of the second feature from the second sensor by using the intersection point, the center point, the near point and the endpoint includes: Predetermining the actual length of the lane segment; Determining the pixel length of the lane segment in the two-dimensional image; Determine a mapping relationship between the two-dimensional image and the actual according to the actual length and the pixel length; Determine a first pixel distance of a lane segment in the two-dimensional image by using the mapping relationship; Determine a second pixel distance between the endpoint and the near point in the two-dimensional image using the endpoint and the near point; Using the center point and the intersection point, determining a third pixel distance between the center point and the intersection point along the lane line trajectory direction in the two-dimensional image; Determine a fourth pixel distance between the near point and the intersection point in the two-dimensional image using the near point and the intersection point; Determine the distance of the second sensor of the moving object in the second characteristic distance according to the actual length of the lane segment, the first pixel distance, the second pixel distance, the third pixel distance, the fourth pixel distance and the mapping relationship; The distance of the second feature from the second sensor is expressed as: Right now in, Indicates the actual distance corresponding to the second pixel distance, represents the second pixel distance, represents the third pixel distance, represents the first pixel distance, Indicates the fourth pixel distance.

5. The method according to claim 1, characterized in that The similarity is expressed as: Among them, d r represents the distance between the first feature and the first sensor, d i represents the distance between the second feature and the second sensor, f r Indicates the frame number of the first feature in the 3D point cloud image, f i represents the number of frames representing the second feature in the two-dimensional image, and λ represents the scaling factor.

6. The method according to claim 1, characterized in that The external parameter matrix between the first sensor and the second sensor is expressed as: in, represents the optimization result, K represents the internal parameter matrix of the second sensor, T represents the external parameter matrix between the first sensor and the second sensor, and u i , x i Respectively represent the first feature and the second feature associated in the matching result.

7. The method according to claim 1, characterized in that The three-dimensional point cloud image is projected onto the two-dimensional image, and the matching result and the intrinsic parameter matrix of the first sensor are used to optimize until the projection error is minimized, so as to obtain the extrinsic parameter matrix between the first sensor and the second sensor, and then the following steps are further included: Determining a maximum projection error, and monitoring the maximum projection error in real time; In response to the maximum projection error being greater than a preset error threshold, determining that drift occurs between the first sensor and the second sensor, and re-determining an extrinsic parameter matrix between the first sensor and the second sensor; The maximum projection error is expressed as: Where K represents the internal parameter matrix of the second sensor, represents the optimization result, u j , x j Respectively represent the first and second features associated in the matching results after re-matching, z d Indicates a first distance of a new first feature from the first sensor.

8. A trajectory-based automatic external parameter calibration device, characterized in that: include: An acquisition module, used to acquire a three-dimensional point cloud image acquired by a first sensor and a two-dimensional image acquired by a second sensor at the same time; A feature extraction module, used to extract features from the three-dimensional point cloud image and the two-dimensional image respectively, and determine a first feature of the three-dimensional point cloud image and a second feature of the two-dimensional image; a feature matching module, configured to perform similarity matching using a first distance between the first feature and the first sensor and the number of frames in which the first feature is located in the three-dimensional point cloud image, and a second distance between the second feature and the second sensor and a second distance between the second feature and the two-dimensional image, to obtain a matching result; An external parameter matrix module is used to project the three-dimensional point cloud image to the two-dimensional image, optimize the projection error using the matching result and the internal parameter matrix of the first sensor until the projection error is minimized, and obtain the external parameter matrix between the first sensor and the second sensor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images