Sensor external parameter calibration method and device for rail transit

Abstract

The invention relates to the technical field of multi-sensor fusion, and provides a rail transit-oriented sensor external parameter calibration method and device, and the method comprises the steps: determining a 2D linear equation of a first linear element in a rail transit scene image which is shot by a camera and corresponds to a current environment; calculating depth information of each point in the 2D linear equation, and determining a first 3D linear equation based on the depth information and the 2D linear equation; determining a second 3D linear equation based on first point cloud data of a first linear element in a radar scanning current environment; and registering a plurality of first to-be-registered point sets in the first 3D linear equation and second to-be-registered point sets with the same depth as the first to-be-registered point sets in the second 3D linear equation to obtain an external parameter matrix between the camera and the radar. According to the method, high-precision external parameter calibration between the camera and the radar is realized by utilizing the linear characteristic that the rail transit scene has more linear elements, the precision of the calibrated external parameter matrix is high, and the calculated amount is small.

Description

Technical Field

[0001] The present invention relates to the field of multi-sensor fusion technology, and in particular to a sensor extrinsic parameter calibration method and device for rail transit. Background Art

[0002] At present, perception systems that rely on multi-sensor fusion have been used in urban rail transit scenarios. With the increasing application of sensors, the need to accurately estimate the external parameters between sensors has become increasingly important, directly affecting the safety and reliability of railway transportation systems. Existing cross-sensor external parameter calibration methods can be roughly divided into two categories: traditional methods and deep learning methods.

[0003] Although the external parameter calibration method based on deep learning can accurately calibrate, it has a large amount of calculation and requires a lot of computing resources. Although the traditional calibration method has a relatively small amount of calculation, it requires a lot of manpower to manually adjust the parameters. This process not only relies on professional knowledge and experience, but also has low accuracy when dealing with scenes with repeated or weak textures in rail transit (referring to scenes where the surface of an object lacks obvious textures, or the texture pattern is complex and unclear), making it more difficult to apply in different environments. Therefore, how to achieve a method for automatic calibration of external parameters of lidar and cameras with relatively small amount of calculation, high accuracy, and suitable for rail transit is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a sensor external parameter calibration method and device for rail transit, which are used to solve the above-mentioned technical problems existing in the prior art.

[0005] The present invention provides a sensor external parameter calibration method for rail transit, comprising the following steps.

[0006] Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0007] Depth information of each point in the 2D straight line equation is calculated, and a first 3D straight line equation is determined based on the depth information and the 2D straight line equation.

[0008] A second 3D straight line equation is determined based on first point cloud data of the first straight line element in the current environment scanned by radar.

[0009] The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar.

[0010] According to a sensor extrinsic parameter calibration method for rail transit provided by the present invention, a 2D straight line equation of a first straight line element in a rail transit scene image corresponding to a current environment captured by a camera is determined, including the following steps.

[0011] Extracting a plurality of key points on the first straight line element; Performing perspective transformation on the rail transit scene image based on the plurality of key points to obtain a top view with the first straight line element as the center line; Performing rail transit scene image enhancement processing on the top view to extract a target point set on the first straight line element; The target point set is fitted to obtain a 2D straight line equation of the first straight line element.

[0012] According to a sensor extrinsic parameter calibration method for rail transit provided by the present invention, after performing rail transit scene image enhancement processing on the overhead view, extracting the target point set on the first straight line element, and before fitting the target point set to obtain the 2D straight line equation of the first straight line element, it also includes: clustering and smoothing the target point set on the first straight line element using a sliding window technology.

[0013] According to a sensor extrinsic parameter calibration method for rail transit provided by the present invention, the first straight line element is a track, and the depth information of each point in the 2D straight line equation is calculated, including the following steps.

[0014] A reference point is respectively determined on the two tracks, and a line connecting the two reference points is perpendicular to the tracks.

[0015] Depth information of the two reference points is determined based on the respective 2D coordinates of the two reference points, the actual distance between the two reference points, and a coordinate conversion matrix, where the conversion matrix is ​​a conversion matrix from an image coordinate system to a camera coordinate system.

[0016] According to a sensor external parameter calibration method for rail transit provided by the present invention, the first straight line element is a track, and the second 3D straight line equation is determined based on the first point cloud data of the first straight line element in the current environment scanned by radar, including the following steps.

[0017] Based on the global features of the overall point cloud scanned by the radar, the ground area is identified.

[0018] In the ground area, a search is performed in a direction perpendicular to the track with a certain search step length, the point cloud density within each search step length is calculated, and the point cloud density extreme point is found, and the search step length is smaller than the track width.

[0019] Within the range of the track width, two points with the highest point cloud density are selected, and the track line area is determined based on the two points with the highest point cloud density.

[0020] The point cloud data of the track line area is obtained to obtain the first point cloud data of the track.

[0021] The first point cloud data is fitted to obtain the second 3D straight line equation.

[0022] According to a sensor extrinsic parameter calibration method for rail transit provided by the present invention, multiple first point sets to be registered in the first 3D straight line equation and second point sets to be registered in the second 3D straight line equation with the same depth as each first point set to be registered are registered to obtain an extrinsic parameter matrix between the camera and the radar, including the following steps.

[0023] Within a preset depth range, the first set of points to be registered is obtained from the first 3D straight line equation at intervals of a certain depth step.

[0024] Within the preset depth range, at intervals of the same depth step, the second set of points to be registered is obtained from the second 3D straight line equation.

[0025] The first set of points to be registered and the second set of points to be registered are registered to obtain an extrinsic parameter matrix between the camera and the radar.

[0026] According to a sensor extrinsic parameter calibration method for rail transit provided by the present invention, after aligning multiple first point sets to be aligned in the first 3D straight line equation and second point sets to be aligned in the second 3D straight line equation with the same depth as each first point set to be aligned, and obtaining the extrinsic parameter matrix between the camera and the radar, it also includes the following steps.

[0027] A target area where the second straight line element is located is extracted from the rail transit scene image of the current environment captured by the camera.

[0028] The target area is projected onto the overall point cloud data scanned by the radar using the external parameter matrix to extract the second point cloud data of the second straight line element.

[0029] A cluster analysis is performed on the second point cloud data to screen out valid second point cloud data of the second straight line element.

[0030] Converting the effective second point cloud data into an image of the area to be compared through the external parameter matrix; The angle and translation of the extrinsic parameter matrix are adjusted to reduce the edge error between the image of the area to be compared and the target area, and when the edge error is less than an error threshold, the adjusted extrinsic parameter matrix is ​​output.

[0031] The present invention also provides a sensor external parameter calibration device for rail transit, comprising the following modules.

[0032] The 2D equation determination module is used to determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0033] The first 3D equation determination module is used to calculate the depth information of each point in the 2D straight line equation, and determine the first 3D straight line equation based on the depth information and the 2D straight line equation.

[0034] The second 3D equation determination module is used to determine the second 3D straight line equation based on the first point cloud data of the first straight line element in the current environment scanned by the radar.

[0035] The registration module is used to register multiple first to-be-registered point sets in the first 3D straight line equation and second to-be-registered point sets in the second 3D straight line equation with the same depth as each first to-be-registered point set, so as to obtain an external parameter matrix between the camera and the radar.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the sensor external parameter calibration method for rail transit as described in any one of the above-mentioned methods is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the sensor external parameter calibration method for rail transit as described in any one of the above is implemented.

[0038] The sensor extrinsic parameter calibration method and device for rail transit provided by the present invention determines the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera, converts the 2D straight line equation into a first 3D straight line equation, determines the second 3D straight line equation based on the first point cloud data of the first straight line element in the current environment scanned by the radar, and aligns the multiple first to-be-aligned point sets in the first 3D straight line equation and the second to-be-aligned point sets in the second 3D straight line equation that have the same depth as the first to-be-aligned point sets, so as to obtain the extrinsic parameter matrix between the camera and the radar. In this embodiment, since the rail transit scene has more straight line elements, based on the straight line features of the straight line elements, the straight line elements can be accurately identified and located in the image and the 3D point cloud, so that the extrinsic parameter calibration between the camera and the radar can be realized based on the first to-be-aligned point set and the second to-be-aligned point set on the straight line elements. The calibrated extrinsic parameter matrix has high accuracy and small calculation amount compared with the calibration method of deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the sensor external parameter calibration method for rail transit provided by the present invention.

[0041] Figure 2 It is a schematic diagram of using the track where the vehicle is located as the first straight line element in the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0042] Figure 3 It is a schematic diagram of restoring depth information from an image in the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0043] Figure 4 It is a schematic diagram of the overall point cloud of radar scanning in the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0044] Figure 5 It is a schematic diagram of extracting a track point cloud from an overall point cloud in the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0045] Figure 6 It is one of the registration effect diagrams in the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0046] Figure 7 This is the second alignment effect diagram of the sensor extrinsic parameter calibration method for rail transit provided by the present invention.

[0047] Figure 8 It is a structural schematic diagram of a sensor external parameter calibration device for rail transit provided by the present invention.

[0048] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The sensor extrinsic parameter calibration method for rail transit according to an embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps S110 to S140.

[0051] Step S110: determining the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0052] Specifically, the camera is a camera installed at the front end of the train head, and the rail transit scene image of the current environment captured by the camera is the image in front of the train head, wherein the rail transit scene image has very obvious straight line elements, such as: the two tracks where the train is currently located, the tracks near the train, the electric poles next to the two tracks where the train is currently located, and the edges of the carriages of other nearby trains. Preferably, the two tracks where the train is currently located can be selected as the first straight line element, because the rail transit scene image is captured by the camera in front of the train head, and the two tracks where the train is currently located are basically located in the middle of the rail transit scene image, which is convenient for identification. In this embodiment, the points on the first straight line element can be identified by image recognition technology, and the identified points can be fitted to obtain the 2D straight line equation of the first straight line element.

[0053] Step S120: Calculate the depth information of each point in the 2D straight line equation, and determine a first 3D straight line equation based on the depth information and the 2D straight line equation.

[0054] Specifically, the depth information of each point in the 2D straight line equation can be calculated through the conversion relationship between the image coordinate system and the camera coordinate system. The depth information is combined with the 2D coordinates of each point in the 2D straight line equation to obtain the 3D coordinates in the camera coordinate system, thereby obtaining the first 3D straight line equation.

[0055] Step S130: Determine the second 3D straight line equation based on the first point cloud data of the first straight line element in the current environment scanned by the radar. Specifically, the first point cloud data of the first straight line element scanned by the radar is the 3D coordinate of the first straight line element in the camera coordinate system, and therefore, the second 3D straight line equation can be obtained by fitting the 3D coordinate.

[0056] Step S140: align the multiple first to-be-aligned point sets in the first 3D straight line equation and the second to-be-aligned point sets in the second 3D straight line equation that have the same depth as the first to-be-aligned point sets, to obtain the external parameter matrix between the camera and the radar, that is, to calibrate the external parameters between the camera and the radar. The alignment method may adopt an iterative closest point (ICP) algorithm, which minimizes the difference between the two to-be-aligned point sets through iterative optimization, thereby calculating the relative posture relationship between the radar and the camera, and finally obtaining the external parameter matrix between the camera and the radar.

[0057] The sensor extrinsic parameter calibration method for rail transit in this embodiment determines the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera, converts the 2D straight line equation into a first 3D straight line equation, determines the second 3D straight line equation based on the first point cloud data of the first straight line element in the current environment scanned by the radar, and aligns the multiple first point sets to be registered in the first 3D straight line equation and the second point sets to be registered in the second 3D straight line equation that have the same depth as the first point sets to be registered, so as to obtain the extrinsic parameter matrix between the camera and the radar. In this embodiment, since the rail transit scene has more straight line elements, the straight line elements can be accurately identified and located in the image and the 3D point cloud based on the straight line features of the straight line elements, so that the high-precision extrinsic parameter calibration between the camera and the radar can be achieved according to the first point set to be registered and the second point set to be registered on the straight line elements. The calibrated extrinsic parameter matrix has high accuracy and small computational complexity compared to the calibration method of deep learning.

[0058] It should be noted that: in the sensor extrinsic calibration method for rail transit of this embodiment, the extrinsic calibration between the camera and the radar is realized based on the straight line elements in the rail transit scene image. Even in the texture repetition or weak texture scene, as long as there are straight line elements, high-precision extrinsic calibration between the camera and the radar can be realized. The sensor extrinsic calibration method for rail transit of this embodiment is applicable to various scenes such as viaduct railways, tunnel railways and outdoor railways, and can realize high-precision extrinsic calibration in complex and weak-texture environments.

[0059] In some embodiments, determining the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera includes the following steps.

[0060] Extract multiple key points on the first straight line element. Specifically, the key points of the first straight line element can be extracted from the rail transit scene image by using a line segment detector LSD algorithm. LSD is an efficient and accurate straight line detection technology that can reliably identify the line segment features of the first straight line element in a complex environment. The extracted key points include the starting point and the end point of the first straight line element.

[0061] Based on the multiple key points, the rail transit scene image is perspective transformed to obtain a top view with the first straight line element as the center line, thereby reducing the interference of the environment on the extraction of the target point set on the first straight line element. In the case where the first straight line element is the two tracks where the train is currently located, after the image is corrected by perspective transformation, the multiple first straight line elements are in a parallel state, which is more conducive to the subsequent extraction of the target point set on the track.

[0062] The overhead view is subjected to rail transit scene image enhancement processing, and the target point set on the first straight line element is extracted. Specifically, first, the contrast of the rail transit scene image is improved by histogram equalization to make the first straight line element more prominent; then, Gaussian blur is applied to smooth the rail transit scene image to remove noise and details, in preparation for edge detection and feature extraction; then, pixel threshold segmentation is used to binarize the image, and only the highlight part of the first straight line element is retained, thereby further highlighting the position of the first straight line element. The above image enhancement processing effectively enhances the visibility and continuity of the target point set of the first straight line element in the image. Especially in the case where the first straight line element is the two tracks where the train is currently located, the tracks rub against the train wheels, and the track brightness is high. After image enhancement processing, it is easier to extract the target point set on the track.

[0063] The target point set is fitted to obtain the 2D straight line equation of the first straight line element. Specifically, a first-order Bezier curve can be used to fit the first straight line element to obtain the 2D straight line equation of the first straight line element in the rail transit scene image. The Bezier curve is a smooth parametric curve suitable for straight line features with a certain curvature and continuity. By fitting the Bezier curve to the target point set on the extracted first straight line element, the position and shape of the first straight line element can be accurately obtained.

[0064] like Figure 2 As shown in the figure, the first straight line element is the two tracks where the train is currently located, and the track line obtained by fitting is as follows Figure 2 Shown in red and green lines.

[0065] Preferably, due to the camera shooting angle, the lower half of the rail transit scene image may include the front part of the train. The upper half of the rail transit scene image can be masked, and the target point set of the first straight line element can be extracted from the upper half of the rail transit scene image to avoid interference from the front part of the train.

[0066] In some embodiments, after the rail transit scene image enhancement processing is performed on the top view, and the target point set on the first straight line element is extracted, before the target point set is fitted to obtain the 2D straight line equation of the first straight line element, it also includes: clustering and smoothing the target point set on the first straight line element using a sliding window technology. Specifically, by sliding a fixed-size window on the image, the points on the first straight line element in the window are clustered and smoothed, thereby obtaining a target point set with better continuity of the first straight line element, and effectively reducing the influence of noise and discrete points, so that the extracted target point set of the first straight line element has better stability and accuracy.

[0067] In some embodiments, the first straight line element is a track, and calculating the depth information of each point in the 2D straight line equation includes the following steps.

[0068] A reference point is determined on each of the two tracks, and the line connecting the two reference points is perpendicular to the track. Figure 3 As shown, take a point on each of the two tracks, namely point P l and Point P r ,point P l and Point P r The line connecting the two reference points is perpendicular to the track, that is, the distances from the two reference points to the camera focal plane are equal, that is, the two reference points have the same depth information.

[0069] Depth information of the two reference points is determined based on the respective 2D coordinates of the two reference points, the actual distance between the two reference points, and a coordinate conversion matrix, where the conversion matrix is ​​a conversion matrix from an image coordinate system to a camera coordinate system.

[0070] Specifically, the conversion is performed according to the following formula.

[0071] .

[0072] Among them, (u, v) represents the coordinates of the pixel point in the image coordinate system UV, ( X , Y , Z ) represents the coordinates of the pixel point (u, v) in the camera coordinate system XYZ. The imaging area of ​​the camera is a rectangle. f x and f y They are the x-axis and y-axis focal lengths in the corresponding rectangular imaging area in the camera coordinate system XYZ, c x and c y They respectively represent the coordinates of the optical center of the camera in the image coordinate system.

[0073] For two points on the track, P l and Point P r , convert the coordinates of the two points in the image coordinate system to the corresponding coordinates in the camera coordinate system, and do the following calculations: .

[0074] .

[0075] .

[0076] Among them, Δ u is the horizontal pixel difference between the two reference points in the rail transit scene image (i.e., in the image coordinate system), Δ X is the distance between the two reference points in the actual scene (in the camera coordinate system), which is the track gauge (1.51m). Z l and Z r Points P l and Point P r The Z-axis coordinate in the camera coordinate system is the depth information. According to the above formula, the depth information lost during the camera projection process can be obtained Z = Z l = Z r .

[0077] In some embodiments, the first straight line element is a track, and step S130 includes the following steps.

[0078] Based on the global features of the overall point cloud scanned by the radar, the ground area is identified, wherein the global features include: point cloud density and / or reflectivity. Figure 4 As shown, according to prior knowledge, the ground where the track is located usually has a high point cloud density. Therefore, in this embodiment, a density threshold of the point cloud density can be set to identify the ground area. For example, the density threshold can be set to 30%, and the area with a point cloud density greater than 30% is determined as the ground area. It should be noted that the point cloud returned by the radar is stored in a certain format (such as PCD format), so the number of point clouds in each PCD format is used as the basis for calculation, for example: 10,000, then in each sampling interval, the location with more than 3,300 points is considered to be the ground area.

[0079] In the ground area, search in a direction perpendicular to the track with a certain search step length, calculate the point cloud density within each search step length, and find the extreme point of point cloud density (maximum point cloud density), where the search step length is less than the track width. Due to the long-term friction between the track and the train wheels, the track surface is smooth and flat, and the point cloud density at the track surface position is higher than that of other areas on the ground. Therefore, the extreme point of the point cloud density corresponds to the position of the track.

[0080] Within the range of the track width, two points with the highest point cloud density are selected, and the track line area is determined based on the two points with the highest point cloud density.

[0081] The point cloud data of the track line area is obtained to obtain the first point cloud data of the track. Specifically, for the point cloud data of each track line area, filtering technology is applied to remove noise and unnecessary points. The filtering process can use methods such as mean filtering or median filtering to effectively reduce the random noise in the point cloud data, thereby improving the smoothness and accuracy of the data. Through filtering, clearer and continuous track line point cloud data, that is, the first point cloud data, can be obtained.

[0082] The first point cloud data is fitted to obtain the second 3D line equation. Specifically, the principal component analysis (PCA) method can be used to fit the point cloud data of each track line, and the 3D equation of the track line, i.e., the second 3D line equation, is calculated based on the track line direction vector and starting point obtained by PCA fitting. This equation describes the position and direction of the track line in three-dimensional space, and provides a basis for track line alignment and sensor calibration. The final processing effect is as follows: Figure 5 As shown, the position and shape of the track line extracted by the above method in the point cloud data are demonstrated.

[0083] In some embodiments, the step S140 specifically includes the following steps.

[0084] Within the preset depth range, at intervals of a certain depth step, the first set of points to be registered is obtained from the first 3D straight line equation. , thereby obtaining multiple first point sets to be registered. Specifically, the preset depth range can be selected in the first 3D straight line equation, the depth Z is in the range of 10 meters to 80 meters, the depth step can be 0.05~0.2 meters, the smaller the depth step, the more the number of first point sets to be registered is collected, that is, more first point sets to be registered are obtained, and the smaller the depth step, for example: collecting the first point set to be registered once at an interval of 0.1 meters, ensures the accurate description of the first straight line elements in the three-dimensional space, thereby improving the accuracy and robustness of the subsequent registration process.

[0085] Within the preset depth range, at intervals of the same depth step, the second set of points to be registered is obtained from the second 3D straight line equation. , that is, the first set of points to be registered and the second set of points to be registered under each same step length correspond to each other, ensuring the spatial consistency of the point cloud data and the image data.

[0086] The first set of points to be registered and the second set of points to be registered are registered to obtain the extrinsic parameter matrix between the camera and the radar. Specifically, the ICP algorithm can be used for registration. The ICP algorithm continuously adjusts the position and posture of the lidar point set to minimize the Euclidean distance between the corresponding point pairs in the image point set. In each iteration, the ICP recalculates the nearest point pair and updates the transformation matrix until it converges to the optimal solution, thereby obtaining the extrinsic parameter matrix between the camera and the radar. The extrinsic parameter matrix describes the rotation and translation relationship between the radar coordinate system and the camera coordinate system, and is the key to realizing the fusion of the two sensor data. Figure 6 As shown in the figure, after the extrinsic parameter matrix is ​​calculated, the overall point cloud scanned by the radar can be projected into the camera image through the extrinsic parameter matrix to achieve spatial alignment of the radar data and the image data. In this way, the corresponding position of the overall point cloud scanned by the radar in the image can be intuitively displayed, thereby verifying the registration effect.

[0087] In some embodiments, after step S140, the following steps are also included.

[0088] The target area where the second straight line element is located is extracted from the rail transit scene image of the current environment captured by the camera. The second straight line element may be a column area of ​​a utility pole in the rail transit scene image. Specifically, semantic segmentation may be used to identify and extract semantic information of the column area of ​​the utility pole, and the column area of ​​the utility pole may be segmented from the background to generate the column area of ​​the utility pole.

[0089] The target area is projected onto the overall point cloud data scanned by the radar using the external parameter matrix to extract the second point cloud data of the second straight line element.

[0090] The second point cloud data is clustered to screen out valid second point cloud data of the second straight line element. For a scene where the second straight line element is a pole of a utility pole, cluster analysis helps to group points belonging to different poles in the second point cloud data, while removing noise and irrelevant point clouds, thereby extracting the second point cloud data of each pole, i.e., valid second point cloud data.

[0091] The effective second point cloud data is converted into the image of the area to be compared through the external parameter matrix, that is, the effective second point cloud data is converted into coordinate points corresponding to the second straight line elements in the rail transit scene image through the external parameter matrix, and these coordinate points form the image of the area to be compared.

[0092] The angle and translation of the extrinsic parameter matrix are adjusted to reduce the edge error between the image of the area to be compared and the target area, thereby further improving the registration precision and accuracy. When the edge error is less than the error threshold, the adjusted extrinsic parameter matrix is ​​output, and the adjusted extrinsic parameter matrix has higher precision. The error threshold can be set according to the precision requirements of the actual application, for example, the error threshold is 3-5 pixels. The effect of using the adjusted extrinsic parameter matrix to project the overall point cloud scanned by the radar into the camera image through the adjusted extrinsic parameter matrix is ​​shown in the figure below. Figure 7 As shown by Figure 6 and Figure 7 It can be seen that Figure 7 In , the adjusted extrinsic matrix is ​​used to project the whole point cloud onto the camera image, which is more effective.

[0093] The sensor external parameter calibration device for rail transit provided by the present invention is described below. The sensor external parameter calibration device for rail transit described below and the sensor external parameter calibration method for rail transit described above can correspond to each other.

[0094] The sensor external parameter calibration device for rail transit according to an embodiment of the present invention is as follows: Figure 8 As shown, it includes the following modules.

[0095] The 2D equation determination module 810 is used to determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0096] The first 3D equation determination module 820 is used to calculate the depth information of each point in the 2D straight line equation, and determine the first 3D straight line equation based on the depth information and the 2D straight line equation.

[0097] The second 3D equation determination module 830 is used to determine the second 3D straight line equation based on the first point cloud data of the first straight line element in the current environment scanned by the radar.

[0098] The registration module 840 is used to align multiple first to-be-registered point sets in the first 3D straight line equation and second to-be-registered point sets in the second 3D straight line equation with the same depth as each first to-be-registered point set, to obtain an external parameter matrix between the camera and the radar.

[0099] In the sensor extrinsic parameter calibration device for rail transit of this embodiment, since rail transit scenes have more straight line elements, the straight line elements can be accurately identified and located in images and 3D point clouds based on the straight line features of the straight line elements, so that high-precision extrinsic parameter calibration between cameras and radars can be achieved based on the first set of points to be registered and the second set of points to be registered on the straight line elements. The calibrated extrinsic parameter matrix has high accuracy and less computational complexity than the deep learning calibration method.

[0100] In some embodiments, the 2D equation determination module 810 includes the following modules.

[0101] A key point extraction module is used to extract multiple key points on the first straight line element.

[0102] The perspective transformation module is used to perform perspective transformation on the rail transit scene image based on the multiple key points to obtain a top view with the first straight line element as the center line.

[0103] The image enhancement processing module is used to perform rail transit scene image enhancement processing on the overhead view and extract a target point set on the first straight line element.

[0104] The 2D straight line equation fitting module is used to fit the target point set to obtain the 2D straight line equation of the first straight line element.

[0105] In some embodiments, the 2D equation determination module 810 also includes: a clustering and smoothing processing module, which is used to cluster and smooth the target point set on the first straight line element using a sliding window technique after performing rail transit scene image enhancement processing on the overhead view and extracting the target point set on the first straight line element, and before fitting the target point set to obtain the 2D straight line equation of the first straight line element.

[0106] In some embodiments, the first straight line element is a track, and the first 3D equation determination module 820 is specifically used to determine a reference point on each of the two tracks, and the line connecting the two reference points is perpendicular to the track; based on the respective 2D coordinates of the two reference points, the actual distance between the two reference points and the coordinate transformation matrix, the depth information of the two reference points is determined, and the transformation matrix is ​​a transformation matrix from the image coordinate system to the camera coordinate system.

[0107] In some embodiments, the first straight line element is a track, and the second 3D equation determination module 830 includes the following modules.

[0108] The ground area recognition module is used to identify the ground area based on the global features of the overall point cloud scanned by the radar.

[0109] The extreme point search module is used to search in a ground area in a direction perpendicular to the track with a certain search step length, calculate the point cloud density within each search step length, and find the extreme point of the point cloud density. The search step length is smaller than the track width.

[0110] The track line area determination module is used to select two points with the highest point cloud density within the range of the track width, and determine the track line area based on the two points with the highest point cloud density.

[0111] The point cloud data acquisition module is used to acquire the point cloud data of the track line area to obtain the first point cloud data of the track.

[0112] The 3D straight line equation fitting module is used to fit the first point cloud data to obtain the second 3D straight line equation.

[0113] In some embodiments, the registration module 840 is specifically used to obtain the first set of points to be registered from the first 3D straight line equation at intervals of a certain depth step within a preset depth range; to obtain the second set of points to be registered from the second 3D straight line equation at intervals of the same depth step within the preset depth range; and to align the first set of points to be registered and the second set of points to be registered to obtain an external parameter matrix between the camera and the radar.

[0114] In some embodiments, the sensor extrinsic parameter calibration device for rail transit also includes: an extrinsic parameter matrix calibration module, which is used to extract a target area corresponding to a second straight line element in the rail transit scene image of the current environment captured by the camera; use the extrinsic parameter matrix to project the target area into the overall point cloud data scanned by the radar to extract the second point cloud data of the second straight line element; perform cluster analysis on the second point cloud data to screen out the valid second point cloud data of the second straight line element; convert the valid second point cloud data into an image of the area to be compared through the extrinsic parameter matrix; adjust the angle and translation of the extrinsic parameter matrix to reduce the edge error between the image of the area to be compared and the target area, and output the adjusted extrinsic parameter matrix when the edge error is less than the error threshold.

[0115] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the sensor external parameter calibration method for rail transit, and the method includes the following steps.

[0116] Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0117] Depth information of each point in the 2D straight line equation is calculated, and a first 3D straight line equation is determined based on the depth information and the 2D straight line equation.

[0118] A second 3D straight line equation is determined based on first point cloud data of the first straight line element in the current environment scanned by radar.

[0119] The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar.

[0120] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sensor external parameter calibration method for rail transit provided by the above methods, which includes the following steps.

[0122] Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0123] Depth information of each point in the 2D straight line equation is calculated, and a first 3D straight line equation is determined based on the depth information and the 2D straight line equation.

[0124] A second 3D straight line equation is determined based on first point cloud data of the first straight line element in the current environment scanned by radar.

[0125] The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar.

[0126] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the sensor external parameter calibration method for rail transit provided by the above methods is implemented, and the method includes the following steps.

[0127] Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera.

[0128] Depth information of each point in the 2D straight line equation is calculated, and a first 3D straight line equation is determined based on the depth information and the 2D straight line equation.

[0129] A second 3D straight line equation is determined based on first point cloud data of the first straight line element in the current environment scanned by radar.

[0130] The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar.

[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sensor extrinsic parameter calibration method for rail transit, characterized in that: include: Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera; Calculating depth information of each point in the 2D straight line equation, and determining a first 3D straight line equation based on the depth information and the 2D straight line equation; Determine a second 3D straight line equation based on first point cloud data of the first straight line element in the current environment scanned by radar; The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar.

2. The sensor external parameter calibration method for rail transit according to claim 1 is characterized in that: Determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera, including: Extracting a plurality of key points on the first straight line element; Performing perspective transformation on the rail transit scene image based on the plurality of key points to obtain a top view with the first straight line element as the center line; Performing rail transit scene image enhancement processing on the top view to extract a target point set on the first straight line element; The target point set is fitted to obtain a 2D straight line equation of the first straight line element.

3. The sensor external parameter calibration method for rail transit according to claim 2 is characterized in that: After performing rail transit scene image enhancement processing on the top view and extracting the target point set on the first straight line element, and before fitting the target point set to obtain the 2D straight line equation of the first straight line element, the method further includes: The sliding window technology is used to cluster and smooth the target point set on the first straight line element.

4. The sensor external parameter calibration method for rail transit according to claim 1, characterized in that: The first straight line element is a track, and the depth information of each point in the 2D straight line equation is calculated, including: Determine a reference point on each of the two tracks, wherein a line connecting the two reference points is perpendicular to the tracks; Depth information of the two reference points is determined based on the respective 2D coordinates of the two reference points, the actual distance between the two reference points, and a coordinate conversion matrix, where the conversion matrix is ​​a conversion matrix from an image coordinate system to a camera coordinate system.

5. The sensor external parameter calibration method for rail transit according to claim 1, characterized in that: The first straight line element is a track, and based on the first point cloud data of the first straight line element in the current environment scanned by a radar, determining the second 3D straight line equation includes: Based on the global features of the overall point cloud scanned by the radar, the ground area is identified; In the ground area, searching is performed in a direction perpendicular to the track with a certain search step length, the point cloud density within each search step length is calculated, and the point cloud density extreme value point is found, wherein the search step length is smaller than the track width; Within the range of the track width, selecting two points with the highest point cloud density, and determining the track line area based on the two points with the highest point cloud density; Acquire point cloud data of the track line area to obtain first point cloud data of the track; The first point cloud data is fitted to obtain the second 3D straight line equation.

6. The sensor external parameter calibration method for rail transit according to claim 1, characterized in that: The plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets are registered to obtain an extrinsic parameter matrix between the camera and the radar, including: Within a preset depth range, at intervals of a certain depth step, obtaining the first set of points to be registered from the first 3D straight line equation; Within the preset depth range, at intervals of the same depth step, obtaining the second set of points to be registered from the second 3D straight line equation; The first set of points to be registered and the second set of points to be registered are registered to obtain an extrinsic parameter matrix between the camera and the radar.

7. The sensor extrinsic parameter calibration method for rail transit according to any one of claims 1 to 6, characterized in that: After registering the plurality of first to-be-registered point sets in the first 3D straight line equation and the second to-be-registered point sets in the second 3D straight line equation having the same depth as the first to-be-registered point sets, and obtaining the external parameter matrix between the camera and the radar, the method further includes: Extracting a target area corresponding to the second straight line element from the rail transit scene image of the current environment captured by the camera; Projecting the target area onto the overall point cloud data scanned by the radar using the external parameter matrix to extract second point cloud data of the second straight line element; Performing cluster analysis on the second point cloud data to screen out valid second point cloud data of the second straight line element; Converting the effective second point cloud data into an image of the area to be compared through the external parameter matrix; The angle and translation of the extrinsic parameter matrix are adjusted to reduce the edge error between the image of the area to be compared and the target area, and when the edge error is less than an error threshold, the adjusted extrinsic parameter matrix is ​​output.

8. A sensor external parameter calibration device for rail transit, characterized in that: include: A 2D equation determination module, used to determine the 2D straight line equation of the first straight line element in the rail transit scene image corresponding to the current environment captured by the camera; A first 3D equation determination module, configured to calculate depth information of each point in the 2D straight line equation, and determine a first 3D straight line equation based on the depth information and the 2D straight line equation; A second 3D equation determination module, configured to determine a second 3D straight line equation based on first point cloud data of the first straight line element in the current environment scanned by a radar; The registration module is used to register multiple first to-be-registered point sets in the first 3D straight line equation and second to-be-registered point sets in the second 3D straight line equation with the same depth as each first to-be-registered point set, so as to obtain an external parameter matrix between the camera and the radar.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the sensor extrinsic parameter calibration method for rail transit according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sensor extrinsic parameter calibration method for rail transit according to any one of claims 1 to 7 is implemented.

Images