Joint calibration method and device for 2D sensor and 3D sensor of steel rail

Abstract

The invention discloses a combined calibration method and device for a 2D sensor and a 3D sensor of a steel rail, which can be used in the field of track detection, and the method comprises the following steps: calibrating internal parameters and external parameters of the 3D sensor, and determining an attitude transformation relationship between a 3D sensor coordinate system and a world coordinate system according to the external parameters of the 3D sensor; calibrating external parameters of the 2D sensor, and determining a reference coordinate system of the 2D sensor; extracting feature points of the target plane target image, and determining an attitude transformation relation between a 3D sensor coordinate system and a 2D sensor reference coordinate system; determining an attitude transformation relation between the 2D sensor coordinate system and the 3D sensor coordinate system through attitude combination; and determining the attitude transformation relation between the 2D sensor coordinate system and the world coordinate system through attitude combination. According to the invention, coordinates of two-dimensional data and three-dimensional data of the steel rail can be unified, and the calibration precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of track detection, and in particular to a method and device for jointly calibrating a rail 2D sensor and a 3D sensor. Background Art

[0002] This section is intended to provide a background or context for embodiments of the present invention. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] At present, rail surface defect detection is mainly divided into methods based on two-dimensional texture images and methods based on three-dimensional point cloud images. The former is a qualitative detection, and the latter is a quantitative detection. Among them, the two-dimensional texture image comes from a 2D sensor, which uses a linear array camera and a lighting source to scan the rail surface, thereby obtaining a two-dimensional texture image of the rail surface. The three-dimensional point cloud comes from a 3D sensor, which uses a triangulation optical path composed of an area array camera and line structured light to obtain rail cross-sectional profile data based on the triangulation principle, and finally obtains the three-dimensional point cloud data of the rail by splicing the equally spaced rail cross-sectional profile data.

[0004] Generally, two-dimensional texture images are easily disturbed by factors such as dirt on the rail surface. Therefore, the rail surface defect detection method based on two-dimensional texture images has the problem of high false alarm rate. Although three-dimensional point cloud can quantitatively describe the damage degree of rail, it cannot identify subtle defects such as rail cracks. Therefore, the current research mainly focuses on the fusion analysis of two-dimensional texture images and three-dimensional point cloud images of rails, and identifies rail surface defects by fusing information to achieve the effect of complementing each other's strengths. The coordinate unification of two-dimensional texture images and three-dimensional point cloud images of rails is the basis of fusion analysis. However, the two types of data come from different sensors, and their imaging principles, imaging fields and installation positions are different, resulting in inconsistent data space coordinate systems. Summary of the invention

[0005] The embodiment of the present invention provides a joint calibration method of a rail 2D sensor and a 3D sensor, which is used to unify the coordinates of the rail 2D data and the 3D data to improve the calibration accuracy. The method includes:

[0006] Calibrate the internal and external parameters of the 3D sensor, and determine the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system based on the external parameters of the 3D sensor;

[0007] Calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor;

[0008] The target plane target image of the rail is collected by a 3D sensor, feature points of the target plane target image are extracted, and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor is determined according to the feature points, internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor;

[0009] According to the external parameters of the 2D sensor and the attitude transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor, the attitude transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system is determined through attitude combination;

[0010] According to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system, and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system, the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system is determined through posture combination.

[0011] The embodiment of the present invention further provides a joint calibration device for a rail 2D sensor and a 3D sensor, which is used to unify the coordinates of the rail 2D data and the 3D data to improve the calibration accuracy. The device includes:

[0012] The 3D calibration module is used to calibrate the internal and external parameters of the 3D sensor and determine the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system according to the external parameters of the 3D sensor;

[0013] 2D calibration module, used to calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor;

[0014] A feature calibration module is used to collect a target plane target image of the rail through a 3D sensor, extract feature points of the target plane target image, and determine the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor based on the feature points, internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor;

[0015] The joint calibration module is used to determine the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor; and to determine the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system through posture combination according to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system.

[0016] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned joint calibration method of the rail 2D sensor and the 3D sensor when executing the computer program.

[0017] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned joint calibration method of the rail 2D sensor and the 3D sensor is implemented.

[0018] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned rail 2D sensor and 3D sensor joint calibration method is implemented.

[0019] In an embodiment of the present invention, internal parameters and external parameters of the 3D sensor are calibrated, and a posture transformation relationship between a 3D sensor coordinate system and a world coordinate system is determined according to the external parameters of the 3D sensor; the external parameters of the 2D sensor are calibrated to determine a reference coordinate system of the 2D sensor; a target plane target image of a rail is collected by the 3D sensor, feature points of the target plane target image are extracted, and a posture transformation relationship between the 3D sensor coordinate system and a reference coordinate system of the 2D sensor is determined according to the feature points, the internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor; a posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system is determined through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor; a posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system is determined through posture combination. In this way, the two-dimensional texture images and three-dimensional point cloud images of the rails from different sensors are represented in the same coordinate system through joint calibration, realizing the coordinate unification of the two-dimensional and three-dimensional data of the rails. This has the advantages of high precision, good real-time performance, and simple operation, which greatly reduces the difficulty of subsequent fusion analysis of rail surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A schematic diagram of a two-dimensional image and a three-dimensional image of a rail provided in an embodiment of the present invention;

[0022] Figure 2 It is a flow chart of a joint calibration method of a rail 2D sensor and a 3D sensor provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a joint calibration device for a rail 2D sensor and a 3D sensor provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the positional relationship between a planar target, a rail and an electrically controlled displacement platform provided in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of calibrating internal parameters of a 3D sensor provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of 3D sensor external parameter calibration provided in an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of 2D sensor calibration provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of external parameter calibration of a 2D sensor and a 3D sensor provided in an embodiment of the present invention;

[0029] Fig. 9 A schematic diagram of a joint calibration device for a rail 2D sensor and a 3D sensor provided in an embodiment of the present invention;

[0030] Fig.10 It is a structural block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0032] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0033] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.

[0034] First, some of the terms in the manual are explained:

[0035] Intrinsic parameters and extrinsic parameters are two concepts often used in digital photogrammetry, which are used to describe the internal performance of the camera and the geometric relationship between the camera and the object, respectively.

[0036] Intrinsic parameters: Intrinsic parameters are the internal performance parameters of the camera, including focal length, principal point position, radial distortion, etc. These parameters determine the camera imaging quality and accuracy. Intrinsic parameters are usually estimated through camera calibration or automatic calibration methods and used in subsequent processing.

[0037] External parameters: External parameters describe the geometric relationship between the camera and the scene, including the camera position, posture, and angle, etc. That is, the position and orientation of the camera in three-dimensional space. External parameters can be obtained through sensors such as GPS and inertial navigation devices, or estimated through methods such as image matching.

[0038] There are many types of damage on the rail surface, with great differences in appearance and location on the rail surface. Under the interference of the on-site environment, it is very difficult to detect. Existing damage detection on the rail surface mostly uses 2D image recognition, such as Figure 1As shown in the figure, the two-dimensional texture image of the rail surface obtained by a 2D camera is used to classify and detect defect information, and the technology is relatively mature. However, under the influence of factors such as light, surface foreign matter (oil stains, water stains) and surface rust, the collected two-dimensional texture image cannot distinguish between normal surfaces and defective surfaces, resulting in a very high false alarm rate. In fact, the depth map is not affected by these factors. In addition, whether it is the high-speed railway line maintenance rules or the conventional railway line repair rules, the judgment criteria for minor and serious damage to rails are based on depth information as the main reference standard. The characteristics of some damage types in the depth direction are more obvious than the two-dimensional geometric features. Therefore, the three-dimensional measurement technology can better improve the recognition rate of damage defects, reduce the false alarm rate, and judge the degree of damage to the rail based on the depth information by obtaining the three-dimensional morphology of the rail surface.

[0039] It can be seen that the detection method based on the two-dimensional texture image of the rail surface cannot obtain depth information and cannot accurately determine the degree of rail damage. Although the detection method based on the three-dimensional point cloud image of the rail can accurately determine the degree of rail damage, it cannot identify subtle defects such as rail cracks. Therefore, the fusion analysis of the two-dimensional grayscale data and three-dimensional point cloud data of the rail surface is helpful for a comprehensive evaluation of the rail surface state and reduces the false alarm rate and missed alarm rate of rail defect detection. The coordinate unification of the two-dimensional and three-dimensional data of the rail is the basis of the fusion analysis, which can ensure the accurate alignment of the 2D detection results with the 3D detection results.

[0040] Prior art 1 proposes a method for pedestrian detection in rail transit scenes based on the fusion of three-dimensional point clouds and images. This method first uses a deep learning model trained with a constructed rail transit scene pedestrian data set to perform pedestrian detection on three-dimensional point clouds and images respectively. On this basis, the rotation and translation matrix between the laser radar and the camera is solved according to the principle of consistency of the spatial position of the target in the three-dimensional point cloud and the image, and finally the three-dimensional point cloud target detection result is projected into the image coordinate system. The purpose of this method is to identify large targets by fusing information, so the fusion error requirements for two-dimensional images and three-dimensional point clouds are relatively low.

[0041] Prior art 2 proposes a calibration method based on the fusion of 3D point cloud and 2D image, using Zhang Zhengyou calibration method to obtain the intrinsic parameters of the camera, and then calculating the extrinsic parameters of the camera by selecting corresponding points and PNP algorithm to complete the calculation of the corresponding mapping relationship between 2D coordinates and 3D coordinates. However, the Astra Mini depth camera used in this method has certain limitations in depth perception range and cannot fully meet the depth range requirements in certain specific applications.

[0042] Existing technology 3 analyzes the data of laser radar to output the three-dimensional information of the object; based on the recognition result of step 2, the data of the monocular high-definition RGB camera is processed and analyzed to output the two-dimensional information of the recognized object; the two-dimensional information is mapped to the three-dimensional information, and the two-dimensional and three-dimensional information are fused by Kalman filtering, and the acceleration model such as three-dimensional Kalman is used to robustly obtain the real-time recognition information of the object. This method fuses the two-dimensional and three-dimensional information by Kalman filtering, which cannot meet the application scenario of high-speed and real-time railway.

[0043] The existing technology 4 requires the coordinated movement of the XYZ three-axis motion stage, and the calibration conditions are not suitable for the application scenario of dynamic detection of railway rails.

[0044] In summary, the traditional method has low accuracy, poor real-time performance, and complex operation, and is not suitable for application scenarios of dynamic detection on railway sites. Based on this, an embodiment of the present invention provides a joint calibration method for rail 2D sensors and 3D sensors, such as Figure 2 As shown, including:

[0045] Step 201: calibrating the internal parameters and external parameters of the 3D sensor, and determining the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system according to the external parameters of the 3D sensor;

[0046] Step 202: calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor;

[0047] Step 203: collecting a target plane target image of the rail through a 3D sensor, extracting feature points of the target plane target image, and determining a posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor according to the feature points, internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor;

[0048] Step 204: determining the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor;

[0049] Step 205: According to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system, and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system, the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system is determined through posture combination.

[0050] The method for jointly calibrating rail 2D sensors and 3D sensors proposed in the embodiment of the present invention represents the two-dimensional texture images and three-dimensional point cloud images of the rails from different sensors in the same coordinate system through joint calibration, thereby realizing the coordinate unification of the two-dimensional data and three-dimensional data of the rails. The method has the advantages of high precision, good real-time performance, and simple operation, which greatly reduces the difficulty of subsequent fusion analysis of rail surface defects.

[0051] When it comes to specific implementation, Figure 3 The schematic diagram of the joint calibration device of the rail 2D sensor and 3D sensor provided in the embodiment of the present invention. Among them, ①: 2D sensor, ②: 3D sensor, ③: 3D camera, ④: line laser, ⑤ 2D camera, ⑥: line laser, ⑦: rail, ⑧: plane target. 2D camera ⑤ and line laser ⑥ form a 2D sensor, 3D camera ③ and line laser ④ form a 3D sensor, and the plane target can be placed above the rail through an electric control displacement platform. Figure 4 FIG. 1 is a schematic diagram of the positional relationship between the planar target, the rail and the electric-controlled displacement platform provided in an embodiment of the present invention. Figure 4 As shown, the planar target is driven by the electric-controlled displacement platform to move along the extension direction of the rail.

[0052] The rail 2D and 3D sensor joint calibration equipment is mainly divided into three modules. They are 3D sensor calibration module, 2D sensor calibration module and 2D / 3D sensor external parameter calibration module. The coordinate systems involved are as follows: Figure 8 As shown, Figure 8 This is a schematic diagram of the external parameter calibration of the 2D sensor and the 3D sensor provided in an embodiment of the present invention, wherein the 3D camera coordinate system is denoted as ccs1, the 2D camera coordinate system is denoted as ccs2, tcs is the reference coordinate system of the 2D camera, and wcs is the reference coordinate system of the 3D camera, also known as the world coordinate system, which is the coordinate system in which the 2D sensor data and the 3D sensor data are finally normalized.

[0053] In one embodiment, calibrating the internal parameters and external parameters of the 3D sensor includes:

[0054] The 3D sensor collects images of the calibration plate at different positions and postures, and uses the images of the calibration plate at different positions and postures to calculate the internal parameters of the 3D sensor;

[0055] Through the intersection of multiple calibration plates at different heights with the laser plane, the light strip images of multiple intersection lines are collected, and the points on the multiple intersection lines are fitted to obtain the actual laser plane;

[0056] Collect calibration plate images corresponding to calibration plates at different heights;

[0057] The external parameters of the 3D sensor are calculated according to the actual laser plane, the light strip images of multiple intersection lines and the calibration plate images corresponding to the calibration plates at different heights.

[0058] During specific implementation, the 3D sensor calibration module is specifically used for: Step 1, 3D camera internal parameter calibration. Figure 5 FIG. 1 is a schematic diagram of calibrating the internal parameters of a 3D sensor provided in an embodiment of the present invention. Figure 5 As shown in Figure 2, a plane target is used to calibrate the internal parameters of the 3D camera. A set of calibration plate images in different poses are collected by the 3D camera, and the camera internal parameters are calculated using the calibration plate images in different poses. Step 2: 3D camera external parameter calibration. Figure 6 Schematic diagram of 3D sensor external parameter calibration provided in an embodiment of the present invention. The calibration method is as follows Figure 6 As shown in the figure, two calibration plates at different heights are used to intersect with the laser plane, and the light stripe images of the two intersection lines are collected, and the laser plane is fitted by the points on the intersection line. Subsequently, the laser is turned off, the exposure time of the 3D camera is increased, and the calibration plate images at high and low positions are collected. Finally, the external parameters ccs1Pwcs of the 3D camera are calculated using the calibration plate images at high and low positions and the light stripe images, which are recorded as the attitude transformation relationship between the 3D camera coordinate system and the world coordinate system.

[0059] In one embodiment, calibrating the external parameters of the 2D sensor includes:

[0060] The plane target images of the rails in different postures are collected by 2D sensors;

[0061] Determine the internal parameters and reference coordinate system of the 2D sensor based on the planar target images in different postures of the rail;

[0062] The external parameters of the 2D sensor are determined according to the internal parameters of the 2D sensor and the reference coordinate system.

[0063] In one embodiment, the planar target images of the rails in different postures are collected by a 2D sensor, including:

[0064] Control the planar target of the rail to move along the scanning direction of the 2D sensor;

[0065] Change the posture of the planar target and collect images of the planar target in different postures through a 2D sensor.

[0066] The 2D sensor calibration module is specifically used to place a planar target on an electric-controlled translation stage, which controls the planar target to move along the scanning direction. Figure 7 FIG. 2 is a schematic diagram of a 2D sensor calibration provided in an embodiment of the present invention, as shown in FIG. Figure 7As shown. Turn on the line laser and collect the plane target image through the 2D camera. Change the posture of the plane target on the electric control translation stage, and repeatedly collect the plane target images in different postures through the 2D camera. Finally, calibrate the 2D camera intrinsic parameters by the plane target images in different postures, and use the target coordinate system corresponding to the last plane target image as the reference coordinate system tcs of the 2D camera. The external parameters of the 2D camera are recorded as ccs2Ptcs.

[0067] In one embodiment, determining the internal parameters and reference coordinate system of the 2D sensor according to the planar target images in different postures of the rail includes:

[0068] Determine the internal parameters of the 2D sensor based on the planar target images in different postures of the rail;

[0069] A target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is determined, and the target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is used as a reference coordinate system of the 2D sensor.

[0070] In one embodiment, a target plane target image of a rail is collected by a 3D sensor, and feature points of the target plane target image are extracted, including:

[0071] After the 2D sensor acquires the plane target image for the last time, the 3D sensor acquires the plane target image at this moment as the target plane target image of the rail;

[0072] Extract feature points of the target plane image.

[0073] The 2D / 3D sensor external parameter calibration module is specifically used for: After the 2D camera has collected the planar target image for the last time, turn off the line laser in the 3D sensor, and let the 3D camera collect the planar target image at this time, which is recorded as image Image. Extract the feature points in the image Image, and combine the 3D camera internal parameters in module 1 to obtain the posture transformation relationship ccs1Ptcs between the 3D camera coordinate system ccs1 and the reference coordinate system tcs of the 2D camera. ccs1Htcs is recorded as the homogeneous transformation matrix corresponding to the posture transformation relationship ccs1Ptcs (subsequent naming is similar).

[0074] Then, the 2D sensor calibration module is used to obtain the posture transformation relationship ccs2Ptcs between the 2D camera coordinate system ccs3 and the reference coordinate system tcs of the 2D camera. Through posture combination, the posture transformation relationship ccs2Pccs1 between the 2D camera coordinate system ccs2 and the 3D camera coordinate system ccs1 is obtained.

[0075] Posture combination steps:

[0076] First, calculate the homogeneous transformation matrix corresponding to the posture transformation relationship ccs2Pccs1

[0077]

[0078] in, is the homogeneous transformation matrix corresponding to the posture transformation relationship ccs2Ptcs, It is the homogeneous transformation matrix corresponding to the posture transformation relationship ccs1Ptcs.

[0079] Then, the homogeneous transformation matrix Transform into posture transformation relationship ccs2Pccs1.

[0080] Finally, through posture combination of ccs2Pccs1 and ccs1Pwcs, the posture transformation relationship ccs2Pwcs between the 2D camera coordinate system ccs2 and the world coordinate system wcs is obtained, thus completing the 2D / 3D sensor external parameter calibration.

[0081] Posture combination steps:

[0082] First, calculate the homogeneous transformation matrix corresponding to the posture transformation relationship ccs2Pwcs

[0083]

[0084] in, It is the homogeneous transformation matrix corresponding to the posture transformation relationship ccs1Pwcs.

[0085] Then, the homogeneous transformation matrix Transformed into posture transformation relationship ccs2Pwcs.

[0086] The embodiment of the present invention also provides a rail 2D sensor and 3D sensor joint calibration device, as described in the following embodiment. Since the principle of solving the problem by the device is similar to the joint calibration method of the rail 2D sensor and 3D sensor, the implementation of the device can refer to the implementation of the joint calibration method of the rail 2D sensor and 3D sensor, and the repeated parts will not be repeated.

[0087] Fig. 9 Schematic diagram of a joint calibration device for a rail 2D sensor and a 3D sensor provided in an embodiment of the present invention, as shown in Fig. 9 As shown, the device comprises:

[0088] The 3D calibration module 901 is used to calibrate the internal and external parameters of the 3D sensor, and determine the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system according to the external parameters of the 3D sensor;

[0089] A 2D calibration module 902 is used to calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor;

[0090] The feature calibration module 903 is used to collect the target plane target image of the rail through the 3D sensor, extract the feature points of the target plane target image, and determine the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor according to the feature points, the internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor;

[0091] The joint calibration module 904 is used to determine the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor; determine the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system through posture combination according to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system.

[0092] In one embodiment, the 3D calibration module 901 is specifically used for:

[0093] The 3D sensor collects images of the calibration plate at different positions and postures, and uses the images of the calibration plate at different positions and postures to calculate the internal parameters of the 3D sensor;

[0094] Through the intersection of multiple calibration plates at different heights with the laser plane, the light strip images of multiple intersection lines are collected, and the points on the multiple intersection lines are fitted to obtain the actual laser plane;

[0095] Collect calibration plate images corresponding to calibration plates at different heights;

[0096] The external parameters of the 3D sensor are calculated according to the actual laser plane, the light strip images of multiple intersection lines and the calibration plate images corresponding to the calibration plates at different heights.

[0097] In one embodiment, the 2D calibration module 902 is specifically used for:

[0098] The plane target images of the rails in different postures are collected by 2D sensors;

[0099] Determine the internal parameters and reference coordinate system of the 2D sensor based on the planar target images in different postures of the rail;

[0100] The external parameters of the 2D sensor are determined according to the internal parameters of the 2D sensor and the reference coordinate system.

[0101] In one embodiment, the 2D calibration module 902 is specifically used for:

[0102] Control the planar target of the rail to move along the scanning direction of the 2D sensor;

[0103] Change the posture of the planar target and collect images of the planar target in different postures through a 2D sensor.

[0104] In one embodiment, the 2D calibration module 902 is specifically used for:

[0105] Determine the internal parameters of the 2D sensor based on the planar target images in different postures of the rail;

[0106] A target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is determined, and the target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is used as a reference coordinate system of the 2D sensor.

[0107] In one embodiment, the feature calibration module 903 is specifically used for:

[0108] After the 2D sensor acquires the plane target image for the last time, the 3D sensor acquires the plane target image at this moment as the target plane target image of the rail;

[0109] Extract feature points of the target plane image.

[0110] Based on the above invention concept, Fig.10 As shown, the present invention also proposes a computer device 1000, including a memory 1010, a processor 1020, and a computer program 1030 stored in the memory 1010 and executable on the processor 1020, wherein the processor 1020 implements the aforementioned joint calibration method of the rail 2D sensor and the 3D sensor when executing the computer program 1030.

[0111] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned joint calibration method of the rail 2D sensor and the 3D sensor is implemented.

[0112] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned rail 2D sensor and 3D sensor joint calibration method is implemented.

[0113] In summary, in an embodiment of the present invention, the internal parameters and external parameters of the 3D sensor are calibrated, and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system is determined according to the external parameters of the 3D sensor; the external parameters of the 2D sensor are calibrated to determine the reference coordinate system of the 2D sensor; the target plane target image of the rail is collected by the 3D sensor, the feature points of the target plane target image are extracted, and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor is determined according to the feature points, the internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor; the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system is determined through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor; the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system is determined through posture combination. In this way, the two-dimensional texture images and three-dimensional point cloud images of the rails from different sensors are represented in the same coordinate system through joint calibration, realizing the coordinate unification of the two-dimensional and three-dimensional data of the rails. This has the advantages of high precision, good real-time performance, and simple operation, which greatly reduces the difficulty of subsequent fusion analysis of rail surface defects.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A joint calibration method for rail 2D sensor and 3D sensor, characterized in that: include: Calibrate the internal and external parameters of the 3D sensor, and determine the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system based on the external parameters of the 3D sensor; Calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor; The target plane target image of the rail is collected by a 3D sensor, feature points of the target plane target image are extracted, and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor is determined according to the feature points, internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor; According to the external parameters of the 2D sensor and the attitude transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor, the attitude transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system is determined through attitude combination; According to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system, and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system, the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system is determined through posture combination.

2. The method according to claim 1, characterized in that Calibrate the internal and external parameters of the 3D sensor, including: The 3D sensor collects images of the calibration plate at different positions and postures, and uses the images of the calibration plate at different positions and postures to calculate the internal parameters of the 3D sensor; Through the intersection of multiple calibration plates at different heights with the laser plane, the light strip images of multiple intersection lines are collected, and the points on the multiple intersection lines are fitted to obtain the actual laser plane; Collect calibration plate images corresponding to calibration plates at different heights; The external parameters of the 3D sensor are calculated according to the actual laser plane, the light strip images of multiple intersection lines and the calibration plate images corresponding to the calibration plates at different heights.

3. The method according to claim 1, characterized in that Calibrate the external parameters of the 2D sensor, including: The plane target images of the rails in different postures are collected by 2D sensors; Determine the internal parameters and reference coordinate system of the 2D sensor based on the planar target images in different postures of the rail; The external parameters of the 2D sensor are determined according to the internal parameters of the 2D sensor and the reference coordinate system.

4. The method according to claim 3, characterized in that The 2D sensor collects the planar target images of the rail in different postures, including: Control the planar target of the rail to move along the scanning direction of the 2D sensor; Change the posture of the planar target and collect images of the planar target in different postures through a 2D sensor.

5. The method according to claim 4, characterized in that The internal parameters and reference coordinate system of the 2D sensor are determined based on the planar target images in different postures of the rail, including: Determine the internal parameters of the 2D sensor based on the planar target images in different postures of the rail; A target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is determined, and the target coordinate system corresponding to the planar target image acquired by the 2D sensor for the last time is used as a reference coordinate system of the 2D sensor.

6. The method according to claim 4, characterized in that The target plane target image of the rail is collected by a 3D sensor, and the feature points of the target plane target image are extracted, including: After the 2D sensor acquires the plane target image for the last time, the 3D sensor acquires the plane target image at this moment as the target plane target image of the rail; Extract feature points of the target plane image.

7. A joint calibration device for rail 2D sensor and 3D sensor, characterized in that: include: The 3D calibration module is used to calibrate the internal and external parameters of the 3D sensor and determine the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system according to the external parameters of the 3D sensor; 2D calibration module, used to calibrate the external parameters of the 2D sensor and determine the reference coordinate system of the 2D sensor; A feature calibration module is used to collect a target plane target image of the rail through a 3D sensor, extract feature points of the target plane target image, and determine the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor based on the feature points, internal parameters of the 3D sensor and the reference coordinate system of the 2D sensor; The joint calibration module is used to determine the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system through posture combination according to the external parameters of the 2D sensor and the posture transformation relationship between the 3D sensor coordinate system and the reference coordinate system of the 2D sensor; and to determine the posture transformation relationship between the 2D sensor coordinate system and the world coordinate system through posture combination according to the posture transformation relationship between the 2D sensor coordinate system and the 3D sensor coordinate system and the posture transformation relationship between the 3D sensor coordinate system and the world coordinate system.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images