Rail transit clearance detection method

Abstract

The invention provides a rail transit clearance detection method. The method comprises the following steps: processing acquired train operation environment image data to generate a depth map; extracting an orbit region in the depth map by using a semantic segmentation algorithm, and calculating three-dimensional space coordinates of an orbit according to the orbit region and the depth map; performing convolution optimal transformation processing on the three-dimensional space coordinates of the orbit by using the orbit prior constraint equation to form orbit three-dimensional point cloud data; on the basis of the three-dimensional point cloud data of the track and prior knowledge, track center lines of a straight line and a curve section are fitted through a method of minimizing the sum of squares of errors; extracting a multi-scene track plane from the track three-dimensional point cloud data through a variance vector minimization method; and a three-dimensional track clearance bounding box is constructed by combining the track center line and the track plane, so that three-dimensional detection of the vehicle clearance and the building clearance is realized. According to the method, the three-dimensional point cloud data and the clearance bounding box are constructed, so that rapid detection of the vehicle clearance and the building clearance is realized.

Description

Technical Field

[0001] The present invention relates to the field of clearance detection, and in particular to a rail transit clearance detection method. Background Art

[0002] The scope of railway operation is constantly expanding, and railway safety issues are becoming a focus of attention. Among them, railway clearance detection, as a key link to ensure the safety of train operation, has received widespread attention. The so-called railway clearance detection refers to ensuring that the space required for trains and their cargo is not disturbed during the operation of rail transit, including the safe distance on both sides and above the track to prevent the train from colliding with surrounding facilities, vegetation and other trains.

[0003] Traditional clearance detection methods mainly rely on manual inspection and regular mechanical scanning. These methods are not only time-consuming and labor-intensive, but also often have unsatisfactory detection accuracy. With the rapid development of computer vision and artificial intelligence technologies, automated detection technology has begun to play an important role in railway clearance detection. Through image processing and deep learning algorithms, railway clearance ranges can be efficiently and accurately identified and evaluated, thereby significantly improving detection efficiency and safety.

[0004] Current clearance detection technologies mainly rely on visual devices such as lidar, depth cameras and 3D laser devices, or their combination. However, the number of rail trains in my country is huge, and the large-scale application of these high-cost devices will bring huge economic costs. Therefore, it is particularly important to find a clearance detection method that is low-cost, fast, accurate and adaptable. Summary of the invention

[0005] The present invention provides a rail transit clearance detection method to effectively detect the rail transit clearance.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] According to one aspect of the present invention, a rail transit clearance detection method is provided, comprising:

[0008] The camera installed at the front end of the train collects image data of the operating environment in front of the train, processes the image data of the operating environment, and generates a depth map;

[0009] Extracting the track area in the depth map using a semantic segmentation algorithm, and calculating the three-dimensional spatial coordinates of the track according to the track area and the depth map;

[0010] Using the track prior constraint equation, the three-dimensional spatial coordinates of the track are subjected to convolution optimal transformation processing to form track three-dimensional point cloud data;

[0011] Based on the three-dimensional point cloud data of the track and prior knowledge, the track centerline of the straight line and the curve segment is fitted by minimizing the sum of square errors;

[0012] Extracting multi-scenario track planes from the track three-dimensional point cloud data by using a variance vector minimization method;

[0013] A three-dimensional track limit bounding box is constructed by combining the track centerline and track plane to realize three-dimensional detection of vehicle limits and building limits.

[0014] Preferably, the step of collecting the operating environment image data in front of the train by a camera installed at the front end of the train, processing the operating environment image data, and generating a depth map includes:

[0015] Set up a calibration board with a checkerboard pattern whose black-and-white spacing is known and can be photographed by the camera. Before installing the camera, use the calibration board to calibrate and calculate the camera's intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2. The intrinsic parameter matrix K includes the focal length (f x ,f y ) and the pixel position coordinates corresponding to the camera center (c x ,c y );

[0016] The calibrated camera is installed at the center of the front of the train. The train keeps moving forward. The camera collects the image data of the running environment in front of the train. The device coordinate system is set as follows: the coordinates of the camera device are (0,0,0), the train travel direction is the y-axis, the direction perpendicular to the track plane is the z-axis, and the x-axis is perpendicular to the plane yoz;

[0017] The monocular estimation algorithm based on convolutional neural network is selected to process the collected operating environment image data to obtain each frame of color image I rgb The corresponding initial depth map D pred ;

[0018] For each frame depth map D pred and color image I rgb , use the depth completion algorithm to generate the depth map D completed .

[0019] Preferably, the extracting the track area in the depth map by using a semantic segmentation algorithm, and calculating the three-dimensional spatial coordinates of the track according to the track area and the depth map, comprises:

[0020] The depth map D is segmented using a semantic segmentation algorithm. completed Process, segment the track area, and output the pixel coordinates (u, v) of the track area;

[0021] Obtain the camera's intrinsic parameter matrix and distortion coefficient, and use formula (1) to perform dedistortion processing on the pixel coordinates (u, v) of the track area to obtain the corrected pixel coordinates (u c ,v c );

[0022]

[0023] Where r is the radial distance, which represents the distance from the point to the center of the image, (u,v), (u c ,v c ) are the pixel coordinates before and after correction, (c x ,c y ) is the pixel position coordinate corresponding to the camera center, which is a parameter included in the internal parameter matrix K, k1, k2, k3 and p1, p2 are the radial distortion coefficient and tangential distortion coefficient;

[0024] From the depth map D completed Extract the depth value d corresponding to the pixel coordinates of the track area;

[0025] Using the corrected pixel coordinates (u c ,v c ) and the corresponding depth value d, and the three-dimensional spatial coordinates (X, Y, Z) of the track in the camera coordinate system are calculated by formula (2);

[0026]

[0027] Perform visual odometer calculation based on the operating environment image data by a feature point matching method to obtain the camera's external parameter matrix [R|t], including a rotation matrix R and a translation vector t;

[0028] The three-dimensional space coordinates (X, Y, Z) of the orbit in the world coordinate system are calculated by formula (3) using the camera external parameter matrix and the three-dimensional space coordinates (X, Y, Z) in the camera coordinate system. w ,Y w ,Z w );

[0029]

[0030] Preferably, the method of performing convolution optimal transformation processing on the three-dimensional space coordinates of the track using the track prior constraint equation to form the track three-dimensional point cloud data includes:

[0031] The three-dimensional space coordinate point data in the world coordinate system on both sides of the track are divided into left and right area voxels by voxelization method to obtain the left track voxel area and the right track voxel area, calculate the center of mass of the points in each voxel, and construct a specific track prior constraint equation by formula (4), filter out the voxel areas that do not meet the prior constraints, and perform convolution optimal transformation smoothing and alignment processing on the three-dimensional space coordinate point data to form track three-dimensional point cloud data;

[0032]

[0033] In the formula, Voxel i represents the spatial range of the i-th voxel, |V i | represents the number of points inside the voxel, represents the centroid of the left region of the i-th voxel, represents the distance calculation from the center of mass of the left region of the ith voxel to the center of mass of the right region of the ith voxel, W 轨 is the known track width and ε is the allowable error range.

[0034] Preferably, the method of fitting the track centerline of the straight line and the curve segment by minimizing the sum of square errors based on the track three-dimensional point cloud data and prior knowledge includes:

[0035] Using the track three-dimensional point cloud data to obtain point cloud coordinates and track prior knowledge, the track prior knowledge including track straightness, curvature radius and track spacing;

[0036] Use low-order polynomials to fit straight line segments, high-order polynomials or specific curve equations to fit curve segments, define the function E that minimizes the sum of squared errors by using formulas (5) and (6), determine the track parameters of the fitted straight line segments and curve segments by using formulas (7) and (8), substitute them into formulas (5) and (6) to fit the track center lines of the straight line and curve segments, and use an optimization algorithm to make the curvature of the fit close to the given a priori curvature radius, while ensuring that the spacing of the fitted curves is consistent with the track design standard;

[0037]

[0038] In the formula, t is a parameter, (x0, y0, z0) is the coordinate of the starting point of the straight line segment, a, b, c represent the direction vector components of the straight line in the x, y, z directions respectively;

[0039]

[0040] Where f(t), g(t), and h(t) are quadratic polynomials of the x, y, and z components;

[0041]

[0042] Where, t i is the parameter t value corresponding to each point, (x0, y0, z0) is the coordinate of the starting point of the line segment, a, b, c represent the direction vector components of the line in the x, y, z directions respectively, f(ti), g(ti), h(ti) are the parameter t i The three-dimensional coordinates of the corresponding fitting curve are shown below.

[0043] Preferably, the method of extracting multi-scene track planes from the track three-dimensional point cloud data by minimizing the variance vector includes:

[0044] Using the track 3D point cloud data P, the point cloud data is centralized and the covariance matrix is ​​calculated;

[0045] Based on the covariance matrix, the eigenvalue decomposition is performed using formula (9), and the normal vector n of the orbital plane is selected by minimizing the variance vector. The multi-scenario orbital plane equation is constructed using the normal vector n using formula (10);

[0046]

[0047] In the formula, λ i is the ith eigenvalue, v i is the eigenvector corresponding to the i-th eigenvalue, P i =(x i ,y i ,z i ) is the coordinate of the ith point, The formula for representing the centralized point cloud data is, represents a covariance matrix, and by performing eigenvalue decomposition on the covariance matrix, the eigenvector corresponding to the smallest eigenvalue is the normal vector n of the plane;

[0048] The normal vector n is used to construct the multi-scenario orbit plane equation through formula (10);

[0049]

[0050] In the formula, are the components of the normal vector n.

[0051] Preferably, the three-dimensional track limit bounding box is constructed by combining the track centerline and the track plane to realize the three-dimensional detection of vehicle limits and building limits, including:

[0052] Using the track centerline and track plane equations, the width and height of the track limit bounding box are defined as the safe width W of the vehicle limit and the building limit respectively. v , W b and height H v , H b, and supports dynamic adjustment of the width and height of the track bounding box;

[0053] Using the information of the width and height of the track limit bounding box, a limit frame of the vehicle limit and the building limit of the section perpendicular to the track centerline is constructed for any point c(t)=(x(t), y(t), z(t)) on the track centerline through formulas (11) and (12);

[0054]

[0055]

[0056] In the formula, Represents the two endpoints of the positive and negative semi-axes in the x-axis direction of the vehicle limit. Represents the two endpoints of the positive and negative semi-axes of the vehicle limit in the z-axis direction, x(t), z(t) are the coordinates of any point on the track centerline, W v , W b and H v , H b It is the safe width and height of vehicle limits and building limits;

[0057] The boundary frames of the vehicle limit and the building limit are used to construct a three-dimensional track limit bounding box along the track plane to achieve three-dimensional detection of the vehicle limit and the building limit.

[0058] According to another aspect of the present invention, there is provided a rail transit clearance detection device, comprising: a depth map generation module, a three-dimensional space coordinate calculation module, a track three-dimensional point cloud data acquisition module, a track centerline acquisition module and a track clearance detection module;

[0059] The depth map generation module is used to collect the operating environment image data in front of the train through a camera installed at the front end of the train, process the operating environment image data, and generate a depth map;

[0060] The three-dimensional space coordinate calculation module is used to extract the track area in the depth map using a semantic segmentation algorithm, and calculate the three-dimensional space coordinates of the track based on the track area and the depth map;

[0061] The track three-dimensional point cloud data acquisition module is used to perform convolution optimal transformation processing on the three-dimensional space coordinates of the track using the track prior constraint equation to form track three-dimensional point cloud data;

[0062] The track centerline acquisition module is used to fit the track centerline of the straight line and curve segments by minimizing the square error sum method based on the track three-dimensional point cloud data and prior knowledge;

[0063] The track plane acquisition module is used to extract multi-scene track planes from the track three-dimensional point cloud data by using a variance vector minimization method;

[0064] The track clearance detection module is used to construct a three-dimensional track clearance bounding box in combination with the track centerline and the track plane, so as to realize three-dimensional detection of vehicle clearance and building clearance.

[0065] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the rail transit clearance detection method is implemented.

[0066] A computer program product includes a computer program, and when the computer program is run on one or more processors, it is used to implement the rail transit clearance detection method.

[0067] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the rail transit clearance detection method.

[0068] It can be seen from the technical solutions provided by the embodiments of the present invention that the rail transit clearance detection method provided by the present invention is innovative in that it utilizes a monocular depth estimation algorithm and semantic segmentation technology, combined with track prior knowledge, and constructs three-dimensional point cloud data and bounding boxes to achieve rapid detection of vehicle limits and building limits. This method not only reduces the detection cost and simplifies the operation process, but also has extremely high adaptability and can achieve efficient detection in different scenarios and track types. Through self-learning and optimization algorithms, the system can continuously improve detection accuracy, ensure the safe operation of rail transit, and effectively reduce the occurrence of accidents.

[0069] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0071] Figure 1 A processing flow chart of a rail transit clearance detection method provided in an embodiment of the present invention.

[0072] Figure 2 A schematic diagram of a camera imaging coordinate transformation principle provided by an embodiment of the present invention.

[0073] Figure 3 A schematic diagram of a two-dimensional limit frame of a train and a building in a vertical section at any point on the center line provided by an embodiment of the present invention.

[0074] Figure 4 A schematic diagram of a three-dimensional rail vehicle clearance and building clearance bounding box provided in an embodiment of the present invention.

[0075] Figure 5 A structural diagram of a rail transit clearance detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0077] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0078] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0079] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0080] Example 1

[0081] The processing flow of a rail transit clearance detection method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the following processing steps are included:

[0082] Step S1: Before installing the camera, obtain the camera's intrinsic parameter matrix and distortion coefficient through calibration calculation on the calibration board, and install a single camera or a combination of calibrated monocular cameras, infrared cameras, depth cameras, industrial cameras, etc. at the front center of the train. The train keeps moving forward. The camera installed at the front of the train collects image data of the operating environment in front of the train. The device coordinate system is set as follows: the coordinates of the camera device are (0,0,0), the train travel direction is the y-axis, the direction perpendicular to the track plane is the z-axis, and the x-axis is perpendicular to the plane yoz.

[0083] A monocular depth estimation algorithm is used to process the collected image data of the operating environment in front of the train to generate a depth map.

[0084] Step S2: extract the track area in the depth map using a semantic segmentation algorithm, obtain the track depth value according to the track area and the depth map, and calculate the three-dimensional spatial coordinates of the track based on the track depth value.

[0085] Step S3: construct a track priori constraint equation, and use the track priori constraint equation to perform convolution optimal transformation processing on the three-dimensional space coordinates of the track to form track three-dimensional point cloud data.

[0086] Step S4: Based on the track 3D point cloud data and prior knowledge, the track centerline of the straight line and curve segments is fitted by minimizing the sum of square errors.

[0087] Step S5: extracting multi-scene track planes from the track three-dimensional point cloud data by using the variance vector minimization method.

[0088] Step S6: Construct a three-dimensional track limit bounding box in combination with the track centerline and the track plane to achieve three-dimensional detection of vehicle limits and building limits.

[0089] Furthermore, step S1 specifically includes the following steps:

[0090] S11: Before installing the camera, prepare a calibration plate with a checkerboard pattern whose black-white spacing is known and can be photographed by the camera for camera calibration;

[0091] S12: placing a calibration chessboard on a plane, ensuring that the surface of the chessboard is parallel or perpendicular to the plane, taking multiple images of the calibration plate at different angles and positions, and ensuring that the calibration plate covers all areas of the image;

[0092] S13: Use the findChessboardCorners computer vision algorithm in OpenCV to detect the corners in each calibration board image. For inaccurately detected corners, you can manually mark the corners.

[0093] S14: Using the detected corner points and the actual size of the calibration plate, apply the calibrateCamera camera calibration algorithm in OpenCV to calculate the camera intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2. The intrinsic parameter matrix K includes the focal length (f x ,f y ) and the pixel position coordinates corresponding to the camera center (c x ,c y );

[0094] S15: A single camera or a combination of calibrated monocular cameras, infrared cameras, depth cameras, industrial cameras, etc. is installed at the center of the front end of the train. The train keeps moving forward. The camera installed at the front end of the train collects image data of the running environment in front of the train. The device coordinate system is set as follows: the coordinate of the camera device is (0,0,0), the train travel direction is the y-axis, the direction perpendicular to the track plane is the z-axis, and the x-axis is perpendicular to the plane yoz;

[0095] S16: Select a monocular estimation algorithm based on a convolutional neural network with a U-Net structure to process the collected image data, and obtain a color image I for each frame. rgb The corresponding initial depth map D pred ;

[0096] S17: For each frame depth map D pred and color image I rgb , using the depth completion algorithm based on the Depth Completion Network to generate a high-precision depth map D completed .

[0097] Furthermore, step S2 specifically includes the following steps:

[0098] S21: Use the semantic segmentation algorithm of the U-Net structure to obtain the high-precision depth map D in step S1 completed Processing, accurately identifying and segmenting the track area, and outputting the pixel coordinates (u, v) of the track area;

[0099] S22: Obtain the camera intrinsic parameter matrix and distortion coefficient obtained in step S1, and use formula (1) to perform dedistortion processing on the track pixel coordinates to obtain the corrected pixel coordinates (u c ,v c );

[0100]

[0101] Where r is the radial distance, which represents the distance from the point to the center of the image, (u,v), (u c ,v c ) are the pixel coordinates before and after correction, (c x ,c y ) is the pixel position coordinate corresponding to the camera center, which is a parameter included in the internal parameter matrix K, k1, k2, k3 and p1, p2 are the radial distortion coefficient and tangential distortion coefficient;

[0102] S23: The high-precision depth map D obtained from step S1 completed , extract the depth value d corresponding to the track pixel coordinate;

[0103] S24: Using the corrected pixel coordinates (u c ,v c ) and the corresponding depth value d, and the three-dimensional spatial coordinates (X, Y, Z) of the track in the camera coordinate system are calculated by formula (2);

[0104]

[0105] S25: Using the image data collected in step S1, visual odometer calculation is performed by a feature point matching method to obtain the camera's external parameter matrix [R|t], including a rotation matrix R and a translation vector t;

[0106] S26: Using the camera external parameter matrix and the three-dimensional space coordinates (X, Y, Z) in the camera coordinate system obtained in S24, the three-dimensional space coordinates (X, Y, Z) in the world coordinate system are calculated by formula (3) w ,Y w ,Z w ). Figure 2 A camera imaging coordinate transformation principle diagram provided by an embodiment of the present invention;

[0107]

[0108] Furthermore, step S3 specifically includes the following steps:

[0109] S31: dividing the three-dimensional space coordinate point data on both sides of the track into left and right area voxels by voxelization method;

[0110] S32: Calculate the centroid of each point in each voxel, construct a specific orbit prior constraint equation through formula (4), filter out the voxel area that does not meet the prior constraint, and use a 3×3 Gaussian convolution kernel to perform spatial domain filtering convolution smoothing processing on the three-dimensional space coordinates. Iteratively perform each point in the left orbit voxel area that does not meet the prior constraint, find the nearest point in the point cloud of the right orbit voxel area as its corresponding point, calculate the square sum error of the Euclidean distance between all corresponding point pairs, and the rotation and translation transformation matrix to minimize the square sum error. Apply the transformation matrix to a point cloud and update its position to minimize the distance between the point clouds to achieve optimal transformation fine alignment processing, thereby forming high-precision orbit three-dimensional point cloud data;

[0111]

[0112] In the formula, Voxel i represents the spatial range of the i-th voxel, |V i | represents the number of points inside the voxel, represents the centroid of the left region of the i-th voxel, represents the distance calculation from the center of mass of the left region of the ith voxel to the center of mass of the right region of the ith voxel, W 轨 is the known track width and ε is the allowable error range.

[0113] Furthermore, step S4 specifically includes the following steps:

[0114] S41: using the track three-dimensional point cloud data obtained in step S3, obtaining point cloud coordinates and track prior knowledge, where the track prior knowledge includes track straightness, curvature radius and track spacing;

[0115] S42: The track centerline in three-dimensional space needs to be described by three coordinates x, y, and z to describe its geometric shape. For the straight line segment, in the process of fitting the track centerline, the track is divided into straight line segments and curved segments for separate processing. The straight line segment track is fitted using the straight line equation or a lower-order first-order or second-order polynomial using formula (5), and the curved segment track is fitted using a higher-order polynomial, arc or spline curve equation using formula (6);

[0116]

[0117] In the formula, t is a parameter, (x0, y0, z0) is the coordinate of the starting point of the straight line segment, a, b, c represent the direction vector components of the straight line in the x, y, z directions respectively;

[0118]

[0119] Where f(t), g(t), and h(t) are quadratic polynomials of the x, y, and z components;

[0120] S43: define a function E that minimizes the sum of squared errors, determine the track parameters of the fitted straight line segment and the curved line segment through formulas (7) and (8), and fit the track center lines of the straight line and the curved line segment;

[0121]

[0122] Where, t i is the parameter t value corresponding to each point, (x0, y0, z0) is the coordinate of the starting point of the line segment, a, b, c represent the direction vector components of the line in the x, y, z directions respectively, f(t i ),g(t i ),h(t i ) is the parameter t i The three-dimensional space coordinates of the corresponding fitting curve;

[0123] For linear straight line fitting problems, the normal equations of linear algebra are used to solve them. For nonlinear curve fitting problems, the gradient descent method or Newton's method is used for iterative optimization. By using the optimization algorithm, the curvature of the fitting is close to the given prior curvature radius, while ensuring that the spacing of the fitting curve is consistent with the track design standards.

[0124] Furthermore, step S5 specifically includes the following steps:

[0125] S51: Using the track three-dimensional point cloud data P obtained in step S3, centralize the point cloud data and calculate the covariance matrix;

[0126] S52: Based on the covariance matrix, perform eigenvalue decomposition using formula (9), and select the normal vector n of the orbital plane by minimizing the variance vector;

[0127]

[0128] In the formula, λ i is the ith eigenvalue, v i is the eigenvector corresponding to the i-th eigenvalue, P i =(x i ,y i ,z i ) is the coordinate of the ith point, The formula for representing the centralized point cloud data is, represents the covariance matrix;

[0129] Perform eigenvalue decomposition on the covariance matrix, and use the eigenvector corresponding to the smallest eigenvalue obtained by decomposition as the normal vector n, v of the plane iRepresents the eigenvector corresponding to the i-th eigenvalue. The variance vector is minimized by performing eigenvalue decomposition on the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is the normal vector n of the plane. For example, if there are 5 results for the i-th eigenvalue, there must be a smallest eigenvalue, such as the third one. Then the eigenvector v3 corresponding to this smallest eigenvalue is the normal vector n we are looking for.

[0130] S53: construct the multi-scenario orbit plane equation using the normal vector n through formula (10);

[0131]

[0132] In the formula, are the components of the normal vector n.

[0133] Furthermore, S6 specifically includes the following steps:

[0134] S61: Using the track centerline obtained in step S4 and the track plane equation constructed in step S5, define the width and height of the bounding box as the safety width W of the vehicle limit and the building limit respectively. v , W b and height H v , H b , and supports dynamic adjustment to adapt to different track types and scene requirements;

[0135] S62: Using the information obtained in S61, the vehicle limit and building limit boundary frame of the section perpendicular to the center line of the track are constructed for any point c(t)=(x(t), y(t), z(t)) on the center line of the track through formulas (11) and (12). Figure 3 A schematic diagram of a boundary frame of vehicle limits and building limits of a vertical section at any point on the center line of a line provided in an embodiment of the present invention.

[0136]

[0137] In the formula, Represents the two endpoints of the positive and negative semi-axes in the x-axis direction of the vehicle limit. Represents the two endpoints of the positive and negative semi-axes of the vehicle limit in the z-axis direction, x(t), z(t) are the coordinates of any point on the track centerline, W v , W b and H v , H b It is the safe width and height of vehicle limits and building limits;

[0138] Constructing a three-dimensional track limit bounding box along the track plane with the bounding boxes of the vehicle limit and the building limit to achieve three-dimensional detection of the vehicle limit and the building limit;

[0139] S63: Figure 4 A schematic diagram of a three-dimensional rail vehicle limit and building limit bounding box provided by an embodiment of the present invention. The two-dimensional limit frame constructed in S62 is used to construct a three-dimensional rail limit bounding box along the rail plane to achieve three-dimensional detection of vehicle limits and building limits. Railway limits include locomotive vehicle limits and building limits.

[0140] Example 2

[0141] The structure of a rail transit clearance detection device provided by an embodiment of the present invention is as follows Figure 5 As shown, the following modules are included: a depth map generation module 1, a three-dimensional space coordinate calculation module 2, a track three-dimensional point cloud data acquisition module 3, a track centerline acquisition module 4, a track plane acquisition module 5 and a track limit detection module 5.

[0142] Specifically, the depth map generation module 1 is used to set a calibration plate with a checkerboard pattern whose black-and-white spacing is known and can be photographed by the camera. Before installing the camera, the calibration plate is used to calibrate and calculate the camera intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2. The intrinsic parameter matrix K includes the focal length (f x ,f y ) and the pixel position coordinates corresponding to the camera center (c x ,c y );

[0143] The calibrated camera is installed at the center of the front of the train. The train keeps moving forward. The camera collects the image data of the running environment in front of the train. The device coordinate system is set as follows: the coordinates of the camera device are (0,0,0), the train travel direction is the y-axis, the direction perpendicular to the track plane is the z-axis, and the x-axis is perpendicular to the plane yoz;

[0144] The monocular estimation algorithm based on convolutional neural network is selected to process the collected operating environment image data to obtain each frame of color image I rgb The corresponding initial depth map D pred ;

[0145] For each frame depth map D pred and color image I rgb , use the depth completion algorithm to generate the depth map D completed .

[0146] Specifically, the three-dimensional space coordinate calculation module 2 is used to use a semantic segmentation algorithm to calculate the depth map D completed Process, segment the track area, and output the pixel coordinates (u, v) of the track area;

[0147] Obtain the camera's intrinsic parameter matrix and distortion coefficient, and use formula (1) to perform dedistortion processing on the pixel coordinates (u, v) of the track area to obtain the corrected pixel coordinates (u c ,v c );

[0148]

[0149] Where r is the radial distance, which represents the distance from the point to the center of the image, (u,v), (u c ,v c ) are the pixel coordinates before and after correction, (c x ,c y ) is the pixel position coordinate corresponding to the camera center, which is a parameter included in the internal parameter matrix K, k1, k2, k3 and p1, p2 are the radial distortion coefficient and tangential distortion coefficient;

[0150] From the depth map D completed Extract the depth value d corresponding to the pixel coordinates of the track area;

[0151] Using the corrected pixel coordinates (u c ,v c ) and the corresponding depth value d, and the three-dimensional spatial coordinates (X, Y, Z) of the track in the camera coordinate system are calculated by formula (2);

[0152]

[0153] Perform visual odometer calculation based on the operating environment image data by a feature point matching method to obtain the camera's external parameter matrix [R|t], including a rotation matrix R and a translation vector t;

[0154] The three-dimensional space coordinates (X, Y, Z) of the orbit in the world coordinate system are calculated by formula (3) using the camera external parameter matrix and the three-dimensional space coordinates (X, Y, Z) in the camera coordinate system. w ,Y w ,Z w );

[0155]

[0156] Specifically, the track three-dimensional point cloud data acquisition module 3 is used to divide the three-dimensional space coordinate point data in the world coordinate system on both sides of the track into left and right area voxels by voxelization, obtain the left track voxel area and the right track voxel area, calculate the center of mass of the points in each voxel, construct a specific track prior constraint equation by formula (4), filter out the voxel areas that do not meet the prior constraints, perform convolution optimal transformation smoothing and alignment processing on the three-dimensional space coordinate point data, and form track three-dimensional point cloud data;

[0157]

[0158] In the formula, Voxel i represents the spatial range of the i-th voxel, |V i | represents the number of points inside the voxel, represents the centroid of the left region of the i-th voxel, represents the distance calculation from the center of mass of the left region of the ith voxel to the center of mass of the right region of the ith voxel, W 轨 is the known track width and ε is the allowable error range.

[0159] Specifically, the track centerline acquisition module 4 is used to obtain point cloud coordinates and track prior knowledge using the track three-dimensional point cloud data, where the track prior knowledge includes track straightness, curvature radius and track spacing;

[0160] Use low-order polynomials to fit straight line segments, high-order polynomials or specific curve equations to fit curve segments, define the function E that minimizes the sum of squared errors by using formulas (5) and (6), determine the track parameters of the fitted straight line segments and curve segments by using formulas (7) and (8), substitute them into formulas (5) and (6) to fit the track center lines of the straight line and curve segments, and use an optimization algorithm to make the curvature of the fit close to the given a priori curvature radius, while ensuring that the spacing of the fitted curves is consistent with the track design standard;

[0161]

[0162] In the formula, t is a parameter, (x0, y0, z0) is the coordinate of the starting point of the straight line segment, a, b, c represent the direction vector components of the straight line in the x, y, z directions respectively;

[0163]

[0164] Where f(t), g(t), and h(t) are quadratic polynomials of the x, y, and z components;

[0165]

[0166] Where, t i is the parameter t value corresponding to each point, (x0, y0, z0) is the coordinate of the starting point of the line segment, a, b, c represent the direction vector components of the line in the x, y, z directions respectively, f(ti), g(ti), h(ti) are the parameter t i Specifically, the track plane acquisition module 5 is used to use the track three-dimensional point cloud data P to centralize the point cloud data and calculate the covariance matrix;

[0167] Based on the covariance matrix, the eigenvalue decomposition is performed using formula (9), and the normal vector n of the orbital plane is selected by minimizing the variance vector. The multi-scenario orbital plane equation is constructed using the normal vector n using formula (10);

[0168]

[0169] In the formula, λ i is the ith eigenvalue, v i is the eigenvector corresponding to the i-th eigenvalue, P i =(x i ,y i ,z i ) is the coordinate of the ith point, The formula for centralizing point cloud data is: represents a covariance matrix, and by performing eigenvalue decomposition on the covariance matrix, the eigenvector corresponding to the smallest eigenvalue is the normal vector n of the plane;

[0170] The normal vector n is used to construct the multi-scenario orbit plane equation through formula (10);

[0171]

[0172] In the formula, are the components of the normal vector n.

[0173] Specifically, the track limit detection module 6 is used to define the track limit bounding box with a width and a height as the safety width W of the vehicle limit and the building limit respectively by using the track centerline and the track plane equation. v , W b and height H v , H b , and supports dynamic adjustment of the width and height of the track bounding box;

[0174] Using the information of the width and height of the track limit bounding box, a limit frame of the vehicle limit and the building limit of the section perpendicular to the track centerline is constructed for any point c(t)=(x(t), y(t), z(t)) on the track centerline through formulas (11) and (12);

[0175]

[0176]

[0177] In the formula, Represents the two endpoints of the positive and negative semi-axes in the x-axis direction of the vehicle limit. Represents the two endpoints of the positive and negative semi-axes of the vehicle limit in the z-axis direction, x(t), z(t) are the coordinates of any point on the track centerline, Wv , W b and H v , H b It is the safe width and height of vehicle limits and building limits;

[0178] The boundary frames of the vehicle limit and the building limit are used to construct a three-dimensional track limit bounding box along the track plane to achieve three-dimensional detection of the vehicle limit and the building limit.

[0179] The specific process of using the device of the present invention to perform rail transit clearance detection is similar to the aforementioned method embodiment and will not be repeated here.

[0180] Example 3

[0181] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the rail transit clearance detection method described in Example 1 is implemented.

[0182] A computer program product includes a computer program, which, when running on one or more processors, is used to implement the rail transit clearance detection method described in Example 1.

[0183] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the rail transit clearance detection method described in Example 1.

[0184] To sum up, the present invention adopts industrial cameras for data collection, which greatly reduces the hardware cost compared with the traditional detection method that relies on high-cost equipment such as lidar, depth cameras, etc., and is suitable for the promotion and application of large-scale rail transit systems, thereby significantly reducing the investment in the overall detection system; secondly, by utilizing advanced computer vision technology and artificial intelligence algorithms, the present invention can realize real-time and automated track limit detection, and adopts monocular estimation algorithms and semantic segmentation algorithms in deep learning to quickly process and analyze large amounts of image data, effectively improving detection efficiency and reducing manual operation time and costs; in addition, the present invention combines depth estimation and semantic segmentation technology to accurately identify track areas and calculate the three-dimensional spatial coordinates of the track, and provides high-precision track model reconstruction through fine point cloud data alignment and smoothing processing, ensuring the accuracy of limit detection and the safety of train operation.

[0185] The method of the present invention is highly adaptable and suitable for various detection environments and track types. It can maintain high detection accuracy in complex environments and can be adjusted according to different needs to adapt to a variety of track design parameters and limit standards. In addition, the method simplifies the complexity of equipment calibration and maintenance, and the detection task can be completed using a single or combined camera. The flexible camera calibration and installation process makes it adaptable to a variety of train model configurations and easy to operate. Finally, with the help of deep learning and machine learning algorithms, the present invention has self-learning and self-adaptation capabilities. The system can continuously optimize the algorithm according to the collected data, improve the recognition and adaptability to various scenarios, ensure the detection accuracy and efficiency in long-term use, effectively prevent the collision between trains and obstacles, ensure operation safety, and reduce accidents.

[0186] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0187] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0188] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0189] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A rail transit clearance detection method, characterized in that: include: The camera installed at the front end of the train collects image data of the operating environment in front of the train, processes the image data of the operating environment, and generates a depth map; Extracting the track area in the depth map using a semantic segmentation algorithm, and calculating the three-dimensional spatial coordinates of the track according to the track area and the depth map; Using the track prior constraint equation, the three-dimensional spatial coordinates of the track are subjected to convolution optimal transformation processing to form track three-dimensional point cloud data; Based on the three-dimensional point cloud data of the track and prior knowledge, the track centerline of the straight line and the curve segment is fitted by minimizing the sum of square errors; Extracting multi-scenario track planes from the track three-dimensional point cloud data by using a variance vector minimization method; A three-dimensional track limit bounding box is constructed by combining the track centerline and track plane to realize three-dimensional detection of vehicle limits and building limits.

2. The method according to claim 1, characterized in that: The method of collecting the operating environment image data in front of the train by means of a camera installed at the front end of the train, processing the operating environment image data, and generating a depth map includes: Set up a calibration plate with a checkerboard pattern whose black-and-white spacing is known and can be captured by the camera. Before installing the camera, use the calibration plate to calibrate and calculate the camera's intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2. The intrinsic parameter matrix K includes the focal length (fx, fy) and the pixel position coordinates (cx, cy) corresponding to the camera center. The calibrated camera is installed at the center of the front of the train. The train keeps moving forward. The camera collects the image data of the running environment in front of the train. The device coordinate system is set as follows: the coordinates of the camera device are (0,0,0), the train travel direction is the y-axis, the direction perpendicular to the track plane is the z-axis, and the x-axis is perpendicular to the plane yoz; The monocular estimation algorithm based on convolutional neural network is selected to process the collected operating environment image data to obtain each frame of color image I rgb The corresponding initial depth map D pred ; For each frame depth map D pred and color image I rgb , use the depth completion algorithm to generate the depth map D completed .

3. The method according to claim 2, characterized in that The method of extracting the track area in the depth map by using a semantic segmentation algorithm and calculating the three-dimensional spatial coordinates of the track according to the track area and the depth map includes: The depth map D is segmented using a semantic segmentation algorithm. completed Process, segment the track area, and output the pixel coordinates (u, v) of the track area; Obtain the camera's intrinsic parameter matrix and distortion coefficient, and use formula (1) to perform dedistortion processing on the pixel coordinates (u, v) of the track area to obtain the corrected pixel coordinates (u c ,v c ); Where r is the radial distance, which represents the distance from the point to the center of the image, (u,v), (u c ,v c ) are the pixel coordinates before and after correction, (c x ,c y ) is the pixel position coordinate corresponding to the camera center, and is a parameter included in the intrinsic parameter matrix K. k1, k2, k3 and p1, p2 are the radial distortion coefficient and tangential distortion coefficient; From the depth map D completed Extract the depth value d corresponding to the pixel coordinates of the track area; Using the corrected pixel coordinates (u c ,v c ) and the corresponding depth value d, and the three-dimensional spatial coordinates (X, Y, Z) of the track in the camera coordinate system are calculated by formula (2); Perform visual odometer calculation based on the operating environment image data by a feature point matching method to obtain the camera's external parameter matrix [R|t], including a rotation matrix R and a translation vector t; The three-dimensional space coordinates (X, Y, Z) of the orbit in the world coordinate system are calculated by formula (3) using the camera external parameter matrix and the three-dimensional space coordinates (X, Y, Z) in the camera coordinate system. w ,Y w ,Z w ); 4. The method according to claim 3, characterized in that: The method of using the track priori constraint equation to perform convolution optimal transformation processing on the three-dimensional space coordinates of the track to form track three-dimensional point cloud data includes: The three-dimensional space coordinate point data in the world coordinate system on both sides of the track are divided into left and right area voxels by voxelization method to obtain the left track voxel area and the right track voxel area, calculate the center of mass of the points in each voxel, and construct a specific track prior constraint equation by formula (4), filter out the voxel areas that do not meet the prior constraints, and perform convolution optimal transformation smoothing and alignment processing on the three-dimensional space coordinate point data to form track three-dimensional point cloud data; In the formula, Voxel i represents the spatial range of the i-th voxel, |V i | represents the number of points inside the voxel, represents the centroid of the left region of the i-th voxel, represents the distance calculation from the center of mass of the left region of the ith voxel to the center of mass of the right region of the ith voxel, W 轨 is the known track width and ε is the allowable error range.

5. The method according to claim 4, characterized in that The method of fitting the track centerline of the straight line and the curve segment based on the track three-dimensional point cloud data and prior knowledge by minimizing the sum of square errors includes: Using the track three-dimensional point cloud data to obtain point cloud coordinates and track prior knowledge, the track prior knowledge including track straightness, curvature radius and track spacing; Use low-order polynomials to fit straight line segments, high-order polynomials or specific curve equations to fit curve segments, define the function E that minimizes the sum of squared errors by using formulas (5) and (6), determine the track parameters of the fitted straight line segments and curve segments by using formulas (7) and (8), substitute them into formulas (5) and (6) to fit the track center lines of the straight line and curve segments, and use an optimization algorithm to make the curvature of the fit close to the given a priori curvature radius, while ensuring that the spacing of the fitted curves is consistent with the track design standard; In the formula, t is a parameter, (x0, y0, z0) is the coordinate of the starting point of the straight line segment, a, b, c represent the direction vector components of the straight line in the x, y, z directions respectively; Where f(t), g(t), and h(t) are quadratic polynomials of the x, y, and z components; Where, t i is the parameter t value corresponding to each point, (x0, y0, z0) is the coordinate of the starting point of the line segment, a, b, c represent the direction vector components of the line in the x, y, z directions respectively, f(ti), g(ti), h(ti) are the parameter t i The three-dimensional coordinates of the corresponding fitting curve are shown below.

6. The method according to claim 5, characterized in that The method of extracting a multi-scene track plane from the track three-dimensional point cloud data by minimizing the variance vector includes: Using the track 3D point cloud data P, the point cloud data is centralized and the covariance matrix is ​​calculated; Based on the covariance matrix, the eigenvalue decomposition is performed using formula (9), and the normal vector n of the orbital plane is selected by minimizing the variance vector. The multi-scenario orbital plane equation is constructed using the normal vector n using formula (10); In the formula, λ i is the i-th eigenvalue, vi is the eigenvector corresponding to the i-th eigenvalue, P i =(x i ,y i ,z i ) is the coordinate of the ith point, The formula for representing the centralized point cloud data is, represents a covariance matrix, and by performing eigenvalue decomposition on the covariance matrix, the eigenvector corresponding to the smallest eigenvalue is the normal vector n of the plane; The normal vector n is used to construct the multi-scenario orbit plane equation through formula (10); In the formula, are the components of the normal vector n.

7. The method according to claim 6, characterized in that The three-dimensional track limit bounding box is constructed by combining the track centerline and the track plane to realize the three-dimensional detection of vehicle limits and building limits, including: Using the track centerline and track plane equations, the width and height of the track limit bounding box are defined as the safe width W of the vehicle limit and the building limit respectively. v , W b and height H v , H b , and supports dynamic adjustment of the width and height of the track bounding box; Using the information of the width and height of the track limit bounding box, a limit frame of the vehicle limit and the building limit of the section perpendicular to the track centerline is constructed for any point c(t)=(x(t), y(t), z(t)) on the track centerline through formulas (11) and (12); In the formula, Represents the two endpoints of the positive and negative semi-axes in the x-axis direction of the vehicle limit. Represents the two endpoints of the positive and negative semi-axes of the z-axis direction of the vehicle limit, x(t), z(t) are the coordinates of any point on the track centerline, W v , W b and H v , H b It is the safe width and height of vehicle limits and building limits; The boundary frames of the vehicle limit and the building limit are used to construct a three-dimensional track limit bounding box along the track plane to achieve three-dimensional detection of the vehicle limit and the building limit.

8. A rail transit clearance detection device, characterized in that: include: Depth map generation module, three-dimensional space coordinate calculation module, track three-dimensional point cloud data acquisition module, track centerline acquisition module and track limit detection module; The depth map generation module is used to collect the operating environment image data in front of the train through a camera installed at the front end of the train, process the operating environment image data, and generate a depth map; The three-dimensional space coordinate calculation module is used to extract the track area in the depth map using a semantic segmentation algorithm, and calculate the three-dimensional space coordinates of the track based on the track area and the depth map; The track three-dimensional point cloud data acquisition module is used to perform convolution optimal transformation processing on the three-dimensional space coordinates of the track using the track prior constraint equation to form track three-dimensional point cloud data; The track centerline acquisition module is used to fit the track centerline of the straight line and curve segments by minimizing the square error sum method based on the track three-dimensional point cloud data and prior knowledge; The track plane acquisition module is used to extract multi-scene track planes from the track three-dimensional point cloud data by using a variance vector minimization method; The track clearance detection module is used to construct a three-dimensional track clearance bounding box in combination with the track centerline and the track plane, so as to realize three-dimensional detection of vehicle clearance and building clearance.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the rail transit clearance detection method as described in any one of claims 1-7 is implemented.

10. A computer program product, characterized in that It comprises a computer program, which, when running on one or more processors, is used to implement the rail transit clearance detection method as described in any one of claims 1 to 7. An electronic device, comprising: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the rail transit clearance detection method as described in any one of claims 1 to 7.

Images