Intelligent identification method of subway track area facilities and equipment based on point cloud and image map

Abstract

The present invention discloses a method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map, comprising: S1, generating a point cloud layout map I according to point cloud data P0 and image data I0 respectively; pt And image two-dimensional layout diagram I I2 S2, according to the tunnel profile section will be compressed image two-dimensional layout diagram I ' I2 Mapping to the three-dimensional layout of the image I' I3 ; S3, according to the corresponding relationship of the pipe segment, the three-dimensional layout map of the pipe segment and the pipe segment point cloud are registered; S4, the point cloud feature tensor T is extracted Pt , extract the image feature tensor T I ; S5, point cloud feature tensor T Pt and the image feature tensor T I Fusion into point graph feature tensor T PtI The MLP network is used to classify the points and equipment, and the prediction score function L is used to optimize the model to realize intelligent identification of facilities and equipment in the track area. The advantages of the present invention are: by combining point cloud features with image map features, accurate identification of facilities and equipment of different sizes can be achieved in a dark tunnel environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital asset construction of subway track area facilities and equipment, and specifically relates to an intelligent identification method for subway track area facilities and equipment based on point cloud and image map. Background Art

[0002] There are a large number of facilities and equipment in the subway track area, which are highly complex. Accurate identification of these facilities is crucial for the daily operation and maintenance of rail transit and equipment asset management. Existing detection methods mostly use image data to directly detect facilities and equipment, which is not reliable in dark light environments in tunnels.

[0003] Patent CN117893837A discloses a method, device, terminal device and medium for railway infrastructure point cloud recognition, which uses the interpolation enhancement of the original point cloud to input the facility equipment point cloud recognition model to obtain the facility equipment recognition result. This method uses a single point cloud recognition method to make it difficult to recognize the details of small target facilities and equipment, and the recognition accuracy is not high. Patent CN115830416A discloses a method and system for highway equipment identification and early warning, which uses image data to identify the facilities and equipment of the target section of the highway. This method uses an image recognition detection method that makes it difficult to achieve facility equipment recognition in a dark light environment in a subway tunnel.

[0004] In summary, there is an urgent need for an intelligent identification method for subway track area facilities and equipment based on point cloud and image map. Summary of the invention

[0005] The purpose of the present invention is to provide a method for intelligent identification of subway track area facilities and equipment based on point cloud and image map in response to the above-mentioned deficiencies in the prior art, which can realize accurate identification of facilities and equipment of different sizes in a dark tunnel environment by combining point cloud features with image map features.

[0006] The purpose of the present invention is achieved by the following technical solutions:

[0007] A method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map, the method comprising the following steps:

[0008] S1, based on the two-dimensional laser scanner and linear array camera mounted on the track area inspection vehicle, respectively realize the point cloud data P 0 and image data I 0 Collect and according to the point cloud data P 0 and the image data I 0 Generate point cloud layout diagram I pt And image two-dimensional layout diagram I I2 ;

[0009] S2, the image is two-dimensionally spread out in Figure I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 The height and the point cloud layout diagram I pt The height is kept consistent, and the compressed image two-dimensional layout diagram I' is displayed according to the tunnel profile section. I2 Mapping to the three-dimensional layout of the image I' I3 ;

[0010] S3, respectively, the compressed image two-dimensional layout diagram I ' I2 And the point cloud layout diagram I pt Perform annular seam identification to obtain a segment three-dimensional layout diagram and a segment point cloud, and align the segment three-dimensional layout diagram and the segment point cloud according to the segment correspondence relationship;

[0011] S4, inputting the segment point cloud into a PointTransformer network composed of a feature extraction layer and a feature fusion layer to extract the point cloud feature tensor T Pt The three-dimensional layout of the pipe segment is input into the ResNet network composed of residual blocks to extract the image feature tensor T I ;

[0012] S5. The point cloud feature tensor T Pt and the image feature tensor T I Fusion into point graph feature tensor T PtI , and classified through the MLP network; the prediction scoring function L is used to optimize the model to achieve intelligent identification of facilities and equipment in the track area.

[0013] The specific steps of step S1 are as follows:

[0014] (1) Point cloud layout diagram I pt Generate: the point cloud layout diagram I pt It is composed of a plurality of single-ring point cloud images, wherein the single-ring point cloud images are obtained by calculating and sorting the angles of the single-ring point clouds relative to the center point of the two-dimensional laser scanner;

[0015] (2) The two-dimensional layout diagram of the image I I2 Generation: It is generated by matching and stitching SIFT features between images from different adjacent cameras and between adjacent images from the same camera.

[0016] The specific steps of step S2 are as follows:

[0017] (1) Use bilinear interpolation to transform the image into a two-dimensional layout diagram I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 The height and the point cloud layout diagram Ipt Highly consistent;

[0018] (2) The tunnel profile section is sampled at equal angles, and the sampling coordinate value is L ij ;

[0019] (3) According to the coordinate value L ij and the corresponding image information I' I2ij , and obtain the three-dimensional layout diagram I' of the image I3 .

[0020] The specific steps of step S3 are as follows:

[0021] (1) Count the point cloud layout diagram I respectively pt and the compressed two-dimensional layout diagram of the image I' I2 The grayscale cumulative value G of each column of pixels ptj and G I ' 2j ;

[0022] (2) Through the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The corresponding thresholds H pt and H' I2 Control and traverse the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 , and respectively perform the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The segment annular seam identification, the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The grayscale cumulative value of meets the following conditions, which is the segment annular joint:

[0023]

[0024] Where: and They are respectively the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The average value of the grayscale accumulation value in the [jn, j+n] column, W pt and W' I2 Respectively represent the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The image width;

[0025] (3) According to the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I'I2 Segment Circumferential Seam Identification Index pt and Index' I2 , the point cloud data P 0 and the three-dimensional layout diagram I of the image I ' 3 Split and match segment by segment ring.

[0026] The specific steps of step S4 are as follows:

[0027] (1) The point cloud data P 0 The feature extraction layer is formed by sequentially performing farthest point sampling, KNN grouping, the MLP network and the residual Transformer module;

[0028] (2) The feature fusion layer adopts a U-net structure, and the features of the same layer are processed by a fully connected network and concatenated with the features of trilinear interpolation, and then learned through the residual Transformer module;

[0029] (3) The three-dimensional layout diagram of the image I I ' 3 Information can be directly transferred through the cross-layer connection of the residual block to learn and extract deep features of the image. The principle of deep feature extraction is as follows:

[0030]

[0031] Where: x is the initial input of the network, F l (x l-1 ,w l ) represents the nonlinear transformation in the residual block of the lth layer, w l is the parameter of layer l, x l-1 is the input feature of the l-1th layer.

[0032] The specific steps of step S5 are as follows:

[0033] (1) The point cloud feature tensor T Pt Flatten, and then transform the image feature tensor T I Splicing is performed on the feature dimension to obtain the point graph feature tensor T PtI ;

[0034] (2) The point graph feature tensor T PtI Obtaining a three-dimensional bounding box and a classification score for each point through the MLP network, and using the prediction score function L to iteratively optimize the classification model;

[0035] Wherein, the prediction scoring function L is:

[0036]

[0037] Where: N is the number of point clouds, is the offset regression loss of point i, m i ∈{0,1} indicates whether point i is within the three-dimensional bounding box. is the cross entropy loss of point i, L stn is the spatial transformation regularization loss;

[0038] (3) A voting mechanism is adopted to count the classification scores of the points within the three-dimensional bounding box, and the category with the most votes is selected as the final classification label of the three-dimensional bounding box to complete the identification of facilities and equipment in the track area.

[0039] The advantages of the present invention are:

[0040] (1) It can realize automatic processing of the entire process from data collection to facility and equipment identification, reduce manual misjudgment, and improve identification efficiency;

[0041] (2) By combining point cloud features with image features, accurate identification of facilities and equipment of different sizes can be achieved in dark tunnel environments;

[0042] (3) It can realize the intelligent identification of facilities and equipment such as departure indicators in the track area, switches, signal machines, beacons, rails, roadbeds, switches, sleepers, segments, lateral escape platforms in bypass channels, fasteners, oilers, trackside inspection equipment, and online inspection of pantographs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The following is a flowchart of the intelligent identification method for facilities and equipment in a subway track area according to the present invention;

[0044] Figure 2 A schematic diagram of a model framework of a method for intelligently identifying facilities and equipment in a subway track area according to the present invention;

[0045] Figure 3 is a schematic diagram of a feature extraction layer of the present invention;

[0046] Figure 4 Schematic diagram of the characteristic fusion layer of the present invention. DETAILED DESCRIPTION

[0047] The features of the present invention and other related features are further described in detail below through embodiments in conjunction with the accompanying drawings to facilitate understanding by those skilled in the art:

[0048] Example: Figure 1-4As shown, this embodiment specifically relates to a method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map, and the method mainly includes the following steps:

[0049] S1, based on the two-dimensional laser scanner and linear array camera mounted on the track area inspection vehicle, respectively realize the point cloud data P 0 and image data I 0 Collect and based on the point cloud data P 0 and image data I 0 Generate point cloud layout diagram I pt And image two-dimensional layout diagram I I2 .

[0050] The specific steps of step S1 are as follows:

[0051] (1) Point cloud layout diagram I pt Generate: Point Cloud Layout I pt It is composed of multiple single-ring point cloud images. The single-ring point cloud image is obtained by calculating and sorting the angle of the single-ring point cloud relative to the center point of the two-dimensional laser scanner (generated using the point cloud grayscale information).

[0052] (2) Image 2D layout diagram I I2 Generation: It is generated by matching and stitching SIFT features between images from different adjacent cameras and between adjacent images from the same camera.

[0053] S2, the image is two-dimensionally spread out in Figure I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 Height and point cloud layout diagram I pt The height of the compressed image is kept consistent, and the two-dimensional layout diagram I' is displayed according to the tunnel profile section. I2 Mapping to the three-dimensional layout of the image I' I3 .

[0054] The specific steps of step S2 are as follows:

[0055] (1) Use bilinear interpolation to transform the image into a two-dimensional layout diagram I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 Height and point cloud layout diagram I pt The height is consistent.

[0056] (2) The tunnel profile section is sampled at equal angles, and the sampling coordinate value is L ij .

[0057] (3) According to the coordinate value L ij and the corresponding image information I' I2ij , get the three-dimensional layout diagram of the image I'I3 .

[0058] S3, respectively, the compressed image two-dimensional layout diagram I ' I2 And point cloud layout diagram I pt Annular seams are identified to obtain the three-dimensional layout diagram of the segment and the segment point cloud, and the three-dimensional layout diagram of the segment and the segment point cloud are aligned according to the corresponding relationship of the segment.

[0059] The specific steps of step S3 are as follows:

[0060] (1) Statistical point cloud distribution diagram I pt And the compressed image two-dimensional layout diagram I' I2 The grayscale cumulative value G of each column of pixels ptj and G I ' 2j .

[0061] (2) Through point cloud layout I pt And the compressed image two-dimensional layout diagram I' I2 The corresponding thresholds H pt and H' I2 Control and traverse the point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 , and respectively perform point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 Segment circumferential seam identification, point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 The grayscale cumulative value of meets the following conditions, which is the segment annular joint:

[0062]

[0063] Where: and They are point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 The average value of the grayscale accumulation value in the [jn, j+n] column, W pt and W' I2 Respectively represent the point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 In this embodiment, H pt and H' I2 The value of is 0.5, and the value of n is 10.

[0064] (3) Based on the point cloud layout diagram I pt And the compressed image two-dimensional layout diagram I' I2 Segment Circumferential Seam Identification Indexpt and Index' I2 , the point cloud data P 0 And image three-dimensional layout diagram I I ' 3 Split and match segment by segment ring.

[0065] S4. Input the segment point cloud into the PointTransformer network composed of feature extraction layer and feature fusion layer to extract the point cloud feature tensor T Pt , the segment 3D layout diagram is input into the ResNet network composed of residual blocks to extract the image feature tensor T I .

[0066] The specific steps of step S4 are as follows:

[0067] (1) Point cloud data P 0 The feature extraction layer is composed of the farthest point sampling, KNN grouping, MLP network and residual Transformer module in sequence to enhance the feature expression ability of point cloud and alleviate the gradient disappearance problem.

[0068] (2) The feature fusion layer adopts the U-net structure, which processes the features of the same layer through a fully connected network and concatenates them with the features of trilinear interpolation, and then learns them through the residual Transformer module to reduce data loss.

[0069] (3) Image 3D layout diagram I I ' 3 Information can be directly transferred through cross-layer connections of residual blocks to learn and extract deep features of images. The principle of deep feature extraction is as follows:

[0070]

[0071] Where: x is the initial input of the network, F l (x l-1 ,w l ) represents the nonlinear transformation in the residual block of the lth layer, w l is the parameter of layer l, x l-1 is the input feature of the l-1th layer. In this embodiment, the number of feature extraction layers and feature fusion layers in the PointTransformer network are both 4, which can realize the extraction of spatial position features, local geometric features, multi-scale features, and density features in the point cloud of facilities and equipment; the number of residual blocks in the ResNet network is 16, which can realize the extraction of different colors, textures, and contexts of facilities and equipment.

[0072] S5. Convert the point cloud feature tensor T Ptand the image feature tensor T I Fusion into point graph feature tensor T PtI , and classified through the MLP network; the prediction scoring function L is used to optimize the model to achieve intelligent identification of facilities and equipment in the track area.

[0073] The specific steps of step S5 are as follows:

[0074] (1) The point cloud feature tensor T Pt Flatten and then transform the image feature tensor T I Concatenate on the feature dimension to get the point graph feature tensor T PtI .

[0075] (2) Point graph feature tensor T PtI The 3D bounding box and the classification score of each point are obtained through the MLP network, and the prediction score function L is used to iteratively optimize the classification model;

[0076] Among them, the prediction score function L is:

[0077]

[0078] Where: N is the number of point clouds, L i offset is the offset regression loss of point i, m i ∈{0,1} indicates whether point i is within the three-dimensional bounding box, L i score is the cross entropy loss of point i, L stn is the spatial transformation regularization loss.

[0079] (3) A voting mechanism is used to count the classification scores of the points within the three-dimensional bounding box and select the category with the most votes as the final classification label of the three-dimensional bounding box to complete the identification of facilities and equipment in the track area.

[0080] In this embodiment, the facilities and equipment in the track area include departure indicators, switches, signal machines, beacons, rails, roadbeds, sleepers, segments, lateral escape platforms for bypass channels, fasteners, oilers, trackside inspection equipment, and pantograph online inspection equipment.

[0081] For small-sized devices in the track area (fasteners, oilers, beacons): The model can accurately capture the tiny geometric structure and texture details of small devices through a sophisticated local feature extraction mechanism. Multi-scale feature extraction and hierarchical processing allow the model to identify and express the features of small devices at different resolutions, so that even subtle structural differences can be effectively distinguished.

[0082] For large-scale equipment in the track area (turnouts, rails, ballasts, segments): The model uses large-scale information processing capabilities to effectively capture the long-distance dependencies between the various parts of large equipment. Deep feature extraction and information aggregation mechanisms enable the model to gradually build an abstract representation of the overall shape and function of large equipment. At the same time, multi-scale feature extraction ensures that the model does not ignore important local details while paying attention to the overall structure, thereby comprehensively and accurately describing the complex characteristics and functional attributes of large equipment.

[0083] The beneficial effects of this embodiment are: it can realize automatic processing of the entire process from data collection to facility and equipment identification, reduce manual misjudgment, and improve identification efficiency; by combining point cloud features with image features, it can realize accurate identification of facilities and equipment of different sizes in the dark light environment of the tunnel; it can realize intelligent identification of facilities and equipment such as departure indicators, switches, signal machines, beacons, rails, roadbeds, switches, sleepers, segments, lateral escape platforms of bypass channels, fasteners, oilers, trackside detection equipment, and pantograph online detection in the track area.

Claims

1. A method for intelligent identification of facilities and equipment in subway track areas based on point cloud and image map, characterized in that The method comprises the following steps: S1, based on the two-dimensional laser scanner and the linear array camera mounted on the track area detection vehicle, respectively realize the collection of point cloud data P0 and image data I0, and generate a point cloud layout map I0 according to the point cloud data P0 and the image data I0 pt And image two-dimensional layout diagram I I2 ; S2, the image is two-dimensionally spread out in Figure I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 The height and the point cloud layout diagram I pt The height is kept consistent, and the compressed image two-dimensional layout diagram I' is displayed according to the tunnel profile section. I2 Mapping to the three-dimensional layout of the image I' I3 ; S3, respectively, the compressed image two-dimensional layout diagram I ' I2 And the point cloud layout diagram I pt Perform annular seam identification to obtain a segment three-dimensional layout diagram and a segment point cloud, and align the segment three-dimensional layout diagram and the segment point cloud according to the segment correspondence relationship; S4, inputting the segment point cloud into a PointTransformer network composed of a feature extraction layer and a feature fusion layer to extract the point cloud feature tensor T Pt The three-dimensional layout of the pipe segment is input into the ResNet network composed of residual blocks to extract the image feature tensor T I ; S5. The point cloud feature tensor T Pt and the image feature tensor T I Fusion into point graph feature tensor T PtI , and classified through the MLP network; the prediction scoring function L is used to optimize the model to achieve intelligent identification of facilities and equipment in the track area.

2. According to claim 1, a method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map, characterized in that The specific steps of step S1 are as follows: (1) Point cloud layout diagram I pt Generate: the point cloud layout diagram I pt It is composed of a plurality of single-ring point cloud images, wherein the single-ring point cloud images are obtained by calculating and sorting the angles of the single-ring point clouds relative to the center point of the two-dimensional laser scanner; (2) The two-dimensional layout diagram of the image I I2 Generation: It is generated by matching and stitching SIFT features between images from different adjacent cameras and between adjacent images from the same camera.

3. The method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map according to claim 2 is characterized in that The specific steps of step S2 are as follows: (1) Use bilinear interpolation to transform the image into a two-dimensional layout diagram I I2 The height of the compressed image is compressed so that the compressed image two-dimensional layout diagram I' I2 The height and the point cloud layout diagram I pt Highly consistent; (2) The tunnel profile section is sampled at equal angles, and the sampling coordinate value is L ij ; (3) According to the coordinate value L ij and the corresponding image information Get the three-dimensional layout diagram I' of the image I3 .

4. The method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map according to claim 3 is characterized in that The specific steps of step S3 are as follows: (1) Count the point cloud layout diagram I respectively pt and the compressed two-dimensional layout diagram of the image I' I2 The grayscale accumulation value of each column of pixels and (2) Through the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The corresponding thresholds H pt and H' I2 Control and traverse the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 , and respectively perform the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The segment annular seam identification, the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The grayscale cumulative value of meets the following conditions, which is the segment annular joint: Where: and They are respectively the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The average value of the grayscale accumulation value in the [jn, j+n] column, W pt and W I '2 respectively represent the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 The image width; (3) According to the point cloud layout diagram I pt and the compressed two-dimensional layout diagram of the image I' I2 Segment Circumferential Seam Identification Index pt and Index' I2 , the point cloud data P0 and the image three-dimensional layout diagram I I '3 Split and match the segments one by one.

5. The method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map according to claim 4 is characterized in that The specific steps of step S4 are as follows: (1) The point cloud data P0 is sequentially subjected to farthest point sampling, KNN grouping, the MLP network and the residual Transformer module, and the farthest point sampling, the KNN grouping, the MLP network and the residual Transformer module constitute the feature extraction layer; (2) The feature fusion layer adopts a U-net structure, and the features of the same layer are processed by a fully connected network and concatenated with the features of trilinear interpolation, and then learned through the residual Transformer module; (3) The three-dimensional layout diagram of the image I I '3 Through the cross-layer connection of the residual block, information can be directly transferred to learn and extract the deep features of the image. The principle of deep feature extraction is as follows: Where: x is the initial input of the network, F l (x l-1 ,w l ) represents the nonlinear transformation in the residual block of the lth layer, w l is the parameter of layer l, x l-1 is the input feature of the l-1th layer.

6. The method for intelligently identifying facilities and equipment in a subway track area based on point cloud and image map according to claim 5 is characterized in that The specific steps of step S5 are as follows: (1) The point cloud feature tensor T Pt Flatten, and then transform the image feature tensor T I Splicing is performed on the feature dimension to obtain the point graph feature tensor T PtI ; (2) The point graph feature tensor T PtI Obtaining a three-dimensional bounding box and a classification score for each point through the MLP network, and using the prediction score function L to iteratively optimize the classification model; Wherein, the prediction scoring function L is: Where: N is the number of point clouds, is the offset regression loss of point i, m i ∈{0,1} indicates whether point i is within the three-dimensional bounding box. is the cross entropy loss of point i, L stn is the spatial transformation regularization loss; (3) A voting mechanism is adopted to count the classification scores of the points within the three-dimensional bounding box, and the category with the most votes is selected as the final classification label of the three-dimensional bounding box to complete the identification of facilities and equipment in the track area.

Images