Railway dangerous rockfall identification method and system based on three-dimensional point cloud twinning

Abstract

The present application relates to the technical field of image recognition, and particularly relates to a railway dangerous rockfall identification method and system based on three-dimensional point cloud twinning; the present application constructs a three-dimensional point cloud twin reference body of a railway wall and a track safety corridor along the railway based on historical point clouds, receives a current inspection point cloud and a current image, maps the two to a twin coordinate space, generates a signed geometric residual field according to the surface distance, normal variation and point density difference between the current inspection point cloud and the twin reference body, performs image recognition on the current image, obtains a rock mass exposed area, a fissure texture area, a shadow area and a vegetation sheltering area, back projects the image recognition result to the twin coordinate space, and determines a dangerous rockfall candidate area and an uncertain area according to the boundary consistency and spatial continuity of the geometric residual and the image semantics; the present application can distinguish real rock mass changes from pseudo anomalies caused by point cloud loss, shadows and vegetation sheltering.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method and system for identifying dangerous rockfalls and falling rocks along railways based on three-dimensional point cloud twins. Background Technology

[0002] Current methods for identifying dangerous rockfalls along railways typically rely on manual inspections, 2D image recognition, or 3D point cloud comparison. 2D image recognition schemes generally acquire surface textures from inspection images of slopes, rock walls, and adjacent areas of the track. Image segmentation or target detection models are then used to identify pixel regions such as exposed rock areas, fissure texture areas, shadow areas, and suspected rockfall boundaries. The locations of suspected dangerous rocks are then output based on pixel area, texture continuity, and target confidence. This type of scheme has a relatively straightforward implementation process; after image acquisition, grayscale enhancement, edge extraction, semantic segmentation, and candidate box filtering are used to obtain the recognition results, making it suitable for rapid assessment of visible surface features. However, railway slopes exhibit significant topographical undulations, and the rock surfaces are often similar in color. Vegetation obstruction, lighting angles, post-rain watermarks, and shadow variations can alter the image texture, compressing the differences between fissures, edges, and ordinary textures in 2D images. Consequently, the image recognition results struggle to reflect the true spatial changes of the rock mass relative to the railway safety corridor.

[0003] Existing 3D point cloud recognition schemes typically utilize multi-period point cloud registration and differential analysis to determine rock mass changes. The conventional approach is to use historical point clouds as a baseline, transforming the current inspection point cloud to the same coordinate system through feature point matching, iterative nearest-point registration, or local planar constraints. Then, the distance difference, normal angle, curvature changes, or point density changes between the current point cloud and the baseline point cloud are calculated to identify suspected convex, concave, missing, or newly added block areas. Some schemes overlay the differential results onto railway line models or slope models to estimate the location and approximate volume of rockfalls. While these schemes can obtain spatial scale information, point cloud sampling is significantly affected by scanning angle, occlusion location, reflection intensity, and the choice of registration benchmark. Steep slope areas and concave rock mass areas are prone to localized point cloud sparseness. Voids, misalignments, and resampling errors in point cloud differential analysis may manifest as geometric residuals, making it difficult to distinguish between actual rock mass changes and acquisition errors in the 3D differential results.

[0004] The main technical problem with existing technologies is the lack of a unified twin benchmark that can simultaneously constrain 3D geometric changes and image semantic changes during railway rockfall identification. This leads to the inability to reliably interpret the geometric residuals generated by the current inspection point cloud, and the inability to establish a stable correspondence between the crack textures, exposed rock boundaries, shadows, and vegetation occlusion obtained from current image recognition and the 3D residuals. Because the geometric difference results and image recognition results are often processed in independent chains, point cloud registration errors, missing point density, lighting and shadows, vegetation occlusion, and ordinary rock textures can be mixed into the same category of suspected anomalies. The system struggles to determine whether a particular area is a candidate rockfall formed by actual rock mass displacement and boundary expansion, or an uncertain area caused by changes in acquisition conditions. This problem directly results in discontinuous rockfall boundaries, unclear candidate area attributes, unstable volume estimation, and affects the reliability of subsequent rockfall identification results. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for identifying dangerous rocks and falling rocks on railways based on three-dimensional point cloud twins, which can solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A railway rockfall identification system based on 3D point cloud twins includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it constructs a 3D point cloud twin reference body of the rock wall and track safety corridor along the railway based on historical point clouds. It receives the current inspection point cloud and the current image, and maps the current inspection point cloud and the current image to the coordinate space of the 3D point cloud twin reference body. It generates a signed geometric residual field based on the surface distance, normal variation and point density difference between the current inspection point cloud and the 3D point cloud twin reference body. It performs image recognition on the current image to obtain image recognition results including exposed rock areas, fissure texture areas, shadow areas and vegetation occlusion areas. It backprojects the image recognition results to the coordinate space, and determines the candidate areas and uncertain areas of rockfall based on the boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results.

[0007] Preferably, when the processor constructs the three-dimensional point cloud twin reference body, it establishes a unified coordinate framework based on the track surface, slope toe line, and continuous rock wall surface in the historical point cloud. The continuous rock wall surface is divided into two levels of expression: voxel blocks and local surface patches. Each voxel block is associated with a normal vector, curvature, roughness, point density, and historical stability label. Voxel blocks that satisfy the condition that the position residual of multiple periods is less than the stability threshold and do not overlap with the crack texture area obtained by image recognition of historical inspection images are written into a stable registration set. The stable registration set and the track safety corridor are jointly written into the spatial constraints of the three-dimensional point cloud twin reference body.

[0008] Preferably, when the processor generates the signed geometric residual field, it uses the stable registration set as a registration constraint to perform resolution-consistent resampling on the current inspection point cloud, determines the normal projection distance from the current point to the twin reference surface as a signed distance value, writes the local normal rotation amount, curvature change amount, surface convexity mark, surface concavity mark and point density confidence value into the residual vector, generates a residual connected region for residual vectors that span adjacent voxels and have the same sign direction, and retains the index of the residual vector and the corresponding voxel block.

[0009] Preferably, when the processor obtains the image recognition result, it divides the current image into a rock mass exposed area, a fissure texture area, a shadow area, and a vegetation occlusion area. Based on the projection relationship between the current image and the current inspection point cloud, it converts the pixel skeleton of the fissure texture area, the semantic boundary of the rock mass exposed area, the occlusion boundary of the shadow area, and the occlusion boundary of the vegetation occlusion area into voxel semantic evidence in the coordinate space. It then establishes an index relationship between the voxel semantic evidence and the signed geometric residual field, and records the pixel source, projection depth, and visibility marker of the voxel semantic evidence.

[0010] Preferably, the historical stable label is jointly generated by multiple historical inspection point clouds and historical inspection images. The processor jointly encodes the centroid shift, normal shift, point density fluctuation and image recognition semantic changes of the same voxel block in adjacent inspection periods, and marks voxel blocks that continuously show low centroid shift, low normal shift and no crack texture enhancement as stable voxel blocks, and marks voxel blocks that have short-term density reduction and fall into vegetation shading area or shadow area as transient shading voxel blocks, and excludes the transient shading voxel blocks from the stable registration set.

[0011] Preferably, when the processor performs multi-scale filtering on the residual connected components, it acquires the convex residual connected components at the first voxel scale, acquires the normal rotation region overlapping the convex residual connected components at the second voxel scale, and traces the residual symbol continuum along the adjacency direction of the twin reference plane; for residual connected components lacking the residual symbol continuum and with a point density confidence value lower than the density threshold, the processor transfers the residual connected components into the uncertain region; for residual connected components that overlap across scales, the processor retains their boundary voxel set.

[0012] Preferably, after establishing the index relationship, the processor performs neighborhood matching on the pixel skeleton of the crack texture area in the voxel semantic evidence and the residual boundary in the signed geometric residual field, performs closure matching on the semantic boundary of the exposed rock mass area and the outer contour of the residual connected domain, and writes the area that simultaneously satisfies the adjacent residual boundary of the crack texture area pixel skeleton, the semantic boundary of the exposed rock mass area surrounding the residual connected domain, and the shadow area and vegetation occlusion area not covering the main residual position into the dangerous rockfall candidate area, and records the corresponding semantic evidence number.

[0013] Preferably, the processor constructs a cross-modal consistency feature vector for each candidate region of dangerous rockfall. The cross-modal consistency feature vector includes the continuous length of the three-dimensional residual, the proportion of residual sign consistency, the concentration of normal rotation, the overlap rate of the fracture texture area boundary, the closure degree of the exposed rock mass area, the proportion of occlusion semantics, the point density confidence value, and the historical stable label change status. The cross-modal consistency feature vector is input into the candidate discrimination model, and the candidate discrimination model outputs the category code according to the candidate label, the verification label, and the occlusion label.

[0014] Preferably, the processor performs three-dimensional connectivity merging of adjacent candidate regions for dangerous rockfalls based on the candidate labels, maintains the independent boundary of the corresponding region in the uncertain region based on the verification labels, extracts the displacement vector along the normal direction of the twin reference surface for the regions that have completed three-dimensional connectivity merging, calculates the volume estimate value according to the overlap relationship between the residual connected domain formed by the signed geometric residual field and the coordinate space voxel partition, and outputs the dangerous rockfall identification result including three-dimensional boundary, displacement vector, volume estimate value, label category and semantic evidence number.

[0015] A railway rockfall identification method based on 3D point cloud twins includes: constructing a 3D point cloud twin reference body for the rock walls and track safety corridors along the railway line based on historical point clouds; receiving the current inspection point cloud and the current image, and mapping the current inspection point cloud and the current image to the coordinate space of the 3D point cloud twin reference body; generating a signed geometric residual field based on the surface distance, normal variation, and point density difference between the current inspection point cloud and the 3D point cloud twin reference body; performing image recognition on the current image to obtain image recognition results including exposed rock areas, fracture texture areas, shadow areas, and vegetation occlusion areas; backprojecting the image recognition results to the coordinate space, and determining candidate areas and uncertain areas of rockfall based on the boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. A 3D point cloud twin reference body for the rock face and track safety corridor along the railway line is constructed based on historical point clouds. The current inspection point cloud and the current image are uniformly mapped to the same coordinate space, allowing geometric changes and image semantic changes to be processed accordingly under the same spatial index. After the surface distance, normal variation, and point density difference between the current inspection point cloud and the 3D point cloud twin reference body are converted into a signed geometric residual field, the rock bulges, depressions, normal rotations, and point cloud missing parts are no longer just isolated distance differences, but are recorded as spatial residuals with direction, continuity, and density confidence. After the current image obtains the exposed rock mass area, fracture texture area, shadow area, and vegetation occlusion area through image recognition, it is back-projected to the same coordinate space, and an index relationship is established with the residual boundaries and residual connected domains. Thus, areas with 3D residual continuity, fracture texture adjacency, and exposed rock mass boundary closure are identified as candidate areas for dangerous rockfalls; residuals falling into shadow, vegetation occlusion, or low point density areas and lacking spatial continuity are classified as uncertain areas. This processing chain enables a calculable distinction between actual rock mass changes and acquisition errors, provides spatial continuity constraints for dangerous rock boundary extraction, and ensures that the three-dimensional location, boundary, and direction of change of candidate rockfall areas have a consistent data basis.

[0017] 2. By employing stable registration sets, voxel semantic evidence, residual connected components, multi-scale filtering, and continuous constraints of cross-modal consistent feature vectors, the intermediate data structure of the identification results is more suitable for long-term inspection and verification. Stable voxel blocks in historical point clouds are used to constrain the registration of current inspection point clouds, reducing coordinate drift caused by slope change areas participating in registration. Voxel blocks record normal vectors, curvature, roughness, point density, and historical stability labels, enabling subsequent residual judgments to distinguish stable rock face surfaces, transient occlusion areas, and suspected change areas. Residual connected components retain spatial chains with consistent sign directions at different voxel scales and, combined with point density confidence values, filter out local anomalies lacking continuity. The fracture skeleton, exposed rock mass boundaries, shadow boundaries, and vegetation occlusion boundaries formed by image recognition are converted into voxel semantic evidence, which can then be used for neighborhood matching and closure matching with the residual outer contour. After classification by candidate labels, verification labels and occlusion labels, adjacent candidate areas of dangerous rocks and rocks can be merged according to the three-dimensional connectivity relationship, while the verification areas still maintain independent boundaries. Finally, the three-dimensional boundary, displacement vector, volume estimate and semantic evidence number are output, so that the identification results of dangerous rocks and rocks include traceable geometric evidence and image recognition evidence. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the overall processing flow of the railway dangerous rock and falling rock identification system based on three-dimensional point cloud twins according to the present invention. Figure 2 This is a flowchart illustrating the construction process of the 3D point cloud twin reference body and stable registration set of the present invention. Figure 3This is a flowchart of the signed geometric residual field and image recognition semantic backprojection of the present invention; Figure 4 This is a flowchart of the cross-modal consistency discrimination and rockfall result output of the present invention. Detailed Implementation

[0019] refer to Figure 1 In one embodiment, a railway rockfall identification system based on 3D point cloud twins is provided. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it constructs a 3D point cloud twin reference body of the rock face and track safety corridor along the railway line based on historical point clouds. It receives the current inspection point cloud and the current image, and maps the current inspection point cloud and the current image to the coordinate space of the 3D point cloud twin reference body. It generates a signed geometric residual field based on the surface distance, normal variation, and point density difference between the current inspection point cloud and the 3D point cloud twin reference body. It performs image recognition on the current image to obtain image recognition results including exposed rock areas, fissure texture areas, shadow areas, and vegetation-covered areas. It back-projects the image recognition results onto the coordinate space. The boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results determine candidate areas and uncertain areas for rockfalls. Specifically, historical point clouds are used for... A spatial reference that does not drift with changes in single inspection conditions is formed. The track safety corridor is used to define the relevant spatial range of railway operation. The current inspection point cloud is used to reflect the three-dimensional geometric changes between the rock wall surface and the reference. The current image is used to provide image recognition evidence of rock mass exposure, fissure texture, shadows and vegetation occlusion. After the processor unifies the two-dimensional image coordinates, point cloud coordinates and twin reference coordinates into the same coordinate space, it simultaneously reads the geometric residuals and semantic evidence for the same voxel position. Then, it uses whether the residual direction is continuous, whether the residual boundary is adjacent to the fissure texture or rock mass exposure boundary, and whether the residual area is covered by shadows or vegetation occlusion as the judgment criteria. Areas with rock mass outward displacement or boundary expansion characteristics are output as candidate areas for dangerous rockfalls, and areas lacking spatial continuity or within the semantic range of occlusion are output as uncertain areas. The advantage of this embodiment is that it constrains the point cloud difference through the three-dimensional point cloud twin reference and interprets the source of the residual through image recognition, so that the single point cloud error and single frame image texture interference are included in the same discrimination chain.

[0020] refer to Figure 2In this embodiment, the 3D point cloud twin reference volume uses historical point clouds of the rock wall along the railway as the initial data source. After coordinate unification, outlier removal, local surface fitting, and voxelization, the historical point cloud forms reference data that can be called by the current inspection point cloud. The coordinate unification process uses stable geometric elements in the track surface, toe line, and continuous rock wall surface as constraints. The track surface provides the corridor direction and elevation reference, the toe line provides the transition constraint between the slope and the track, and the continuous rock wall surface provides the spatial constraint of the rock mass surface. During local surface fitting, a set of neighboring points is selected in the neighborhood of each voxel and the principal direction is calculated. The direction of minimum change of the principal direction is used as the normal vector, and the average deviation of the neighborhood points from the fitted surface is used as the roughness. The relationship between the number of neighboring points and the space occupied by voxels is used as the point density. The curvature is calculated from the change of the main direction of the local neighborhood. The historical stable state is determined by the changes of residuals in multiple periods and the results of historical image recognition. If a voxel does not show continuous outward movement, abrupt change in normal or enhanced crack texture in multiple historical inspections, it is written into the stable registration set. If a voxel is frequently covered by vegetation or shadows in historical images, it does not participate in the construction of the stable registration set. This avoids using occluded areas or potentially changed areas as constraints for the current inspection point cloud registration. The advantage of this embodiment is that the stable registration set is selected from the joint screening of geometric stability and image semantic stability, which can reduce the coordinate offset caused by the participation of changed rock masses in registration.

[0021] Table 1. Data fields and generation rules of the 3D point cloud twin reference body The fields listed in Table 1 are stored in the computer program using voxel numbers as the primary index. When reading the current inspection point cloud, the processor obtains the reference surface normal vector, historical stability label, and point density confidence value based on the voxel number. When reading the current image recognition result, it searches for historical semantic records at the same spatial location based on the semantic association number. If the current region has geometric residuals but was historically a transient occlusion voxel, the processor does not directly incorporate it into the candidate region for dangerous rockfalls. Instead, it requires it to form an adjacency relationship with the current fracture texture area or the exposed rock boundary before entering the candidate judgment. If the current region has geometric residuals and was historically a stable voxel, the processor further checks whether the residual sign is consistent along the adjacent voxels to determine whether it is a new three-dimensional change region. There is no isolated use relationship between the table fields. The normal vector, point density, historical stability label, and semantic association number jointly participate in the residual interpretation, transforming the twin reference body from a simple historical point cloud set into a spatial reference object with geometric and image recognition attributes. The advantage of this embodiment is that the reference data can be traced item by item in subsequent recognition, making it easy to distinguish between real rock mass changes, occlusion changes, and sampling changes.

[0022] refer to Figure 3In a preferred embodiment, when the processor generates the signed geometric residual field, it uses a stable registration set as the registration constraint between the current inspected point cloud and the 3D point cloud twin reference. The current inspected point cloud is first resampled according to the voxel scale of the twin reference. The resampled current point is not directly subtracted from the most recent historical point, but is projected onto the reference surface patch of the corresponding voxel. The normal projection distance from the current point to the reference surface patch is used as the signed distance value. The angle between the normal vector of the current point's neighborhood and the normal vector of the reference voxel is used as the normal rotation amount. The relative difference between the point density of the current point's neighborhood and the point density of the reference voxel is used as the point density change amount. The signed distance value is... A positive value indicates that the current rock mass surface has shifted outward or bulged outward along the reference normal. A negative value indicates that the current rock mass surface has retreated inward or formed a depression along the reference normal. The point density change is used to distinguish between real surface changes and sampling omissions. The processor writes the signed distance value, normal rotation amount, curvature change amount, surface bulge mark, surface depression mark, and point density confidence value into the residual vector. Then, it aggregates residual vectors with consistent sign direction and spatial adjacency along the voxel adjacency relationship, generates residual connected domains, and retains the index of the residual vector and the corresponding voxel block. The advantage of this embodiment is that the residual result contains direction, structure, and density information, which can avoid judging rock mass changes based solely on point-to-point distance.

[0023] In this embodiment, the signed geometric residual field can be calculated using the following formula, assuming the current inspection point is... ,and The corresponding projection points on the twin reference surface are The normal vector of the twin reference voxel is The neighborhood normal vector of the current point is The density of neighboring points of the current point is The reference voxel density is The constant used to avoid a denominator of 0 is The signed distance value is The normal rotation is The change in point density is The residual vector is The calculation formula is: in, Represents the dot product of vectors. Represents the inverse cosine operation, when , hour, The value is 0.04, indicating that the current point deviates outward by 0.04 units along the reference line. hour, Approximately 0.283 radians indicates that the current local surface has undergone a normal rotation relative to the reference plane. , , hour, The value is approximately -0.100, indicating that the current neighborhood sampling density is lower than the reference density. The advantage of this embodiment is that the same formula simultaneously provides geometric displacement, attitude change, and sampling reliability, so that subsequent candidate region judgment does not depend on a single difference component.

[0024] Furthermore, the image recognition results of the current image are not output as separate two-dimensional candidate boxes. The processor divides the current image into exposed rock areas, fracture texture areas, shadow areas, and vegetation occlusion areas. Based on the projection relationship between the current image and the current inspection point cloud, it transforms the image pixels into the coordinate space of a three-dimensional point cloud twin reference volume. The fracture texture area is first refined to form a pixel skeleton. Skeleton pixels fall into corresponding voxels according to the projection depth to form fracture skeleton evidence. The semantic boundaries of the exposed rock areas are converted into voxel boundary chains after boundary tracing. The occlusion boundaries of the shadow and vegetation occlusion areas are converted into occlusion voxel sets. For each voxel semantic evidence, the pixel source, projection depth, and visibility marker are recorded. The pixel source is used to trace the original region in image recognition, the projection depth is used to exclude mismatches that are projected onto the back side of the rock wall or non-visible surfaces, and the visibility marker is used to describe whether the voxel is semantically covered by occlusion in the current image. Subsequently, the processor establishes an index relationship between the voxel semantic evidence and the signed geometric residual field through voxel numbering, so that the same rock mass spatial location can be referenced by both geometric residual and image recognition semantics at the same time. The advantage of this embodiment is that after the two-dimensional image recognition result is converted into three-dimensional voxel evidence, it can be spatially matched with the point cloud residual boundary.

[0025] In this embodiment, the semantic segmentation of the current image can be achieved using a pixel-level classification network or a boundary enhancement segmentation network. The training data comes from the exposed rock texture, fissure linear texture, shadow areas, and vegetation-occluded areas in images of rock walls along railway lines. After the image is input, scale normalization and brightness distribution correction are performed first, and then multi-scale convolutional feature extraction is used to obtain the texture response map. The fissure texture area is jointly determined by the linear high-response area and the adjacent dark texture area. The exposed rock area is jointly determined by the surface color of the rock and the rough texture area. The shadow area is determined by the low brightness area and the boundary continuously changing with the direction of illumination. The vegetation-occluded area is determined by the green or branch and leaf texture. Once the region is determined, the processor performs connected component processing on the segmentation results, deletes isolated pixel blocks, retains semantic boundaries adjacent to the rock wall region, retains the center line and endpoints of the crack texture region after skeletonization, retains the outer contour of the exposed rock area, and retains the coverage area of ​​the shadow area and vegetation occlusion area. When the image recognition results are written into the three-dimensional voxel semantic evidence, if the same voxel corresponds to multiple pixel semantics, the main semantic is determined according to the pixel whose projection depth is closest to the current point cloud surface, and the auxiliary semantic record is formed according to other pixels. The advantage of this embodiment is that the image recognition results will not remain at the two-dimensional image level, but will become spatial evidence that can explain the source of the three-dimensional residual.

[0026] Preferably, the historical stable labels are jointly generated by multiple historical inspection point clouds and historical inspection images. The processor jointly encodes the centroid shift, normal shift, point density fluctuation, and image recognition semantic changes of the same voxel block in adjacent inspection periods. Voxel blocks that continuously exhibit low centroid shift and low normal shift without crack texture enhancement are marked as stable voxel blocks. Voxel blocks that show short-term point density decreases and simultaneously fall into vegetation shading or shadow areas are marked as transient shading voxel blocks, and these transient shading voxel blocks are excluded from the stable registration set. Specifically, the centroid shift is calculated from the difference in centroid coordinates of the same voxel in adjacent historical periods, and the normal shift is calculated from the reference... The angle between the surface normal vectors is calculated. The point density fluctuation is calculated from the relationship between the number of points within a voxel and the space occupied by its neighborhood. The semantic changes in image recognition are obtained from the changes in the proportion of fracture texture areas, exposed rock areas, shadow areas, and vegetation occlusion areas in historical images. If both the centroid offset and normal offset remain within a stable range, and the fracture texture area does not extend along adjacent voxels, then the voxel block is marked as a stable voxel block. If the point density decreases and occlusion semantics occur simultaneously and recover in subsequent historical periods, then it is marked as a transient occlusion voxel block. Transient occlusion voxel blocks are not used for the current registration. The advantage of this embodiment is that the registration datum excludes the area affected by occlusion, reducing the sources of spurious changes in the subsequent residual field.

[0027] In this embodiment, the historical stability label can be calculated using stability evaluation values. Let the centroid offset of the j-th voxel between adjacent historical inspection periods be denoted as . The normal offset angle is Point density fluctuation is The semantic change in image recognition is The stability evaluation value is The weighting coefficients are a, b, c, and e, respectively, and a + b + c + e = 1. The calculation formula is: in, This represents the absolute amount of centroid shift. Indicates the change in the included angle of the normal. It represents the absolute quantity indicating the relative change in point density. This represents the change in the proportion of semantic categories in image recognition between the current historical period and adjacent historical periods, where a=0.30, b=0.30, c=0.20, e=0.20. , , , hour, This indicates that the voxel block exhibits relatively small intertemporal variations, and that when the same voxel block appears... and When the proportion of shaded areas or areas obscured by vegetation increases, even If the suspected change range is not reached, it will not be written into the stable registration set. The advantage of this embodiment is that the stability label is given by both geometric change and image recognition semantics, which can prevent the occlusion recovery process from being mistaken for a stable change in the rock mass.

[0028] In a preferred embodiment, the processor performs multi-scale filtering on the residual connected components. At the first voxel scale, it acquires convex residual connected components; at the second voxel scale, it acquires normal rotation regions overlapping with the convex residual connected components; and it traces the residual sign continuum along the adjacent direction of the twin datum. For residual connected components lacking a residual sign continuum and with point density confidence values ​​below a density threshold, the processor transfers the residual connected component to an uncertain region. For residual connected components overlapping across scales, the processor retains their boundary voxel sets. Specifically, the first voxel scale is used to identify local regions where the rock mass surface has shifted outward relative to the datum, and the second voxel scale is used to examine the periphery of this shifted region. Whether there is continuous normal rotation on the edge, if a convex region exists at a fine scale but disappears at a coarse scale, and the corresponding point density confidence value is low, then the region is more likely to be formed by sparse sampling or occlusion. If the convex region appears at both scales and the residual sign extends along the adjacent direction of the rock wall surface, then it is retained as a region with continuous spatial variation. The boundary voxel set is formed by the position where the residual sign turns from positive to close to zero, and serves as the geometric basis for subsequent matching with the boundary of the exposed rock mass area and the fracture skeleton. The advantage of this embodiment is that the residual determination is changed from single-scale point difference to cross-scale connectivity analysis, which can suppress the interference of local voids and isolated noise on the candidate region.

[0029] refer to Figure 4Furthermore, after establishing the index relationship between voxel semantic evidence and the signed geometric residual field, the processor performs neighborhood matching on the pixel skeleton of the fracture texture region in the voxel semantic evidence and the residual boundary in the signed geometric residual field. It also performs closure matching on the semantic boundary of the exposed rock mass region and the outer contour of the residual connected domain. Regions that simultaneously satisfy the following conditions—adjacent residual boundaries of the fracture texture region pixel skeleton, the semantic boundary of the exposed rock mass region surrounding the residual connected domain, and the main residual location not covered by shadow areas and vegetation-covered areas—are written into the candidate region for dangerous rockfalls, and the corresponding semantic evidence number is recorded. Specifically, during neighborhood matching, fracture skeleton evidence is searched in adjacent voxels centered on the residual boundary voxel. If the fracture skeleton is along the residual boundary... If the extension extends to the boundary of the exposed rock mass area at the endpoint, it is assumed that there is a spatial correspondence between the three-dimensional geometric residual and the two-dimensional fracture texture. During closure matching, a closed voxel chain is formed along the semantic boundary of the exposed rock mass area, and it is checked whether the outer contour of the residual connected domain is located inside or adjacent to the closed voxel chain. If the main residual position is covered by the shadow area or vegetation cover area, it is not directly written into the candidate area of ​​dangerous rockfall, but is entered into the review or uncertainty processing together with the low-density residual connected domain. The semantic evidence number records the image pixel source and the three-dimensional voxel position. The advantage of this embodiment is that the candidate area must have both three-dimensional change boundary and image recognition semantic support at the same time, avoiding ordinary texture or isolated residual being directly output as dangerous rockfall.

[0030] Table 2. Fields and Calculation Sources of Cross-Modal Consistency Feature Vectors The fields listed in Table 2 are written into the cross-modal consistency feature vector after the candidate regions are formed. The processor constructs the cross-modal consistency feature vector for each candidate region of dangerous rockfall. The cross-modal consistency feature vector includes the continuous length of the three-dimensional residual, the proportion of residual sign consistency, the concentration of normal rotation, the overlap rate of the fracture texture zone boundary, the closure degree of the exposed rock mass zone, the proportion of occlusion semantics, the point density confidence value, and the historical stable label change status. The cross-modal consistency feature vector is then input into the candidate discrimination model. The candidate discrimination model outputs category codes according to the candidate label, the verification label, and the occlusion label. In specific implementation, the candidate... The discrimination model can be a classification tree with threshold branches or a shallow neural classifier. The geometric continuity field in the input vector is used to determine whether the residual is a change in the connected space. The image recognition field is used to determine whether the fracture texture and exposed rock mass boundary support the residual boundary. The occlusion field and point density confidence value are used to determine whether the residual is affected by the acquisition conditions. The historical stable label change status is used to determine whether the current area deviates from the historical stable record. The advantage of this embodiment is that the discrimination input is composed of geometric, semantic, occlusion and historical stability information, which can provide a clear data source for candidate labels, verification labels and occlusion labels.

[0031] In this embodiment, cross-modal consistency can be assisted by a consistency score to perform category encoding for the candidate discrimination model. Let the normalized value of the continuous length of the three-dimensional residual of the k-th candidate region be... The proportion of residuals with consistent signs is Normal rotation concentration is The overlap rate of the crack texture region boundary is The closure degree of the exposed rock mass area is The proportion of occlusion semantics is The missing value of the point density confidence value is The historical stable label change volume is Consistency score: The weighting coefficient is The calculation formula is: in It has a positive correlation with the candidate judgment. It is inversely related to the candidate judgment, when , , , , , , , When the weights are 0.15, 0.15, 0.10, 0.20, 0.15, 0.10, 0.10, and 0.05 respectively, The candidate discrimination model uses this score in conjunction with the constraints of each field. If the proportion of occluded semantics is too high or the number of missing point density confidence values ​​is too high, then even if... Once the candidate range is reached, a verification label or occlusion label is also output. The advantage of this embodiment is that the candidate judgment has both a comprehensive score and retains the restrictions on occlusion and low-density anomalies.

[0032] Furthermore, the processor performs 3D connectivity merging on adjacent candidate rockfall regions based on candidate labels, maintains the independent boundaries of corresponding regions within uncertain regions based on verification labels, and extracts displacement vectors along the normal direction of the twin reference surface for the regions that have completed 3D connectivity merging. It calculates volume estimates based on the overlap relationship between the residual connected domain formed by the signed geometric residual field and the coordinate space voxel partitioning, and outputs the rockfall identification result including 3D boundaries, displacement vectors, volume estimates, label categories, and semantic evidence numbers. Specifically, 3D connectivity merging is based on the adjacency relationship between boundary voxels of the residual connected domain. If the boundary voxels of two candidate regions are spatially adjacent... If adjacent fracture skeleton evidence or exposed rock mass boundaries have a continuous relationship, they are merged into the same candidate object of dangerous rockfall. If there is obvious shadow coverage or low point density interval between two areas, they are kept separate and a verification label is output. The displacement vector is obtained by weighted average of the normal offset of each residual voxel in the candidate object. The volume estimate is calculated by accumulating the positive outward convex residual on the voxel area. The label category is used to indicate whether the object belongs to the candidate, verification or occlusion type. The semantic evidence number is used to backtrack the corresponding image recognition area. The advantage of this embodiment is that the output results retain the correlation between the three-dimensional boundary, displacement direction and image recognition evidence, which is convenient for locating the same spatial object during subsequent verification.

[0033] In this embodiment, the volume estimate can be calculated using the voxel integral of the residual connected domain. Let the set of voxels occupied by the k-th candidate object that has completed 3D connected merging be . The projected area of ​​voxel v on the twin reference plane is The signed distance of voxel v represents the value The estimated size of the candidate object is The calculation formula is: in, This represents the summation of all voxels within the candidate object. This indicates that only the residuals shifted or convex outward along the reference method are accumulated. When a candidate object contains 3 voxels, the 3 voxels... They are 0.25, 0.25, and 0.20 respectively. When the values ​​are 0.04, 0.03, and -0.02 respectively, The result indicates that the estimated convex volume of the candidate object relative to the twin reference surface is not included in the convex volume due to the negative concave residual, but it is still retained in the residual field for boundary judgment. The advantage of this embodiment is that the volume estimation comes from the overlapping relationship between the three-dimensional residual and the voxel area, and does not rely on the two-dimensional image area inference.

[0034] In a preferred embodiment, the computer program in the memory is organized according to the data flow into processing logics such as historical data database construction, current data access, 3D mapping, geometric residual calculation, image recognition, semantic backprojection, cross-modal discrimination, and result output. During the historical data database construction stage, a 3D point cloud twin reference volume is written. During the current data access stage, the current inspection point cloud and the current image are read. During the 3D mapping stage, coordinate transformation is performed based on the current inspection point cloud and the stable registration set, and the projection relationship from the current image to the point cloud is determined simultaneously. During the geometric residual calculation stage, a signed geometric residual field is generated. During the image recognition stage, the exposed rock mass area and fracture texture are output. The semantic back-projection stage converts image recognition results into voxel semantic evidence in the areas of shadow, vegetation cover, and other areas. The cross-modal discrimination stage matches the residual connected domain with the voxel semantic evidence for boundary consistency and spatial continuity. The result output stage saves candidate areas for dangerous rocks and uncertain areas. The data stored in the memory are indexed by voxel numbers in coordinate space, image pixel sources, and inspection period numbers, so that the data from multiple periods, the current data, and the output results are traceable. The advantage of this embodiment is that each processing logic runs around the same three-dimensional point cloud twin reference body, avoiding the point cloud differential link and the image recognition link from being separated.

[0035] Furthermore, when processing the unified mapping between the current inspection point cloud and the current image, the processor does not directly stretch the image recognition result to three-dimensional space. Instead, it first establishes a visibility mapping based on the projection relationship between visible rock wall points and image pixels in the current inspection point cloud. If an image pixel corresponds to multiple depth candidate points, the candidate point with the same projection depth as the rock wall surface is selected as the main mapping point, and the other candidate points are recorded as occlusion-related points. If a current point cloud voxel does not have a corresponding image pixel, an unobserved marker is written into the voxel semantic evidence. The unobserved marker is not equivalent to a shadow area or a vegetation occlusion area. It is only treated as a semantic missing point in subsequent discrimination. If an image pixel falls into a non-rock wall area outside the track safety corridor, no rock wall voxel semantic evidence is written. The processor determines the correspondence between the fracture texture skeleton, the exposed rock boundary, and the occlusion boundary in three-dimensional space through the visibility mapping, avoiding the misprojection of foreground occlusions or non-rock wall textures into the dangerous rock candidate area. The advantage of this embodiment is that the two-dimensional semantic back projection process retains the visibility judgment, which can reduce the propagation of image recognition errors to three-dimensional residual discrimination.

[0036] In this embodiment, the formation of uncertain regions follows the same data index as the candidate regions for dangerous rocks and falls, rather than being directly deleted as discarded results. The processor writes regions lacking continuous chains of residual symbols, with low point density confidence values, where the main residual locations are covered by shadow areas or vegetation occlusion areas, and where the image recognition semantics and geometric residual boundaries are not closed into uncertain regions. It retains the boundaries of the residual connected domains, the source of the occlusion semantics, the point density confidence value, and the historical stability label. Uncertain regions can be re-matched with new image recognition results and new geometric residual fields when subsequent inspection data is accessed. If uncertain regions continuously shift outward, have enhanced crack textures, or have closed rock mass exposed boundaries in subsequent inspections, they are transferred to the candidate regions for dangerous rocks and falls. If the point density recovers and the residuals disappear in subsequent inspections, they are recorded as transient occlusion or sampling anomalies. The advantage of this embodiment is that the system does not lose abnormal clues due to single image occlusion or sparse point clouds, nor does it directly output residuals with insufficient evidence as dangerous rocks.

[0037] In a preferred embodiment, the training samples for the candidate discriminant model consist of stable rock face regions, verified dangerous rock face regions, shaded regions, and vegetation-covered regions from historical inspection data. The input fields in the training samples are consistent with the cross-modal consistency feature vectors in Table 2. Stable rock face regions provide samples with low residual continuity, low fracture overlap, and high historical stability; verified dangerous rock face regions provide samples with high residual continuity, high boundary closure, and adjacent fractures; and shaded and vegetation-covered regions provide samples with high occlusion semantic proportion and low point density confidence. During training, candidate labels, verification labels, and occlusion labels are used as output categories. Constraint branches are set for the proportion of occlusion semantics and the confidence value of point density, so that samples with strong occlusion evidence are preferentially included in the verification label or occlusion label. When implementing the classification tree, each branch node corresponds to the judgment condition of a feature field. When implementing the shallow neural classifier, the input layer receives the normalized feature vector, and the output layer gives the category code. The advantage of this embodiment is that the input of the discrimination model is consistent with the front-end geometric residual and image recognition evidence, and the model output can be traced back to specific voxel and pixel evidence.

[0038] In this embodiment, the railway safety corridor participates in spatial constraints as part of the 3D point cloud twin reference body, but does not change the core data processing logic of dangerous rockfall identification. When constructing the reference body, the processor writes the railway safety corridor into the coordinate space, so that the spatial relationship between the candidate region and the corridor can be recorded when the candidate region is output. Rock wall voxels, residual connected domains and image recognition semantic evidence are still processed according to the same voxel index. When the 3D boundary of a certain dangerous rockfall candidate region is close to the spatial range of the corridor, the output result retains its 3D boundary and displacement vector for subsequent risk judgment. When the candidate region is far away from the corridor range, the candidate label or verification label is still output in the same way. The image recognition and point cloud residual discrimination results are not directly changed due to the business level. The processor records the relative position between the candidate object and the corridor in the result data, so that the dangerous rockfall identification result has spatial reference. The advantage of this embodiment is that the railway safety corridor participates in the calculation as a coordinate constraint and result index, and does not replace the 3D residual and image recognition discrimination of the actual changes in the rock mass.

[0039] In one embodiment, a method for identifying dangerous rockfalls along railways based on 3D point cloud twins is provided. The method includes constructing a 3D point cloud twin reference body for rock walls and track safety corridors along the railway line based on historical point clouds; receiving current inspection point clouds and current images; mapping the current inspection point clouds and the current image to the coordinate space of the 3D point cloud twin reference body; generating a signed geometric residual field based on the surface distance, normal variation, and point density difference between the current inspection point clouds and the 3D point cloud twin reference body; performing image recognition on the current image to obtain image recognition results including exposed rock areas, fracture texture areas, shadow areas, and vegetation-covered areas; backprojecting the image recognition results to the coordinate space; and determining the dangerous rockfall based on the boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results. In the specific execution of the process, the historical point cloud first forms a twin reference volume containing voxel numbers, normal vectors, curvature, roughness, point density, and historical stable labels. The current inspection point cloud enters a unified coordinate space based on the stable registration set and generates a residual vector. After image recognition, the current image forms a semantic segmentation result. The semantic segmentation result is converted into voxel semantic evidence through projection depth and visibility markers. The residual connected components and voxel semantic evidence are matched with neighborhood and closure to form candidate labels, verification labels, or occlusion labels. Finally, the three-dimensional boundary, displacement vector, volume estimate, label category, and semantic evidence number are output. The advantage of this embodiment is that the method processing flow is consistent with the system operation logic, and the closed-loop processing of three-dimensional point cloud twinning, image recognition, residual interpretation, and candidate output can be completed with the same inventive concept.

Claims

1. A railway rockfall identification system based on 3D point cloud twins, characterized in that, The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it constructs a three-dimensional point cloud twin reference body of the rock wall and track safety corridor along the railway based on historical point clouds. It receives the current inspection point cloud and the current image, maps the current inspection point cloud and the current image to the coordinate space of the three-dimensional point cloud twin reference body, generates a signed geometric residual field based on the surface distance, normal variation and point density difference between the current inspection point cloud and the three-dimensional point cloud twin reference body, performs image recognition on the current image, and obtains image recognition results including exposed rock areas, fracture texture areas, shadow areas and vegetation occlusion areas. It backprojects the image recognition results to the coordinate space, and determines the candidate areas and uncertain areas of dangerous rockfalls based on the boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results.

2. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 1, characterized in that, When the processor constructs the three-dimensional point cloud twin reference body, it establishes a unified coordinate framework based on the track surface, slope toe line, and continuous rock wall surface in the historical point cloud. The continuous rock wall surface is divided into two levels of expression: voxel blocks and local surface patches. Each voxel block is associated with a normal vector, curvature, roughness, point density, and historical stability label. Voxel blocks that satisfy the condition that the position residual of multiple periods is less than the stability threshold and do not overlap with the crack texture area obtained by image recognition of historical inspection images are written into a stable registration set. The stable registration set and the track safety corridor are jointly written into the spatial constraints of the three-dimensional point cloud twin reference body.

3. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 2, characterized in that, When the processor generates the signed geometric residual field, it uses the stable registration set as the registration constraint to perform resolution-consistent resampling on the current inspection point cloud, determines the normal projection distance from the current point to the twin reference surface as the signed distance value, writes the local normal rotation amount, curvature change amount, surface convexity mark, surface concavity mark and point density confidence value into the residual vector, generates residual connected components for residual vectors that span adjacent voxels and have the same sign direction, and retains the index of the residual vector and the corresponding voxel block.

4. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 3, characterized in that, When the processor obtains the image recognition result, it divides the current image into exposed rock areas, fissure texture areas, shadow areas, and vegetation occlusion areas. Based on the projection relationship between the current image and the current inspection point cloud, it converts the pixel skeleton of the fissure texture area, the semantic boundary of the exposed rock area, the occlusion boundary of the shadow area, and the occlusion boundary of the vegetation occlusion area into voxel semantic evidence in the coordinate space. It then establishes an index relationship between the voxel semantic evidence and the signed geometric residual field, and records the pixel source, projection depth, and visibility marker of the voxel semantic evidence.

5. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 2, characterized in that, The historical stable labels are jointly generated by multiple historical inspection point clouds and historical inspection images. The processor jointly encodes the centroid shift, normal shift, point density fluctuation and image recognition semantic changes of the same voxel block in adjacent inspection periods. Voxel blocks that continuously show low centroid shift, low normal shift and no crack texture enhancement are marked as stable voxel blocks. Voxel blocks that have short-term density reduction and fall into vegetation shading area or shadow area are marked as transient shading voxel blocks. The transient shading voxel blocks are excluded from the stable registration set.

6. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 3, characterized in that, When the processor performs multi-scale filtering on the residual connected domain, it obtains the convex residual connected domain at the first voxel scale, obtains the normal rotation region that overlaps with the convex residual connected domain at the second voxel scale, and traces the continuous chain of residual symbols along the adjacency direction of the twin reference plane. For a residual connected region that lacks the continuous chain of residual symbols and whose point density confidence value is lower than the density threshold, the processor transfers the residual connected region into the uncertain region; For residual connected domains that overlap across scales, the processor preserves their boundary voxel sets.

7. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 4, characterized in that, After establishing the index relationship, the processor performs neighborhood matching between the pixel skeleton of the crack texture area in the voxel semantic evidence and the residual boundary in the signed geometric residual field, performs closure matching between the semantic boundary of the exposed rock mass area and the outer contour of the residual connected domain, and writes the area that simultaneously satisfies the adjacent residual boundary of the crack texture area pixel skeleton, the semantic boundary of the exposed rock mass area surrounding the residual connected domain, and the area where the main residual position is not covered by the shadow area and the vegetation occlusion area into the candidate area of ​​dangerous rockfall, and records the corresponding semantic evidence number.

8. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 7, characterized in that, The processor constructs a cross-modal consistency feature vector for each candidate region of dangerous rockfall. The cross-modal consistency feature vector includes the continuous length of the three-dimensional residual, the proportion of residual sign consistency, the concentration of normal rotation, the overlap rate of the fracture texture area boundary, the closure degree of the exposed rock mass area, the proportion of occlusion semantics, the point density confidence value, and the historical stable label change status. The processor inputs the cross-modal consistency feature vector into the candidate discrimination model, and the candidate discrimination model outputs the category code according to the candidate label, the verification label, and the occlusion label.

9. The railway rockfall identification system based on three-dimensional point cloud twins according to claim 8, characterized in that, The processor performs three-dimensional connectivity merging of adjacent candidate regions for dangerous rockfalls based on the candidate labels, maintains the independent boundary of the corresponding region in the uncertain region based on the verification labels, extracts the displacement vector of the region that has completed three-dimensional connectivity merging along the normal direction of the twin reference surface, calculates the volume estimate value according to the overlap relationship between the residual connected domain formed by the signed geometric residual field and the coordinate space voxel partition, and outputs the dangerous rockfall identification result including three-dimensional boundary, displacement vector, volume estimate value, label category and semantic evidence number.

10. A method for identifying dangerous rockfalls and falling rocks along railways based on 3D point cloud twins, characterized in that, include: A three-dimensional point cloud twin reference body was constructed based on historical point clouds to build safety corridors for rock walls and tracks along the railway line. Receive the current inspection point cloud and the current image, and map the current inspection point cloud and the current image to the coordinate space of the three-dimensional point cloud twin reference body; A signed geometric residual field is generated based on the surface distance, normal variation, and point density difference between the current inspected point cloud and the three-dimensional point cloud twin reference. Image recognition is performed on the current image to obtain image recognition results including exposed rock areas, fracture texture areas, shadow areas, and vegetation-covered areas; The image recognition results are back-projected into the coordinate space, and the candidate regions and uncertain regions of dangerous rockfalls are determined by the boundary consistency and spatial continuity between the signed geometric residual field and the image recognition results.

Images