Railway external environment hidden danger dynamic screening method based on unmanned aerial vehicle video

Abstract

The invention discloses a railway external environment hidden danger dynamic screening method based on an unmanned aerial vehicle video, and the method comprises the steps: building a railway geographic grid based on a route, cutting a video frame image, achieving the discretization of a video image space in a railway direction, and reducing the image distortion; then, performing image matching on the unmanned aerial vehicle video and the reference image by adopting a GNN method based on large-kernel 3D convolution, and realizing geocoding and mileage positioning of the unmanned aerial vehicle video; and finally, performing multi-temporal change detection on the image with the geographic coordinates to realize automatic troubleshooting of hidden dangers and geographic information statistics. According to the method, geographic coding processing is carried out on the unmanned aerial vehicle video data, the incidence relation between any pixel of the video frame and the railway mileage is established, geographic coordinate positioning of the pixels of the unmanned aerial vehicle video frame is achieved, and a high-precision and high-frequency data source is provided for railway external environment remote sensing quantitative dynamic monitoring; and the universal application of the unmanned aerial vehicle video in railway external environment safety monitoring is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of railway external environment monitoring, and in particular to a method for dynamically screening hidden dangers in the railway external environment based on drone videos. Background Art

[0002] Efficient screening for hidden dangers in the railway's external environment is a crucial component of ensuring public safety and plays a significant role in developing the national economy and safeguarding people's well-being. With the increasing frequency of extreme weather and natural disasters, the complexity of external environmental hazards along railway lines and the cost of manual screening are increasing. This leads to difficulties in quantitative analysis and long screening times during railway operations and maintenance, posing a serious threat to the safety of people's property. To effectively improve the accuracy and efficiency of screening for hidden dangers in the railway's external environment, the state and railway companies are vigorously promoting the development of a "three-in-one" safety assurance system encompassing human, physical, and technical defenses, and are actively exploring new technologies for efficient and accurate dynamic monitoring of the railway's external environment.

[0003] With the continuous advancement of drone technology, multi-sensor drones have found significant applications in railway surveying and mapping, construction progress monitoring, and equipment safety patrols and supervision. Drone inspection technology enables high-frequency monitoring and dynamic intelligent inspections. Multi-phase drone imagery is a key data source for drone inspections. Intelligent interpretation of multi-phase aerial video data offers advantages over traditional manual ground inspections in the inspection of critical construction site equipment and facilities, flood control and geological hazards, and external environmental hazards, offering greater efficiency and coverage.

[0004] However, the existing methods for screening railway operation and maintenance hidden dangers based on drone videos still have some problems, as follows:

[0005] (1) Single frames of drone videos lack geographic information and cannot be quantitatively analyzed.

[0006] Currently, many railway bureaus' engineering sections are equipped with drones, and drone video has become a crucial means of recording on-site inspections. Existing drone-based hazard screening methods mostly rely on multi-frame video change detection at fixed work sites. These methods screen for hazards solely within the pixel space of the video. The relatively fixed video scenes and the lack of geographic information within a single frame make it impossible to dynamically correlate change detection results with railway mileage, hindering subsequent quantitative analysis.

[0007] (2) The video screening results of off-road environmental hazards lack accurate entity attribute information, making it impossible to quickly manage and rectify the hazards.

[0008] Existing drone video hazard screening methods struggle to convert hazard screening results into geographic entities, making it impossible to calculate hazard-related entity attributes. Most drone video hazard screening methods lack geographic information and are not dynamically linked to railway mileage. Screening results are limited to merging changing pixels and displaying results across pixel space. These methods struggle to directly obtain attribute information such as the railway mileage, distance from the railway center, and area of ​​the hazard, hindering the refined management and remediation of external environmental hazards.

[0009] (3) The existing drone video processing process is complex, requiring high hardware processing capabilities of the drone, and the video geocoding cycle is long.

[0010] Drone videos can be used to create three-dimensional geographic scenes and achieve video geolocation based on existing GIS methods. However, this process requires high-precision camera position and attitude information and long-term complex calculations, which places high demands on the hardware processing capabilities of the drone equipment. Existing drone equipment has low configuration and cannot meet the requirements of video three-dimensional reconstruction, making it difficult to quickly achieve video geolocation. Summary of the Invention

[0011] In order to solve the problems in the background technology, the present invention provides a dynamic screening method for hidden dangers in the railway external environment based on drone video, which can use low-configuration drone equipment to quickly and accurately locate changing entities.

[0012] To this end, the present invention adopts the following technical solutions:

[0013] A method for dynamically screening hidden dangers in the railway external environment based on drone videos includes the following steps:

[0014] S1, prepare the reference image of the area to be screened and obtain the first phase of drone video of the area to be screened;

[0015] S2, dividing the area to be screened into a railway grid consisting of a plurality of grids of the same size, wherein the length and width of the grids in the railway grid are both smaller than the length and width of the geographical range corresponding to each frame of the drone video;

[0016] S3, determining the corresponding video frame image in the drone video according to the position of each grid in S2;

[0017] S4, registering the video frame image with the reference image based on a large-core three-dimensional convolutional attention network to obtain the same-name points of the video frame image and the reference image;

[0018] S5, converting the video frame image to the geographic coordinate system of the reference image according to the same-name points obtained in S4;

[0019] S6, cropping the converted video frame image to obtain a first phase image that matches the corresponding grid range;

[0020] S7, obtain the second phase of drone video of the area to be screened, process it using the methods of steps S2 to S6 to obtain the second phase of images, perform change detection on the second phase of images and the first phase of images to obtain a pixel set of the changed area; S8, locate the changed entity in the area to be screened based on the pixel set of the changed area, and determine the specific position of the changed entity.

[0021] The method for dividing the railway grid in step S2 is:

[0022] According to each route of the UAV, the intersection of the normal line at the minimum mileage of the railway in the area to be screened and the projection of the route is taken as the starting point, and the UAV is detected every M. x / 2Set a grid center point G i , i is the number of the grid center point, and finally the grid center point set is obtained. The grid center point G i The coordinates of a railway are determined by its mileage value and perpendicular distance from the railway; M x N is the length of the geographical range corresponding to the image of a single frame of drone video along the route direction; x is the number of pixels of a single-frame drone video along the flight path; f is the focal length of the drone video; P s is the pixel size of the drone's camera; H is the drone's flight altitude;

[0023] Each of the grid center points G i As the grid B defined around it i The center of the screened area is divided into several grids B of the same size. i Railway grid.

[0024] The reference image in S1 is a satellite image or orthophoto image of the monitoring range covering both sides of the railway in the area to be screened; the shooting range between the routes of the drone video overlaps by more than half to reduce the impact of image distortion; the angle between the direction of the drone's lens and the route direction is between 75° and 90°.

[0025] Step S3 includes the following sub-steps:

[0026] S31, the grid center point G i The coordinates are transformed into the geographic coordinate system;

[0027] S32, using the position information recorded when the drone shoots each frame to filter the distance from the grid center point G i The number of the most recent video frame F Gi ;

[0028] S33, using the number F of the video frame Gi Capture video frame image P in drone video i .

[0029] Step S4 includes the following sub-steps:

[0030] S41, searching for the reference image corresponding to the current grid;

[0031] S42, for the video frame image and the reference image, respectively calculating their feature points and feature descriptors using a neural network based on a large-kernel three-dimensional convolution, the neural network comprising: a shared encoder network Encoder for processing input images and extracting image features, a decoder DecoderP for feature point detection, and a decoder DecoderD for feature point description;

[0032] S43, obtaining the same-name points between the corresponding video frame image and the reference image through feature matching of the attention graph neural network.

[0033] Step S6 includes the following sub-steps:

[0034] S61, determining the boundaries of the grid corresponding to each grid center point in the geographic coordinate system;

[0035] S62, using the four-dimensional range to geocode the video frame image P i After clipping, the resulting image is the image that matches the grid range. i .

[0036] Preferably, when laying out the drone routes, take the railway centerline as the center, calculate the parallel lines offset by a distance of D1 / 2 on both sides of the railway, and use them as the first route on the left and right sides of the railway. Other routes are laid out in parallel in sequence according to the route distance until the entire railway monitoring range is covered; the distance from the outermost route on both sides of the railway to the railway centerline is greater than the distance from the railway monitoring range boundary to the railway centerline; wherein D1≤0.5*M y , M y =P s *H*N y / f;M y is the width of the geographical range corresponding to the image of a single-frame drone video in the direction perpendicular to the flight path, P s is the pixel size of the video camera; H is the flying height of the drone; N y is the number of pixels of a single-frame drone video image perpendicular to the flight path; f is the focal length of the drone video camera.

[0037] In step S7, change detection is performed on images of different periods using the Mamba spatiotemporal state model.

[0038] The specific location of the changed entity is determined by the attribute information of the changed entity, and the attribute information includes: the line to which it belongs, mileage, distance from the railway, location, area, time and railway up and down line type.

[0039] Preferably, the process further includes S9, notifying the person in charge to conduct on-site review based on the specific location of the changed entity determined in step S8.

[0040] Use the same method to obtain each subsequent image, perform change detection on it compared with the previous image, and obtain a pixel set of the changed area; locate the changed entity in the area to be screened based on the pixel set of the changed area, and determine the specific position of the changed entity, thereby realizing continuous screening of the area to be screened.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. By geocoding drone video data, establishing a correlation between any pixel in the video frame and railway mileage, and realizing the geographic coordinate positioning of drone video frame pixels, this method provides a high-precision and high-frequency data source for quantitative dynamic remote sensing monitoring of the railway external environment, and expands the universal application of drone video in railway external environment safety monitoring.

[0043] 2. Compared with the traditional method of stitching the entire video frame image, the method of the present invention first uses coordinate grid cropping and then geocoding, and then stitches adjacent grid images. It can maximize the reduction of distorted pixels in the video frame geocoding and improve the geographic accuracy of the geocoding image results.

[0044] 3. Compared with traditional CNN and Transformer models, the railway hidden danger change detection model based on the time-state-space model constructed in the present invention has better long-distance information interaction capabilities, can more efficiently process long sequence data, reduce memory requirements, and realize the spatiotemporal interaction of multi-temporal railway hidden danger features and the rapid and accurate extraction of hidden danger pixels.

[0045] 4. Compared with other deep learning image matching methods, the method of the present invention replaces traditional convolution with three-dimensional large-kernel convolution, which increases the receptive field while taking into account three-dimensional information. It can flexibly mine the long-range dependencies of matching features with lower computational pressure, and realizes rapid matching of video frame images with reference images, thereby reducing the requirements for drone hardware performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an embodiment of the present invention;

[0047] Figure 2A schematic diagram of a UAV route layout scheme according to an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the structure of a large-core three-dimensional convolutional attention network in an embodiment of the present invention;

[0049] Figure 4 It is the overall framework of the change detection network based on the Mamba model in the embodiment of the present invention;

[0050] Figure 5 This is the specific structure of the encoder and decoder in the Mamba spatiotemporal state model in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0052] like Figure 1 As shown, the method for dynamic screening of hidden dangers in the railway external environment based on drone video of the present invention includes the following steps:

[0053] S1, prepare reference images of the area to be screened and obtain drone video:

[0054] First, use satellite images or orthophotos covering the monitoring range on both sides of the railway in the area to be screened as reference images;

[0055] Then, if Figure 2 As shown, a drone is used to perform aerial photography according to the drone route to obtain drone videos.

[0056] The drone routes are set in parallel, the distance between two adjacent drone routes is D1, and the projection distance between the drone route closest to the railway and the railway is D1 / 2.

[0057] In order to reduce the impact of image distortion, the shooting ranges between routes should overlap by more than half, that is, D1≤0.5*M y , where: M y =P s *H*N y / f,M y is the width of the geographical range corresponding to the image of a single-frame drone video in the direction perpendicular to the flight path, P s is the pixel size of the video camera, in microns; H is the flying height of the drone; N yis the number of pixels of a single-frame drone video image perpendicular to the flight path; f is the focal length of the drone video camera.

[0058] When laying out drone flight paths, use the railway centerline as the center and draw parallel lines offset by a distance of D1 / 2 on either side of the railway. These lines serve as the first flight path (projection) on the left and right sides of the railway. Additional flight paths are laid out in parallel, following the flight path distances, until the entire railway monitoring range is covered. The distance from the outermost flight paths on either side of the railway to the railway centerline must be greater than the distance from the railway monitoring range boundary to the railway centerline. To minimize the effects of lens distortion, the angle between the drone's lens and the flight path during flight should be between 75° and 90°.

[0059] S2, divide the railway grid:

[0060] First, determine several grid center points G on the UAV route i The grid center point set is composed of i, where i is the number of the grid center point. Specifically: for each drone route, the intersection of the normal line and the route at the minimum mileage of the railway in the area to be screened is taken as the starting point, and every M x / 2Set a grid center point, M x is the length of the geographical range corresponding to the image of a single frame of drone video along the route direction, and finally the grid center point set is obtained. When the grid center point is located on the left side facing the direction of the long mileage of the railway, its coordinates are (L i ,D i ); When the grid center point is on the right side facing the direction of the long railway mileage, its coordinates are (L i ,-D i ).

[0061] Among them, L i is the line mileage value of the i-th grid center point, D i is the perpendicular distance from the center point of the i-th grid to the railway line, M x =P s *H*N x / f. Where, N x The number of pixels along the flight path of a single-frame drone video.

[0062] Then, divide the grid B with each grid center point as the center i , divide the area to be screened into several grids B of the same size i Railway grid composed of grid B i The width in the direction perpendicular to the railway is D1, and the length in the direction parallel to the railway is M x / 2;

[0063] In the railway grid divided according to the above method, the length and width of the grid are both about half of the video image range, which can ensure that only the image with less deformation in the center of the key video frame is used, reducing the impact of geometric distortion.

[0064] S3, calculating the video key frame, includes the following steps:

[0065] S31, performing coordinate transformation of the grid center point, including:

[0066] S311, according to the grid center point G i The mileage value is located at the nearest pile point C n , n is the number of the pile point.

[0067] If you click G i The mileage value is greater than that of point C n The mileage value is G i Located near pile point C n The direction of the long mileage is located at stake C n and C n+1 Otherwise, it is located in the direction of small mileage and at stake point C n-1 and C n between.

[0068] S312, calculate G i The perpendicular coordinates of a point:

[0069] Let point G i The line segment formed by connecting two adjacent pile points is V L V L+1 The mileage at both ends is L L and L L+1 , find the grid center point G by linear interpolation i On the Vertical Point G′ of the Railway i The coordinates (G′ ix ,G′ iy ),have:

[0070]

[0071] Where d is V L V L+1 The straight-line distance between the two endpoints, (V Lx ,V Ly ) is V L The plane coordinates in the geographic coordinate system, (V (L+1)x ,V (L+1)y ) is V L+1 The plane coordinates in the geographic coordinate system.

[0072] S313, calculate point G i The plane coordinates in the geographic coordinate system (Gix ,G iy ).

[0073] First determine the line segment V L V L+1 The azimuth of point G i The coordinates of the vertical point and point G i Calculation of the position relationship with the railway G i G i ′’s coordinate azimuth, and finally use the polar coordinate method to solve point G i The plane coordinates of the point in the geographic coordinate system (G ix ,G iy ).

[0074] S32, calculating the numbers of the video frames corresponding to the grids.

[0075] Determine each grid B i The target video frame corresponding to the center of G is the premise for establishing the spatiotemporal relationship. In order to ensure that the video frame image is the best match with the grid, the present invention iteratively searches and determines G according to the principle that the distance between the center of the video frame image and the center of the grid is the shortest. i The video frame F corresponding to the point Gi The SRT file of the drone flight records the number of video frames. N POS with camera station (drone) N The corresponding relationship, where POS N Contains the three-dimensional space coordinates (X, Y, Z) of the camera station and the aerial posture expressed by the HPR angle system (Heading, Pitch, Roll), so the video frame and the camera point coordinates have Frame N =f(X, Y, Z). On the other hand, the grid coordinates with elevation can be used to solve the image plane coordinates according to the collinearity equation, thereby calculating the distance to the image center. This step specifically includes:

[0076] S321, first according to the grid center point G i Plane coordinates (G ix ,G iy ) and the DEM model with elevation information obtained by the drone, and calculate the three-dimensional coordinates (G ix ,G iy ,G iz ).

[0077] S322, then calculate the distance between the drone and the grid center point G i Video frame F when the distance is closest G1 As the initial value, and calculate the point G according to the collinear equation L of formula (2) i The projection point P on the frame imageGi (x Gi ,y Gi )=L[(Gx,Gy,Gz),POS(F G1 )], then the projection point P can be calculated Gi and the frame image center point P c The distance D.

[0078]

[0079] Where (X S ,Y S ,Z S ) is the line element of the image exterior orientation element, that is, the imaging position of each image, which can be obtained from POS N (x0, y0) is the coordinate of the principal point in the image plane coordinate system, F is the principal distance, and F, x0, y0 are the intrinsic parameters of the camera system, which can be obtained through camera calibration.

[0080] The rotation matrix A can be obtained by performing matrix operations on the image's attitude angles. When the HPR rotation system (Heading, Pitch, Roll) is used, the calculation process is as follows:

[0081]

[0082] a1=cos H·cos R+sin H·sin P·sin R,

[0083] a2=sin H·cos P,

[0084] a3=cos H·sin R-sin H·sin P·cos R,

[0085] b1=cos H·sin P·sin R-sin H·cos R,

[0086] b2=cos H·cos P,

[0087] b3=-sin H·sin R-cos H·sin P·cos R,

[0088] c1=-cos P·sin R,

[0089] c2=sin P,

[0090] c3=cos P·cos R;

[0091] Among them, H, P and R are the attitude angles of the image, A H 、A P and A Ris the attitude angle matrix.

[0092] S323, use the iterative method to obtain the grid center point G i The closest target video frame.

[0093] Iteratively add S frames on the timeline of the drone video until the distance D converges to the minimum. The step length s follows the following rules: if the distance D increases after moving, the step length s is halved; if the distance D decreases or remains unchanged after moving, continue moving; when the step length s is 1 and the distance D increases after moving, the iteration ends. The distance D is calculated continuously until the iteration stops, and the distance to the grid center point G is obtained. i The number of the video frame with the smallest distance F Gi .

[0094] S33, generate video frame image:

[0095] Use the video frame number F Gi Capture video frame image P in drone video i :For example, the first frame F G1 The corresponding picture is P1, then extract the video F Gi The video frame image P corresponding to the frame i .

[0096] S4, registering the video frame with the reference image, including the following steps:

[0097] S41, search for reference images:

[0098] Each grid center point G i As the center, with 2*MAX(Mx,My) as the radius, search the reference image corresponding to the current grid and determine the grid center point G i The corresponding reference image R i .

[0099] S42, extracting feature points and descriptors based on a large-core 3D convolutional attention network:

[0100] For the video frame image P i With reference image R i , using a neural network based on large-kernel three-dimensional convolution to calculate its feature points and feature descriptors respectively. The network consists of three parts: a shared encoder network Encoder for processing input images and extracting image features, a decoder DecoderP for feature point detection, and a decoder DecoderD for feature point description, where:

[0101] Large kernel 3D convolution can flexibly mine the long-range dependencies of 3D features with low computational pressure. The detailed structure is as follows: Figure 3As shown in the figure. The Encoder uses convolutional layers and large-kernel 3D convolution to exploit the spatial correlation of the image, and uses three pooling layers to reduce the input image to one-eighth and increase the number of channels to 128. DecoderP uses the Encoder output and performs a large-kernel 3D convolution attention operation, outputting a probability map with the same size as the original image and 1 channel. Each pixel corresponds to the probability that the point in the input image is a feature point. Non-maximum suppression and threshold filtering are then performed to avoid the aggregation of high-confidence feature points and retain feature points of better quality. Similar to DecoderP, DecoderD uses convolution, interpolation, and L2 normalization to obtain a descriptor for each pixel, with a feature vector length of 256. After the above processing, the feature points and descriptors of the video frame and reference image are extracted.

[0102] S43, feature matching based on attention graph neural network to obtain the video frame image P i With reference image R i The points of similarity between them are:

[0103] Based on the feature points and descriptors obtained in step S42, key point encoding is performed so that the feature points and descriptors can be added together to form a high-dimensional feature vector. Based on the feature vector, a graph neural network (GNN) is constructed and optimized using the self-attention mechanism and the cross-attention mechanism to obtain a description vector for feature matching.

[0104] Using the optimal transmission theory, a matching score matrix is ​​established and solved to obtain the matching relationship between feature points. Specifically, the similarity between the feature description vectors generated by GNN is converted into a cost matrix B, where B xy Indicates that the feature point x is moved from P i Match to R i The cost of feature point y. The cost is calculated using the cosine distance between feature vectors. Find the transmission matrix T so that the total transmission cost ∑ x,y T xy B xy To minimize, the transfer matrix T must satisfy the marginal constraint that the total transfer amount of each feature point is equal to its weight. The optimal transfer problem can be viewed as a linear programming problem. Solving this problem yields the optimal transfer matrix T, which can be directly used as the matching score matrix S. Based on the matching score matrix, the feature point pairs with the highest scores are selected as matching pairs. Thresholds or other strategies can be used to determine the final matching relationship.

[0105] Finally, the matching results are adjusted using local geometric constraints to obtain the video frame image P i With reference image R i The points of the same name between .

[0106] S5, geocode the video frame image:

[0107] According to the same-name points obtained in S4, P is established i With R i The conversion matrix C between i , according to the transformation matrix C i , P i Convert to R i The coordinate system where the video frame image P is located is completed i The geographic coding is performed to obtain the coordinates of all pixels in the video frame image in the geographic coordinate system; specifically, the following steps are included:

[0108] S51, determine the same-name points:

[0109] First, according to each grid video frame image P determined in S4 i With reference image R i The points of the same name between them, let P i and R i The points with the same name are {p i1 ,p i2 ,…,p in} and {r i1 ,r i2 ,…,r in}, n is P i and R i The total number of synonyms.

[0110] S52, construct the transformation matrix:

[0111] Transformation matrix C i For a 3x3 matrix:

[0112]

[0113] S53, establish the equation system:

[0114] For each pair of points with the same name (p ij ,r ij ),have:

[0115]

[0116] Where (x ij ,y ij ) is P i Point (x′ ij ,y′ ij ) is R i Points in . Expanding this equation, we get:

[0117]

[0118] S54, least squares method solution:

[0119] Since there are n points of the same name, we can establish a linear equation system and then use the least squares method to solve C i The goal of the least squares method is to minimize the sum of squares of the transformation errors of all point pairs.

[0120] S55, calculate the transformation matrix:

[0121] By solving the least squares problem, we can get the transformation matrix C i Parameter a i11 ,a i12 ,…,a i33 .

[0122] S56, apply the transformation matrix:

[0123] Finally, use the obtained transformation matrix C i , P i All points in R are converted to i In the coordinate system where the video frame image P is located, i Geocoding:

[0124]

[0125] Formula (5-4) shows how to use the transformation matrix C i P i The point (x ij ,y ij ) to R i In the coordinate system where it is located, the new coordinate (x′ ij ,y′ ij ).

[0126] S6, the converted video frame image P i Cut and get the image o that matches the corresponding grid range i , including the following steps:

[0127] S61, determine each grid center point G i Corresponding grid B i The four-dimensional range S in the geographic coordinate system Gi .

[0128] S62, geocoding the video frame image P i To perform cropping:

[0129] Since the video frame corresponding to the grid is determined in step S32, and the video frame image P is completed through coordinate conversion in step S5, i Geocoding, so use the range SGi The geocoded video frame image P i By clipping, we can get the first phase image O that matches the grid range. i This step ensures that the image is geographically aligned with the grid, facilitating subsequent geographic analysis and applications.

[0130] S7, based on the Mamba spatiotemporal state model, performs change detection on images of different periods. Figure 4 and Figure 5 , the specific steps are as follows:

[0131] S71, obtain the second phase of drone video of the area to be screened, and use the method of steps S2-S6 to obtain the same image O as the first phase. i Second phase images at different times O i ';

[0132] S72, perform spatiotemporal state feature encoding and basic decoding:

[0133] In order to achieve complete modeling of the spatial information around the hidden danger and improve the efficiency of model detection, the present invention uses the cross-scanning spatial state module (VSS) to construct a feature encoder. i And the second issue of the image O i ′ as input.

[0134] The input data then goes through four identical encoding stages for spatial modeling. In each stage, the input data is first downsampled and then the spatial context information is fully modeled using a cross-scan spatial state module.

[0135] After feature encoding, in order to learn the spatiotemporal relationship of time series images, the spatiotemporal relationship modeling mechanism is used to encode the feature F encoder Parallel decoding is performed in both time and space. Three spatiotemporal modeling mechanisms are used: sequential modeling, cross-modeling, and parallel modeling. These three mechanisms, along with the Mamba architecture, fully exploit the inherent spatiotemporal relationships of multi-temporal features, helping the decoder achieve accurate change detection results.

[0136] S73, perform secondary decoding of changes and semantics:

[0137] The change decoder is used to determine the pixels that have changed before and after, and the semantic decoder is used to determine the type of surface objects in multi-time series data, and the pixel range and specific category of the changed area are obtained.

[0138] Specifically, the change decoder consists of four stages. At the beginning of each stage, the spatiotemporal relationships between multiple temporal features are modeled using the spatiotemporal state model (STSS). After processing by the change decoder, a resulting image of changed and unchanged pixels is obtained.

[0139] The semantic decoder focuses on spatial information and performs spatial modeling exclusively, consisting of four stages. At the beginning of each stage, the cross-scan spatial state module models the global spatial context of the input data. The fusion module then upsamples the feature maps and fuses them with the higher-resolution underlying encoded features. Finally, the module's output features are upsampled four times to predict the corresponding hidden danger coverage type.

[0140] For details of the Mamba spatiotemporal state model, see Hongruixuan Chen, Jian Song, Chengdu Han, et al. ChangeMamba: Remote Sensing Change Detection with Spatiotemporal StateSpace Model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-20.

[0141] S74, after performing change detection on the images of different periods, the pixel values ​​of the images are divided into changed pixel values ​​and non-changed pixel values, and then the areas where the changed pixel values ​​of all adjacent grids are located are merged to obtain a changed area pixel set.

[0142] S8, locate the changed entity and upload it:

[0143] The vector boundaries of the entire change area are automatically calculated through the edge detection algorithm, and each independent vector box is recorded as a change entity A. k , k is the number of the changed entity, according to the railway mileage data, the changed entity A is obtained k After calculation, each change entity A k The attribute information is used to determine the changed entity A. k The specific location of the railway. Attribute information includes: line, mileage, distance from the railway, location, area, time and railway type.

[0144] S9, system push and on-site review:

[0145] The monitoring system will change each entity A k The mileage position in the attribute information, combined with the jurisdiction mileage range of each unit, will be used to change each entity A k The mobile inspection APP is pushed to the person in charge of the railway section to remind the railway responsible person to conduct on-site review.

[0146] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for dynamic screening of hidden dangers in railway external environment based on drone video, characterized in that: The following steps are involved: S1, prepare reference images of the area to be screened; obtain the first phase of drone video of the area to be screened; S2, dividing the area to be screened into a railway grid consisting of a plurality of grids of the same size, wherein the length and width of each grid are respectively smaller than the length and width of the geographical range corresponding to each frame of the drone video; S3, determining the corresponding video frame image in the drone video according to the position of each grid in S2; S4, registering the video frame image with the reference image based on a large-core three-dimensional convolutional attention network to obtain the same-name points of the video frame image and the reference image; S5, converting the video frame image to the geographic coordinate system of the reference image according to the same-name points obtained in S4; S6, cropping the video frame image converted in S5 to obtain a first phase image matching the corresponding grid range; S7, obtaining a second phase of drone video of the area to be screened, processing it using the methods of steps S2 to S6 to obtain a second phase of imagery, and performing change detection on the second phase of imagery and the first phase of imagery to obtain a pixel set of the changed area; S8, locating the changed entity in the area to be screened according to the changed area pixel set, and determining the specific position of the changed entity.

2. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 1 is characterized in that: The method for dividing the railway grid in step S2 is: According to each route of the UAV, the intersection of the normal line at the minimum mileage of the railway in the area to be screened and the projection of the route is taken as the starting point, and the UAV is detected every M. x / 2Set a grid center point G i , i is the number of the grid center point, and finally the grid center point set is obtained. The grid center point G i The coordinates of a railway station are determined by its mileage value and perpendicular distance from the railway. in: M x N is the length of the geographical range corresponding to the image of a single frame of drone video along the route direction; x is the number of pixels of a single-frame drone video along the flight path; f is the focal length of the drone video; P s is the pixel size of the drone's camera; H is the drone's flight altitude; Each of the grid center points G i As the grid B defined around it i The center of the screened area is divided into several grids B of the same size. i Railway grid.

3. The method for dynamic screening of hidden dangers in the railway external environment based on drone video according to claim 1 is characterized by: The reference image in S1 is a satellite image or orthophoto image of the monitoring range covering both sides of the railway in the area to be screened; the shooting range between the routes of the drone video overlaps by more than half to reduce the impact of image distortion; the angle between the direction of the drone's lens and the route direction is between 75° and 90°.

4. The method for dynamic screening of hidden dangers in the railway external environment based on drone video according to claim 2 is characterized by: Step S3 includes the following sub-steps: S31, the grid center point G i The coordinates are transformed into the geographic coordinate system; S32, using the position information recorded when the drone shoots each frame to filter the distance from the grid center point G i The number of the most recent video frame F Gi ; S33, using the number F of the video frame Gi Capture video frame image P in drone video i .

5. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 1 is characterized in that: Step S4 includes the following sub-steps: S41, searching for the reference image corresponding to the current grid; S42, for the video frame image and the reference image, respectively calculating their feature points and feature descriptors using a neural network based on a large-kernel three-dimensional convolution, the neural network comprising: a shared encoder network Encoder for processing input images and extracting image features, a decoder DecoderP for feature point detection, and a decoder DecoderD for feature point description; S43, obtaining the same-name points between the corresponding video frame image and the reference image through feature matching of the attention graph neural network.

6. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 5 is characterized in that: Step S6 includes the following sub-steps: S61, determining the boundaries of the grid corresponding to each grid center point in the geographic coordinate system; S62, using the four-dimensional range to geocode the video frame image P i After clipping, the resulting image is the image that matches the grid range. i .

7. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 1 is characterized by: When laying out the drone route, take the railway centerline as the center and calculate the parallel lines offset by D1 / 2 on both sides of the railway as the first route on the left and right sides of the railway. Other routes are laid out in parallel in accordance with the route distance until the entire railway monitoring range is covered. The distance from the outermost route on both sides of the railway to the railway centerline is greater than the distance from the railway monitoring range boundary to the railway centerline. Among them, D1≤0.5*M y , M y =P s *H*N y / f;M y is the width of the geographical range corresponding to the image of a single-frame drone video in the direction perpendicular to the flight path, P s is the pixel size of the video camera; H is the flying height of the drone; N y is the number of pixels of a single-frame drone video image perpendicular to the flight path; f is the focal length of the drone video camera.

8. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 1 is characterized by: In step S7, change detection is performed on images of different periods using the Mamba spatiotemporal state model.

9. The method for dynamic screening of hidden dangers in railway external environment based on drone video according to claim 1, characterized in that: The specific location of the changed entity is determined by the attribute information of the changed entity, and the attribute information includes: the line to which it belongs, mileage, distance from the railway, location, area, time and railway up and down line type.

10. The method for dynamic screening of hidden dangers in the railway external environment based on drone video according to any one of claims 1 to 9, characterized in that: The process further includes S9, notifying the person in charge to conduct on-site review based on the specific location of the changed entity determined in step S8.

Images