Millimeter wave radar map convolution-based urban rail vehicle-mounted autonomous obstacle detection method and system

Abstract

The application provides a kind of city rail vehicle autonomous obstacle detection method and system based on millimeter wave radar graph convolution, belongs to the technical field of rail transit obstacle detection, millimeter wave radar initialization, constantly input collected radar frame to queue;Read several frames of radar point cloud data from the queue, convert the processed point cloud data into a graph structure;The trained graph convolutional neural network is used to process the graph structure, and finally outputs the obstacle detection and classification result, and makes a warning for train travel.The high-dimensional embedding helps to realize information reorganization and high-dimensional aggregation of channel characteristics, and the rotation and translation invariance ensures the accuracy and robustness of obstacle detection;The jump connection structure effectively solves the over-smoothing problem of graph neural network;Through unique network structure and feature selection division, the feature relationship between different points is focused, and global information is obtained by combining attention mechanism, to realize more efficient and accurate detection of rail transit obstacle classification and detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rail transit obstacle detection, and particularly relates to a metro rail vehicle-mounted autonomous obstacle detection method and system based on millimeter wave radar graph convolution. BACKGROUND

[0002] Urban rail transit is a large-capacity public transportation infrastructure and is the backbone of the transportation network of large cities. It has the advantages of large capacity and fast speed and can effectively alleviate urban traffic congestion. The safety and efficiency of the urban rail transit system are related to the life safety and health of passengers and the convenience of travel. Train active obstacle detection is an important research and development direction for realizing the automatic operation of urban rail transit trains and is of great significance for effectively detecting abnormal intrusion obstacles outside the train, improving the obstacle detection capability of unmanned trains, and enhancing the vehicle-mounted autonomous perception capability.

[0003] Metro trains operate in complex and often challenging environments, such as darkness, heavy rain, fog, and other adverse weather conditions. Traditional obstacle detection methods mainly rely on manual detection or ground-based obstacle detection. The former has low detection efficiency and high labor intensity, while the latter has limited detection range and high deployment cost if the entire line is monitored. Train active obstacle detection is an important research and development direction for realizing the automatic operation of urban rail transit trains and is a key technology for improving train operation safety. Vehicle-mounted active obstacle detection installs environmental perception sensors, such as visible light cameras, laser radars, and millimeter wave radars, and computing resources on the vehicle side. Visible light cameras are greatly affected by the environment, and images will degrade to varying degrees in dark environments, resulting in image distortion, detail loss, and low contrast. Laser radars use near-infrared light as a detection medium, and the power attenuates during atmospheric propagation. Using laser radars in rain, snow, fog, and other "adverse weather" conditions not only significantly reduces product performance but also easily causes safety accidents due to "false obstacles".

[0004] Compared with optical cameras and laser radars, millimeter wave radars exhibit strong robustness in rain, snow, fog, and dense dust environments. At the same time, the speed detection accuracy of moving objects based on the Doppler effect is much higher than that of other same-type speed sensors. Therefore, high-reliability and high-precision position and speed detection lay a foundation for the all-weather operation of metro train autonomous obstacle detection. However, compared with camera and laser radar-based sensors, millimeter wave radar point clouds are sparse, have low angular resolution, and have obvious disorder and irregularity. Multi-path interference and noise points affect the real emission point cloud. These deficiencies restrict the ability of traditional radars to accurately detect and locate objects in the surrounding environment.

[0005] Currently, the millimeter wave radar point cloud processing method has the following shortcomings: first, the speed, position and RCS characteristic information of the millimeter wave radar are not fully utilized; second, it is difficult to deal with the sparsity of data. Due to the low angular resolution, multipath interference, reduced echo strength at long distances and mirror reflection caused by the limited number of antennas, the millimeter wave radar point cloud is more sparse than the laser radar, especially the reflected targets at long distances. The millimeter wave radar data is sparse in features, and the existing clustering and detection algorithms cannot effectively extract the obstacle features due to the limited number of features in a frame of data. Third, the millimeter wave radar point cloud shows disordered and irregular characteristics, and the use of traditional convolutional neural network (CNN) will cause the network to have bias (according to the data distribution, it will be considered that there is more likely to be an obstacle in a certain area). At the same time, after the millimeter wave radar collects information and is labeled, a large number of non-class background points cause serious sample imbalance, resulting in obvious model bias and affecting the detection effect.

[0006] An existing millimeter wave radar point cloud 3D target detection method based on a graph neural network preprocesses millimeter wave radar point cloud data, constructs a graph data structure point cloud data, constructs a neighborhood-aware graph neural network NAGNN, and based on the extracted feature representation, predicts the target class, regresses the target position and the target detection confidence through a multi-task head detection network, and determines the 3D target position, size and orientation of the millimeter wave radar point cloud. This technology does not consider the improvement of the graph attention mechanism in terms of computation, storage and reduction ability, and lacks consideration in terms of rotation and translation invariance of the graph convolution network.

[0007] A radar obstacle detection system based on pre-built graphs, which uses a set of customized high-resolution, long-range millimeter wave radars suitable for rail transit scenarios as perception sensors, and filters out ROI (track boundary) obstacles based on pre-built graphs. This method filters out obstacles outside the ROI by pre-built graphs, uses the DBSCAN algorithm for clustering and Kalman tracking of radar point clouds, and does not propose a targeted processing method for the sparsity and disordered irregularity of point clouds. It also does not involve semantic recognition of millimeter wave radar target detection objects.

[0008] A millimeter wave radar-based long-range perception system for detecting road boundaries and trains. The system uses high-resolution, long-range millimeter wave radar customized for strong scattering environments of rail transit, uses azimuth scattering characteristics to eliminate false detections; according to the speed of the train itself, the radar detection point cloud is divided into static target point cloud and dynamic target point cloud. Then, using the road boundary curvature, global geometric parallel information and multi-frame information fusion, the static target point is extracted and the boundary is fitted. Finally, the radar trajectory information is clustered and the shape is estimated, and the train is identified according to the position and speed of the detected train and the extracted boundary information to judge the collision risk. However, it does not consider the sparsity and unordered irregularity of point clouds, does not effectively utilize the structured information between points and their neighbors in the point cloud information, and cannot fully utilize all features.

[0009] In summary, the prior art has the following challenges: trains operate in complex and often challenging environments, such as darkness, heavy rain, fog, and other adverse weather conditions. Traditional obstacle detection methods are mainly based on visible light cameras and laser radar sensors. However, visible light cameras are greatly affected by the environment, and images in dark environments will degrade to varying degrees, resulting in image distortion, loss of detail, and reduced contrast, making it difficult to recognize. Laser radar uses near-infrared light as a detection medium, and the power attenuates during atmospheric propagation. Using laser radar in "adverse weather" such as rain, snow, and fog not only significantly reduces product performance, but also easily causes "false obstacles" leading to safety accidents. Existing methods based on fusion of vision and radar have complex networks, alignment errors of heterogeneous sensor data, and lengthy detection steps; there is a slow response speed problem, which cannot meet the real-time requirements. Millimeter wave radar point cloud data is irregular, and using traditional convolutional neural networks (CNN) can cause network bias (according to data distribution, it is believed that there is a higher likelihood of obstacles in a certain area). At the same time, millimeter wave radar data features are sparse, and the amount of features in a frame of data is limited, and existing clustering and detection algorithms cannot effectively extract obstacle features. After labeling the millimeter wave radar collected information, a large number of non-class background points cause serious sample imbalance, resulting in significant model bias and affecting detection effectiveness. SUMMARY

[0010] The present application aims to provide a millimeter wave radar graph convolution-based urban rail vehicle-mounted autonomous obstacle detection method and system to solve at least one of the technical problems in the above background art.

[0011] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0012] In a first aspect, the present application provides a millimeter wave radar graph convolution-based urban rail vehicle-mounted autonomous obstacle detection method, comprising:

[0013] The millimeter wave radar is initialized, and the collected radar frames are continuously input into the queue; a plurality of frame radar point cloud data are read from the queue, and the processed point cloud data are converted into a graph structure;

[0014] The trained graph convolutional neural network is used for processing the graph structure, and finally an obstacle detection and classification result is output, so as to make a warning for train travel; wherein the trained graph convolutional neural network comprises one regression head and four graph convolutional layers, the regression head and each graph convolutional layer each comprise a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer combined with a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a skip connection structure; before passing through each network layer, vector embedding of features is performed; node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector.

[0015] Further, the graph convolutional calculation is:

[0016] b i,j = f(e i,j ,W j s j ,W i s i ),j∈N(i)

[0017]

[0018] wherein b i,j is the calculation result of the combined edge attribute e i of the node s j and the node s i,j , W i and W j are learnable weights in the training process, the node attribute of the next layer is updated through the aggregation function Aggr and the fully connected layer MLP; s i ' is the node feature output to the next layer network, and N(i) is all nodes having a connection relationship with the point i after the graph data is constructed.

[0019] Further, the automatic calibration mechanism is that a coefficient c i,j is calculated, and automatic calibration is performed by combining Δs i,j to repair errors possibly existing in radar sensor measurement, and the dimensions are aligned through linear projection:

[0020] c i,j = LeakyReLU(s i +s j +e i,j )

[0021] b i,j=a T (c i,j +linear(Δs i,j ))

[0022] wherein a T represents the transpose of the attention coefficient matrix, and linear(Δsi,j) is a linear mapping of the feature difference between point i and point j.

[0023] Further, the structure of the graph convolution layer is:

[0024]

[0025] The jump connection structure Res is added, that is, the original feature is added to the updated feature:

[0026]

[0027] Further, the total loss function of the graph convolution neural network is obtained by adding the classification category loss and the loss of the target anchor frame, wherein the category loss is the cross entropy loss, and the anchor frame loss is the huber loss.

[0028] Further, the cross loss is:

[0029]

[0030] wherein N is the number of samples, C is the number of categories, w i is the weight of the i-th sample, y ic represents that the i-th sample belongs to the category c, and p ic is the probability that the model predicts that the i-th sample belongs to the category c.

[0031] The huber loss is calculated in the following manner:

[0032]

[0033] wherein y is the true value, is the predicted value, and δ is a threshold value for controlling where the loss function switches from L1 loss to L2 loss.

[0034] In a second aspect, the present application provides a city rail vehicle-mounted autonomous obstacle detection system based on a millimeter wave radar graph convolution, comprising:

[0035] An acquisition module is configured to initialize the millimeter wave radar, constantly input the collected radar frames into a queue, read a plurality of frames of radar point cloud data from the queue, and convert the processed point cloud data into a graph structure.

[0036] The processing module is configured to process the graph structure by using the trained graph convolutional neural network, and finally output an obstacle detection and classification result to provide a warning for train travel.

[0037] In a third aspect, the present application provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implement the millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method according to the first aspect.

[0038] In a fourth aspect, the present application provides a computer device including a memory and a processor, wherein the processor and the memory are in communication with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method according to the first aspect.

[0039] In a fifth aspect, the present application provides an electronic device including a processor, a memory and a computer program, wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method according to the first aspect.

[0040] The present application has the following advantages: by using millimeter wave radar point cloud data and a graph convolutional neural network, combining attention mechanisms, multi-layer high-dimensional graph embedding, graph convolution rotation translation invariance, automatic feature alignment, and network structure jump connection, the high-dimensional graph embedding, feature extraction, and accurate classification and regression of unordered irregular point cloud data are solved. The attention mechanism improves the calculation and storage efficiency and the reduction ability, the high-dimensional embedding helps to realize information reorganization and high-dimensional aggregation of channel features, and improves the expression and classification performance of the model. The rotation translation invariance ensures the accuracy and robustness of obstacle detection; the jump connection structure effectively solves the problem of over-smoothing of the graph neural network. Through the unique network structure and feature selection and division, the feature relationship between different points is focused, the global information is obtained by combining the attention mechanism, and more efficient and accurate detection of urban rail obstacle classification and detection is realized.

[0041] The advantages of the additional aspects of the present application will become apparent from the following description, or will be apparent from practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0043] Figure 1 RailRadarGNN-based urban rail train active obstacle detection process described in the embodiments of the present application.

[0044] Figure 2 RailRadarGNN working process schematic diagram described in the embodiments of the present application.

[0045] Figure 3 RailRadarGNN model main body frame schematic diagram described in the embodiments of the present application.

[0046] Figure 4 Single-layer graph convolution structure schematic diagram described in the embodiments of the present application.

[0047] Figure 5 RailRadarGNN target detection result schematic diagram described in the embodiments of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below, and the examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below through the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0049] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the field to which the present application belongs.

[0050] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as such.

[0051] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprising," "including," "containing," and "having" and the like, when used in the specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0052] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0053] In order to facilitate the understanding of the present application, the present application will be further explained and described in specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present application.

[0054] Those skilled in the art should understand that the drawings are only schematic views of the embodiments, and the components in the drawings are not necessarily essential for the implementation of the present application.

[0055] The present application is directed to the feature of all-weather operation of urban rail vehicle autonomous obstacle detection, and proposes a RailRadarGNN deep learning model based on graph convolution. The millimeter wave radar sensor is used, which is less affected by bad weather such as wind, rain, and fog, and dark light environment. By calculating the distance and direction of each reflection point, a point cloud data containing multiple points is generated, which represents the related information of the surrounding objects. Using millimeter wave radar point cloud data and graph convolutional neural network, combining attention mechanism, multi-layer high-dimensional graph embedding, graph convolution rotation translation invariance, automatic feature alignment, and network structure jump connection, the high-dimensional graph embedding, feature extraction, and accurate classification, regression of unordered irregular point cloud data are solved. The attention mechanism improves the calculation and storage efficiency and the reduction ability, the high-dimensional embedding helps to realize the high-dimensional aggregation of information and channel features, and improves the expression and classification performance of the model. The rotation translation invariance ensures the relative universality of the operator of the detection module; the jump connection structure effectively solves the problem of over-smoothing of the graph neural network. Through the unique network structure and feature selection and division, the feature relationship between different points is focused, and the global information is obtained combined with the attention mechanism, and then more efficient and accurate rail transit obstacle classification and detection is realized.

[0056] Embodiment 1

[0057] In this embodiment 1, first provide a kind of city rail vehicle autonomous obstacle detection system based on millimeter wave radar graph convolution, the system includes: acquisition module, for millimeter wave radar initialization, constantly to the queue input collected radar frame;Read several frames of radar point cloud data from queue, convert the point cloud data processed into graph structure.Processing module, for using trained graph convolutional neural network to process the graph structure, finally output obstacle detection and classification result, make early warning to train travel;Wherein, the trained graph convolutional neural network includes a regression head, 4 graph convolutional layers, the regression head and each graph convolutional layer all include a vector embedding layer, a convolution layer, a regularization layer and a ReLU activation layer;The convolution layer in regression head is the convolution layer combined with graph attention mechanism and automatic calibration mechanism, the convolution layer in graph convolutional layer is the convolution layer added with skip connection structure;Before passing through each network, vector embedding of feature will be carried out;Node feature is initially embedded into 32-dimensional vector, and edge feature is embedded into 4-dimensional vector.

[0058] In this embodiment, using the above system, a city rail vehicle autonomous obstacle detection method based on millimeter wave radar graph convolution is realized, including: millimeter wave radar initialization, constantly to the queue input collected radar frame;Read several frames of radar point cloud data from queue, convert the point cloud data processed into graph structure;Using trained graph convolutional neural network to process the graph structure, finally output obstacle detection and classification result, make early warning to train travel;Wherein, the trained graph convolutional neural network includes a regression head, 4 graph convolutional layers, the regression head and each graph convolutional layer all include a vector embedding layer, a convolution layer, a regularization layer and a ReLU activation layer;The convolution layer in regression head is the convolution layer combined with graph attention mechanism and automatic calibration mechanism, the convolution layer in graph convolutional layer is the convolution layer added with skip connection structure;Before passing through each network, vector embedding of feature will be carried out;Node feature is initially embedded into 32-dimensional vector, and edge feature is embedded into 4-dimensional vector.

[0059] Wherein, graph convolution calculation is:

[0060] b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i)

[0061]

[0062] Wherein, b i,j For the node s i In graph structure and node sj the combined edge attribute e i,j the calculation result of W i and W j is a learnable weight in the training process, and the node attribute of the next layer is updated via the aggregation function Aggr and through the fully connected layer MLP; s i is the output of the node feature to the next layer network, and N(i) is all nodes connected with point i after the graph data is constructed.

[0063] The automatic calibration mechanism is to calculate the coefficient c i,j , and automatically calibrate Δs i,j to repair the errors that may exist when the radar sensor measures, and align the dimensions through linear projection:

[0064] c i,j = LeakyReLU(s i + s j + e i,j )

[0065] b i,j = a T (c i,j + linear(Δs i,j ))

[0066] wherein a T represents the transpose of the attention coefficient matrix, and linear(Δsi,j) is the linear mapping of the difference between the node features of point i and point j.

[0067] The structure of the graph convolution layer is as follows:

[0068]

[0069] The jump connection structure Res is added, that is, the original feature is added to the updated feature:

[0070]

[0071] The total loss function of the graph convolution neural network is obtained by adding the classification category loss and the loss of the target anchor frame, wherein the category loss is the cross-entropy loss, and the anchor frame loss is the huber loss.

[0072] The cross loss is:

[0073]

[0074] wherein N is the sample quantity, C is the category quantity, w i is the weight of the i-th sample, y ic represents that the i-th sample belongs to the category c, and p icis the probability that the model predicts the ith sample belongs to class c;

[0075] The huber loss is calculated as:

[0076]

[0077] where y is the true value, is the predicted value, and d is a threshold that controls where the loss function switches from L1 loss to L2 loss.

[0078] Embodiment 2

[0079] In this embodiment 2, a target detection method based on attention self-calibration graph convolutional neural network RailRadarGNN is provided, and the method flow is as shown in Figure 1 After the initialization of the millimeter wave radar, the collected radar frames are continuously input into the queue; the asynchronous acquisition program ends the instruction, if it ends, the occupied resources are cleaned up, and the program ends; otherwise, the data collection continues. Another part of the asynchronous operation will read a number of frame radar point cloud data from the queue every interval, convert the processed point cloud data into a graph structure, send it into the trained graph convolutional neural network, and finally output the detection and classification results to make a pre-warning for train travel.

[0080] In this embodiment, the subway obstacle detection processing flow based on millimeter wave radar graph convolutional neural network is as shown in Figure 2 The specific subdivision is as follows:

[0081] A. Millimeter wave radar point cloud data acquisition and preprocessing

[0082] Point cloud data preprocessing includes millimeter wave radar sensor parameter configuration, point cloud data acquisition, and multi-frame aggregation, to ensure that the data is suitable for further analysis. The specific steps are as follows:

[0083] Data acquisition: long-distance (detection range 500m) millimeter wave radar is used to collect moving target point cloud data on the subway track.

[0084] Data aggregation: due to the weak reflection signal of long-distance radar, the reflection coefficient of the object surface, and the RCS change caused by the movement of the moving target, single-frame data will appear sparse or even disappear. Considering the detection target moving speed, the acquisition frame rate, and the RCS of the object, the optimal fusion parameter N is determined to fuse N frames of point cloud data into 1 frame, to overcome the influence of point cloud sparsity. At the same time, the data is organized into a format suitable for subsequent GCN (graph convolutional neural network) processing, and is labeled with a timestamp by the sensor.

[0085] B. Graph structure data construction

[0086] The graph structure construction includes the following steps: connecting nodes by aggregating frame point cloud data (using k-nearest neighbors (for any point, take its nearest k points, generate connection relationship, and construct graph structure) or radius neighborhood method (for any point, take all points within a certain distance from its center to construct graph structure). We choose the edge feature to be the relative position between two points, and the node feature to include Doppler velocity, RCS reflection area, time index, and node degree, so that the network is not affected by the irregularity of the point cloud.

[0087] C. Feature high-dimensional embedding

[0088] In graph convolutional networks, feature high-dimensional embedding is a key step that allows the network to convert sparse point cloud data into a dense representation in high-dimensional space, better capturing complex patterns and relationships in the data. As shown in Figure 4 each node (blue circle) has an initial feature vector, denoted as s i Before graph convolution, the feature vector of each node is processed through a multi-layer perceptron (MLP). For each edge in the graph (the line connecting two nodes), the difference between the two node features is calculated, denoted as Δs i,j . The edge feature e i,j represents the attribute between nodes. The node feature is combined with the features of neighboring nodes through an aggregation function (such as summation, averaging, or more complex functions). This process considers the local neighborhood information of the node. The aggregated features are processed through another MLP to generate updated node features s i ′. This updated feature will be used for the next layer of graph convolution or as the final output

[0089] In this process, we use multi-layer perceptron (MLP) for feature high-dimensional embedding before and after each graph convolution layer, on the one hand to unify the feature dimension, and on the other hand to facilitate the network to extract features from more dense information. The dimension of the node feature embedding layer is mapped from the input 5-dimensional feature to 32-dimensional, then through 64, 128, 224-dimensional layers in turn. The dimension of the edge feature embedding layer is mapped from the input 2-dimensional feature to 4-dimensional, then through 8, 16-dimensional layers in turn.

[0090] D. Graph convolutional neural network construction combined with attention and automatic calibration mechanism

[0091] Each graph convolution layer in the graph convolutional network follows a specific structure, which goes through a vector embedding layer, a graph convolution layer, a regularization layer, and a ReLU activation layer. The overall network structure is as follows Figure 3As shown, data is used to construct a graph structure, where nodes represent data points and edges represent relationships between data points, and then the data is input to the graph convolution network according to the network structure, and deeper abstraction and learning of features are performed. After the graph convolution layer, the network contains one or more regression heads for obstacle detection or classification. During the training process, the output of the network is compared with the true label, and the loss is calculated. Finally, the network output is used to classify, detect obstacles, or perform other tasks.

[0092] wherein the convolutional layers are divided into two types: one is a graph convolutional layer one combined with a graph attention mechanism and an automatic calibration mechanism; and one is a graph convolutional layer two with a skip connection structure; and the feature division is as described in B. The dimensions of the graph convolutional layers are 224, 224, 128, 64, and 32 from the input layer to the output layer, reflecting the feature extraction process.

[0093] Each time the network passes, vector embedding of features is performed; the node features are initially embedded into a 32-dimensional vector, and the edge features are embedded into a 4-dimensional vector. Then these embedded features are input into the graph convolutional layer combined with the attention mechanism and the modified automatic registration mechanism. Subsequently, after embedding the vectors, the features compressed in dimension are input into the subsequent another type of graph convolutional layer.

[0094] In this embodiment, the attention mechanism is one of the key technologies, which is used to enhance the model's ability to identify key features in radar point cloud data. The attention mechanism allows the model to dynamically focus on the most important parts of the input data, thereby improving the accuracy and efficiency of obstacle detection. The graph convolution calculation can be considered as the content of formula (1) and formula (2), wherein b i,j is the calculation of the point s i in the graph structure j combined with the edge attribute e i,j , W i and W j are weights that can be learned during the training process, which are updated to the node attributes of the next layer through the aggregation function Aggr and the fully connected layer (MLP).

[0095] After combining the automatic registration mechanism, the coefficient c i,j is calculated as shown in formula (3), combined with formula (4) Δs i,jAn automatic calibration is performed to correct errors that may exist in the radar sensor measurement, and the dimensions are aligned through linear projection, where W is a learnable weight, and for the attention coefficient matrix a, it is calculated as shown in equation (5), so as to obtain the relationship between the information aggregated by a certain node and the information aggregated by other related nodes. Through this design, the graph convolution layer can effectively improve the detection accuracy of the model while maintaining the integrity of the data features. The MPNN layer is a simple and general graph convolutional neural network layer, and for the scenario that needs to combine edge features, its structure is shown in equations (6) and (7). For the MPNN layer, the way of passing to the next layer in equation (7) is modified, and a skip connection structure is added, which is finally shown in equation (8) to help the model converge faster while improving the detection accuracy of the model. Where Res is the skip connection structure, and its way is as follows: h updated = h + x, the original feature is added to the updated feature.

[0096] b i,j = f(e i,j ,W j s j ,W i s i ),j∈N(i) (1)

[0097]

[0098] c i,j = LeakyReLU(s i +s j +e i,j ) (3)

[0099] b i,j = a T (c i,j + linear (Δs i,j )) (4)

[0100]

[0101]

[0102]

[0103] E.Obstacle detection and classification

[0104] After the features pass through the last layer of graph convolution, a layer of vector is obtained, which will be used as the input of the classifier and target detection. If the sensor information includes angle features, a 6-dimensional output vector is generated, and if it does not include angle features, a 5-dimensional output vector (class and target box coordinates and width and height) is generated. Thus, the point cloud classification class and the target anchor box are obtained.

[0105] The loss function of model training: as shown in formula (9), it is obtained by the loss of classification category and the loss of target anchor frame. The category loss is cross-entropy loss, and the anchor frame loss is huber loss. The cross-entropy loss is shown in formula (10), N is the number of samples, C is the number of categories, w i is the weight of the ith sample, y ic represents that the ith sample belongs to category c, p ic is the probability that the model predicts that the ith sample belongs to category c. The huber loss calculation method is shown in formula (11), where y is the true value, is the predicted value, and δ is a threshold value for controlling where the loss function switches from L1 loss to L2 loss. In order to solve the problem of sample imbalance, as shown in formula (12), the number of samples is multiplied by the ratio of the number of samples to the number of most categories in the batch of training data to train the model in the back propagation process. The upper and lower parts are calculated to make the data be a floating point tensor in the gpu, keep the type consistent and improve the calculation efficiency. It helps to alleviate the problem of tending to identify all points as background points in the few sample situation.

[0106] L = L cls + λL reg (9)

[0107]

[0108]

[0109] The back propagation process of graph convolutional neural network (GCN) is similar to that of conventional deep neural network, but needs to consider the particularity of graph structure data. In GCN, the feature vector of each node is updated through graph convolutional layer, which usually includes weight matrix, activation function and possibly other operations.

[0110] In combination Figure 5 with the above, in the embodiment, the final target detection and recognition test is carried out on the Radar Scene data set and the actually collected data set of the subway tunnel, and the precision rate (True Positive) of the vehicle, moving pedestrian and crowd 120 meters away is 85%, 78% and 82% respectively, which basically meets the demand of millimeter wave radar sensor for urban rail autonomous obstacle target recognition.

[0111] In this embodiment, other clustering algorithms such as DBSCAN can also be used to generate graph structure data, which can also effectively express the spatial relationship of point cloud data. At the same time, variants based on graph neural networks can also be used. Graph Convolutional Networks (GCN) is a basic architecture of graph neural networks, which learns the representation of nodes by applying convolution operations on the nodes of the graph. Variants of GCN, such as ChebNet and GCNII, can also achieve certain detection effects by improving the design of convolution kernels and optimizing the message passing mechanism.

[0112] In summary, the autonomous obstacle detection and tracking of urban rail trains is restricted by all-weather conditions (day, night, and severe weather conditions such as fog, rain, and snow), and needs to overcome the influence of light and severe weather. At the same time, it has low cost, low computational and storage complexity, and is suitable for real-time edge deployment and application. Urban rail autonomous obstacle detection technology mainly relies on optical sensors such as cameras and laser radars. These technologies are affected by light and adverse weather conditions, and their performance will be greatly reduced. In addition, the existing method based on the fusion of vision and radar has a complex processing procedure and slow response speed, which is difficult to meet the real-time requirements. The irregularity and sparsity of point cloud data make it difficult for existing algorithms to extract features, especially in millimeter wave radar data, where the data points are sparse, making it difficult to effectively identify obstacles. In addition, the sample imbalance problem is also a challenge for existing technologies. Because the number of background points is much larger than that of obstacle points, the model is too sensitive to background points during training, which affects the detection effect of obstacles. The RailRadarGNN model in this embodiment aims to combine graph convolutional neural networks, graph attention mechanisms, multi-layer high-dimensional graph embedding, graph convolution rotation translation invariance, feature automatic calibration mechanisms, and network structure jump connection technologies to solve the high-dimensional graph embedding, feature extraction, and accurate classification and regression of unordered and irregular point cloud data. The attention mechanism enables the model to focus on key features, improving computational efficiency and reduction ability; high-dimensional embedding helps to achieve information reorganization and high-dimensional aggregation of channel features, improving the expression and classification performance of the model. The model's ability to adapt to different data distributions improves the accuracy and robustness of detection; rotation translation invariance ensures that obstacle detection classification and tracking are independent of the absolute position and rotation angle of the object; the jump connection structure effectively solves the over-smoothing problem of graph neural networks. Through unique network structure and feature selection and division, the model focuses on the feature relationship between different points, obtains global information through the attention mechanism, and then realizes more efficient and accurate detection of rail transit obstacle classification and detection. Not only is it suitable for rail transit obstacle detection, but its principles and techniques can also be applied to other fields such as autonomous driving and intelligent monitoring, with wide application prospects and market potential.

[0113] In this embodiment, the graph convolutional neural network (GCN) is the core of the obstacle classification and detection. GCN can process graph structure data and extract features of point cloud data through graph convolutional layers. GCN can not only capture local features of point cloud data, but also capture global features through graph structure, thereby improving the accuracy of obstacle detection. In addition, the introduction of GCN enables the model to adaptively learn complex patterns in point cloud data, improving the generalization ability of the model. The application of graph convolutional neural network makes the obstacle classification in complex environment more accurate. The graph attention mechanism improves the ability of the model to capture important features, and the automatic calibration mechanism enhances the robustness of the system. To solve the problem of sample imbalance, this embodiment proposes a method based on sample weight adjustment. During model training, different weights are assigned to samples of different categories, so that the model pays more attention to samples of minority categories during training, thereby effectively solving the problem of sample imbalance and improving the detection performance of the model for minority category obstacles.

[0114] Embodiment 3

[0115] Embodiment 3 provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implement a millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method as described above. The method comprises:

[0116] The millimeter wave radar is initialized, and the collected radar frames are continuously input into the queue. A plurality of frames of radar point cloud data are read from the queue, and the processed point cloud data is converted into a graph structure.

[0117] The trained graph convolutional neural network is used to process the graph structure, and finally outputs the obstacle detection and classification results, and gives a warning for train travel. The trained graph convolutional neural network includes a regression head and four graph convolutional layers. The regression head and each graph convolutional layer each include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer. The convolutional layer in the regression head is a convolutional layer combined with a graph attention mechanism and an automatic calibration mechanism. The convolutional layer in the graph convolutional layer is a convolutional layer with a skip connection structure. Vector embedding of features is performed before each network. The node features are initially embedded in a 32-dimensional vector, and the edge features are embedded in a 4-dimensional vector.

[0118] Embodiment 4

[0119] The embodiment 4 provides a computer device, comprising a memory and a processor, the processor and the memory are in communication with each other, the memory stores program instructions which can be executed by the processor, and the processor calls the program instructions to execute the millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method as described above, the method comprises the following steps:

[0120] The millimeter wave radar is initialized, and collected radar frames are continuously input into the queue; a plurality of frame radar point cloud data are read from the queue, and the processed point cloud data are converted into a graph structure;

[0121] The trained graph convolutional neural network is used for processing the graph structure, and finally obstacle detection and classification results are output, and a warning is given to train travel; wherein the trained graph convolutional neural network comprises a regression head and four graph convolutional layers, the regression head and each graph convolutional layer comprise a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer combined with a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a skip connection structure; before passing through each layer of network, vector embedding of features is performed; node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector.

[0122] Embodiment 5

[0123] The embodiment 5 provides an electronic device, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the instructions for realizing the millimeter wave radar graph convolution-based urban rail vehicle autonomous obstacle detection method as described above, and the method comprises the following steps:

[0124] The millimeter wave radar is initialized, and collected radar frames are continuously input into the queue; a plurality of frame radar point cloud data are read from the queue, and the processed point cloud data are converted into a graph structure;

[0125] The trained graph convolutional neural network is used for processing the graph structure, and finally obstacle detection and classification results are output, and a warning is given to train travel; wherein the trained graph convolutional neural network comprises a regression head and four graph convolutional layers, the regression head and each graph convolutional layer comprise a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer combined with a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a skip connection structure; before passing through each layer of network, vector embedding of features is performed; node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector.

[0126] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the present application can be implemented with computer programs (also called software) that include program instructions, which are executed by one or more processors. The software programs can be structured in any of a variety of programming languages, such as assembly language, machine language, a high-level procedural language, or one or more object-oriented languages. The software programs, when executed, can read instructions from a computer-readable medium, such as a floppy disk, a CD-ROM, a DVD, a memory stick, a hard disk drive, or a solid state drive. The software programs can also be downloaded from the Internet, a local area network, or a wide area network.

[0127] The present application is described in relation to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It is understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus. Figure 1 The means can include any apparatus with a processor that performs functions specified by instructions stored by computer-readable storage media. Such instructions can be read into the processor from other computer-readable storage media. Such other computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disk drives), solid state memory (e.g., random access memory), compact disc read only memories, etc. The means can be one or more computer program products. The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus.

[0128] The means can include any apparatus with a processor that performs functions specified by instructions stored by computer-readable storage media. Such instructions can be read into the processor from other computer-readable storage media. Such other computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disk drives), solid state memory (e.g., random access memory), compact disc read only memories, etc. The means can be one or more computer program products. Figure 1 The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus. Figure 1 The means can include any apparatus with a processor that performs functions specified by instructions stored by computer-readable storage media. Such instructions can be read into the processor from other computer-readable storage media. Such other computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disk drives), solid state memory (e.g., random access memory), compact disc read only memories, etc. The means can be one or more computer program products. The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus.

[0129] The means can include any apparatus with a processor that performs functions specified by instructions stored by computer-readable storage media. Such instructions can be read into the processor from other computer-readable storage media. Such other computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disk drives), solid state memory (e.g., random access memory), compact disc read only memories, etc. The means can be one or more computer program products. Figure 1 The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus. Figure 1 The means can include any apparatus with a processor that performs functions specified by instructions stored by computer-readable storage media. Such instructions can be read into the processor from other computer-readable storage media. Such other computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disk drives), solid state memory (e.g., random access memory), compact disc read only memories, etc. The means can be one or more computer program products. The flow diagrams and / or block diagrams in the flow diagrams and / or block diagrams illustrate the functions and / or operations that can be performed by one or more computer program instructions. The computer program instructions can be executed by one or more processors of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus.

[0130] The above description is only for the specific embodiments of the present application, and is not intended to limit the scope of the present application. Those skilled in the art should understand that various modifications or variations can be made to the technical solutions disclosed in the present application without departing from the spirit and scope of the present application.

Claims

1. A method for autonomous obstacle detection on urban rail vehicles based on millimeter-wave radar image convolution, characterized in that, include: The millimeter-wave radar is initialized by continuously inputting the acquired radar frames into the queue; Read several frames of radar point cloud data from the queue, and convert the processed point cloud data into a graph structure; A pre-trained graph convolutional neural network (GCNN) is used to process the graph structure, ultimately outputting obstacle detection and classification results to provide early warnings for train operation. The pre-trained GCNN includes a regression head and four graph convolutional layers. Each regression head and each graph convolutional layer includes a vector embedding layer, a convolutional layer, a regularization layer, and a ReLU activation layer. The convolutional layer in the regression head combines graph attention and automatic calibration mechanisms, while the convolutional layers in the graph convolutional layers incorporate jump connections. Before each network layer, feature vector embedding is performed; node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector. The automatic calibration mechanism calculates coefficient c. i,j Combined with Δs i,j Automatic calibration is performed to correct potential errors in radar sensor measurements, and dimensions are aligned via linear projection. c i,j =LeakyReLU(s i +s j +e i,j ) b i,j =a T (c i,j +linear(Δs i,j )) Among them, a T Represents the transpose of the attention coefficient matrix, linear(Δs) i,j () represents a linear mapping of the feature differences between points i and j. The structure of the convolutional layer is shown in the figure: Add a jump connection structure Res, which means adding the original feature to the updated feature:

2. The method for autonomous obstacle detection on urban rail vehicles based on millimeter-wave radar image convolution as described in claim 1, characterized in that, Graph convolution is calculated as follows: b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i) Among them, b i,j For node s in the graph structure i With node s j The combined edge attribute e i,j The calculation results, W i With W j The weights, which are learnable during training, are aggregated via the Aggro function and then updated to update the node attributes of the next layer through a fully connected MLP layer; i ′ represents the node features output to the next layer of the network, and N(i) represents all nodes connected to point i after the graph data is constructed.

3. The method for autonomous obstacle detection on urban rail vehicles based on millimeter-wave radar image convolution as described in claim 2, characterized in that, The total loss function of the graph convolutional neural network is obtained by adding the classification loss and the target anchor box loss, where the classification loss is the cross-entropy loss and the anchor box loss is the Huber loss.

4. The method for autonomous obstacle detection on urban rail vehicles based on millimeter-wave radar image convolution according to claim 3, characterized in that, The cross loss is: Where N is the number of samples, C is the number of categories, and w i It is the weight of the i-th sample, y ic p indicates that the i-th sample belongs to category c. ic It is the probability that the model predicts the i-th sample belongs to class c; The Huber loss is calculated as follows: Where y is the true value. δ is the predicted value, and δ is a threshold used to control where the loss function switches from L1 loss to L2 loss.

5. A vehicle-mounted autonomous obstacle detection system for urban rail transit based on millimeter-wave radar image convolution, characterized in that, include: The acquisition module is used for millimeter-wave radar initialization, continuously inputting the acquired radar frames into the queue; Read several frames of radar point cloud data from the queue, and convert the processed point cloud data into a graph structure; The processing module is used to process the graph structure using a trained graph convolutional neural network (GCNN) and ultimately output obstacle detection and classification results to provide early warnings for train operation. The trained GCNN includes a regression head and four graph convolutional layers. Each regression head and each graph convolutional layer includes a vector embedding layer, a convolutional layer, a regularization layer, and a ReLU activation layer. The convolutional layers in the regression head combine graph attention and automatic calibration mechanisms, while the convolutional layers in the graph convolutional layers incorporate jump connections. Before each network layer, feature vector embedding is performed. Node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector. The automatic calibration mechanism calculates coefficient c. i,j Combined with Δs i,j Automatic calibration is performed to correct potential errors in radar sensor measurements, and dimensions are aligned via linear projection. c i,j =LeakyReLU(s i +s j +e i,j ) b i,j =a T (c i,j +linear(Δs i,j )) Among them, a T Represents the transpose of the attention coefficient matrix, linear(Δs) i,j () represents a linear mapping of the feature differences between points i and j. The structure of the convolutional layer is shown in the figure: Add a jump connection structure Res, which means adding the original feature to the updated feature:

6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in any one of claims 1-4.

7. A computer device, characterized in that, The system includes a memory and a processor, which communicate with each other. The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in any one of claims 1-4.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions to implement the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in any one of claims 1-4.

Images