Urban rail vehicle-mounted autonomous obstacle detection method and system based on millimeter wave radar map convolution

Abstract

The invention provides an urban rail vehicle-mounted autonomous obstacle detection method and system based on millimeter wave radar map convolution, and belongs to the technical field of rail traffic obstacle detection. Millimeter wave radar is initialized, and collected radar frames are continuously input into a queue; reading a plurality of frames of radar point cloud data from the queue, and converting the processed point cloud data into a graph structure; and processing the graph structure by using the trained graph convolutional neural network, finally outputting an obstacle detection and classification result, and performing early warning on train driving. According to the invention, high-dimensional embedding helps to realize information recombination and high-dimensional aggregation of channel features, and rotation and translation invariance ensures the accuracy and robustness of obstacle detection; the jump connection structure effectively solves the problem of excessive smoothness of the graph neural network; through a unique network structure and feature selection and division, the feature relationship between different points is concerned, global information is acquired in combination with an attention mechanism, and more efficient and accurate rail transit obstacle classification and detection are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit obstacle detection, and in particular to an urban rail vehicle-mounted autonomous obstacle detection method and system based on millimeter-wave radar image convolution. Background Art

[0002] Urban rail transit is a large-capacity public transportation infrastructure and a backbone transportation network in large cities. It has large capacity, high speed, and can effectively alleviate urban traffic congestion. The safety and operation efficiency of urban rail transit systems are related to the life, health, and travel convenience of passengers. Active obstacle detection on trains is an important research and development direction for realizing the automated operation of urban rail transit trains. It is of great significance to effectively detect abnormal intrusive obstacles outside the train, improve the obstacle detection capability of unmanned trains, and enhance the autonomous perception capability of vehicles.

[0003] Urban rail trains operate in complex and often challenging environments, such as darkness, heavy rain, fog and other severe weather conditions. Traditional obstacle detection methods mainly rely on manual or ground-based obstacle detection. The former has low detection efficiency and high labor intensity; the latter has a limited detection range, and if the entire line is covered by monitoring, the deployment cost is high. Train active obstacle detection is an important research and development direction for realizing the automated operation of urban rail transit trains and a key technology for improving the safety of train operation. On-board active obstacle detection installs environmental perception sensors, such as visible light cameras, lidar and millimeter wave radar, and computing resources on the vehicle side. Visible light cameras are greatly affected by the environment, and images in dark light environments will degrade to varying degrees, resulting in image distortion, loss of details, decreased contrast, and low recognition rate. Lidar uses near-infrared light as a detection medium, and power attenuation will occur during atmospheric propagation. Using lidar in "bad weather" such as rain, snow, and fog will not only greatly reduce product performance, but also easily cause "false obstacles" and lead to safety accidents.

[0004] Compared with optical cameras and lidars, millimeter-wave radars show strong robustness in environmental conditions such as rain, snow, fog, and concentrated dust. At the same time, the speed detection accuracy of moving objects based on the Doppler effect is much higher than other similar speed sensors. Therefore, high reliability and high precision position and speed detection lay the foundation for the all-weather operation of autonomous obstacle detection of urban rail trains. However, compared with sensors based on cameras and lidars, millimeter-wave radar point clouds are sparse, have low angular resolution, are obviously disordered and irregular, and multipath interference noise affects the real emission point cloud. These shortcomings restrict the ability of traditional radars to accurately detect and locate objects in the surrounding environment.

[0005] At present, the millimeter-wave radar point cloud processing method has the following shortcomings: first, it fails to fully utilize the speed, position and RCS feature information of the millimeter-wave radar; second, it is difficult to deal with the sparsity of the data. Due to the low angular resolution caused by the limited number of antennas, multipath interference, reduced echo intensity at long distances and mirror reflection, the millimeter-wave radar point cloud is more sparse than the laser radar, especially the reflective targets at long distances. The millimeter-wave radar data features are sparse, and the number of features in a frame of data is limited. The existing clustering and detection algorithms cannot effectively extract obstacle features. Third, for the millimeter-wave radar point cloud, which exhibits characteristics such as disorder and irregularity, the use of traditional convolutional neural networks (CNN) will lead to bias in the network (according to the data distribution, it is believed that there are more obstacles in a certain area). At the same time, after the millimeter-wave radar collects and annotates the information, a large number of unclassified background points lead to serious sample imbalance, resulting in obvious model deviation and affecting the detection effect.

[0006] An existing 3D target detection method for millimeter-wave radar point cloud based on graph neural network preprocesses millimeter-wave radar point cloud data, constructs point cloud data with graph data structure, and constructs neighborhood-aware graph neural network NAGNN. Based on the extracted feature representation, the multi-task head detection network predicts the target category, regresses the target position and target detection confidence, and determines the 3D target position, size and orientation of the millimeter-wave radar point cloud. This technology does not consider the improvement of the computation, storage and reduction capabilities of the graph attention mechanism, and does not take into account the rotation and translation invariance of the graph convolutional network.

[0007] A radar obstacle detection system based on pre-built maps, which uses a customized high-resolution, long-range millimeter-wave radar that can adapt to rail transit scenarios as a perception sensor, and filters out obstacles outside the ROI (track limit) based on pre-built maps. This method uses pre-built maps to filter out obstacles outside the ROI, and uses the DBSCAN algorithm to cluster and Karaman track radar point clouds. It does not propose targeted processing methods for the sparse and disordered irregular characteristics of point clouds, and does not involve the semantic recognition of millimeter-wave radar target detection objects.

[0008] A long-range perception system based on millimeter-wave radar is used to detect road boundaries and trains. The system uses a high-resolution, long-range millimeter-wave radar customized for the strong scattering environment of rail transit, and uses the azimuth scattering characteristics to eliminate false detections; the radar detection point cloud is divided into a static target point cloud and a dynamic target point cloud according to the train's own speed. Then, the road boundary curvature, global geometric parallel information and multi-frame information fusion are used to stably extract and fit the boundaries at the static target point. Finally, the radar trajectory information is clustered and the shape is estimated, and the train is identified and the collision risk is judged based on the detected train position and speed and the extracted boundary information. However, it does not take into account the sparse and disordered irregular characteristics of the point cloud, does not effectively utilize the structured information between the point and the neighboring points and the global points in the point cloud information, and cannot fully utilize all the features.

[0009] In summary, the existing technologies face the following challenges: Trains operate in complex and often challenging environments, such as darkness, heavy rain, fog, and other severe weather conditions. Traditional obstacle detection methods are mainly based on visible light cameras and lidar sensors. However, visible light cameras are greatly affected by the environment, and images in dark environments will degrade to varying degrees, resulting in image distortion, loss of details, decreased contrast, and low recognition rate. Lidar uses near-infrared light as a detection medium, and power attenuation occurs during atmospheric propagation. Using lidar in "bad weather" such as rain, snow, and fog not only greatly reduces product performance, but also easily leads to "false obstacles" and safety accidents. Existing methods based on vision and radar fusion have complex networks, heterogeneous sensor data alignment errors, and lengthy detection links; there are problems such as slow response speed and cannot meet real-time requirements. Millimeter-wave radar point cloud data has irregularities, and the use of traditional convolutional neural networks (CNNs) will lead to biased networks (according to data distribution, it is believed that there are more obstacles in a certain area). At the same time, the features of millimeter-wave radar data are sparse, and the number of features in a frame of data is limited. Existing clustering and detection algorithms cannot effectively extract obstacle features. After the millimeter-wave radar collects information and annotates it, a large number of unclassified background points lead to serious sample imbalance, resulting in obvious model deviation and affecting the detection effect. Summary of the invention

[0010] The object of the present invention is to provide an urban rail vehicle-mounted autonomous obstacle detection method and system based on millimeter-wave radar image convolution to solve at least one technical problem existing in the above-mentioned background technology.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] In a first aspect, the present invention provides an urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution, comprising:

[0013] Initialize the millimeter-wave radar and continuously input the collected radar frames into the queue; read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure;

[0014] The graph structure is processed by using a trained graph convolutional neural network, and obstacle detection and classification results are finally output to give an early warning for train operation; wherein, the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in 32-dimensional vectors, and edge features are embedded in 4-dimensional vectors.

[0015] Furthermore, the graph convolution is calculated as:

[0016] b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i)

[0017]

[0018] Among them, b i,j is the node s in the graph structure i With node s j The combined edge attribute e i,j The calculation result, W i With W j It is the learnable weight in the training process, which updates the node attributes of the next layer through the aggregation function Aggr and the fully connected layer MLP; i ′ is the node feature output to the next layer of the network, and N(i) is all the nodes that are connected to point i after the graph data is constructed.

[0019] Furthermore, the automatic calibration mechanism is to calculate the coefficient c i,j , combined with Δs i,j Perform automatic calibration to correct possible errors in radar sensor measurements and align dimensions via linear projection:

[0020] c i,j =LeakyReLU(s i +s j +e i,j )

[0021] b i,j=a T (c i,j +linear(Δs i,j ))

[0022] Among them, a T represents the transpose of the attention coefficient matrix, and linear(Δsi,j) is the linear mapping of the feature difference between point i and point j.

[0023] Furthermore, the graph convolution layer structure is:

[0024]

[0025] Add the jump connection structure Res, that is, add the original features to the updated features:

[0026]

[0027] Furthermore, the total loss function of the graph convolutional neural network is the sum of the classification category loss and the target anchor box loss, where the category loss is the cross entropy loss and the anchor box loss is the Huber loss.

[0028] Furthermore, the cross loss is:

[0029]

[0030] Where N is the number of samples, C is the number of categories, and w i is the weight of the i-th sample, y ic Indicates that the i-th sample belongs to category c, p ic is the probability that the model predicts that the i-th sample belongs to category c;

[0031] The huber loss is calculated as:

[0032]

[0033] Among them, y is the true value, is the predicted value, and δ is a threshold that controls where the loss function switches from L1 loss to L2 loss.

[0034] In a second aspect, the present invention provides an urban rail vehicle-mounted autonomous obstacle detection system based on millimeter wave radar image convolution, comprising:

[0035] The acquisition module is used to initialize the millimeter-wave radar and continuously input the collected radar frames into the queue; it reads several frames of radar point cloud data from the queue and converts the processed point cloud data into a graph structure;

[0036] A processing module is used to process the graph structure using a trained graph convolutional neural network, and finally output obstacle detection and classification results to issue an early warning for train operation; wherein the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in 32-dimensional vectors, and edge features are embedded in 4-dimensional vectors.

[0037] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in the first aspect is implemented.

[0038] In a fourth aspect, the present invention provides a computer device, comprising a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in the first aspect.

[0039] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in the first aspect.

[0040] The beneficial effects of the present invention are as follows: by utilizing millimeter-wave radar point cloud data and graph convolutional neural networks, combining attention mechanisms, multi-layer high-dimensional graph embedding, graph convolution rotation and translation invariance, automatic feature alignment and other technologies and jump connections of network structures, the high-dimensional graph embedding, feature extraction and accurate classification and regression of disordered and irregular point cloud data are solved. The attention mechanism improves the computing storage efficiency and specification capabilities, and high-dimensional embedding helps to achieve information reorganization and high-dimensional aggregation of channel features, improving the expression and classification performance of the model. Rotation and translation invariance ensures the accuracy and robustness of obstacle detection; the jump connection structure effectively solves the problem of over-smoothing of graph neural networks. Through the unique network structure and feature selection division, attention is paid to the feature relationship between different points, and the global information is obtained in combination with the attention mechanism, thereby achieving more efficient and accurate detection of rail transit obstacle classification and detection.

[0041] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0043] Figure 1 This is the process of active obstacle detection for urban rail trains based on RailRadarGNN described in an embodiment of the present invention.

[0044] Figure 2 The figure is a schematic diagram of the RailRadarGNN workflow described in an embodiment of the present invention.

[0045] Figure 3 Schematic diagram of the main framework of the RailRadarGNN model described in an embodiment of the present invention.

[0046] Figure 4 Schematic diagram of a single-layer graph convolution structure according to an embodiment of the present invention.

[0047] Figure 5 A schematic diagram of the RailRadarGNN target detection result described in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0049] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0050] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0051] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0052] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0053] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0054] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0055] Aiming at the feature of all-weather operation of autonomous obstacle detection on urban rail vehicles, the present invention proposes a RailRadarGNN deep learning model based on graph convolution. The millimeter-wave radar sensor that is less affected by bad weather such as wind, rain and fog and dark light environment is used to generate a point cloud data containing multiple points by calculating the distance and direction of each reflection point. These points represent the relevant information of surrounding objects. By using millimeter-wave radar point cloud data and graph convolutional neural network, combining attention mechanism, multi-layer high-dimensional graph embedding, graph convolution rotation and translation invariance, automatic feature alignment and other technologies and network structure jump connection, the high-dimensional graph embedding, feature extraction and accurate classification and regression of disordered and irregular point cloud data are solved. The attention mechanism improves the computing storage efficiency and specification ability, and the high-dimensional embedding helps to realize the high-dimensional aggregation of information reorganization and channel features, and improves the expression and classification performance of the model. The rotation and translation invariance ensures the relative universality of the operators of the detection module; the jump connection structure effectively solves the problem of excessive smoothing of the graph neural network. Through the unique network structure and feature selection division, attention is paid to the feature relationship between different points, and the global information is obtained by combining the attention mechanism, thereby achieving more efficient and accurate detection of rail transit obstacle classification and detection.

[0056] Example 1

[0057] In this embodiment 1, firstly, an urban rail vehicle-mounted autonomous obstacle detection system based on millimeter wave radar graph convolution is provided, the system comprising: an acquisition module for initializing the millimeter wave radar, continuously inputting the collected radar frames into the queue; reading several frames of radar point cloud data from the queue, and converting the processed point cloud data into a graph structure. A processing module is used to process the graph structure using a trained graph convolutional neural network, and finally output obstacle detection and classification results to give an early warning for train travel; wherein, the trained graph convolutional neural network comprises a regression head and 4 graph convolutional layers, the regression head and each graph convolutional layer each comprises a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer combining a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded into a 32-dimensional vector, and edge features are embedded into a 4-dimensional vector.

[0058] In this embodiment, the above system is used to implement an urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar graph convolution, including: initializing the millimeter-wave radar, continuously inputting the collected radar frames into the queue; reading several frames of radar point cloud data from the queue, and converting the processed point cloud data into a graph structure; processing the graph structure using a trained graph convolutional neural network, and finally outputting obstacle detection and classification results to warn the train; wherein the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer combining a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in a 32-dimensional vector, and edge features are embedded in a 4-dimensional vector.

[0059] Among them, the graph convolution is calculated as:

[0060] b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i)

[0061]

[0062] Among them, b i,j is the node s in the graph structure i With node sj The combined edge attribute e i,j The calculation result, W i With W j It is the learnable weight in the training process, which updates the node attributes of the next layer through the aggregation function Aggr and the fully connected layer MLP; i ′ is the node feature output to the next layer of the network, and N(i) is all the nodes that are connected to point i after the graph data is constructed.

[0063] The automatic calibration mechanism is to calculate the coefficient c i,j , combined with Δs i,j Perform automatic calibration to correct possible errors in radar sensor measurements and align dimensions via linear projection:

[0064] c i,j =LeakyReLU(s i +s j +e i,j )

[0065] b i,j =a T (c i,j +linear(Δs i,j ))

[0066] Among them, a T represents the transpose of the attention coefficient matrix, and linear(Δsi,j) is the linear mapping of the feature difference between point i and point j.

[0067] The structure of the graph convolution layer is as follows:

[0068]

[0069] Add the jump connection structure Res, that is, add the original features to the updated features:

[0070]

[0071] The total loss function of the graph convolutional neural network is the sum of the classification category loss and the target anchor box loss, where the category loss is the cross entropy loss and the anchor box loss is the Huber loss.

[0072] The cross loss is:

[0073]

[0074] Where N is the number of samples, C is the number of categories, and w i is the weight of the i-th sample, y ic Indicates that the i-th sample belongs to category c, p icis the probability that the model predicts that the i-th sample belongs to category c;

[0075] The huber loss is calculated as:

[0076]

[0077] Among them, y is the true value, is the predicted value, and δ is a threshold that controls where the loss function switches from L1 loss to L2 loss.

[0078] Example 2

[0079] In this embodiment 2, a target detection method based on an attention self-calibration graph convolutional neural network RailRadarGNN is provided. The method flow is as follows: Figure 1 As shown in the figure. After the millimeter-wave radar is initialized, it continuously inputs the collected radar frames into the queue; it asynchronously obtains the program end instruction. If it ends, it cleans up the occupied resources and ends the program; otherwise, it continues to collect data. The other part of the asynchronous operation will read several frames of radar point cloud data from the queue at intervals, convert the processed point cloud data into a graph structure, and send it to the trained graph convolutional neural network, and finally output the detection and classification results to give an early warning for train driving.

[0080] In this embodiment, the subway obstacle detection process based on the millimeter wave radar image convolutional neural network is as follows: Figure 2 As shown, it is divided into the following processes:

[0081] A. Millimeter-wave radar point cloud data acquisition and preprocessing

[0082] Point cloud data preprocessing includes millimeter wave radar sensor parameter configuration, point cloud data acquisition and multi-frame aggregation to ensure that the data is suitable for further analysis. The specific steps are as follows:

[0083] Data collection: Use long-range (detection range 500m) millimeter-wave radar to collect point cloud data of moving targets on subway tracks.

[0084] Data aggregation: Due to the weak reflection signal of long-distance radar, the reflection coefficient of the surface of the object and the change of RCS caused by the movement of the moving target, the single frame data will appear sparse or even disappear. Considering the moving speed of the detection target, the acquisition frame rate and the RCS of the object, the optimized fusion parameter N is determined, and the N frames of point cloud data are fused into 1 frame to overcome the effect of sparse point cloud. At the same time, the data is organized into a format suitable for subsequent GCN (graph convolutional neural network) processing and timestamped by the sensor.

[0085] B. Graph structure data construction

[0086] The construction of the graph structure includes the following steps: the aggregated frame point cloud data (by connecting the nodes using the k-nearest neighbor method (for any point, take the nearest k points to generate a connection relationship to build a graph structure) or the radius neighborhood method (for any point, take it as the center of the circle, and build a graph structure with all points within a certain distance) . We select the edge feature as the relative position between the two points, and the node features include Doppler velocity, RCS reflection area, time index, and node degree, so that the network is not affected by the irregularity of the point cloud.

[0087] C. Feature High-Dimensional Embedding

[0088] In graph convolutional networks, high-dimensional embedding of features is a key step that allows the network to transform sparse point cloud data into dense representations in high-dimensional space, thereby better capturing complex patterns and relationships in the data. Figure 4 As shown, each node (blue circle) has an initial eigenvector, denoted as s i Before graph convolution, the feature vector of each node is processed through a multi-layer perceptron (MLP). For each edge in the graph (the line connecting two nodes), the difference between the features of the two nodes is calculated, denoted as Δs i,j . Edge feature e i,j Represents the attributes between nodes. The node features are combined with the features of neighboring nodes through an aggregation function (such as sum, average, or more complex functions). This process takes into account the local neighborhood information of the node. The aggregated features are processed by another MLP to generate updated node features s i ′. This updated feature will be used for the next layer of graph convolution or as the final output

[0089] In this process, we use multi-layer perceptrons (MLP) before and after each graph convolution layer to embed features in high dimensions, which not only unifies the feature dimensions, but also facilitates the network to extract features from denser information. The dimension of the node feature embedding layer is mapped from the input 5-dimensional feature to 32-dimensional, and then passes through 64-, 128-, and 224-dimensional layers in sequence. The dimension of the edge feature embedding layer is mapped from the input 2-dimensional feature to 4-dimensional, and then passes through 8- and 16-dimensional layers in sequence.

[0090] D. Construction of graph convolutional neural network combining attention and automatic calibration mechanism

[0091] Each graph convolution layer in the graph convolution network follows a specific structure, which goes through a vector embedding layer, a graph convolution layer, a regularization layer, and a ReLU activation layer. The overall network structure is as follows Figure 3As shown in the figure, the data is used to construct a graph structure, where nodes represent data points and edges represent the relationship between data points. The data is then input into the graph convolutional network according to the network structure to perform deeper abstraction and learning of features. After the graph convolution layer, the network contains one or more regression heads for obstacle detection or classification. During the training process, the output of the network is compared with the true label and the loss is calculated. Finally, the network output is used for classification, obstacle detection or other tasks.

[0092] Among them, there are two types of convolutional layers: one is the graph convolutional layer 1 that combines the graph attention mechanism and the automatic calibration mechanism; the other is the graph convolutional layer 2 with a jump connection structure; their feature divisions are as introduced in B. The dimensions of the graph convolutional layer are 224, 224, 128, 64, and 32 from the input layer to the output layer, respectively, reflecting the process of feature extraction.

[0093] Before each layer of the network, the features are embedded into vectors; node features are initially embedded into 32-dimensional vectors, and edge features are embedded into 4-dimensional vectors. These embedded features are then input into a graph convolutional layer that combines an attention mechanism with a modified automatic registration mechanism. After embedding the vectors, the dimensionally compressed features are then input into another type of graph convolutional layer.

[0094] In this embodiment, the attention mechanism is one of the key technologies used to enhance the model's ability to identify key features in radar point cloud data. The attention mechanism allows the model to dynamically focus on the most important parts of the input data, thereby improving the accuracy and efficiency of obstacle detection. The graph convolution calculation can be viewed as the contents of formula (1) and formula (2), where b i,j Calculate the midpoint s for the graph structure i With s j Combined edge attribute e i,j The calculation result, W i With W j It is a learnable weight during the training process, which updates the node attributes of the next layer through the aggregation function Aggr and the fully connected layer (MLP).

[0095] After combining the automatic registration mechanism, the coefficient c is calculated i,j As shown in formula (3), combined with formula (4), Δs i,jAutomatic calibration is performed to repair the errors that may exist in the radar sensor during measurement. After linear projection, the dimensions are aligned. W is the learnable weight. For the attention coefficient matrix a, its calculation method is shown in formula (5). In this way, the relationship between the information aggregated from a certain node and the information aggregated from other related nodes can be obtained. Through this design, the graph convolution layer can effectively improve the detection accuracy of the model while maintaining the integrity of data features. The MPNN layer is a simple and general graph convolutional neural network layer. For scenarios that need to combine edge features, its structure is shown in formulas (6) and (7). For the MPNN layer, the method of passing to the next layer in formula (7) is modified, and a jump connection structure is added. Finally, as shown in formula (8), it helps the model converge faster while improving the detection accuracy of the model. Among them, Res is the jump connection structure, and its method is as shown in h updated =h+x, add the original features to the updated features.

[0096] b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i)(1)

[0097]

[0098] c i,j =LeakyReLU(s i +s j +e i,j ) (3)

[0099] b i,j =a T (c i,j +linear(Δs i,j )) (4)

[0100]

[0101]

[0102]

[0103] E. Obstacle Detection and Classification

[0104] After the features pass through the last layer of graph convolution, a layer of vectors is obtained, which will be used as the input of the classifier and target detection. If the sensor information includes angle features, a 6-dimensional output vector is generated. If it does not include angle features, a 5-dimensional output vector (category and target frame coordinates and width and height) is generated. Thus, the point cloud classification category and target anchor frame are obtained.

[0105] The loss function of model training is as shown in formula (9), which is obtained by the loss of classification category and the loss of target anchor box. The category loss is cross entropy loss and the anchor box loss is huber loss. The cross entropy loss is shown in formula (10), where N is the number of samples, C is the number of categories, and w i is the weight of the i-th sample, y ic Indicates that the i-th sample belongs to category c, p ic is the probability that the model predicts that the i-th sample belongs to category c. , Huber loss is calculated as shown in formula (11), where y is the true value, is the predicted value, and δ is a threshold used to control where the loss function switches from L1 loss to L2 loss. In order to solve the problem of sample imbalance, as shown in formula (12), the number of samples is multiplied by the ratio of the number of samples with the largest number of types in the batch of training data to train the model in the back propagation process. The calculation is divided into two parts to make the data all floating-point tensors in the GPU, maintain type consistency and computational efficiency. It helps alleviate the problem of tending to identify all points as background points in the case of few samples.

[0106] L=L cls +λL reg (9)

[0107]

[0108]

[0109] The backpropagation process of a graph convolutional neural network (GCN) is similar to that of a regular deep neural network, but the particularity of graph structured data needs to be taken into account. In a GCN, the feature vector of each node is updated through graph convolutional layers, which typically include weight matrices, activation functions, and possibly other operations.

[0110] Combination Figure 5 As shown, in this embodiment, target detection and recognition tests are finally performed on the Radar Scene dataset and the dataset actually collected in the subway tunnel. The accuracy rates (True Positive) for vehicles, moving pedestrians and crowds 120 meters away are 85%, 78% and 82% respectively, which basically meet the needs of millimeter wave radar sensors for autonomous obstacle target recognition in urban rail.

[0111] In this embodiment, other clustering algorithms such as DBSCAN can also be used to generate graph structure data, which can also effectively express the spatial relationship of point cloud data. At the same time, variants based on graph neural networks can also be used. Graph Convolutional Networks (GCN) is a basic architecture of graph neural networks, which learns the representation of nodes by applying convolution operations on the nodes of the graph. GCN variants, such as ChebNet and GCNII, can also achieve certain detection effects by improving the design of convolution kernels and optimizing the message passing mechanism.

[0112] In summary, autonomous obstacle detection and tracking of urban rail trains are subject to all-weather constraints (day, night, and severe weather conditions such as fog, rain, snow, etc.), and need to overcome the influence of lighting and severe weather. At the same time, they need to have low cost, low computational and storage complexity, and are suitable for real-time edge deployment and application. Autonomous obstacle detection technology on urban rail vehicles mainly relies on optical sensors such as cameras and lidars. These technologies are greatly affected by harsh environments such as light and rain and fog, and their performance will be greatly reduced. In addition, the existing methods based on vision and radar fusion have complex processing procedures and slow response speeds, making it difficult to meet real-time requirements. The irregularity and sparsity of point cloud data lead to difficulties in feature extraction in existing algorithms, especially in millimeter-wave radar data. Due to the sparse data points, it is difficult to effectively identify obstacles. In addition, the problem of sample imbalance is also a challenge for existing technologies. Since the number of background points is much larger than the number of obstacle points, the model is overly sensitive to background points during training, which affects the detection effect of obstacles. The RailRadarGNN model in this embodiment aims to combine graph convolutional neural networks, graph attention mechanisms, multi-layer high-dimensional graph embedding, graph convolution rotation and translation invariance, feature automatic calibration mechanisms and network structure jump connection technology to solve the high-dimensional graph embedding, feature extraction and accurate classification and regression of disordered irregular point cloud data. The attention mechanism enables the model to focus on key features, improves the computing storage efficiency and specification capabilities; high-dimensional embedding helps to achieve information reorganization and high-dimensional aggregation of channel features, and improves the expression and classification performance of the model. The model's adaptability to different data distributions improves the accuracy and robustness of detection; rotation and translation invariance ensures that the detection, classification and tracking of obstacles are independent of the absolute position and rotation angle of the object; the jump connection structure effectively solves the problem of excessive smoothing of graph neural networks. Through the unique network structure and feature selection division, attention is paid to the feature relationship between different points, and the global information is obtained by combining the attention mechanism, thereby achieving more efficient and accurate detection of rail transit obstacle classification and detection. Not only is it suitable for rail transit obstacle detection, its principles and technical means can also be extended to other fields, such as autonomous driving, intelligent monitoring, etc., with broad application prospects and market potential.

[0113] In this embodiment, the graph convolutional neural network (GCN) is the core of obstacle classification and detection. GCN can process graph structure data and extract the features of point cloud data through the graph convolution layer. GCN can not only capture the local features of point cloud data, but also capture global features through the graph structure, thereby improving the accuracy of obstacle detection. In addition, the introduction of GCN enables the model to adaptively learn complex patterns in point cloud data, improving the generalization ability of the model. The application of graph convolutional neural networks makes the classification of obstacles in complex environments more accurate. The graph attention mechanism improves the model's ability to capture important features, while the automatic calibration mechanism enhances the robustness of the system. In response to the problem of sample imbalance, this embodiment proposes a method based on sample weight adjustment. During the model training process, by assigning different weights to samples of different categories, the model pays more attention to samples of minority categories during training, thereby effectively solving the problem of sample imbalance and improving the model's detection performance for minority category obstacles.

[0114] Example 3

[0115] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution is implemented. The method includes:

[0116] Initialize the millimeter-wave radar and continuously input the collected radar frames into the queue; read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure;

[0117] The graph structure is processed by using a trained graph convolutional neural network, and obstacle detection and classification results are finally output to give an early warning for train operation; wherein, the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in 32-dimensional vectors, and edge features are embedded in 4-dimensional vectors.

[0118] Example 4

[0119] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the above-mentioned urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution, the method comprising:

[0120] Initialize the millimeter-wave radar and continuously input the collected radar frames into the queue; read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure;

[0121] The graph structure is processed by using a trained graph convolutional neural network, and obstacle detection and classification results are finally output to give an early warning for train operation; wherein, the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in 32-dimensional vectors, and edge features are embedded in 4-dimensional vectors.

[0122] Example 5

[0123] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution, the method comprising:

[0124] Initialize the millimeter-wave radar and continuously input the collected radar frames into the queue; read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure;

[0125] The graph structure is processed by using a trained graph convolutional neural network, and obstacle detection and classification results are finally output to give an early warning for train operation; wherein, the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; node features are initially embedded in 32-dimensional vectors, and edge features are embedded in 4-dimensional vectors.

[0126] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. An urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution, characterized in that: include: The millimeter-wave radar is initialized and the collected radar frames are continuously input into the queue; Read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure; The graph structure is processed by using a trained graph convolutional neural network, and finally obstacle detection and classification results are output to give an early warning for train travel; wherein, the trained graph convolutional neural network includes a regression head and 4 graph convolutional layers, and the regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer; the convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure added; before each layer of the network is passed, feature vector embedding is performed; Node features are initially embedded into 32-dimensional vectors and edge features are embedded into 4-dimensional vectors.

2. The method for autonomous obstacle detection on urban rail vehicles based on millimeter wave radar image convolution according to claim 1 is characterized in that: The graph convolution is calculated as: b i,j =f(e i,j ,W j s j ,W i s i ),j∈N(i) Among them, b i,j is the node s in the graph structure i With node s j The combined edge attribute e i,j The calculation result, W i With W j It is the learnable weight in the training process, which updates the node attributes of the next layer through the aggregation function Aggr and the fully connected layer MLP; i ′ is the node feature output to the next layer of the network, and N(i) is all the nodes that are connected to point i after the graph data is constructed.

3. The urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution according to claim 2 is characterized in that: The automatic calibration mechanism is to calculate the coefficient c i,j , combined with Δs i,j Perform automatic calibration to correct possible errors in radar sensor measurements and align dimensions via linear projection: c i,j =LeakyReLU(s i +s j +e i,j ) b i,j =a T (c i,j +linear(Δs i,j )) Among them, a T represents the transpose of the attention coefficient matrix, and linear(Δsi,j) is the linear mapping of the feature difference between point i and point j.

4. The urban rail vehicle-mounted autonomous obstacle detection method based on millimeter wave radar image convolution according to claim 3 is characterized in that: The structure of the graph convolution layer is as follows: Add the jump connection structure Res, that is, add the original features to the updated features:

5. The method for autonomous obstacle detection on urban rail vehicles based on millimeter wave radar image convolution according to claim 4 is characterized in that: The total loss function of the graph convolutional neural network is the sum of the classification category loss and the target anchor box loss, where the category loss is the cross entropy loss and the anchor box loss is the Huber loss.

6. The method for autonomous obstacle detection on urban rail vehicles based on millimeter wave radar image convolution according to claim 5 is characterized in that: The cross loss is: Where N is the number of samples, C is the number of categories, and w i is the weight of the i-th sample, y ic Indicates that the i-th sample belongs to category c, p ic is the probability that the model predicts that the i-th sample belongs to category c; The huber loss is calculated as: Among them, y is the true value, is the predicted value, and δ is a threshold that controls where the loss function switches from L1 loss to L2 loss.

7. An urban rail vehicle-mounted autonomous obstacle detection system based on millimeter wave radar image convolution, characterized in that: include: The acquisition module is used for initializing the millimeter-wave radar and continuously inputs the collected radar frames into the queue; Read several frames of radar point cloud data from the queue and convert the processed point cloud data into a graph structure; A processing module is used to process the graph structure using a trained graph convolutional neural network, and finally output obstacle detection and classification results to warn the train. The trained graph convolutional neural network includes a regression head and four graph convolutional layers. The regression head and each graph convolutional layer include a vector embedding layer, a convolutional layer, a regularization layer and a ReLU activation layer. The convolutional layer in the regression head is a convolutional layer that combines a graph attention mechanism and an automatic calibration mechanism, and the convolutional layer in the graph convolutional layer is a convolutional layer with a jump connection structure. Before each layer of the network is passed, feature vector embedding is performed. Node features are initially embedded into 32-dimensional vectors and edge features are embedded into 4-dimensional vectors.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the urban rail vehicle-mounted autonomous obstacle detection method based on millimeter-wave radar image convolution as described in any one of claims 1 to 6.

Images