A bird's-eye view semantic segmentation method for track scenarios and a train control system

By employing a bird's-eye view semantic segmentation method for rail scenarios and utilizing multiple networks to process train front view images, a comprehensive and accurate understanding of rail transit scenarios is achieved. This solves the problems of large recognition errors and low efficiency in existing technologies, thereby improving the safety and reliability of train operation.

CN118196419BActive Publication Date: 2026-11-10BEIHANG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410376498.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-29
Publication Date
2026-11-10
Estimated Expiration
2044-03-29

AI Technical Summary

Technical Problem

In rail transit scenarios, existing technologies fail to capture comprehensive bird's-eye view features and have sparse edge features, resulting in large recognition errors and low efficiency, which affects train operation safety.

Method used

A bird's-eye view semantic segmentation method for track scenes is adopted. By acquiring the front view image of the train in real time, a feature extraction network, a depth prediction network, a deformable attention network, and a cross attention network are used in combination with a semantic segmentation network to achieve comprehensive and accurate extraction and fusion of image features, thereby obtaining bird's-eye view visual features.

Benefits of technology

It improves the recognition accuracy of traffic participants in rail scenarios, enhances the safety and reliability of intelligent transportation systems, and enables accurate control and monitoring of trains.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118196419B_ABST
    Figure CN118196419B_ABST
Patent Text Reader

Abstract

This invention discloses a bird's-eye view semantic segmentation method and train control system for rail scenarios, comprising: real-time acquisition of a front-view image captured by a camera deployed above the train; processing the front-view image using a feature extraction network to obtain first feature data; processing the first feature data using a depth prediction network to obtain a first bird's-eye view visual feature; processing the first feature data using a deformable attention network to obtain a second bird's-eye view visual feature; fusing the first and second bird's-eye view visual features using a cross-attention network to obtain a fused feature; and processing the fused feature using a semantic segmentation network to obtain a semantic segmentation result. This invention achieves comprehensive and accurate extraction of bird's-eye view features by fusing sparse and dense bird's-eye view visual features, thereby accurately identifying traffic participation elements and effectively improving the safety and reliability of intelligent transportation in rail scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing. Specifically, it relates to a bird's-eye view semantic segmentation method for railway scenarios and a train control system. Background Technology

[0002] With the widespread adoption of rail transit, trains have reached countless cities and towns, becoming an indispensable mode of transportation and a crucial part of the transportation system. To achieve intelligent train control, existing technologies collect image information from traffic scenes in real time, utilizing computer vision algorithms to recognize and segment these images, thus enabling real-time understanding of the traffic environment.

[0003] However, due to the complex and ever-changing environment around the track, and the need for rapid identification to achieve real-time detection of trains traveling at high speeds, the existing technologies do not fully acquire bird's-eye view features, and the extracted edge features are sparse. There is also a large amount of irrelevant information that affects the processing results. As a result, the identification results will not only have large errors, but the identification efficiency will also be low, affecting traffic safety.

[0004] Therefore, this invention urgently requires a bird's-eye view semantic segmentation method for accurate and rapid understanding of rail transit scenarios. Summary of the Invention

[0005] The present invention is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present invention is to provide a bird's-eye view semantic segmentation method and train control system for track scenarios to improve the accuracy of identifying traffic participants.

[0006] To solve the above problems, the present invention is implemented using the following technical solution:

[0007] A method for bird's-eye view semantic segmentation in railway scenarios is provided. The method includes: real-time acquisition of a front-view image captured by a camera deployed above a train; processing the front-view image using a feature extraction network to obtain first feature data; processing the first feature data using a depth prediction network to obtain first bird's-eye view features, including: presetting a feature depth distribution threshold based on the installation position of the camera; converting the first feature information to a world coordinate system; predicting the depth distribution corresponding to the converted first feature data using a depth prediction network based on the feature depth distribution threshold; performing outer product, projection, and max pooling aggregation on the converted first feature information and depth distribution to obtain the first bird's-eye view features; and processing the first feature data using a deformable attention network to obtain second bird's-eye view features, including: presetting a two-dimensional bird's-eye view space and converting the coordinates in the two-dimensional bird's-eye view space to a world coordinate system; randomly selecting multiple three-dimensional reference points above the converted two-dimensional bird's-eye view space and projecting them onto an image coordinate system to obtain second bird's-eye view features. Projection points; feature points around the projection points are collected in the first feature data; the feature points are processed through a deformable attention network to determine the feature value corresponding to each projection point; a second bird's-eye view feature is determined based on the feature value; the first bird's-eye view feature and the second bird's-eye view feature are fused based on a cross-attention network to obtain a fused feature, including: applying linear transformations to the first bird's-eye view feature and the second bird's-eye view feature respectively to obtain the corresponding query, key, and value, wherein the first bird's-eye view feature corresponds to the first query, the first key, and the first value, and the second bird's-eye view feature corresponds to the second query, the second key, and the second value; a first attention matrix is ​​calculated based on the first key and the second query; a second attention matrix is ​​calculated based on the first query and the second key; the first value and the second value are processed based on the first attention matrix and the second attention matrix to obtain the fused feature; the fused feature is processed using a semantic segmentation network to obtain a semantic segmentation result, wherein the semantic segmentation network sequentially includes multiple convolutional layers, deconvolutional layers, upsampling layers, and pixel-wise classification layers.

[0008] Optionally, the front view image is processed using a feature extraction network to obtain first feature data, including: processing the front view image using a ResNet50 neural network to obtain multi-scale features, wherein the multi-scale features include multiple feature data with different resolutions; fusing the multi-scale features sequentially according to resolution from low to high, including: performing 1×1 convolution and upsampling processing on the feature data of the first resolution, and fusing the processing result with the feature data of the second resolution after performing 1×1 convolution; the first resolution is the feature data with lower resolution among the multi-scale features; the second resolution is the feature data with higher resolution among the multi-scale features; and aggregating the multi-scale feature fusion result using a convolutional neural network to obtain the first feature data.

[0009] Optionally, the transformed first feature information and depth distribution are subjected to outer product, projection, and max pooling aggregation to obtain the first bird's-eye view visual feature, including: performing an outer product between the transformed first feature information and the depth distribution to obtain virtual points; projecting the virtual points onto a stereoscopic bird's-eye view space; dividing the stereoscopic bird's-eye view space into multiple units, aggregating the projected virtual points belonging to the same unit, and performing max pooling on the aggregation result to obtain the first bird's-eye view visual feature.

[0010] Optionally, the feature points are processed by a deformable attention network to determine the feature value corresponding to each projection point, including: processing the feature points by the deformable attention network to determine the corresponding weight value; performing a weighted summation on the feature points corresponding to each projection point to obtain the corresponding feature value, and determining the position of the feature value in the two-dimensional bird's-eye view space.

[0011] Optionally, semantic segmentation network processing is applied to the fused features to obtain semantic segmentation results, including: performing multi-layer convolution processing on the fused features to obtain first data; performing deconvolution processing on the first data to obtain second data; upsampling the second data to obtain third data with the same resolution as the fused features; and performing pixel-by-pixel classification on the third data to obtain semantic segmentation results.

[0012] Optionally, the method further includes: constructing a semantic segmentation model, wherein the semantic segmentation model includes a feature extraction network, a deep prediction network, a deformable attention network, a cross attention network, and a semantic segmentation network; collecting multiple image data and performing polygon annotation on the collected image data, including: starting from a point on the edge of the track in the image data, extending along the edge of the track region in the image data until a closed polygon is formed; and inputting the annotated image data into the semantic segmentation model for training to adjust the variable parameters in the semantic segmentation model.

[0013] Optionally, the semantic segmentation model can be optimized using the cross-entropy loss function.

[0014] Optionally, the first and second values ​​are processed based on the first and second attention matrices to obtain fused features, including: Among them, F fuse Q represents the fusion feature. BEV-D This indicates the second query. Indicates the first key, V BEV-S Let Q represent the first value. BEV-S Indicates the first query. Indicates the second key, V BEV-D This represents the second value, and softmax() represents the normalization function.

[0015] Optionally, the expression for the cross-entropy loss function includes: Among them, L cross_entropy Let N represent the loss value, M represent the number of pixels in the image data, i represent the pixel identifier, c represent the class identifier, and y represent the loss value. ic The sign function is defined as follows: if the true class of pixel i is the same as class c, then y... ic =1; otherwise, y ic p is 0; ic This represents the probability that pixel i belongs to category c.

[0016] A train control system is provided, comprising: an acquisition module for real-time acquisition of a frontal view image captured by a camera deployed above the train; a feature extraction module for processing the frontal view image acquired by the acquisition module using a feature extraction network to obtain first feature data; a first processing module for processing the first feature data using a depth prediction network to obtain a first bird's-eye view feature; a second processing module for processing the first feature data using a deformable attention network to obtain a second bird's-eye view feature; a feature fusion module for fusing the first bird's-eye view feature output by the first processing module and the second bird's-eye view feature output by the second processing module using a cross-attention network to obtain a fused feature; a third processing module for processing the fused feature output by the feature fusion module using a semantic segmentation network to obtain a semantic segmentation result, wherein the semantic segmentation network sequentially includes multiple convolutional layers, deconvolutional layers, upsampling layers, and pixel-by-pixel classification layers; and a control module for controlling train movement based on the semantic segmentation result.

[0017] Compared with existing technologies, this invention provides a bird's-eye view semantic segmentation method and train control system for rail scenarios. By using a depth prediction network, the depth distribution of image features is estimated based on discrete thresholds to obtain effective image depth. By fusing sparse and dense bird's-eye view visual features, comprehensive and accurate extraction of bird's-eye view features is achieved, thereby accurately identifying traffic participation elements and effectively improving the safety and reliability of intelligent transportation in rail scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of a bird's-eye view semantic segmentation method for track scenes provided by a specific embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the scope of protection of the present invention.

[0022] Example 1

[0023] This embodiment provides a bird's-eye view semantic segmentation method for track scenes, the process of which is as follows: Figure 1 As shown, it includes:

[0024] S1 acquires real-time front-view images captured by cameras deployed above the train.

[0025] In this embodiment, a forward-looking camera is deployed above the train's front end. The intrinsic and extrinsic parameters of the camera are calibrated to acquire image data during train operation. The intrinsic parameter matrix of the camera is as follows: The extrinsic parameter matrix is Furthermore, the MLP algorithm is used to display and encode the internal and external parameters of the shooting device, and its expression includes P. e =MLP(concat(E, I)), where P e This indicates the encoding result.

[0026] S2 performs feature extraction network processing on the front view image to obtain the first feature data.

[0027] Step S2 includes:

[0028] S200 uses the ResNet50 neural network to process the front view image and obtain multi-scale features.

[0029] The multi-scale features include multiple feature data with different resolutions.

[0030] Assuming the resolution in the multi-scale features, from high to low, are respectively and

[0031] S210 fuses multi-scale features sequentially from low to high resolution to obtain the first feature data.

[0032] In this step, the feature data of the first resolution is subjected to 1×1 convolution and upsampling, and the processing result is fused with the feature data of the second resolution after 1×1 convolution. The first resolution is the feature data with lower resolution in the multi-scale features; the second resolution is the feature data with higher resolution in the multi-scale features.

[0033] Assuming the minimum resolution Performing a 1×1 convolution operation yields And Perform 2x upsampling and adjacent The results of 1×1 convolution are fused to obtain By analogy, the final result obtained is a multi-scale feature fusion result with rich semantic information in all four layers. The multi-scale fusion results are aggregated using a convolutional neural network to obtain the first feature data F. I =Conv(F I1 F I2 F I3 F I4 ),in

[0034] S3 processes the first feature data based on a deep prediction network to obtain the first bird's-eye view feature.

[0035] Step S3 includes:

[0036] The S300 presets a feature depth distribution threshold based on the installation location of the shooting device.

[0037] Based on the pose of the forward-looking camera deployed at the front of the train, preset depth distribution thresholds D1, D2, ..., D1 for pixel features in the image are used. 40 .

[0038] S310 converts the first feature information to the world coordinate system.

[0039] Based on the intrinsic and extrinsic parameters of the shooting device, a rigid coordinate transformation is performed on the first feature information to obtain the coordinates of each pixel in the first feature information in the world coordinate system.

[0040] S320 uses a depth prediction network to predict the depth distribution corresponding to the transformed first feature data based on a preset feature depth threshold.

[0041] A depth prediction network is used to predict the depth distribution of each pixel coordinate in the transformed first feature information. Each pixel coordinate corresponds to a set of preset feature depth distribution thresholds, and each depth in each set of preset feature depth distribution thresholds corresponds to a probability value, which represents the probability that the pixel's height is at this preset depth. The expression includes: K = DepthNet(T(FI P e In this context, K represents the probability value of the discrete depth distribution, T represents the rigid coordinate transformation of the first feature information, DepthNet represents the depth prediction network, and F... I This represents the first feature data.

[0042] S330 performs outer product, projection, and max pooling aggregation on the transformed first feature information and depth distribution to obtain the first bird's-eye view feature.

[0043] This step includes:

[0044] Step 1: Perform an outer product between the transformed first feature information and the depth distribution to obtain virtual points.

[0045] Depth D i The feature information at the location is in, Depth D i The probability value at that location. The result of the outer product is used to obtain virtual points by combining the feature information of each depth.

[0046] Step 2: Project the virtual points onto the 3D bird's-eye view space.

[0047] This embodiment assumes that virtual points are projected onto a three-dimensional bird's-eye view space to obtain...

[0048] Step 3: Divide the 3D bird's-eye view space into multiple units, and aggregate the projected virtual points belonging to the same unit.

[0049] The bird's-eye view space is divided into bev l It consists of cuboid units, and the virtual point features belonging to the same cuboid after projection are aggregated according to the virtual point projection position.

[0050] Step 4: Perform max pooling on the aggregation results to obtain the first bird's-eye view features.

[0051] Assuming the first bird's-eye view features are

[0052] In this embodiment, since the depth is a set of pre-set data and depths outside the set are not considered, the resulting first bird's-eye view features are sparse, resulting in the loss of a lot of information in the image features.

[0053] S4 processes the first feature data using a deformable attention network to obtain the second bird's-eye view feature.

[0054] Step S4 includes:

[0055] The S400 presets a two-dimensional bird's-eye view space and transforms the coordinates in the two-dimensional bird's-eye view space to the world coordinate system.

[0056] The grid size of the two-dimensional bird's-eye view space is preset, and the size of the bird's-eye view space is scaled proportionally to the world coordinate system. Specifically, the bird's-eye view space query is set. For a pixel at position (u, v) in a two-dimensional bird's-eye view space, its corresponding world coordinates (u′, v′) are calculated based on the size of the bird's-eye view grid. Where u′ represents the x-coordinate in the world coordinate system, u represents the x-coordinate in the shooting device system, s represents the actual spatial size represented by a unit grid in the bird's-eye view, and bev′ w v' represents the width of the two-dimensional bird's-eye view space, v' represents the ordinate in the world coordinate system, v' represents the ordinate in the camera's coordinate system, and bev' represents the width of the two-dimensional bird's-eye view space. h This represents the height of the two-dimensional bird's-eye view space. For each grid cell within each bird's-eye view space, the above bird's-eye view space query Q is performed. pi .

[0057] S410 randomly selects multiple three-dimensional reference points above the converted two-dimensional bird's-eye view space and projects the three-dimensional reference points onto the image coordinate system to obtain projection points.

[0058] Assuming this embodiment samples N 3D reference points along the z-axis, for each 3D reference point, it is projected onto the camera image coordinate system based on the camera's intrinsic and extrinsic parameters to obtain the 2D reference point coordinates p. ref .

[0059] S420 acquires feature points around the projection point in the first feature data.

[0060] The S430 uses a deformable attention network to process feature points and determine the feature value corresponding to each projection point.

[0061] This step includes: processing feature points through a deformable attention network to determine the corresponding weight values; performing a weighted summation of the feature points corresponding to each projection point to obtain the corresponding feature value, and determining the position of the feature value in the two-dimensional bird's-eye view space.

[0062] S440 determines the second bird's-eye view feature based on eigenvalues.

[0063] In this embodiment, the second bird's-eye view feature can be directly composed of the feature values ​​at each location; alternatively, adjacent feature values ​​can be interacted to determine the second bird's-eye view feature. Through interaction, the relationship between adjacent pixels can be understood, leading to a more accurate understanding of image information.

[0064] Assuming that the second bird's-eye view features are obtained through the above steps... Its expressions include Where n represents the identification number of the 3D reference point, and DeformAttn represents the deformable attention network.

[0065] In this embodiment, it is possible to ensure that each grid in the two-dimensional bird's-eye view space has information, but the obtained second bird's-eye view visual feature information is relatively dense, resulting in a large computational load. However, in this embodiment, only the front view image is acquired, which can greatly reduce the computational complexity.

[0066] S5 is based on a cross-attention network, which fuses first and second bird's-eye view features to obtain fused features.

[0067] Step S5 includes:

[0068] S500 applies linear transformations to the first bird's-eye view feature and the second bird's-eye view feature respectively to obtain the corresponding query, key and value, wherein the first bird's-eye view feature corresponds to the first query, the first key and the first value, and the second bird's-eye view feature corresponds to the second query, the second key and the second value.

[0069] Assuming the first query in this embodiment is Q BEV-S The second query is Q. BEV-D The first key is K BEV-S The second key is K BEV-D The first value is V BEV-S The second value is V BEV-D .

[0070] S510 calculates a first attention matrix based on the first key and the second query, whereby the first attention matrix represents the attention distribution of the second bird's-eye view feature on the first bird's-eye view feature.

[0071] Based on the second query Q BEV-D With the first key K BEV-S Calculate the first attention matrix w D-S To characterize F BEV-D Facebook EV-S Attention distribution.

[0072] S520 calculates a second attention matrix based on the first query and the second key, the second attention matrix representing the attention distribution of the first bird's-eye view feature on the second bird's-eye view feature.

[0073] Based on the first query Q BEV-S With the second key K BEV-D Calculate the second attention matrix w S-D To characterize F BEV-S For F BEV-D Attention distribution.

[0074] S530 processes the first and second values ​​based on the first and second attention matrices to obtain fused features.

[0075] Apply the attention weight matrix to the first value V. BEV-S Second value V BEV-D Obtain the corresponding weighted values ​​and add them together to obtain the fusion features. Its expression is Here, softmax() represents the normalization function.

[0076] In this embodiment, sparse first bird's-eye view features and dense second bird's-eye view features are fused to give full play to the advantages of the two bird's-eye view features and accurately extract the fused features, which helps to achieve semantic segmentation of the train's forward view.

[0077] S6 uses a semantic segmentation network to process the fused features to obtain semantic segmentation results.

[0078] The semantic segmentation network sequentially comprises multiple convolutional layers, deconvolutional layers, upsampling layers, and pixel-by-pixel classification layers. In this embodiment, a fully convolutional network is used as the semantic segmentation output head to output the semantic segmentation results of the track scene in bird's-eye view space. Specifically:

[0079] The S600 performs multi-layer convolution processing on the fused features to obtain the first data.

[0080] S610 performs deconvolution processing on the first data to obtain the second data.

[0081] S620 upsamples the second data to obtain third data with the same resolution as the fused feature.

[0082] Assume the third data is

[0083] The S630 performs pixel-by-pixel classification on the third data to obtain semantic segmentation results.

[0084] For F seg Perform pixel-by-pixel classification to obtain semantic segmentation results from a bird's-eye view of the orbital scene.

[0085] During training, the cross-entropy loss function is used to measure the difference between the pixel categories predicted by the model and the true labels, and to guide model optimization. Specifically, the cross-entropy loss function can be expressed as:

[0086]

[0087] Among them, L cross_entropyThe loss value is represented by M, which represents the number of categories (7 in this embodiment), representing the categories of track, trackside lane, traffic light, sign, roadblock, pedestrian, and others. N represents the number of pixels, i represents the current pixel index, c represents the category identifier, and p represents the loss value. ic y represents the probability that pixel i belongs to category c. ic It is a symbolic function, specifically: When the true class corresponding to pixel i is the same as c, y ic Select 1 if the value is 1, otherwise select 0.

[0088] By minimizing the loss function, the model gradually learns more accurate semantic segmentation predictions during training, thus achieving better performance.

[0089] Compared with existing technologies, this embodiment provides a bird's-eye view semantic segmentation method for rail scenarios. It converts the image features from the train's forward-looking perspective into a bird's-eye view, allowing viewers to obtain an overview of the traffic scene. This enables a better overall understanding of a larger area of ​​tracks and roads, acquiring broader, more complete, and more global traffic information, and a better understanding of the surrounding traffic environment. Deploying this bird's-eye view semantic segmentation method for rail scenarios on a train allows for the accurate identification of traffic elements such as tracks, pedestrians, and signs from the forward-looking images collected during the train journey. This enables an accurate understanding of the rail transit scene, allowing intelligent transportation systems to better control and monitor trains, thus improving the safety and reliability of the intelligent transportation system. Through a depth prediction network, depth distribution estimation is performed on image features based on discrete thresholds, obtaining effective image depth. By fusing sparse and dense bird's-eye view visual features, comprehensive and accurate extraction of bird's-eye view features is achieved, thereby accurately identifying traffic participants and effectively improving the safety and reliability of intelligent transportation in rail scenarios.

[0090] Example 2

[0091] This embodiment provides a train control system that implements the bird's-eye view semantic segmentation method for track scenes described in Embodiment 1, including:

[0092] The acquisition module acquires front-view images in real time from the camera deployed above the train.

[0093] The feature extraction module processes the front view image acquired by the acquisition module using a feature extraction network to obtain the first feature data.

[0094] The first processing module processes the first feature data based on a deep prediction network to obtain the first bird's-eye view feature.

[0095] The second processing module processes the first feature data based on a deformable attention network to obtain the second bird's-eye view feature.

[0096] The feature fusion module, based on a cross-attention network, fuses the first bird's-eye view features output by the first processing module and the second bird's-eye view features output by the second processing module to obtain fused features.

[0097] The third processing module uses a semantic segmentation network to process the fused features output by the feature fusion module to obtain semantic segmentation results. The semantic segmentation network includes multiple convolutional layers, deconvolutional layers, upsampling layers, and pixel-by-pixel classification layers.

[0098] The control module controls the train's movement based on the semantic segmentation results.

[0099] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A bird's-eye view semantic segmentation method for track scenes, characterized in that, include: Real-time acquisition of front view images captured by cameras deployed above the train; The front view image is processed using a feature extraction network to obtain first feature data; The first bird's-eye view feature is obtained by processing the first feature data based on the depth prediction network, including: setting a feature depth distribution threshold based on the installation position of the shooting device; converting the first feature information to the world coordinate system; predicting the depth distribution corresponding to the converted first feature data using the depth prediction network based on the feature depth distribution threshold; and performing outer product, projection and max pooling aggregation on the converted first feature information and depth distribution to obtain the first bird's-eye view feature. The second bird's-eye view feature is obtained by processing the first feature data using a deformable attention network, including: pre-setting a two-dimensional bird's-eye view space and transforming the coordinates of each coordinate in the two-dimensional bird's-eye view space to the world coordinate system; randomly selecting multiple three-dimensional reference points above the transformed two-dimensional bird's-eye view space and projecting them onto the image coordinate system to obtain projection points; collecting feature points around the projection points in the first feature data; processing the feature points through the deformable attention network to determine the feature value corresponding to each projection point; and determining the second bird's-eye view feature based on the feature value. Based on a cross-attention network, a first bird's-eye view visual feature and a second bird's-eye view visual feature are fused to obtain a fused feature. This includes: applying linear transformations to the first and second bird's-eye view visual features respectively to obtain corresponding queries, keys, and values, where the first bird's-eye view visual feature corresponds to a first query, a first key, and a first value, and the second bird's-eye view visual feature corresponds to a second query, a second key, and a second value; calculating a first attention matrix based on the first key and the second query; calculating a second attention matrix based on the first query and the second key; and processing the first and second values ​​based on the first and second attention matrices to obtain the fused feature. The semantic segmentation network is used to process the fused features to obtain the semantic segmentation result. The semantic segmentation network includes multiple convolutional layers, deconvolutional layers, upsampling layers and pixel-by-pixel classification layers.

2. The bird's-eye view semantic segmentation method for track scenes according to claim 1, characterized in that, The front view image is processed using a feature extraction network to obtain first feature data, including: The front view image is processed using a ResNet50 neural network to obtain multi-scale features, which include feature data with multiple different resolutions. The multi-scale features are fused sequentially according to their resolution from low to high, including: performing 1×1 convolution and upsampling on the feature data of the first resolution, and fusing the processing result with the feature data of the second resolution after performing 1×1 convolution; the first resolution is the feature data with the lower resolution among the multi-scale features; the second resolution is the feature data with the higher resolution among the multi-scale features; The first feature data is obtained by aggregating the multi-scale feature fusion results using a convolutional neural network.

3. The bird's-eye view semantic segmentation method for track scenes according to claim 1, characterized in that, The transformed first feature information and depth distribution are aggregated by outer product, projection, and max pooling to obtain the first bird's-eye view features, including: The transformed first feature information is multiplied by the depth distribution to obtain virtual points; the virtual points are projected onto the stereoscopic bird's-eye view space; the stereoscopic bird's-eye view space is divided into multiple units, the projected virtual points belonging to the same unit are aggregated, and the aggregation result is max-pooled to obtain the first bird's-eye view visual feature.

4. The bird's-eye view semantic segmentation method for track scenes according to claim 1, characterized in that, Feature points are processed using a deformable attention network to determine the feature values ​​corresponding to each projection point, including: The feature points are processed by a deformable attention network to determine the corresponding weight values; the feature points corresponding to each projection point are weighted and summed to obtain the corresponding feature values, and the position of the feature values ​​in the two-dimensional bird's-eye view space is determined.

5. A bird's-eye view semantic segmentation method for track scenes according to claim 1, characterized in that, The fused features are processed by a semantic segmentation network to obtain semantic segmentation results, including: The fused features are processed by multiple convolutional layers to obtain the first data. The first data is processed by a deconvolution layer to obtain the second data; The second data is upsampled to obtain a third data with the same resolution as the fused feature. The third data is classified pixel by pixel to obtain semantic segmentation results.

6. A bird's-eye view semantic segmentation method for track scenes according to claim 1, characterized in that, Also includes: Construct a semantic segmentation model, which includes a feature extraction network, a deep prediction network, a deformable attention network, a cross attention network, and a semantic segmentation network; Collect multiple image data and perform polygon annotation on the collected image data, including: starting from a point on the edge of the track in the image data, extending along the edge of the track region in the image data until a closed polygon is formed; The labeled image data is input into the semantic segmentation model for training, so as to adjust the variable parameters in the semantic segmentation model.

7. A bird's-eye view semantic segmentation method for track scenes according to claim 6, characterized in that, The semantic segmentation model is optimized using the cross-entropy loss function.

8. A bird's-eye view semantic segmentation method for track scenes according to claim 3, characterized in that, The first and second values ​​are processed based on the first and second attention matrices to obtain fused features, including: Among them, F fuse Q represents the fusion feature. BEV-D This indicates the second query. Indicates the first key, V BEV-S Let Q represent the first value. BEV-S Indicates the first query. Indicates the second key, V BEV-D This represents the second value, and softmax() represents the normalization function.

9. A bird's-eye view semantic segmentation method for track scenes according to claim 7, characterized in that, The expression for the cross-entropy loss function includes: Among them, L cross_entropy Let N represent the loss value, M represent the number of pixels in the image data, i represent the pixel identifier, c represent the class identifier, and y represent the loss value. ic The sign function is defined as follows: if the true class of pixel i is the same as class c, then y... ic =1; otherwise, y ic p is 0; ic This represents the probability that pixel i belongs to category c.

10. A train control system, characterized in that, The bird's-eye view semantic segmentation method for track scenes according to any one of claims 1-9, wherein the train control system comprises: The acquisition module acquires front-view images in real time from the camera deployed above the train. The feature extraction module processes the front view image acquired by the acquisition module based on the feature extraction network to obtain the first feature data; The first processing module processes the first feature data based on a deep prediction network to obtain the first bird's-eye view feature; The second processing module processes the first feature data based on a deformable attention network to obtain the second bird's-eye view feature; The feature fusion module, based on a cross-attention network, fuses the first bird's-eye view features output by the first processing module and the second bird's-eye view features output by the second processing module to obtain fused features. The third processing module uses a semantic segmentation network to process the fused features output by the feature fusion module to obtain semantic segmentation results. The semantic segmentation network includes multiple convolutional layers, deconvolutional layers, upsampling layers, and pixel-by-pixel classification layers in sequence. The control module controls the train's movement based on the semantic segmentation results.

Citation Information

Patent Citations

  • Image classification method and device based on aerial view, equipment and storage medium

    CN114692720A

  • Multi-view fusion scene reconstruction method, sensing network training method and device

    CN114913290A