Target semantic segmentation method based on radar data and image data

Abstract

The invention discloses a target semantic segmentation method based on radar data and image data, and the method comprises the following steps: 1, carrying out the preprocessing of radar point cloud data: obtaining the radar point cloud data through a radar sensor, and carrying out the preprocessing of the radar point cloud data; 2, projecting the preprocessed point cloud data, and converting the point cloud data into a pseudo-depth map form; the method specifically comprises the following sub-steps: step 2-1, determining a projection plane; step 2-2, carrying out coordinate conversion; step 2-3, carrying out projection calculation; step 3, performing feature interaction: performing feature interaction on the radar data in the pseudo-depth map form and the input image data; and step 4, inputting the fused multi-modal features into a decoder network to generate a semantic segmentation result. By combining the advantages of the radar point cloud data and the image data, the invention provides a semantic segmentation method which can still keep high precision in severe environments such as low light, strong light, rain and fog and the like.

Description

Technical Field

[0001] The invention relates to a target semantic segmentation method based on radar data and image data, which is used for monitoring and identifying aircraft and ground equipment in an airport. Background Art

[0002] In modern airport management, it is crucial to accurately monitor and identify aircraft and ground equipment within the airport. Traditional monitoring systems rely mainly on image data, but image data often performs poorly under adverse weather conditions such as low light, strong light, rain and fog, which can easily lead to misidentification and missed identification. In contrast, radar data has the advantages of strong anti-interference ability and all-weather operation, but is relatively insufficient in the recognition of fine structures and textures of targets. Therefore, a multimodal semantic segmentation method combining radar data and image data has emerged to make up for the shortcomings of a single data source and provide more accurate and robust environmental perception capabilities.

[0003] At present, airport image segmentation technology has made certain progress, especially driven by deep learning technology. Semantic segmentation models such as UNet, SegNet, DeepLab, etc. have performed well in the identification of aircraft and ground equipment. Semantic segmentation technology achieves a refined understanding of the image by classifying each pixel in the image into different semantic categories. In the airport environment, these technologies can be used to identify and locate various aircraft, vehicles, and equipment. However, the complex airport environment and the variety of aircraft types pose challenges to the segmentation task. There may be occlusions and overlaps between aircraft and ground equipment, and changes in weather and lighting conditions make it difficult for segmentation methods that rely solely on image data to maintain high accuracy in all cases. In addition, airport operations require real-time monitoring and processing, which places higher requirements on the speed and accuracy of segmentation methods.

[0004] The main problems of existing semantic segmentation algorithms for aircraft in airport scenes are as follows:

[0005] 1) Image data performs poorly in harsh environments: Image data is easily disturbed in low light, strong light, rain and fog, leading to misidentification and missed identification. This situation is particularly prominent in complex environments such as airports, requiring a segmentation method that can operate stably under various lighting and weather conditions.

[0006] 2) Sparsity and noise issues of point cloud data: Although radar point cloud data provides three-dimensional spatial information, its sparseness, uneven density and noise limit its ability to identify details. How to effectively process and utilize this data is an important technical challenge.

[0007] 3) Alignment and fusion of multimodal data: Image data and point cloud data come from different sensors and have different spatial resolutions and information types. How to accurately align the two types of data in the same coordinate system and effectively fuse them to improve the semantic segmentation effect is an urgent problem to be solved.

[0008] 4) Detail preservation and edge restoration: In the process of multimodal data fusion, how to preserve the fine structure and edge information of the image and accurately reflect these details in the final semantic segmentation results is a technical difficulty. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a target semantic segmentation method based on radar data and image data, which can maintain high-precision semantic segmentation in harsh environments such as low light, strong light, rain and fog by combining the advantages of radar point cloud data and image data.

[0010] The object of the present invention is achieved through the following technical solution: a target semantic segmentation method based on radar data and image data, comprising the following steps:

[0011] Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained through the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. The radar point cloud data is preprocessed as follows:

[0012] Step 1-1, denoising: Use a filter to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, and eliminate isolated points and noise points;

[0013] Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms, increase the density of the point cloud data, and use PointNet++ for completion;

[0014] Step 1-3, data registration: Use the iterative closest point algorithm to align point cloud data from different perspectives, and perform iterative optimization by minimizing the distance between the closest point pairs of the two sets of point clouds to ensure the consistency of the point cloud data;

[0015] Step 2: Project the preprocessed point cloud data and convert it into a pseudo depth map; specifically, the following sub-steps are included:

[0016] Step 2-1, determine the projection plane: select the projection plane according to the angle when the point cloud data is collected;

[0017] Step 2-2, coordinate conversion: convert the 3D point cloud data to the camera coordinate system; first, obtain the camera's external parameters including the rotation matrix R and the translation vector T; then convert the point cloud data from the radar coordinate system to the camera coordinate system:

[0018] P c =R·P1+T

[0019] Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system;

[0020] Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map:

[0021] u=f x ·X / Z+c x

[0022] v=f y ·Y / Z+c y

[0023] Where (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y is the focal length parameter of the camera in the x and y axis directions, (c x ,c y ) is the camera principal point coordinate;

[0024] Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data; the specific method is as follows:

[0025] Step 3-1, feature extraction: use convolutional neural network to extract features of image data and pseudo depth map data respectively;

[0026] Step 3-2, multimodal alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism; contrastive learning ensures the consistency of feature expression, and the cross-attention mechanism realizes the complementarity and interaction of features; the cross-attention formula is as follows:

[0027] Attention(Q,K,V)=softmax(Q·K T / sqrt(d k ))·V

[0028] Q is the query matrix, which is the processed point cloud feature; K is the key matrix, V is the value matrix, K and V are the features extracted from the image data; d k is the scale factor;

[0029] Step 3-3, feature fusion: The aligned image features and pseudo depth map features are fused to form a multimodal feature representation; after splicing the aligned image features and pseudo depth map features, the multi-head self-attention mechanism is used for feature fusion. The formula is as follows:

[0030] Z=Attention(head1,…,head i ,…,head n )·W o

[0031] Where W o is the output weight matrix, head i =Attention(Q i ,K i ,V i );

[0032] Step 4: Input the fused multimodal features into the decoder network to generate semantic segmentation results.

[0033] The beneficial effects of the present invention are:

[0034] 1) Improving segmentation accuracy in harsh environments: The present invention aims to propose a semantic segmentation method that can maintain high accuracy in harsh environments such as low light, strong light, rain and fog by combining the advantages of radar point cloud data and image data.

[0035] 2) Solve the problem of sparse and noisy point cloud data: Through the point cloud completion module, the radar point cloud data is denoised and interpolated to improve the density and quality of the point cloud data, thereby enhancing its ability to recognize details.

[0036] 3) Accurate alignment and fusion of multimodal data: Through the compression projection module and the multimodal alignment module, contrastive learning and cross-attention mechanisms are used to achieve effective alignment and fusion of point cloud data and image data, give full play to the complementary advantages of the two types of data, and improve the overall segmentation performance.

[0037] 4) Enhanced detail preservation and edge restoration: After multimodal data fusion, an edge preservation and recovery module is introduced to utilize the rich edge information of image data to ensure that the generated semantic segmentation mask has high-precision details and edges, thereby improving the accuracy and reliability of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart for performing semantic segmentation of aircraft based on radar data and image data;

[0039] Figure 2 It is the projection of the original radar data at different angles;

[0040] Figure 3 A schematic diagram of a depth map obtained by projecting point cloud data;

[0041] Figure 4 This is a schematic diagram of the compression projection module principle;

[0042] Figure 5 This is the overall structure diagram of the algorithm network of the present invention;

[0043] Figure 6 This is the semantic segmentation result diagram of this embodiment. DETAILED DESCRIPTION

[0044] Compared with image data, radar point cloud data provides three-dimensional coordinate information of the target and is less affected by weather and light and shadow changes. Even in harsh environments such as insufficient light or strong light, point cloud data can still provide stable and accurate geometric information. Combining the advantages of radar point cloud data and image data, taking advantage of their strengths and making up for their weaknesses, is an effective idea for semantic segmentation. The present invention proposes a semantic segmentation method based on radar data and image data, aiming to improve segmentation accuracy and system robustness.

[0045] The present invention proposes a new algorithm for semantic segmentation of aircraft based on radar-visual fusion and detail preservation. First, the radar point cloud data is preprocessed by denoising, completion, etc. to obtain point cloud data containing sufficient effective information and a high signal-to-noise ratio, and then transformed through a neural network to obtain a pseudo-depth map for interaction with image data. Next, the image data and point cloud data are combined to perform attention and other operations in a high-dimensional space to obtain features with rich semantic information after interaction. Finally, the features after interaction and the point cloud features are input into the decoding end together, so as to ensure local smoothness while making full use of the semantic information and obtain a better semantic segmentation effect. The technical solution of the present invention is further illustrated below with reference to the accompanying drawings.

[0046] The present invention provides a method for semantic segmentation of a target based on radar data and image data, the process is as follows: Figure 1 As shown in Figure 2, the overall network architecture is as follows: Figure 5 As shown. Specifically including the following steps:

[0047] Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained through the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. Since the original point cloud data often has problems such as sparseness, uneven density and noise, it cannot be used directly. Preprocessing is a necessary step. Therefore, the radar point cloud data is preprocessed as follows:

[0048] Step 1-1, denoising: Use filters, including but not limited to Gaussian filtering, mean filtering, etc. to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, eliminate isolated points and noise points, and improve data quality;

[0049] Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms to increase the density of the point cloud data, and use PointNet++ to complete it; PointNet++ uses layered sampling and local feature extraction to refine and complete the point cloud data. PointNet++ uses the U-Net structure, including an encoder and a decoder, such as Figure 5 As shown in the point cloud completion module in the lower left corner, the encoder uses a series of SetAbstraction (SA) modules to hierarchically abstract the features of the point cloud, while the decoder uses the same number of feature propagation to upsample the features.

[0050] Step 1-3, data registration: Point cloud data is often collected from different angles or different positions. Data registration is performed to ensure the consistency of point cloud data, such as Figure 2 The point cloud images are shown in Figure 1. The point cloud data from different perspectives are aligned using the iterative closest point algorithm. The distance between the closest point pairs of the two sets of point clouds is minimized and iterative optimization is performed to ensure the consistency of the point cloud data.

[0051] Step 2: Project the preprocessed point cloud data and convert it into a pseudo depth map. The point cloud data and image data come from different acquisition methods and cannot be directly spliced ​​or interacted. Because there are rich semantic segmentation research results in the image field, we consider converting the point cloud data into a certain image format and then introducing it into the image semantic segmentation framework. The depth map is a two-dimensional image that provides distance information between objects in the scene and the imaging device. The depth map can be easily obtained based on the projection transformation of the point cloud data, such as Figure 3 shown. Figure 3 for Figure 2 The depth map obtained from the two point cloud images shown. In the present invention, the pre-processed point cloud data is projected and converted into a pseudo depth map form so as to be fused with the image data. Specifically, the following sub-steps are included:

[0052] Step 2-1, determine the projection plane: select the projection plane according to the angle when the point cloud data is collected. Usually, a plane parallel to the radar sensor is selected for projection.

[0053] Step 2-2, coordinate conversion: convert the 3D point cloud data to the camera coordinate system; first, obtain the camera's external parameters (position and orientation), including the rotation matrix R and the translation vector T; then convert the point cloud data from the radar coordinate system to the camera coordinate system:

[0054] P c =R·P1+T

[0055] Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system;

[0056] Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map:

[0057] u=f x ·X / Z+c x

[0058] v=f y ·Y / Z+c y

[0059] Where (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y is the focal length parameter of the camera in the x and y axis directions, usually given in the intrinsic parameters of the camera; (c x ,c y ) is the camera principal point coordinate. Figure 4 As shown in Figure 2, the depth information of each projected point corresponds to the distance information in the original point cloud data.

[0060] Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data; the specific method is as follows:

[0061] Step 3-1, feature extraction: Use convolutional neural network to extract features of image data and pseudo depth map data respectively; the image data uses a general semantic segmentation backbone network as an encoder to extract features. Since the pseudo depth map data is already in the form of a depth map, there is no need to extract features through the same backbone network as the image data. It only needs to be transformed through a multi-layer perceptron (MLP) and then aligned with the features of the image data in a high-dimensional space.

[0062] Step 3-2, multimodal alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism; contrastive learning ensures the consistency of feature expression, such as Figure 5 As shown in the multimodal alignment module below; the cross-attention mechanism realizes the complementarity and interaction of features;

[0063] The cross attention formula is as follows:

[0064] Attention(Q,K,V)=softmax(Q·K T / sqrt(d k ))·V

[0065] Q is the query matrix, which is the processed point cloud feature; K is the key matrix, V is the value matrix, K and V are the features extracted from the image data; d k is the scale factor, which acts as a scaling factor to prevent the dot product value from being too large.

[0066] Step 3-3, feature fusion: The aligned image features and pseudo-depth map features are fused to form a multimodal feature representation to enhance the edge information brought by the radar point cloud data; after splicing the aligned image features and pseudo-depth map features, the multi-head self-attention mechanism is used for feature fusion. The formula is as follows:

[0067] Z=Attention(head1,…,head i ,…,head n )·W o

[0068] Where W o is the output weight matrix, head i =Attention(Q i ,K i ,V i ), i represents the i-th head in the multi-attention mechanism;

[0069] Step 4: Decode the interactive multimodal features and radar data features simultaneously to ensure the edge accuracy of the segmentation results. Input the fused multimodal features into the decoder network for step-by-step upsampling and decoding to generate semantic segmentation results. Select the Segformer decoder. The Segformer decoder is mainly composed of the Attention operation and the feedforward neural network. The Attention operation is as described above:

[0070] Attention(Q,K,V)=softmax(Q·K T / sqrt(d k ))·V

[0071] Q is the query matrix, K is the key matrix, and V is the value matrix.

[0072] Next, the features are transformed nonlinearly through the feed-forward neural network (FFN), and the formula is as follows:

[0073] FFN(X)=GELU(x·W1+b1)·W2+b2

[0074] W1 and W2 are weight matrices, b1 and b2 are biases; GELU is an activation function used to introduce nonlinear characteristics. In this process, skip connections and layer normalization (LN) are used to maintain the consistency and stability of features:

[0075] z l =LN(x l-1 +Attention(Q,K,V)

[0076] x l =LN(z l +FFN(z l ))

[0077] x is the feature map input to the Segformer decoder, x l Represents the feature maps at different layers of the decoder.

[0078] Through the processing of the multi-layer Transformer structure, the spatial resolution of the feature map is gradually upsampled and restored, and finally a high-resolution feature map is obtained. These feature maps contain multi-scale information and can help with accurate positioning and classification. The result of this method is shown in the figure Figure 6 As shown, Figure 6 The two pictures on the left are two airport scene images under the same circumstances, and the two pictures on the right are the corresponding semantic segmentation results.

[0079] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A target semantic segmentation method based on radar data and image data, characterized in that: The following steps are involved: Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained through the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. The radar point cloud data is preprocessed as follows: Step 1-1, denoising: Use a filter to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, and eliminate isolated points and noise points; Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms to increase the density of the point cloud data; use PointNet++ for completion; Step 1-3, data registration: Use the iterative closest point algorithm to align the point cloud data of different perspectives, and perform iterative optimization by minimizing the distance between the closest point pairs of the two sets of point clouds; Step 2: Project the preprocessed point cloud data and convert it into a pseudo depth map; specifically, the following sub-steps are included: Step 2-1, determine the projection plane: select the projection plane according to the angle when the point cloud data is collected; Step 2-2, coordinate conversion: convert the 3D point cloud data to the camera coordinate system; first, obtain the camera's external parameters including the rotation matrix R and the translation vector T; then convert the point cloud data from the radar coordinate system to the camera coordinate system: P c =R·P1+T Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system; Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map: u=f x ·X / Z+c x v=f y ·Y / Z+c y Where (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y is the focal length parameter of the camera in the x and y axis directions, (c x ,c y ) is the camera principal point coordinate; Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data; the specific method is as follows: Step 3-1, feature extraction: use convolutional neural network to extract features of image data and pseudo depth map data respectively; Step 3-2, multimodal alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism; contrastive learning ensures the consistency of feature expression, and cross-attention mechanism realizes the complementarity and interaction of features; The cross attention formula is as follows: Attention(Q,K,V)=softmax(QK T / sqrt(d k ))·V Q is the query matrix, which is the processed point cloud feature; K is the key matrix, V is the value matrix, K and V are the features extracted from the image data; d k is the scale factor; Step 3-3, feature fusion: The aligned image features and pseudo depth map features are fused to form a multimodal feature representation; after splicing the aligned image features and pseudo depth map features, the multi-head self-attention mechanism is used for feature fusion. The formula is as follows: Z=Attention(head1,…,head i ,…,head n )·W o Where W o is the output weight matrix, head i =Attention(Q i ,K i ,V i ); Step 4: Input the fused multimodal features into the decoder network to generate semantic segmentation results.

Images