Target semantic segmentation method based on radar data and image data

Abstract

The application discloses a target semantic segmentation method based on radar data and image data, comprising the following steps: step 1, pre-processing of radar point cloud data: obtaining radar point cloud data through a radar sensor, and pre-processing the radar point cloud data; step 2, projecting the pre-processed point cloud data into a pseudo-depth map form; specifically comprising the following sub-steps: step 2-1, determining a projection plane; step 2-2, performing coordinate conversion; step 2-3, projection calculation; step 3, feature interaction: performing feature interaction on the radar data in the pseudo-depth map form and input image data; step 4, inputting the fused multi-modal features into a decoder network to generate a semantic segmentation result. The application combines the advantages of radar point cloud data and image data, and proposes a semantic segmentation method which can still maintain high precision in low light, strong light, rain and fog and other harsh environments.

Description

Technical Field

[0001] The present invention relates to a target semantic segmentation method based on radar data and image data, which is used for monitoring and identifying aircraft and ground equipment in an airport. Background Art

[0002] Accurately monitoring and identifying aircraft and ground equipment within an airport is crucial for modern airport management. Traditional monitoring systems rely primarily on image data, but image data often performs poorly in adverse weather conditions such as low light, strong sunlight, rain, and fog, easily leading to misidentification and missed identification. In contrast, radar data offers the advantages of strong anti-interference capabilities and all-weather operation, but is relatively inadequate for recognizing fine structures and textures of targets. Therefore, multimodal semantic segmentation methods that combine radar and image data have emerged to address the shortcomings of a single data source and provide more accurate and robust environmental perception capabilities.

[0003] At present, airport image segmentation technology has made certain progress. In particular, driven by deep learning technology, semantic segmentation models such as UNet, SegNet, and DeepLab have performed well in the identification of aircraft and ground equipment. Semantic segmentation technology achieves a refined understanding of the image by classifying each pixel in the image into different semantic categories. In the airport environment, these technologies can be used to identify and locate various aircraft, vehicles, and equipment. However, the complex airport environment and the variety of aircraft types pose challenges to the segmentation task. There may be occlusion and overlap between aircraft and ground equipment, coupled with changes in weather and lighting conditions, making it difficult for segmentation methods that rely solely on image data to maintain high accuracy in all cases. In addition, airport operations require real-time monitoring and processing, which places higher demands on the speed and accuracy of segmentation methods.

[0004] The main problems with existing semantic segmentation algorithms for aircraft in airport scenarios are as follows:

[0005] 1) Image data performs poorly in harsh environments: Image data is easily disturbed in low light, strong light, rain, fog, and other conditions, leading to misidentification and missed recognition. This is particularly prominent in complex environments such as airports, requiring a segmentation method that can operate stably under various lighting and weather conditions.

[0006] 2) Sparsity and noise issues with point cloud data: Although radar point cloud data provides three-dimensional spatial information, its sparseness, uneven density, and noise limit its ability to identify detailed information. Effectively processing and utilizing this data is a significant technical challenge.

[0007] 3) Alignment and Fusion of Multimodal Data: Image data and point cloud data originate from different sensors, with varying spatial resolutions and information types. Accurately aligning and effectively fusing these two types of data in the same coordinate system to improve semantic segmentation performance is a pressing challenge.

[0008] 4) Detail Preservation and Edge Restoration: In the process of multimodal data fusion, how to preserve the fine structure and edge information of the image and accurately reflect these details in the final semantic segmentation results is a technical difficulty. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a target semantic segmentation method based on radar data and image data, which can maintain high-precision semantic segmentation in harsh environments such as low light, strong light, rain and fog by combining the advantages of radar point cloud data and image data.

[0010] The object of the present invention is achieved through the following technical solution: a target semantic segmentation method based on radar data and image data, comprising the following steps:

[0011] Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained through the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. The radar point cloud data is preprocessed as follows:

[0012] Step 1-1, denoising: Use a filter to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, and eliminate isolated points and noise points;

[0013] Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms to increase the density of the point cloud data, and use PointNet++ for completion;

[0014] Steps 1-3, data registration: Use the iterative closest point algorithm to align point cloud data from different perspectives. By minimizing the distance between the closest point pairs of the two sets of point clouds, iterative optimization is performed to ensure the consistency of the point cloud data.

[0015] Step 2: Project the pre-processed point cloud data and convert it into a pseudo depth map. This includes the following sub-steps:

[0016] Step 2-1, determine the projection plane: select the projection plane according to the angle when the point cloud data is collected;

[0017] Step 2-2, coordinate conversion: convert the 3D point cloud data to the camera coordinate system; first, obtain the camera's external parameters including the rotation matrix R and the translation vector T; then convert the point cloud data from the radar coordinate system to the camera coordinate system:

[0018] P c =R·P1+T

[0019] Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system;

[0020] Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map:

[0021] u=f x ·X / Z+c x

[0022] v=f y ·Y / Z+c y

[0023] Among them, (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y is the focal length parameter of the camera in the x and y axis directions, (c x ,c y ) is the camera principal point coordinate;

[0024] Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data. The specific method is as follows:

[0025] Step 3-1, feature extraction: Use convolutional neural network to extract features of image data and pseudo depth map data respectively;

[0026] Step 3-2, Multimodal Alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism. Contrastive learning ensures the consistency of feature expression, and the cross-attention mechanism realizes the complementarity and interaction of features. The cross-attention formula is as follows:

[0027] Attention(Q,K,V)=softmax(Q·K T / sqrt(d k ))·V

[0028] Q is the query matrix, which is the feature of the processed point cloud; K is the key matrix, V is the value matrix, K and V are the features extracted from the image data; d k is the scale factor;

[0029] Step 3-3, feature fusion: Fuse the aligned image features and pseudo depth map features to form a multimodal feature representation; after splicing the aligned image features and pseudo depth map features, use the multi-head self-attention mechanism for feature fusion. The formula is as follows:

[0030] Z = Attention(head1, …, head i ,…,head n )·W o

[0031] where W o is the output weight matrix, head i = Attention(Q i ,K i ,V i );

[0032] Step 4, input the fused multi-modal features into the decoder network to generate the semantic segmentation result.

[0033] The beneficial effects of the present application are:

[0034] 1) Improve segmentation accuracy in harsh environments: The present application aims to combine the advantages of radar point cloud data and image data, and proposes a semantic segmentation method that can still maintain high accuracy in low light, strong light, rain and fog, etc. Harsh environment.

[0035] 2) Solve the problem of sparse and noisy point cloud data: Through the point cloud completion module, the radar point cloud data is denoised and interpolated, improving the density and quality of the point cloud data, thereby enhancing its ability to identify details.

[0036] 3) Realize accurate alignment and fusion of multi-modal data: Through the compression projection module and multi-modal alignment module, use contrast learning and cross attention mechanism to realize effective alignment and fusion of point cloud data and image data, fully exert the complementary advantages of the two kinds of data, and improve the overall segmentation performance.

[0037] 4) Enhance detail preservation and edge recovery: After multi-modal data fusion, introduce the edge preservation and recovery module, use the rich edge information of image data to ensure that the generated semantic segmentation mask has high-precision details and edges, thereby improving the accuracy and reliability of the segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart for aircraft semantic segmentation based on radar data and image data;

[0039] Figure 2 is the projection of the original radar data at different angles;

[0040] Figure 3 is a schematic diagram of the depth map obtained from the point cloud data projection;

[0041] Figure 4 is a schematic diagram of the compression projection module principle;

[0042] Figure 5 This is the overall structure diagram of the algorithm network of the present invention;

[0043] Figure 6 This is the semantic segmentation result diagram of this embodiment. DETAILED DESCRIPTION

[0044] Compared to image data, radar point cloud data provides three-dimensional coordinate information of targets and is less affected by weather and lighting variations. Even in harsh environments such as low or high light levels, point cloud data can still provide stable and accurate geometric information. Combining the advantages of radar point cloud data and image data, leveraging their strengths to overcome their weaknesses, is an effective approach for semantic segmentation. This paper proposes a semantic segmentation method based on radar and image data, aiming to improve segmentation accuracy and system robustness.

[0045] The present invention proposes a new algorithm for semantic segmentation of aircraft based on radar-visual fusion and detail preservation. First, the radar point cloud data is pre-processed by denoising, completion, etc. to obtain point cloud data containing sufficient effective information and a high signal-to-noise ratio, and then transformed through a neural network to obtain a pseudo-depth map for interaction with image data. Next, the image data and point cloud data are combined to perform attention and other operations in a high-dimensional space to obtain features with rich semantic information after interaction. Finally, the features after interaction and the point cloud features are input into the decoding end together, ensuring local smoothness while making full use of the semantic information, and obtaining a better semantic segmentation effect. The technical solution of the present invention is further illustrated below with reference to the accompanying drawings.

[0046] The present invention provides a target semantic segmentation method based on radar data and image data, the process is as follows Figure 1 As shown, the overall network architecture is as follows Figure 5 The specific steps include:

[0047] Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained by the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. Since the original point cloud data is often sparse, has uneven density and noise problems, it cannot be used directly. Preprocessing is a necessary step. Therefore, the radar point cloud data is preprocessed as follows:

[0048] Step 1-1, denoising: Use filters, including but not limited to Gaussian filtering and mean filtering, to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, eliminate isolated points and noise points, and improve data quality;

[0049] Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms to increase the density of the point cloud data, and use PointNet++ for completion; PointNet++ uses layered sampling and local feature extraction to refine and complete the point cloud data. PointNet++ uses the U-Net structure, including encoder and decoder, such as Figure 5 As shown in the point cloud completion module in the lower left corner, the encoder uses a series of SetAbstraction (SA) modules to hierarchically abstract the features of the point cloud, while the decoder uses the same number of feature propagation to upsample the features.

[0050] Steps 1-3, data registration: Point cloud data is often collected from different angles or positions, and data registration is performed to ensure the consistency of point cloud data, such as Figure 2 The following figure shows point cloud images viewed from different angles. An iterative closest point algorithm is used to align point cloud data from different perspectives. This algorithm iteratively optimizes the point cloud data by minimizing the distance between the closest point pairs in each point cloud, ensuring consistency.

[0051] Step 2: Project the pre-processed point cloud data and convert it into a pseudo depth map. Point cloud data and image data come from different acquisition methods and cannot be directly spliced ​​or interacted. Because there are rich semantic segmentation research results in the image field, we consider converting the point cloud data into a certain image format and then introducing it into the image semantic segmentation framework. The depth map is a two-dimensional image that provides distance information between objects in the scene and the imaging device. It is easy to obtain a depth map based on the projection transformation of point cloud data, such as Figure 3 shown. Figure 3 for Figure 2 The depth map obtained from the two point cloud images shown. In the present invention, the pre-processed point cloud data is projected and converted into a pseudo depth map form so as to be fused with the image data. Specifically, it includes the following sub-steps:

[0052] Step 2-1. Determine the projection plane: Select the projection plane based on the angle at which the point cloud data is collected. Usually, a plane parallel to the radar sensor is selected for projection.

[0053] Step 2-2, coordinate conversion: Convert the 3D point cloud data to the camera coordinate system. First, obtain the camera's external parameters (position and orientation), including the rotation matrix R and translation vector T. Then convert the point cloud data from the radar coordinate system to the camera coordinate system:

[0054] P c =R·P1+T

[0055] Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system;

[0056] Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map:

[0057] u=f x ·X / Z+c x

[0058] v=f y ·Y / Z+c y

[0059] Among them, (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y It is the focal length parameter of the camera in the x and y axis directions, which is usually given in the intrinsic parameters of the camera; (c x ,c y ) is the coordinate of the camera principal point. Figure 4 As shown in Figure 2, the depth information of each projected point corresponds to the distance information in the original point cloud data.

[0060] Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data. The specific method is as follows:

[0061] Step 3-1, Feature Extraction: Use a convolutional neural network to extract features of image data and pseudo-depth map data respectively; the image data uses a general semantic segmentation backbone network as an encoder for feature extraction. Since the pseudo-depth map data is already in the form of a depth map, it does not need to be extracted through the same backbone network as the image data. It only needs to be transformed through a multi-layer perceptron (MLP) and aligned with the features of the image data in a high-dimensional space.

[0062] Step 3-2, multimodal alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism; contrastive learning ensures the consistency of feature expression, such as Figure 5 As shown in the multimodal alignment module below; the cross-attention mechanism realizes the complementarity and interaction of features;

[0063] The cross attention formula is as follows:

[0064] Attention(Q,K,V)=softmax(Q·K T / sqrt(d k ))·V

[0065] Q is a query matrix, K is a key matrix, V is a value matrix, K and V are features extracted from image data; d k is a scale factor, which plays a scaling role to prevent the point product value from being too large.

[0066] Step 3-3, feature fusion: fuse the aligned image features and pseudo depth map features to form a multi-modal feature representation to enhance the edge information brought by the radar point cloud data; after splicing the aligned image features and pseudo depth map features, use multi-head self-attention mechanism for feature fusion, the formula is as follows:

[0067] Z = Attention (head1, …, head i , …, head n )·W o

[0068] Where W o is the output weight matrix, head i = Attention (Q i , K i , V i ), i represents the i-th head in the multi-attention mechanism;

[0069] Step 4, decode the multi-modal features after interaction and radar data features at the same time to ensure the edge accuracy of the segmentation result. The fused multi-modal features are input into the decoder network for step-by-step upsampling and decoding to generate the semantic segmentation result. The Segformer decoder is selected. The Segformer decoder is mainly composed of Attention operation and feed-forward neural network, and the Attention operation is as follows:

[0070] Attention (Q, K, V) = softmax (Q·K T / sqrt (d k ))·V

[0071] Q is a query matrix, K is a key matrix, V is a value matrix.

[0072] Then, the feature is nonlinearly transformed by the feed-forward neural network (FFN), and the formula is as follows:

[0073] FFN (X) = GELU (x·W1+b1)·W2+b2

[0074] W1 and W2 are weight matrices, b1 and b2 are biases; GELU is an activation function used to introduce nonlinear characteristics. In this process, skip connections and layer normalization (LN) are used to maintain feature consistency and stability:

[0075] z l =LN(x l-1 +Attention(Q,K,V)

[0076] x l =LN(z l +FFN(z l ))

[0077] x is the feature map input to the Segformer decoder, x l Represents the feature maps at different layers of the decoder.

[0078] Through the processing of the multi-layer Transformer structure, the spatial resolution of the feature map is gradually restored by upsampling, and finally a high-resolution feature map is obtained. These feature maps contain multi-scale information, which can help with accurate positioning and classification. The result of this method is shown in the following figure. Figure 6 As shown, Figure 6 The two pictures on the left are two airport scene images under the same circumstances, and the two pictures on the right are the corresponding semantic segmentation results.

[0079] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A target semantic segmentation method based on radar data and image data, characterized in that: The following steps are involved: Step 1: Preprocessing of radar point cloud data: The radar point cloud data is obtained through the radar sensor, which contains the three-dimensional coordinate information of objects in the environment. The radar point cloud data is preprocessed as follows: Step 1-1, denoising: Use a filter to denoise the point cloud data, perform weighted averaging on the points in the neighborhood of each point, and eliminate isolated points and noise points; Step 1-2, density completion: fill the sparse areas in the point cloud data through interpolation and reconstruction algorithms to increase the density of the point cloud data; use PointNet++ for completion; Steps 1-3, data registration: Use the iterative closest point algorithm to align point cloud data from different perspectives, and perform iterative optimization by minimizing the distance between the closest point pairs of the two sets of point clouds; Step 2: Project the pre-processed point cloud data and convert it into a pseudo depth map. This includes the following sub-steps: Step 2-1, determine the projection plane: select the projection plane according to the angle when the point cloud data is collected; Step 2-2, coordinate conversion: convert the 3D point cloud data to the camera coordinate system; first, obtain the camera's external parameters including the rotation matrix R and the translation vector T; then convert the point cloud data from the radar coordinate system to the camera coordinate system: P c =R·P1+T Among them, P c is a point in the camera coordinate system, and P1 is a point in the radar coordinate system; Step 2-3, projection calculation: Use the camera's intrinsic parameters to project the 3D point cloud data onto a 2D plane to generate a pseudo depth map: u=f x ·X / Z+c x v=f y ·Y / Z+c y Among them, (u, v) is the pixel coordinate on the image plane, (X, Y, Z) is the three-dimensional point coordinate in the camera coordinate system; f x 、f y is the focal length parameter of the camera in the x and y axis directions, (c x ,c y ) is the camera principal point coordinate; Step 3: Feature interaction: Use a camera to capture the scene in the same direction as the radar point cloud data to obtain image data; perform feature interaction between the radar data in the form of a pseudo-depth map and the input image data. The specific method is as follows: Step 3-1, feature extraction: Use convolutional neural network to extract features of image data and pseudo depth map data respectively; Step 3-2, Multimodal Alignment: Align the data features of two different modalities through contrastive learning and cross-attention mechanism. Contrastive learning ensures the consistency of feature expression, and the cross-attention mechanism realizes the complementarity and interaction of features. The cross attention formula is as follows: Attention(Q,K,V)=softmax(QK T / sqrt(d k ))·V Q is the query matrix, which is the feature of the processed point cloud; K is the key matrix, V is the value matrix, K and V are the features extracted from the image data; d k is the scale factor; Step 3-3, feature fusion: Fuse the aligned image features and pseudo depth map features to form a multimodal feature representation; after splicing the aligned image features and pseudo depth map features, use the multi-head self-attention mechanism for feature fusion. The formula is as follows: Z=Attention(head1,…,head i ,…,head n )·W o Where W o is the output weight matrix, head i =Attention(Q i ,K i ,V i ); Step 4: Input the fused multimodal features into the decoder network to generate semantic segmentation results.

Images