A hyperspectral-based automatic driving scene segmentation method

Abstract

The application discloses an automatic driving scene segmentation method based on hyperspectrum. The method comprises the following steps: S1, simultaneously collecting an RGB image and a hyperspectrum image, ensuring that the RGB image and the hyperspectrum image are synchronous in time and registered in space; S2, respectively processing the RGB image and the hyperspectrum image, performing semantic segmentation on the RGB image, and performing feature dimension reduction and material estimation on the hyperspectrum image; and S3, on the basis of the semantic segmentation result of the RGB image, using the material estimation result of the hyperspectrum image to optimize the semantic segmentation result of the RGB image, so as to obtain a more accurate and fine scene segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to a hyperspectral-based automatic driving scene segmentation method. BACKGROUND

[0002] In recent years, with the development of computer vision and machine learning technology, the automatic driving scene understanding algorithm based on RGB camera is gradually mature. However, the RGB camera cannot distinguish the same color different spectrum objects, for example, the same green leaves and cars. Although technologies such as deep learning provide the possibility to distinguish the same color different objects, but still cannot break through the physical limit. Therefore, the introduction of hyperspectral information will bring a new breakthrough to the automatic driving technology.

[0003] The current common hyperspectral acquisition methods are mainly divided into two categories, push-broom and snapshot. The push-broom scans wave by wave in the spectral dimension or line by line in the spatial dimension, while the snapshot can obtain the spectral information of the scene in one imaging. In the automatic driving scene, real-time feedback of the road condition is required, and the speed of the push-broom is not enough to meet the demand.

[0004] The existing snapshot spectral camera for automatic driving is mainly based on a mosaic array, which generally contains only 16 to 25 wave bands. The snapshot hyperspectral camera based on computational photography can obtain high spatial and high spectral resolution video signals (such as CN105338326A), and the spectral information contains 128 wave bands, while meeting the imaging speed required by automatic driving. Compared with the push-broom hyperspectral camera and the mosaic spectral camera, the snapshot hyperspectral camera based on computational photography has the advantages of both, and best meets the needs of automatic driving.

[0005] However, the huge amount of data brought by high spatial and high spectral resolution video information brings new challenges to data processing. Traditional image segmentation algorithms are generally based on threshold, region, edge detection and probability model, while the development of neural network greatly improves the accuracy of image segmentation algorithm. Convolutional neural network (CNN) provides powerful feature extraction capability, but using deep neural network for semantic segmentation of hyperspectral data often means a huge model, which is limited by the limited computing power of the mobile terminal of the car, and it is also difficult to keep up with the demand of automatic driving in terms of computing speed. Therefore, how to reduce the amount of calculation without reducing the segmentation accuracy, and how to make full use of hyperspectral data, has become a problem to be solved. SUMMARY

[0006] In view of the defects of the prior art, the purpose of the present application is to provide a hyperspectral-based automatic driving scene segmentation method.

[0007] The technical scheme adopted by the present application is:

[0008] A hyperspectral-based automatic driving scene segmentation method, comprising the following steps:

[0009] S1, simultaneously collecting the RGB image and the hyperspectral image, ensuring that the RGB image and the hyperspectral image are time-synchronized and spatially registered;

[0010] S2, processing the RGB image and the hyperspectral image respectively, performing semantic segmentation on the RGB image, and performing feature dimension reduction and material estimation on the hyperspectral image;

[0011] S3, based on the semantic segmentation result of the RGB image, using the material estimation result of the hyperspectral image to optimize the semantic segmentation result of the RGB image, to obtain a more accurate and fine scene segmentation result.

[0012] Further, in step S2, the RGB image is subjected to semantic segmentation, specifically: a lightweight RGB image segmentation model is used to quickly extract spatial semantic features, and regions with clear structure and texture features are segmented.

[0013] Further, the image segmentation model uses a full convolutional network.

[0014] Further, in step S2, the hyperspectral image is subjected to feature dimension reduction and material estimation, specifically: first input the hyperspectral image into a dimension reduction model based on manifold learning, then extract a certain number of spectral blocks of a certain size from the dimension-reduced spectral image, and input them into a material estimation model for material estimation.

[0015] Further, the material estimation model uses a dual-stream convolutional neural network.

[0016] Further, in step S3, the material estimation of the target whose confidence of semantic segmentation prediction is lower than the threshold is performed, the result of semantic segmentation is corrected with a certain weight, so as to re-evaluate the category of the target in the scene.

[0017] Further, in step S3, the result of semantic segmentation is corrected with a certain weight, specifically: when the material estimation result and the semantic segmentation result are contrary, the respective weight is multiplied by the larger one of the respective confidence to obtain the final output result of semantic segmentation.

[0018] The present application fuses the advantages of RGB image in spatial feature extraction and the advantages of spectral image in spectral feature extraction, adopts a multi-modal idea, and proposes a lightweight urban road scene segmentation method assisted by material estimation. In image acquisition, RGB images and hyperspectral images are collected simultaneously, and the hyperspectral data is compressed in dimension for the limited storage of mobile terminals. In algorithm processing, on the one hand, a lightweight RGB image segmentation model is used to quickly extract spatial semantic features and segment regions with clear structure and texture features; on the other hand, specific material prior information is used, and the feature dimension reduction and classification of the hyperspectral image are used to realize high-precision material estimation, which is used as an auxiliary constraint for semantic segmentation, to refine and correct the poor results of RBG segmentation, so as to achieve higher segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of the method of the present application is shown.

[0020] Figure 2 A schematic diagram of a vehicle-mounted multi-sensor platform in an embodiment of the present application is shown.

[0021] Figure 3 A schematic diagram of distinguishing targets in an embodiment of the present application is shown, (a) is two points with the same color in an RGB image, and (b) is the spectral curve of the two points.

[0022] Figure 4 A material estimation map predicted by the HSI cube-based method is shown, (a) is an RGB image, (b) is a manually annotated map, (c) is an SVM model result, (d) is a CNN-HSI model result, (e) is a HybirdSN model result, and (f) is a two-flow convolutional neural network Two-CNN model result.

[0023] Figure 5 An effect diagram of the method of the present application is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0025] 1. Dual-channel image acquisition

[0026] This embodiment first provides a vehicle-mounted multi-sensor platform scheme, as shown in Figure 2The system is shown in FIG. 1. It includes a hyperspectral camera 1, a solid-state laser radar 3 (ZVISION ML-30s), and a fisheye camera 4. The hyperspectral camera 1 can capture 1889x1422 spatial resolution, 20fps hyperspectral video covering 128 spectral bands from 450nm to 950nm, with a field of view of 16°x12°, and is placed horizontally. The hyperspectral camera 1 also captures RGB images, which are synchronized in time and registered in space with the hyperspectral images. Of course, other cameras can also be used to capture RGB images. In addition, a diffuse white board 2 is included in the field of view of the hyperspectral camera 1 to record the spectral information of ambient light. The information from the laser radar 3 and the fisheye camera 4 is not currently being used.

[0027] Although deep learning has better performance than traditional algorithms, it is mostly dependent on data sets. Although the earliest urban scene data set was published in 1999, it was not widely studied due to its small data volume and limited computer power at the time. The large-scale data set CityScapes published in 2015 has promoted further research on urban scene understanding. Unlike previous urban data sets, this embodiment includes 1330 hyperspectral images with a spatial resolution of 1889x1422, covering 128 spectral bands from 450nm to 950nm, and aims to promote the application of hyperspectral in the field of autonomous driving. Unlike the approximately several MB of RGB images, for each approximately 1GB size of hyperspectral image, feature extraction and dimensionality reduction can effectively remove redundant information in hyperspectral data, and principal component analysis can reduce 128 spectral channels to 13 channels, while retaining 99% of the effective information and compressing nearly 10 times, providing convenience for the transmission and processing of hyperspectral information.

[0028] 2. Dual-path image processing

[0029] The RGB images and hyperspectral images captured by the image acquisition system will be processed in two paths in the algorithm. The RGB images will be trained in a lightweight semantic segmentation model, and the hyperspectral images will be trained in a manifold learning dimensionality reduction and material estimation model. Since the two are aligned, they can share the manually labeled segmentation labels. The details of the model are discussed as follows:

[0030] 2.1. Lightweight semantic segmentation based on RGB images

[0031] With the development of deep learning in the field of image, the research of hyperspectral image (HSI) classification task also moves from traditional image processing methods to deep learning methods. Many works focus on carefully designing various spectral-spatial networks, among which convolutional neural network (CNN) is one of the most popular structures. To explore the spatial information of HSI classification, in the CNN-based methods, the pixels with adjacent pixels are usually cropped directly from the hyperspectral data to form HSI cubes. However, the spatial land cover distribution of the cropped HSI cubes is usually complex. The land cover label of the cropped HSI cubes cannot be simply determined by the center pixel. In addition, the spatial land cover distribution of the cropped HSI cubes is fixed and less diverse. For the CNN-based methods, training with the cropped HSI cubes will lead to poor generalization to the changes of spatial land cover distribution. Furthermore, the attention of the HSI cubes is focused on the local of the image, lacking global information, although the transformer-based methods have recently also been used for HSI classification (He X, Chen Y, Lin Z. Spatial-spectral transformer for hyperspectral image classification [J]. Remote Sensing, 2021, 13(3): 498), to some extent, the problem of lack of global information is solved, but due to the large amount of hyperspectral information, the transformer model itself is larger than the CNN model, which will be more difficult to deploy in practical application.

[0032] In this embodiment, the RGB image is segmented by using a lightweight semantic segmentation network (such as Fan M, Lai S, Huang J, et al. Rethinking BiSeNet for real-time semantic segmentation [C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9716-9725.), and the predicted effect as shown in FIG. 4 can be obtained. Figure 5

[0033] 2.2. Hyperspectral nonlinear manifold dimensionality reduction for specific materials

[0034] ​Different from linear dimensionality reduction methods such as principal component analysis (PCA) and classical multidimensional scaling (CMDS), manifold learning, as a representative of nonlinear dimensionality reduction methods, has fewer assumptions about the model and is more general. The main idea of manifold learning can be summarized as follows: it is assumed that high-dimensional data can be represented by a low-dimensional manifold, and therefore the geometric structure of the low-dimensional manifold can be learned from the neighborhood structure of the high-dimensional data. The specific approach is to create a kernel to represent the learned manifold geometry, and use the spectral decomposition of the kernel to embed the low-dimensional manifold into the high-dimensional data (Wang J. Geometric structure of high-dimensional data and dimensionality reduction [M]. Heidelberg: Springer, 2012).

[0035] Existing graph learning algorithms usually use a distance-based weight matrix to reflect the similarity relationship between sample pairs and construct a graph structure to map high-dimensional data into a low-dimensional space after dimensionality reduction. However, due to various factors, subtle differences in spectral curves often lead to inaccurate weight matrices. Therefore, according to the collaborative representation model, the intra-class compactness and inter-class separability can be enhanced by constructing intra-class and inter-class collaborative graphs, and the local neighborhood structure of HSI can be maintained by constructing a local neighborhood graph. An optimal objective function is designed to solve the discriminant projection matrix, and finally the discriminant features of different material types are obtained.

[0036] Compared with unsupervised dimensionality reduction methods, supervised algorithms can fully explore the spectral-location-spatial multi-dimensional information and better reflect the complex internal structure in HSI. In particular, in the task described in this embodiment, by learning the semantic information classification in the RGB image, different semantic targets with the same color can be separated, such as the green leaves and cars in Figure 3 Then, the intra-class and inter-class collaborative method described above is used to learn the discriminant features of the intra-class and inter-class, and the spectral information is introduced at the same time of dimensionality reduction, further improving the accuracy of material classification.

[0037] In this embodiment, 19 materials are included in the urban road scene, which are road, sidewalk, building, wall, fence, power pole, traffic light, traffic sign, vegetation, terrain, sky, pedestrian, cyclist, car, truck, bus, train, motorcycle, and bicycle. The hyperspectral camera 1 used in this embodiment can collect a spatial-spectral manifold in the spatial dimension, and the dimensionality reduction for materials can further compress the spatial-spectral manifold in the spectral dimension to obtain a spatial-spectral feature manifold that is more suitable for lightweight use.

[0038] As Figure 4As shown, the HSI cube-based material estimation method is affected by complex external conditions, and the predicted segmentation map target edge is not smooth and has defects inside; while the semantic segmentation-based method has the effect as Figure 5 As shown, there is a visible improvement with the naked eye. However, in the scene in the first row in Figure 5 , the upper semicircular structure of the roadside railing is incorrectly predicted as a pedestrian because the head of the pedestrian is similar to the semicircular railing; in the scene in the second row in Figure 5 , there is also an error of identifying the flowers of the small herbaceous plants in the flower bed as terrain because the plants in the data set are mostly trees or shrubs. The semantic segmentation-based method has the above-mentioned shortcomings, and the introduction of the spectrum can solve it, as shown in Figure 3 , the green plants and the green car have obviously different spectral characteristics, which is a supplement to the RGB image.

[0039] 3. Dual-path algorithm fusion

[0040] Unlike the idea of directly performing 3D convolution on the spectral image, the multi-modal idea of the present application separates the RGB content and the spectral content in the hyperspectral image, fully utilizes the existing lightweight image segmentation algorithm based on RGB, and then introduces the hyperspectral material estimation result for assistance, thereby effectively improving the accuracy of the autonomous driving scene segmentation.

[0041] Specifically, based on the semantic segmentation result of the RGB image, according to the confidence of the prediction of the model (reflecting the probability of the predicted target class to each class of the given label), the target with a confidence lower than a certain threshold is input into the hyperspectral material estimation model for re-evaluation, and the result of the semantic segmentation is corrected with a certain weight (when the results given by the dual-path algorithm are inconsistent, the larger one is selected as the final result by multiplying the respective confidence with the respective weight). The above threshold and weight are obtained by training the overall model (semantic segmentation, material estimation, threshold and weight). The input of the hyperspectral material estimation model is the HSI cube around the target, and the output is the redefined target boundary and the class. However, direct use of the HSI cube will result in a non-smooth boundary, because the hyperspectrum is affected by complex environments, and even the same material will have a huge difference in the hyperspectral space due to the influence of different angles of outdoor natural light and shadows. Therefore, effective hyperspectral dimension reduction not only reduces the model calculation amount, but also improves the robustness of the spectrum to the redundancy in the hyperspectrum.

[0042] Specific algorithm flow, in actual application, after processing RGB image and hyperspectral image in two ways, the prediction result of semantic segmentation will be input into the hyperspectral material estimation model for correction to obtain the scene segmentation result; and in training, in addition to the end-to-end (End-to-End) training of the overall model, the above two ways will be independently trained as the pre-training (Pre training) model of the overall model. The two-way algorithm is trained according to the semantic segmentation graph and the material estimation graph respectively to ensure the effect of strong supervision learning. When training the overall model, three loss functions are set, respectively corresponding to semantic segmentation, material estimation and scene segmentation.

[0043] In the embodiment, SVM, CNN-HSI, HybirdSN, Two-CNN, and a plurality of segmentation network architectures such as FCN50, FCN101, CCNet, PSPNet and HRNet are compared, wherein the material estimation model Two-CNN and the segmentation network FCN101 respectively show better effects.

Claims

1. A hyperspectral-based autonomous driving scene segmentation method, characterized in that, The method includes the following steps: S1 simultaneously acquires RGB and hyperspectral images, ensuring that the RGB and hyperspectral images are synchronized in time and registered in space. S2, process the RGB image and hyperspectral image respectively, perform semantic segmentation on the RGB image, and perform feature dimensionality reduction and material estimation on the hyperspectral image; the semantic segmentation of the RGB image specifically involves: using a lightweight RGB image segmentation model to quickly extract spatial semantic features, and segmenting regions with relatively clear structural and texture features; the feature dimensionality reduction and material estimation of the hyperspectral image specifically involves: first inputting the hyperspectral image into a manifold learning-based dimensionality reduction model, then extracting several spectral blocks of a certain size from the dimensionality-reduced spectral image, and inputting them into a material estimation model for material estimation; S3, based on the semantic segmentation result of the RGB image, optimize the semantic segmentation result of the RGB image using the material estimation result of the hyperspectral image to obtain a more accurate and refined scene segmentation result; specifically, this includes: performing hyperspectral image material estimation on targets whose confidence level in semantic segmentation prediction is lower than a threshold, correcting the semantic segmentation result with a certain weight, thereby re-evaluating the category of the target in the scene.

2. The method for autonomous driving scene segmentation based on hyperspectral imaging according to claim 1, characterized in that, The image segmentation model employs a fully convolutional network.

3. The method for autonomous driving scene segmentation based on hyperspectral imaging according to claim 1, characterized in that, The material estimation model employs a two-stream convolutional neural network.

4. The method for autonomous driving scene segmentation based on hyperspectral imaging according to claim 1, characterized in that, In step S3, the semantic segmentation result is corrected with certain weights. Specifically, when the material estimation result contradicts the semantic segmentation result, the result with the larger weight multiplied by the larger confidence level is selected as the final output result of the semantic segmentation.

Images