Urban scene space base automatic extraction method based on vehicle-mounted multi-modal data

Abstract

The invention provides an urban scene space base automatic extraction method based on vehicle-mounted multi-modal data, which comprises the following steps: acquiring the vehicle-mounted multi-modal data including multi-spectral image data and dual-band laser point cloud data, preprocessing the vehicle-mounted multi-modal data, and extracting the vehicle-mounted multi-modal data to obtain the vehicle-mounted multi-modal data; and urban scene space base extraction is carried out based on the preprocessed vehicle-mounted multi-modal data. According to the method, the urban scene space base is automatically extracted based on the vehicle-mounted multi-modal data, high-robustness feature element extraction and descriptor construction in the dual-band laser point cloud and the multi-spectral image are realized, the expression and processing requirements of an automatic driving system on a large-scale dynamic scene can be met, and the method is convenient to use.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method for automatically extracting urban scene space bases based on vehicle-mounted multimodal data. Background Art

[0002] The rapid development of intelligent driving technology has greatly promoted the research of various environmental perception sensors. Vehicle-mounted laser radar (LiDAR) has an important position in the intelligent driving environment perception system because of its advantages such as accurate acquisition of three-dimensional information of the target, high resolution, strong anti-interference ability, wide detection range, and near all-weather operation. The specific scenario of intelligent driving places high demands on the performance of both the hardware technology and the application algorithms of vehicle-mounted LiDAR. For example, the laser safety level of vehicle-mounted LiDAR used in intelligent driving systems above Level 3 (the intelligent driving automation level proposed by SAE (Society of Automotive Engineers), with a total of 6 levels from Level 0 to 5) should be Level 1 (power less than 0.4mW, low-output laser) to ensure the safety of pedestrians during the driving of intelligent driving cars; the detection distance should be at least 100m, the measurement accuracy should be at the centimeter level, and the scanning frequency should be at least 5Hz to 20Hz to meet the requirements of high-speed driving of intelligent driving cars; and the transmission rate of LiDAR point cloud data, the running speed, accuracy and processing effect of the point cloud processing algorithm must also meet the requirements of high-speed driving of cars; at the same time, the size, weight, cost and driving voltage of the LiDAR should all meet the installation level of the vehicle-mounted system.

[0003] In order to meet the high requirements of intelligent driving for vehicle-mounted laser radar, the relevant research on key technologies and application algorithms of vehicle-mounted laser radar is constantly deepening and expanding. Among the many laser radar hardware technologies such as laser emission, reception, and scanning, it is generally believed that scanning technology is the key to vehicle-mounted laser radar. It directly determines the important parameters of the radar, such as scanning frequency, scanning range, and point cloud data volume. It is the key to reducing the overall volume and cost of vehicle-mounted laser radar, and it also directly affects the realization of vehicle-grade mass production of vehicle-mounted laser radar. In terms of application algorithm development, there are many point cloud processing algorithms for the millions of point cloud data generated per second by high-scanning frequency and resolution vehicle-mounted laser radars. They are all based on different application scenarios selected by application developers, and their characteristics have their own advantages and disadvantages. There is no perfect unified algorithm yet, but the goal pursued by application algorithm research is the same, that is, to continuously pursue real-time and high precision on the basis of effective data processing, so as to provide effective technical support for the control and decision-making of intelligent driving systems. However, most of the existing representations of urban scenes are unstructured geometric representations such as point clouds or raster maps, and the data is redundant, which is not convenient for storage, processing and transmission on vehicle mobile platforms, and it is difficult to meet the representation and processing requirements of autonomous driving systems for large-scale dynamic scenes. Therefore, it is necessary to design an automatic extraction method for urban scene spatial bases based on vehicle-mounted multimodal data. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, an object of the present invention is to provide a method for automatically extracting urban scene spatial bases based on vehicle-mounted multimodal data.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] The present invention provides a method for automatically extracting urban scene space bases based on vehicle-mounted multimodal data, comprising:

[0007] Acquire vehicle-mounted multimodal data, including multispectral imaging data and dual-band laser point cloud data;

[0008] Preprocess the vehicle multimodal data;

[0009] The spatial base of urban scenes is extracted based on the preprocessed vehicle-mounted multimodal data.

[0010] Preferably, the vehicle-mounted multimodal data is preprocessed, specifically:

[0011] Fuse multispectral image data and dual-band laser point cloud data;

[0012] Perform data denoising on the fused data;

[0013] Automatically filter dynamic objects based on the denoised data.

[0014] Preferably, the multispectral image data and the dual-band laser point cloud data are fused, specifically:

[0015] According to the relative position between sensors calibrated by the hardware system, the relative position of the two-band laser scanners that collect dual-band laser point cloud data corresponds to the relative position of the multispectral camera that collects multispectral image data. The point clouds of the two bands are solved and automatically fused with the coordinates in the corresponding multispectral image, which is:

[0016] u i =K[R|t]X i (1)

[0017] Where, X i is any point in the 3D laser point cloud, u i It's point X i The corresponding two-dimensional coordinates in the multispectral image, R and t are the rotation and translation matrices of the virtual camera relative to the laser scanning coordinates.

[0018] Preferably, data denoising is performed on the fused data, specifically:

[0019] The dual-band laser point cloud data is denoised. Based on the sparse and independently distributed characteristics of point cloud noise, the point density of the point cloud is estimated, and low-density flying points are removed. The intensity information of the two-band point clouds is counted separately, and the outliers of the point cloud are manually sampled. The noise with abnormally high and low intensity is filtered out based on the sampling value and combined with the Gaussian mixture model.

[0020] Denoising of multispectral image data is performed, and mixed noise in multispectral image data is removed based on the spatial spectrum gradient network.

[0021] Preferably, the urban scene spatial base is extracted based on the preprocessed vehicle-mounted multimodal data, specifically:

[0022] According to the different target types of urban scenes, urban scenes are divided into road scenes and building facades;

[0023] Extract characteristic elements of road scenes and build models;

[0024] Identify elements on the building facade and perform plane fitting on the facade elements;

[0025] Descriptors are constructed for the constructed feature elements and facade elements.

[0026] Preferably, the characteristic elements of the road scene are extracted and a model is constructed, specifically as follows:

[0027] Build instance segmentation and 3D point cloud feature extraction networks for multispectral image data to extract independent features from vehicle-mounted multimodal data, obtain image features and point cloud features, and fuse them;

[0028] Build a Transformer-based panoramic segmentation network for road traffic facility segmentation, train and predict the network based on fused features, and achieve component-level panoramic segmentation of road traffic facilities;

[0029] A template library is established based on the recognized road traffic signs. Features of the template library and the recognized road traffic signs are extracted based on the pre-trained PointNet. Road traffic signs are retrieved based on the extracted features. Long and continuous road traffic signs are represented by center curves and widths. Curbstones and road traffic poles are modeled using simple Bezier curves and cylinders, respectively.

[0030] The road scene model is constructed based on the identified traffic characteristic elements.

[0031] Preferably, the elements of the building facade are identified, and the facade elements are plane-fitted, specifically:

[0032] The surface growth algorithm is used to segment the point cloud on the road surface to obtain different surface segmentation bodies. The features of each segmentation body are extracted, including: surface feature p, height h, area a, normal vector n, 2D length l, height-to-length ratio r. The rule constraint function f(·) is defined or the surface segmentation body is classified by learning. The surface segmentation bodies that meet the conditions or have a higher probability are classified as building facades, that is:

[0033] F=f(p,h,a,n,l,r)(2)

[0034] The building facade is automatically fitted based on the surface fitting method to obtain the building facade model and skyline.

[0035] Preferably, descriptors are constructed for the constructed characteristic elements and facade elements, specifically:

[0036] Based on the road scene model, the facade model of the building and the skyline, descriptors are constructed for the extracted feature elements, including the category, model, size and retrieval point feature description of the object. The category information, model information and size feature description of the object are used to identify the target information during vehicle positioning. The retrieval point refers to the vehicle trajectory point closest to the model. The retrieval point is used for information retrieval during vehicle positioning. The corresponding target model is quickly located through the retrieval point, and automatic detection and matching are performed on the target model.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0038] The present invention provides a method for automatically extracting urban scene spatial bases based on vehicle-mounted multimodal data, the method comprising obtaining vehicle-mounted multimodal data, including multispectral image data and dual-band laser point cloud data, preprocessing the vehicle-mounted multimodal data, and extracting urban scene spatial bases based on the preprocessed vehicle-mounted multimodal data. The present invention automatically extracts urban scene spatial bases based on vehicle-mounted multimodal data, realizes highly robust feature element extraction and descriptor construction in dual-band laser point clouds and multispectral images, can meet the requirements of autonomous driving systems for the representation and processing of large-scale dynamic scenes, and is easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0041] Figure 2 This is a flowchart for multispectral image denoising;

[0042] Figure 3 This is a schematic diagram of the RoadNet network structure;

[0043] Figure 4 Schematic diagram of the structure of the road traffic facility component-level panoramic segmentation model. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The purpose of the present invention is to provide a method for automatic extraction of urban scene spatial bases based on vehicle-mounted multimodal data, automatic extraction and feature description of urban road surface feature elements, roadside feature elements and building skylines based on deep learning, to achieve highly robust feature element extraction and descriptor construction in dual-band laser point clouds and multispectral images, which is easy to use.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a method for automatically extracting urban scene space bases based on vehicle-mounted multimodal data, comprising:

[0048] Step 100: Acquire vehicle-mounted multimodal data, including multispectral image data and dual-band laser point cloud data;

[0049] Step 200: preprocessing the vehicle-mounted multimodal data;

[0050] Step 300: Extracting the urban scene spatial base based on the preprocessed vehicle-mounted multimodal data.

[0051] In step 200, the vehicle-mounted multimodal data is preprocessed, specifically:

[0052] Step 201: fusing the multispectral image data and the dual-band laser point cloud data;

[0053] Step 202: performing data denoising on the fused data;

[0054] Step 203: Automatically filter dynamic objects based on the denoised data.

[0055] In step 201, the multispectral image data and the dual-band laser point cloud data are fused, specifically:

[0056] According to the relative position between sensors calibrated by the hardware system, the relative position of the two-band laser scanners that collect dual-band laser point cloud data corresponds to the relative position of the multispectral camera that collects multispectral image data. The point clouds of the two bands are solved and automatically fused with the coordinates in the corresponding multispectral image, which is:

[0057] u i =K[R|t]X i (1)

[0058] Where, X i is any point in the 3D laser point cloud, u i It's point X i The corresponding two-dimensional coordinates in the multispectral image, R and t are the rotation and translation matrices of the virtual camera relative to the laser scanning coordinates.

[0059] In step 202, the fused data is subjected to data denoising, specifically:

[0060] Denoise the dual-band laser point cloud data and filter out the noise points generated during the point cloud data acquisition process. Based on the sparse and independently distributed characteristics of point cloud noise, first extract the point cloud to estimate the point density and remove some low-density "flying points". Then, the intensity information of the two-band point clouds is statistically analyzed. The intensity value distribution generally belongs to Gaussian distribution. Due to the influence of noise, the intensity value distribution of the point cloud may become a superposition of multiple Gaussian distributions. By manually sampling some abnormal values ​​of the point cloud, the noise with abnormally high and low intensity is filtered according to the sampling value and combined with the Gaussian Mixture Model.

[0061] Denoising multispectral image data, the presence of mixed noise in multispectral images seriously reduces the data quality, reduces the interpretation accuracy of multispectral images, and limits subsequent applications. This patent uses a spatial spectral gradient network (SSGN) for mixed noise removal in multispectral images. Based on a fully cascaded multi-scale convolutional network, SSGN can simultaneously process different types of noise in different multispectral images or spectra by using the same model. In order to utilize the rich spectral information in multispectral images and the different spatial information of each frequency band, SSGN simultaneously uses spatial data and adjacent spectral data in a fully cascaded multi-scale convolutional neural network block. In addition, spatial gradients and spectral gradients are jointly incorporated into the proposed model. The spatial gradient is used to extract the unique structural directionality of sparse noise in the horizontal and vertical directions, and the spectral gradient is used to obtain additional supplementary information of the spectrum for noise removal. The specific flow chart is as follows Figure 2 As shown;

[0062] In step 300, the urban scene spatial base is extracted based on the preprocessed vehicle-mounted multimodal data, specifically:

[0063] Step 301: Divide the urban scene into road scenes and building facades according to different target types of the urban scene;

[0064] Step 302: extracting characteristic elements of the road scene and constructing a model;

[0065] Step 303: Identify elements of the building facade and perform plane fitting on the facade elements;

[0066] Step 304: construct descriptors for the constructed feature elements and facade elements.

[0067] In step 302, the characteristic elements of the road scene are extracted and a model is constructed, specifically:

[0068] The characteristic elements used for positioning in road scenes include road traffic signs, curbs and road traffic poles on the roadside. The present invention designs a RoadNet to automatically extract road traffic signs, curbs and traffic poles (such as Figure 3 Then, the model of the identified road traffic characteristic elements is automatically constructed. The specific steps are as follows:

[0069] Build an instance segmentation and three-dimensional point cloud feature extraction network for multispectral image data to perform independent feature extraction on-board multimodal data, and maximize the mining of texture and structural information of multi-source data. First, based on the existing network architecture (such as Panoptic FPN), instance segmentation of images is implemented. The training data for instance segmentation uses a manually calibrated multispectral dual-band point cloud dataset. After the instance segmentation network training is completed, the feature extraction module of the instance segmentation network is used for feature extraction of road scene images. For point clouds, the present invention uses the PointGroup network architecture to perform instance segmentation on road scenes. The training phase uses the point cloud data generated in the previous step and the public data point cloud dataset for training. Then, the feature extraction module of the network is used to extract features from point clouds, obtain image features and point cloud features, and fuse them.

[0070] After fusing the features of point cloud and image, a Transformer-based panoramic segmentation network for road traffic facilities segmentation is designed. Figure 4 As shown in the figure, to realize the automatic recognition of road traffic facilities, the first step is to input the fused features into the designed Transformer encoding network. The fused feature sequence can be mapped to the latent space through the encoder through the attention mechanism. The encoding network can capture the connection between different modal features during the learning process. The multimodal attention mechanism cross-integrates the features of different types in the modal data, and can mine the relationship between different modal features and the contextual relationship of the features, realizing the cross-integration of multi-order features and multi-layer relationships. The representation of dot product attention is as follows:

[0071]

[0072] Where Q, K, and V are query values, index values, and eigenvalue vectors, respectively. k is the normalization factor;

[0073] Then the latent space sequence vector is decoded. To achieve panoramic segmentation, the present invention designs two joint decoders, a position decoder and a mask decoder, wherein the position decoder is used to decode the outer bounding box of the detected object, and the mask decoder is used to decode the category of the detected object and the mask information of the object by combining the output of the position decoder with the latent space vector. Finally, the network is trained and predicted to achieve component-level panoramic segmentation of road traffic facilities. Compared with traditional road scene segmentation networks based on point cloud data or image data, this network has better generalization and stronger learning ability.

[0074] A template library is established based on the recognized road traffic signs. Features of the template library and the recognized road traffic signs are extracted based on the pre-trained PointNet. Road traffic signs are retrieved based on the extracted features. Long and continuous road traffic signs are represented by center curves and widths. Curbstones and road traffic poles are modeled using simple Bezier curves and cylinders, respectively.

[0075] The road scene model is constructed based on the identified traffic characteristic elements.

[0076] In step 303, elements of the building facade are identified, and plane fitting is performed on the facade elements, specifically:

[0077] The surface growth algorithm is used to segment the point cloud on the road surface to obtain different surface segmentation bodies. The features of each segmentation body are extracted, including: surface feature p, height h, area a, normal vector n, 2D length l, height-to-length ratio r. The rule constraint function f(·) is defined or the surface segmentation body is classified by learning. The surface segmentation bodies that meet the conditions or have a higher probability are classified as building facades, that is:

[0078] F=f(p,h,a,n,l,r)(6)

[0079] The building facade is automatically fitted based on the surface fitting method to obtain the building facade model and skyline.

[0080] Descriptors are constructed for the characteristic elements and facade elements of the construction, specifically:

[0081] Based on the road scene model, the facade model of the building and the skyline, descriptors are constructed for the extracted feature elements, including the category, model, size and retrieval point feature description of the object. The category information, model information and size feature description of the object are used to identify the target information during vehicle positioning. The retrieval point refers to the vehicle trajectory point closest to the model. The retrieval point is used for information retrieval during vehicle positioning. The corresponding target model is quickly located through the retrieval point, and automatic detection and matching are performed on the target model.

[0082] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0083] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatically extracting urban scene spatial bases based on vehicle-borne multimodal data, characterized in that: include: Acquire vehicle-mounted multimodal data, including multispectral imaging data and dual-band laser point cloud data; Preprocess the vehicle multimodal data; The spatial base of urban scenes is extracted based on the preprocessed vehicle-mounted multimodal data.

2. The method according to claim 1, characterized in that Preprocess the vehicle multimodal data, specifically: Fuse multispectral image data and dual-band laser point cloud data; Perform data denoising on the fused data; Automatically filter dynamic objects based on the denoised data.

3. The method according to claim 2, characterized in that The multispectral image data and dual-band laser point cloud data are fused as follows: According to the relative position between sensors calibrated by the hardware system, the relative position of the two-band laser scanners that collect dual-band laser point cloud data corresponds to the relative position of the multispectral camera that collects multispectral image data. The point clouds of the two bands are solved and automatically fused with the coordinates in the corresponding multispectral image, which is: u i =K[R|t]X i (1) Where, X i is any point in the 3D laser point cloud, u i It's point X i The corresponding two-dimensional coordinates in the multispectral image, R and t are the rotation and translation matrices of the virtual camera relative to the laser scanning coordinates.

4. The method according to claim 2, characterized in that: The fused data is denoised as follows: The dual-band laser point cloud data is denoised. Based on the sparse and independently distributed characteristics of point cloud noise, the point density of the point cloud is estimated, and low-density flying points are removed. The intensity information of the two-band point clouds is counted separately, and the outliers of the point cloud are manually sampled. The noise with abnormally high and low intensity is filtered out based on the sampling value and combined with the Gaussian mixture model. Denoising of multispectral image data is performed, and mixed noise in multispectral image data is removed based on the spatial spectrum gradient network.

5. The method according to claim 1, characterized in that The urban scene spatial base is extracted based on the preprocessed vehicle-mounted multimodal data, specifically: According to the different target types of urban scenes, urban scenes are divided into road scenes and building facades; Extract characteristic elements of road scenes and build models; Identify elements on the building facade and perform plane fitting on the facade elements; Descriptors are constructed for the constructed feature elements and facade elements.

6. The method according to claim 5, characterized in that The characteristic elements of the road scene are extracted and the model is constructed, specifically: Build instance segmentation and 3D point cloud feature extraction networks for multispectral image data to extract independent features from vehicle-mounted multimodal data, obtain image features and point cloud features, and fuse them; Build a Transformer-based panoramic segmentation network for road traffic facility segmentation, train and predict the network based on fused features, and achieve component-level panoramic segmentation of road traffic facilities; A template library is established based on the recognized road traffic signs. Features of the template library and the recognized road traffic signs are extracted based on the pre-trained PointNet. Road traffic signs are retrieved based on the extracted features. Long and continuous road traffic signs are represented by center curves and widths. Curbstones and road traffic poles are modeled using simple Bezier curves and cylinders, respectively. The road scene model is constructed based on the identified traffic characteristic elements.

7. The method according to claim 6, characterized in that Identify the elements of the building facade and perform plane fitting on the facade elements, specifically: The surface growth algorithm is used to segment the point cloud on the road surface to obtain different surface segmentation bodies. The features of each segmentation body are extracted, including: surface feature p, height h, area a, normal vector n, 2D length l, height-to-length ratio r. The rule constraint function f(·) is defined or the surface segmentation body is classified by learning. The surface segmentation bodies that meet the conditions or have a higher probability are classified as building facades, that is: F=f(p,h,a,n,l,r) (2) The building facade is automatically fitted based on the surface fitting method to obtain the building facade model and skyline.

8. The method according to claim 7, characterized in that Descriptors are constructed for the characteristic elements and facade elements of the construction, specifically: Based on the road scene model, the facade model of the building and the skyline, descriptors are constructed for the extracted feature elements, including the category, model, size and retrieval point feature description of the object. The category information, model information and size feature description of the object are used to identify the target information during vehicle positioning. The retrieval point refers to the vehicle trajectory point closest to the model. The retrieval point is used for information retrieval during vehicle positioning. The corresponding target model is quickly located through the retrieval point, and automatic detection and matching are performed on the target model.

Images