Road network information acquisition method fusing remote sensing image and trajectory data

Abstract

The invention relates to the field of road network information acquisition, in particular to a road network information acquisition method fusing remote sensing images and trajectory data. The method comprises the following steps: acquiring high-resolution remote sensing image data and vehicle track data; carrying out preprocessing, rasterization and feature extraction operation on the vehicle trajectory data; constructing a data set for training the convolutional neural network; road network information in a high-resolution remote sensing image and trajectory data is obtained by using a convolutional neural network, a multi-modal feature enhancement module is applied between an image auto-encoder and a trajectory auto-encoder to transmit multi-modal complementary information, and mutual enhancement of image single-modal features and trajectory single-modal features is realized; road prediction images output by an image auto-encoder and a track auto-encoder are connected by adopting a weighted fusion mode based on a gating weight, so that the convolutional neural network can automatically find a proper optimal weight value, and time and energy required for manually selecting the weight are reduced.

Description

Technical Field

[0001] The present invention relates to the field of road network information acquisition, and in particular to a road network information acquisition method that integrates remote sensing images and trajectory data. Background Art

[0002] Complete and detailed road network data is the key to improving the intelligentization of urban transportation. At present, the automatic generation of large-scale urban road networks mainly relies on a single data source, such as high-resolution remote sensing data, point cloud LiDAR data, GNSS trajectory data, etc., which has problems such as fragmented road information extraction, limited coverage, and inconsistent information details. As one of the important data sources for obtaining road network information, high-resolution remote sensing image data has comprehensive and accurate coverage, but it has limitations such as long update cycle and roads are easily blocked by trees and buildings. In addition, the problems of "same spectrum, different objects" and "same object, different spectrum" in the image will seriously affect the extraction effect of roads. Compared with remote sensing images, trajectory data has the advantages of strong real-time, fast update speed, easy acquisition, and low cost. At the same time, it also has limited accuracy, noise points and drift points, and uneven density distribution. These problems greatly increase the difficulty of extracting roads using trajectory data. In addition, the existing methods for obtaining road network information based on convolutional neural networks usually only use one data source, cannot fuse multi-source data to extract road information, and cannot make full use of the complementary information between multi-source data to obtain a more complete and accurate road network structure. Summary of the invention

[0003] In order to solve the problem that the existing technology cannot fuse multi-source data to extract road information and cannot fully utilize the complementary information between multi-source data, the present invention provides a road network information acquisition method that fuses remote sensing images and trajectory data, which mainly includes:

[0004] S1: Acquire high-resolution remote sensing image data and vehicle trajectory data;

[0005] S2: preprocessing, rasterization and feature extraction operations are performed on the acquired vehicle trajectory data;

[0006] S3: Construct a dataset for training convolutional neural networks;

[0007] S4: Use the designed convolutional neural network to obtain road network information from high-resolution remote sensing images and trajectory data. The specific steps are as follows:

[0008] S4.1: Use Python programming language to build a convolutional neural network based on Pytorch framework in Pycharm integrated environment. The convolutional neural network includes an image autoencoder, a trajectory autoencoder, a multimodal feature enhancement module and a weighted fusion method based on gating weights. The multimodal feature enhancement module is used to extract and transmit complementary information of image single modality and trajectory single modality and realize mutual enhancement of image features and trajectory features. The output results of the image autoencoder and the trajectory autoencoder are fused by the weighted fusion method based on gating weights.

[0009] S4.2: The image unimodal features and trajectory unimodal features output by the image encoder and trajectory encoder are as follows:

[0010]

[0011] In the formula, f I is a set of image unimodal features, f T is a set of trajectory unimodal features;

[0012] S4.3: The image unimodal feature information and trajectory unimodal feature information obtained by the AD module in the image autoencoder and trajectory autoencoder are as follows:

[0013]

[0014] In the formula, and are the final outputs of the three CCA branches in each AD module in the image autoencoder and trajectory autoencoder, and are the outputs of the HDC branches in each AD module in the image autoencoder and trajectory autoencoder, i = 1, 2, 3, 4;

[0015] In the multimodal feature enhancement module, the gate weight for dynamically transmitting the trajectory unimodal feature information is:

[0016]

[0017] In the formula, the operator Express and Concatenate by channel, Conv represents 1*1 convolution layer, and It is the parameter of the 1*1 convolution layer. The Sigmoid function ensures that the gate weight range is between [0,1];

[0018] The image features obtained by the image autoencoder after trajectory unimodal feature enhancement are:

[0019]

[0020] In the formula, the operator Represents element-wise multiplication;

[0021] Similarly, the gate weight for dynamically transmitting the image unimodal feature information is:

[0022]

[0023] The trajectory features obtained by the trajectory autoencoder after image unimodal feature enhancement are as follows:

[0024]

[0025] S4.4: The road prediction results of the image autoencoder and trajectory autoencoder are weighted and summed through two learnable gating weights. The calculation formula of the gating weight is:

[0026]

[0027] Where Conv represents a 1*1 convolutional layer, W T With W I is the parameter of the 1*1 convolution layer, Out I With Out T They are the road prediction results of the image autoencoder and trajectory autoencoder respectively;

[0028] Then use a 3*3 convolutional layer to reduce the dimension, and then get the final road prediction fusion result:

[0029] Out = Conv(θ T *Out T +θ I *Out I ).

[0030] A storage device stores instructions and data for implementing a road network information acquisition method that integrates remote sensing images and trajectory data.

[0031] A road network information acquisition device that integrates remote sensing images and trajectory data includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a road network information acquisition method that integrates remote sensing images and trajectory data.

[0032] The technical solution provided by the present invention has the following beneficial effects: the present invention makes full use of the complementary information of remote sensing image data and vehicle trajectory data, and can improve the quality of the acquired road information. The designed convolutional neural network inputs the image data and trajectory data into independent trajectory autoencoders and image autoencoders respectively, effectively obtains and saves the specific information of the two data source modalities, and helps the image autoencoder and trajectory autoencoder to learn the weight parameters suitable for extracting the respective image features and trajectory features, so that the model can more accurately extract the road information contained in the image data and trajectory data. At the same time, a multimodal feature enhancement module is applied between the image autoencoder and the trajectory autoencoder to transmit multimodal complementary information, so as to realize the mutual enhancement of the image unimodal features and the trajectory unimodal features; finally, the road prediction images output by the image autoencoder and the trajectory autoencoder are connected by a weighted fusion method based on gated weights as the final output road network map. This not only allows the convolutional neural network to automatically find the appropriate optimal weight value, but also reduces the time and effort required for manual weight selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0034] Figure 1 is a flow chart of a method for acquiring road network information by integrating remote sensing images and trajectory data in an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of rasterizing discrete trajectory point data in an embodiment of the present invention;

[0036] Figure 3 is a structural diagram of a convolutional neural network in an embodiment of the present invention;

[0037] Figure 4 is a structural diagram of an image autoencoder and a trajectory autoencoder in an embodiment of the present invention;

[0038] Figure 5 is a structural diagram of a multimodal feature enhancement module in an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of the operation of the hardware device in the embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0041] Example 1

[0042] Please refer to Figure 1 , Figure 1: is a flow chart of a method for acquiring road network information by integrating remote sensing images and trajectory data in an embodiment of the present invention, which specifically includes:

[0043] S1: High-resolution remote sensing images are obtained from map software such as Google Earth. For large areas, ENVI software is used to stitch multiple image data downloaded at the same time to generate high-resolution remote sensing images of large areas. Vehicle trajectory data comes from public vehicles equipped with GNSS positioning equipment, such as taxis and Didi operating vehicles;

[0044] S2: Preprocessing, rasterization and feature extraction of vehicle trajectory data. This step includes the following sub-steps:

[0045] S2.1: Delete abnormal trajectory points and trajectory points with a speed of 0 in the acquired vehicle trajectory data;

[0046] S2.2: Filter out drift points in the vehicle trajectory data according to the set maximum speed threshold of 20m / s, that is, remove the trajectory point when the speed of the trajectory point is greater than the maximum speed threshold;

[0047] S2.3: Rasterize discrete trajectory point data, refer to Figure 2 , according to the resolution (W, H) of the remote sensing image, a blank grid image is created, and then the spatial coordinates (x, y) of the grid image corresponding to the longitude and latitude coordinates (lat, lon) of the trajectory point are calculated. The conversion formula between the longitude and latitude coordinates (lat, lon) of the trajectory point and the spatial coordinates (x, y) of the grid image is:

[0048]

[0049] Among them, Lon min ,Lon max ,Lat min ,Lat max Respectively represent the minimum and maximum values ​​of the longitude and latitude of the area where the vehicle trajectory data and remote sensing image data are located, Interval lon ,Interval lat They represent the number of pixels corresponding to 1 longitude and 1 latitude respectively. The function Math.Round is used to round the value.

[0050] S2.4: When rasterizing the vehicle trajectory data, the density feature, speed feature, and distance feature of the trajectory point are used to determine the grayscale value of the pixel where the trajectory point falls:

[0051] For the trajectory density feature, the number of trajectory points falling into each pixel unit is first counted, and then a threshold N is set. In this embodiment, N is 1. If the number of trajectory points is less than the threshold, the gray value of the corresponding pixel is assigned to 0. If the number of trajectory points is equal to the threshold, the gray value of the corresponding pixel is assigned to 128. Otherwise, the gray value of the corresponding pixel is assigned to 255. Finally, a trajectory density grid map is obtained;

[0052] For the trajectory speed feature, the gray value of the pixel is assigned according to the standard deviation of the trajectory point speed in each pixel unit. If the trajectory point speeds of the eight adjacent units of the current pixel unit are all 0, the pixel unit is considered to be isolated, and its gray value is assigned to 0. If the standard deviation of the trajectory point speed is greater than the second preset value or between the first preset value and the second preset value or less than the first preset value, the first preset value and the second preset value can be set according to the variance of the corresponding trajectory data. In this embodiment, the first preset value is 0.5 and the second preset value is 0.8, then the gray value of the pixel is assigned to 0, 128 and 255 respectively; finally, a trajectory speed grid map is obtained;

[0053] For the trajectory distance feature, if two adjacent trajectory points are located in the same pixel unit, the grayscale value of the pixel is assigned to 255. If a trajectory point is located in a pixel unit and its adjacent trajectory point is located in the adjacent unit of this pixel unit, the grayscale value of the pixel is assigned to 128. In other cases, it is assigned to 0. Finally, the trajectory distance grid map is obtained.

[0054] S2.5: splicing the trajectory density grid map, the trajectory speed grid map and the trajectory distance grid map on the channel to form a trajectory feature map;

[0055] S2.6: For the trajectory feature map, an appropriate smoothing strategy is adopted. The basic idea is to smooth the trajectory raster image by calculating the types of other pixels within the eight-neighborhood range of the pixel. In the trajectory raster image, the pixel grayscale values ​​0, 128 and 255 represent "non-road pixels", "potential road pixels" and "road pixels", respectively. When smoothing, if the original type of the pixel is "road pixel" or "potential road pixel" and the total number of pixels belonging to "non-road pixels" within its eight-neighborhood range is greater than the threshold H, it will become a "non-road pixel"; on the contrary, if the original type of the pixel is "non-road pixel" and the total number of pixels belonging to "road pixels" within its eight-neighborhood range is greater than the threshold H, it will become a "potential road pixel".

[0056] S3: Build a dataset for training convolutional neural networks. This step includes the following sub-steps:

[0057] S3.1: Download the OSM road network data corresponding to the image data coverage area from OSM, set buffers of different widths for different roads in the road network, and then manually delete, supplement and modify some of the roads based on the high-resolution remote sensing image data, and then obtain the road label map after rasterization;

[0058] S3.2: The remote sensing image map, trajectory feature map and road label map are uniformly cropped into a size of 1024*1024 pixels to construct a training data set, and a data augmentation method is used on the training set, that is, some images are randomly selected for image shift, image scaling, horizontal flip, vertical flip and diagonal flip operations to expand the training data set.

[0059] S4: Use the designed convolutional neural network to obtain road network information from high-resolution remote sensing images and trajectory data. This step includes the following sub-steps:

[0060] S4.1: Use Python programming language based on Pytorch framework to build convolutional neural network Dual-MFEM-AD-LinkNet in Pycharm integrated environment. Its structure is referenced Figure 3 , which consists of four parts: image autoencoder, trajectory autoencoder, multimodal feature enhancement module (MFEM) and the final weighted fusion method based on gated weights;

[0061] S4.2: Use the encoder part composed of residual units, the AD module composed of mixed hole convolution and cross attention modules, and the decoder part composed of upsampling units to build the image autoencoder and trajectory autoencoder in the Dual-MFEM-AD-LinkNet model. The structure is referenced Figure 4 ;

[0062] S4.3: Build a multimodal feature enhancement module MFEM responsible for transferring the information of the image autoencoder and trajectory autoencoder in stages. Its structure is referenced Figure 5 , where, in the encoder stage, the image encoder and trajectory encoder output the image unimodal features and trajectory unimodal features as follows:

[0063]

[0064] In the formula, f I is a set of image unimodal features, f T is a set of trajectory unimodal features, i = 1, 2, 3, 4;

[0065] The image unimodal feature information and trajectory unimodal feature information obtained by the AD module in the image autoencoder and trajectory autoencoder are as follows:

[0066]

[0067] In the formula, and are the final outputs of the three CCA branches in each AD module in the image autoencoder and trajectory autoencoder, and are the outputs of the HDC branches in each AD module in the image autoencoder and trajectory autoencoder respectively;

[0068] In the multimodal feature enhancement module, the gate weight for dynamically transmitting the trajectory unimodal feature information is:

[0069]

[0070] In the formula, the operator Express and Concatenate by channel, Conv represents 1*1 convolution layer, and It is the parameter of the 1*1 convolution layer. The Sigmoid function ensures that the gate weight range is between [0,1];

[0071] Therefore, the image features obtained by the image autoencoder after trajectory unimodal feature enhancement are:

[0072]

[0073] In the formula, the operator Represents element-wise multiplication;

[0074] Similarly, the gate weight for dynamically transmitting the image unimodal feature information is:

[0075]

[0076] Therefore, the trajectory features obtained by the trajectory autoencoder after image unimodal feature enhancement are as follows:

[0077]

[0078] S4.4: The road prediction results of the image autoencoder and trajectory autoencoder are weighted and summed through two learnable gating weights. The calculation formula of the gating weight is:

[0079]

[0080] Where Conv represents a 1*1 convolutional layer, WT With W I is the parameter of the 1*1 convolution layer, Out I With Out T They are the road prediction results of the image autoencoder and trajectory autoencoder respectively;

[0081] Then use a 3*3 convolutional layer to reduce the dimension, and then get the final road prediction fusion result:

[0082] Out = Conv(θ T *Out T +θ I *Out I );

[0083] S4.5: Use the training dataset expanded in step S3.2 to train the Dual-MFEM-AD-LinkNet model, and then use the trained model to identify road information in high-resolution remote sensing images and trajectory data.

[0084] The multimodal feature enhancement module MFEM in the present invention is composed of an AD module and a gate unit Gate at the encoder stage. The gate unit dynamically transfers the local and global information of the unimodal features obtained by the AD module to another modality; at the decoder stage, it is responsible for statically transferring the complementary information obtained by the AD module in the encoder stage to another modality. After the image autoencoder and trajectory autoencoder of Dual-MFEM-AD-LinkNet use the transmitted multimodal complementary information to enhance each other, the learned unimodal features will become richer. The choice of using learnable gate weights to weightedly fuse the road prediction images of the trajectory autoencoder and the image autoencoder is beneficial to the final road prediction.

[0085] Example 2

[0086] A road network information acquisition device that integrates remote sensing images and trajectory data, characterized in that it includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the road network information acquisition method that integrates remote sensing images and trajectory data.

[0087] Example 3

[0088] A storage device, characterized in that: the storage device stores instructions and data for implementing the road network information acquisition method that integrates remote sensing images and trajectory data.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for acquiring road network information by integrating remote sensing images and trajectory data, characterized in that: include: S1: Acquire high-resolution remote sensing image data and vehicle trajectory data; S2: preprocessing, rasterization and feature extraction operations are performed on the acquired vehicle trajectory data; S3: Construct a dataset for training convolutional neural networks; S4: Use the designed convolutional neural network to obtain road network information from high-resolution remote sensing images and trajectory data. The specific steps are as follows: S4.1: Use Python programming language to build a convolutional neural network based on Pytorch framework in Pycharm integrated environment. The convolutional neural network includes an image autoencoder, a trajectory autoencoder, a multimodal feature enhancement module and a weighted fusion method based on gating weights. The multimodal feature enhancement module is used to extract and transmit complementary information of image single modality and trajectory single modality and realize mutual enhancement of image features and trajectory features. The output results of the image autoencoder and the trajectory autoencoder are fused by the weighted fusion method based on gating weights. S4.2: The image unimodal features and trajectory unimodal features output by the image encoder and trajectory encoder are as follows: In the formula, f I is a set of image unimodal features, f T is a set of trajectory unimodal features, i = 1, 2, 3, 4; S4.3: The image unimodal feature information and trajectory unimodal feature information obtained by the AD module in the image autoencoder and trajectory autoencoder are as follows: In the formula, and are the final outputs of the three CCA branches in each AD module in the image autoencoder and trajectory autoencoder, and are the outputs of the HDC branches in each AD module in the image autoencoder and trajectory autoencoder respectively; In the multimodal feature enhancement module, the gate weight for dynamically transmitting the trajectory unimodal feature information is: In the formula, the operator Express and Concatenate by channel, Conv represents 1*1 convolution layer, and It is the parameter of the 1*1 convolution layer. The Sigmoid function ensures that the gate weight range is between [0,1]; The image features obtained by the image autoencoder after trajectory unimodal feature enhancement are: In the formula, the operator Represents element-wise multiplication; Similarly, the gate weight for dynamically transmitting the image unimodal feature information is: The trajectory features obtained by the trajectory autoencoder after image unimodal feature enhancement are as follows: S4.4: The road prediction results of the image autoencoder and trajectory autoencoder are weighted and summed through two learnable gating weights. The calculation formula of the gating weight is: Where Conv represents a 1*1 convolutional layer, W T With W I is the parameter of the 1*1 convolution layer, Out I With Out T They are the road prediction results of the image autoencoder and trajectory autoencoder respectively; Then use a 3*3 convolutional layer to reduce the dimension, and then get the final road prediction fusion result: Out=Conv(θ T *Out T +θ I *Out I )。 2. A method for acquiring road network information by integrating remote sensing images and trajectory data as claimed in claim 1, characterized in that: In S1, for large areas, ENVI software is used to stitch multiple image data downloaded at the same time to achieve the acquisition of remote sensing image data of high-resolution remote sensing images of large areas. The vehicle trajectory data comes from Volkswagen vehicles equipped with GNSS positioning equipment.

3. The method for acquiring road network information by integrating remote sensing images and trajectory data according to claim 1, characterized in that: In S2, the process of preprocessing, rasterization and feature extraction operations is as follows: S2.1: Delete abnormal trajectory points and trajectory points with a speed of 0 in the acquired vehicle trajectory data; S2.2: Filter out drift points in the original trajectory data according to the set maximum speed threshold, that is, remove the trajectory point when the speed of the trajectory point is greater than the maximum speed threshold; S2.3: Rasterize discrete trajectory point data According to the resolution (W, H) of the remote sensing image, a blank grid image is created, and then the spatial coordinates (x, y) of the grid image corresponding to the longitude and latitude coordinates (lat, lon) of the trajectory point are calculated. The conversion formula between the longitude and latitude coordinates (lat, lon) of the trajectory point and the spatial coordinates (x, y) of the grid image is: Among them, Lon min ,Lon max ,Lat min ,Lat max Respectively represent the minimum and maximum values ​​of the longitude and latitude of the area where the vehicle trajectory data and remote sensing image data are located, Interval lon ,Interval lat They represent the number of pixels corresponding to 1 longitude and 1 latitude respectively. The function Math.Round is used to round the value. S2.4: When rasterizing the vehicle trajectory data, the density feature, speed feature, and distance feature of the trajectory point are used to determine the grayscale value of the pixel where the trajectory point falls: For the trajectory density feature, the number of trajectory points falling into each pixel unit is first counted, and then the threshold N is set. If the number of trajectory points is less than the threshold, the grayscale value of the corresponding pixel is assigned to 0. If the number of trajectory points is equal to the threshold, the grayscale value of the corresponding pixel is assigned to 128. Otherwise, the grayscale value of the corresponding pixel is assigned to 255. Finally, the trajectory density grid map is obtained; 0, 128, and 255 represent "non-road pixels", "potential road pixels", and "road pixels", respectively. For the trajectory speed feature, the gray value of the pixel is assigned according to the standard deviation of the trajectory point speed in each pixel unit. If the trajectory point speeds of the eight adjacent units of the current pixel unit are all 0, the pixel unit is considered to be isolated and its gray value is assigned to 0. If the standard deviation of the trajectory point speed is greater than the second preset value or between the first preset value and the second preset value or less than the first preset value, the gray value of the pixel is assigned to 0, 128 and 255 respectively; finally, a trajectory speed grid map is obtained; For the trajectory distance feature, if two adjacent trajectory points are located in the same pixel unit, the grayscale value of the pixel is assigned to 255. If a trajectory point is located in a pixel unit and its adjacent trajectory point is located in the adjacent unit of this pixel unit, the grayscale value of the pixel is assigned to 128. In other cases, it is assigned to 0. Finally, the trajectory distance grid map is obtained. S2.5: splicing the trajectory density raster map, the trajectory speed raster map and the trajectory distance raster map on the channel to obtain a trajectory feature map; S2.6: When smoothing the trajectory feature map using an appropriate smoothing strategy, if the original type of the pixel is "road pixel" or "potential road pixel" and the total number of pixels belonging to "non-road pixels" within its eight neighborhoods is greater than the threshold H, it will become a "non-road pixel"; conversely, if the original type of the pixel is "non-road pixel" and the total number of pixels belonging to "road pixels" within its eight neighborhoods is greater than the threshold H, it will become a "potential road pixel".

4. The method for acquiring road network information by integrating remote sensing images and trajectory data according to claim 1, characterized in that: The specific implementation process of S3 is: S3.1: Download the OSM road network data corresponding to the image data coverage area, set buffers of different widths for different roads in the road network, and then manually delete, supplement and modify some of the roads based on the high-resolution remote sensing image data, and then obtain the road label map after rasterization; S3.2: The remote sensing image map, trajectory feature map and road label map are cropped into a uniform size to construct a training data set, and a data enhancement method is applied to the training set. The data enhancement method includes randomly selecting part of the images for image shift, image scaling, horizontal flipping, vertical flipping and diagonal flipping to expand the training data set.

5. The method for acquiring road network information by integrating remote sensing images and trajectory data according to claim 1, characterized in that: In S3, both the image autoencoder and trajectory autoencoder include an encoder part, an AD module, and a decoder part. The encoder part is composed of residual units, the AD module is composed of a mixed void convolution and a cross attention module, and the decoder part is composed of upsampling units.

6. The method for acquiring road network information by integrating remote sensing images and trajectory data according to claim 4, characterized in that: The convolutional neural network is trained using the expanded training data set in S3.2, and then the trained convolutional neural network is used to identify road information in high-resolution remote sensing images and trajectory data.

7. A storage device, characterized in that: The storage device stores instructions and data for implementing the road network information acquisition method of integrating remote sensing images and trajectory data as described in any one of claims 1 to 6.

8. A road network information acquisition device integrating remote sensing images and trajectory data, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the road network information acquisition method of integrating remote sensing images and trajectory data as described in any one of claims 1 to 6.

Images