A one-map road asset detection method based on images and laser point clouds

Abstract

The present application relates to the technical field of image processing, in particular to a one-map highway asset detection method based on images and laser point clouds, the method comprising: acquiring image data and three-dimensional point cloud data of a to-be-identified highway and both sides of the to-be-identified highway; obtaining to-be-identified highway asset image recognition data and highway asset point cloud data; fusing the highway asset image recognition data and the highway asset point cloud data based on an attention mechanism fusion strategy; obtaining individualized asset models of each asset on the to-be-identified highway; and fusing with GIS data. The present application obtains individualized asset models of each asset on the to-be-identified highway based on the attention mechanism fusion strategy, improves the accuracy and robustness of the fusion result, realizes a one-map management system, and ensures effective storage of full-factor information in highway operation. The present application automatically and accurately identifies highway assets directly according to images and laser point clouds, and does not need to acquire highway asset data from each operation unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a one-map highway asset detection method based on images and laser point clouds. BACKGROUND

[0002] The highway asset detection is inefficient and prone to errors; in addition, due to the large number of operating units of various highway resources, there is a "data barrier" phenomenon between the business systems of various operating units, and asset information sharing cannot be achieved; leading to difficult acquisition of highway asset data, and traditional asset management mainly adopts manual inspection, manual reporting and regular inspection for management, and the accuracy of data is seriously dependent on the subjective consciousness and checking experience of the staff. SUMMARY

[0003] Therefore, the present application aims to provide a one-map highway asset detection method based on images and laser point clouds to solve the problem of low efficiency of manual detection due to the difficulty in acquiring asset management data in the prior art.

[0004] To achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0005] The present application provides a one-map highway asset detection method based on images and laser point clouds, which comprises the following steps:

[0006] Obtaining image data and three-dimensional point cloud data of a to-be-identified highway and both sides of the to-be-identified highway;

[0007] Inputting the image data into a trained full convolutional network model to obtain to-be-identified highway asset image recognition data;

[0008] Classifying the point cloud data based on the elevation and lateral position information in the three-dimensional point cloud data and based on an alphashapes algorithm to filter out highway asset point cloud data;

[0009] Fusing the highway asset image recognition data and the highway asset point cloud data based on a fusion strategy of an attention mechanism to obtain individualized asset models of each asset on the to-be-identified highway;

[0010] Obtaining GIS data of the to-be-identified highway, and fusing the individualized asset models of each asset of the to-be-identified highway with the GIS data of the to-be-identified highway to construct a real three-dimensional scene.

[0011] Further, the full convolutional network model is a full convolutional network model with 8 times up-sampling capability.

[0012] Further, the training method of the full convolutional network model comprises:

[0013] Obtaining an image set containing highway asset and non-highway asset labels;

[0014] Preprocessing data in the image set;

[0015] The cross-entropy loss function is used as the loss function for training the full convolutional network model;

[0016] The SGD algorithm or Adam algorithm is used as the algorithm for optimizing the parameters of the full convolutional network model;

[0017] The preprocessed image set is divided into a training set and a test set, and the training set is input into the full convolutional network model for training to obtain a trained full convolutional network model.

[0018] Further, the preprocessing of the data in the image set comprises:

[0019] Uniformly adjusting each image in the image set to the input size required for training the full convolutional network model;

[0020] Randomly cropping, randomly flipping, and randomly color jittering the images in the image set;

[0021] Normalizing the pixel values in each image in the image set to a predetermined range.

[0022] Further, when the training set is input into the full convolutional network model for training, the weights of the encoder of the full convolutional network model are initialized by a pre-trained classification network, the weights of the encoder are fixed, and only the decoder of the full convolutional network model is trained; until the decoder part is trained.

[0023] Further, the fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain the individual asset model of each asset on the to-be-identified highway, comprising

[0024] Extracting features of the highway asset image recognition data to obtain asset image features;

[0025] Extracting features of the highway asset point cloud data to obtain asset point cloud features;

[0026] According to the similarity and correlation of the asset image features and the asset point cloud features, the attention weight between the asset image features and the asset point cloud features is calculated;

[0027] According to the attention weight, the asset image features and the asset point cloud features are weighted and fused;

[0028] Further, the asset image features include the shape, texture, and color of the highway asset image.

[0029] Further, the asset point cloud feature comprises a geometric feature of the highway asset point cloud data.

[0030] The geometric feature comprises a position, a normal and a curvature of the point.

[0031] The second aspect of the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as described in the first aspect when executing the program.

[0032] The third aspect of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the method as described in the first aspect.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] In the present application, the image recognition data of the to-be-identified highway asset and the point cloud data of the highway asset are extracted respectively, and the single asset model of each asset on the to-be-identified highway is obtained based on the fusion strategy of the attention mechanism; the fusion strategy of the attention mechanism continuously optimizes the attention weight and the fusion mode in the training process to improve the accuracy and robustness of the fusion result; a one-map management system is realized, which can query, locate, count and analyze all single asset components, and ensures effective storage of full-factor information in highway operation. The highway asset is directly identified from the image and the laser point cloud automatically and accurately, and the highway asset data does not need to be obtained from each operation unit. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments thereof and their descriptions serve to explain the present application. The accompanying drawings in conjunction with the description below are used to provide a further understanding of the present application, and they do not constitute an inappropriate limitation on the present application. In the drawings:

[0036] Figure 1 A flowchart of a one-map highway asset detection method based on an image and a laser point cloud according to the present application;

[0037] Figure 2 A flowchart of a training method of a full convolutional network model according to the present application;

[0038] Figure 3 A flowchart of a pre-processing method of image centralized data according to the present application;

[0039] Figure 4 A flowchart of a single asset model of each asset on the to-be-identified highway obtained based on an attention mechanism according to the present application; DETAILED DESCRIPTION

[0040] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict.

[0041] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "back" and the like indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0042] In addition, in the description of the present application, unless otherwise explicitly limited, the terms "mounting", "connecting", "connection", "connector" should be broadly understood. For example, it can be fixedly connected, or it can be detachably connected, or it can be integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood in conjunction with the specific circumstances.

[0043] The present application will be described in detail below with reference to the accompanying drawings Figures 1 to 4 and in conjunction with the embodiments.

[0044] Overall, the present embodiment provides a one-map road asset detection method based on image and laser point cloud, the method comprising:

[0045] Step S1, acquiring image data and three-dimensional point cloud data of a to-be-identified road and both sides of the to-be-identified road;

[0046] Step S2, inputting the image data into a trained full convolutional network model to obtain road asset image recognition data;

[0047] In the present embodiment, the full convolutional network model processes the input image data and outputs a matrix of the same size as the input image, the number of channels of the matrix being 2, corresponding to the prediction probability or score of the road asset category and the non-road asset category respectively. By comparing each value in the matrix with a preset threshold, the category of each pixel point is determined. Generally, the category with higher probability (such as the category exceeding the threshold) is regarded as the final category of the pixel point, and thus the road asset image recognition data is obtained;

[0048] More specifically, the full convolutional network model is a full convolutional network model with 8 times up-sampling capability (Fully Convolutional Networks-8s, FCN-8s).

[0049] More specifically, the training method of the full convolutional network model comprises:

[0050] Step S21, obtaining an image set containing highway asset and non-highway asset labels;

[0051] Wherein the label means that each image should have a corresponding pixel-level label to indicate whether each pixel belongs to a highway asset.

[0052] Step S22, preprocessing the data in the image set;

[0053] The preprocessing of the data in the image set comprises:

[0054] Step S221, uniformly adjusting each image in the image set to the input size required for training the full convolutional network model;

[0055] Step S222, performing random cropping, random flipping and random color jittering processing on the images in the image set; random cropping, random flipping and random color jittering processing of the images can increase the diversity of the image set, so as to improve the generalization ability of the full convolutional network model.

[0056] Step S223, normalizing the pixel values in each image in the image set to a preset range ， such as [0, 1] or [-1, 1] for model training.

[0057] In the FCN-8s (Fully Convolutional Networks for Semantic Segmentation) model in this embodiment, the layers of score_fr, upscore2, fuse_pool4, upscore_pool4, fuse_pool3, score_pool4, score_pool4c, score_pool3, score_pool3c, upscore8, and score can be mainly divided into several categories: feature extraction layers (such as pool3, pool4, etc.), up-sampling layers (such as upscore2, upscore_pool4, etc.), fusion layers (such as fuse_pool3, fuse_pool4), and the final prediction layer (score).

[0058] For a binary classification task, you first need to modify the output channel number of the score layer from the original number of classes (e.g., 20) to 2, which corresponds to the predicted scores of road assets and non-road assets. To ensure that the output channel number of all layers connected directly to the score layer (such as upscore8, fuse_pool3, etc.) is adjusted accordingly to match the input requirements of the score layer, additional convolutional layers are added before upsampling or fusion to reduce or adjust the channel number.

[0059] Feature extraction layers (such as pool3, pool4) are mainly responsible for extracting useful features from input images; the output channel number of feature extraction layers generally does not need to be changed, as they are designed to extract more abstract feature representations rather than being directly used for classification.

[0060] Step S23, using the cross-entropy loss function as the loss function used in the training of the full convolutional network model;

[0061] The loss function used in the training of the FCN-8s model is usually the cross-entropy loss function, which can measure the difference between the model's predicted results and the true labels. To better optimize the model's performance, sometimes a weighted cross-entropy loss function is used, which assigns different weights to different types of pixels (such as foreground and background) to address the problem of class imbalance.

[0062] Step S24, using the stochastic gradient descent (SGD) algorithm or adaptive moment estimation (Adam) algorithm as the algorithm for optimizing the parameters of the full convolutional network model;

[0063] During training, an optimizer suitable for deep learning tasks is usually selected, such as SGD, Adam, etc. These optimizers can update network weights based on gradient information to minimize the loss function.

[0064] Step S25, dividing the preprocessed image set into a training set and a test set, and inputting the training set into the full convolutional network model for training to obtain a trained full convolutional network model.

[0065] When the training set is input into the full convolutional network model for training, the weights of the encoder of the full convolutional network model are initialized by the pre-trained classification network, the weights of the encoder are fixed, and only the decoder of the full convolutional network model is trained until the decoder part is trained; after the decoder part is trained, the full convolutional network model is adjusted. The FCN-8s model in the present application adopts a phased training strategy, which can speed up the training process and help the model converge to a better solution.

[0066] Step S3, according to the elevation and lateral position information in the three-dimensional point cloud data and based on the alphashapes algorithm, the point cloud data is classified to filter out the highway asset point cloud data;

[0067] Step S4, the highway asset image recognition data and the highway asset point cloud data are fused based on the fusion strategy of attention mechanism; and a single asset model of each asset on the to-be-identified highway is obtained;

[0068] After obtaining the single asset model of each asset on the to-be-identified highway, attribute coding and identity coding are performed on the single asset model of each asset on the highway, and the identity is uniquely represented, which can be used as a primary key for query and identification in the later stage

[0069] The fusion strategy based on attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data, and obtains a single asset model of each asset on the to-be-identified highway, which includes:

[0070] Step S41, features of the highway asset image recognition data are extracted to obtain asset image features;

[0071] Step S42, features of the highway asset point cloud data are extracted to obtain asset point cloud features;

[0072] Step S43, attention weights between the asset image features and the asset point cloud features are calculated according to the similarity and correlation of the asset image features and the asset point cloud features;

[0073] Step S44, the asset image features and the asset point cloud features are weighted and fused according to the attention weights;

[0074] In this embodiment, the method for calculating the attention weights between the asset image features and the asset point cloud features according to the similarity and correlation of the asset image features and the asset point cloud features includes: scoring according to the similarity and correlation of the asset image features and the asset point cloud features; the similarity and correlation scores between the asset image features and the asset point cloud features can be obtained through vector dot product, cosine similarity, neural network, etc.; and the scores are normalized to obtain the final attention weights.

[0075] In this embodiment, the weighted fusion can be realized by simple weighted summation, feature splicing or more complex fusion methods. In the fusion process, a multi-layer fusion strategy can be considered, that is, multiple fusions are performed at different levels of the network to fully utilize the feature information at different levels. The attention weights and the fusion method are continuously optimized through the training process to improve the accuracy and robustness of the fusion results. A suitable loss function can be used to evaluate the quality of the fusion results, and the parameters of the network are updated through the back propagation algorithm.

[0076] Asset image features include shape, texture, and color of the highway asset images.

[0077] In this embodiment, key features are extracted from the highway asset image recognition data, including shape, texture, and color. These features can be automatically extracted using a pre-trained convolutional neural network (CNN) or other deep learning models.

[0078] Asset point cloud features include geometric features of the highway asset point cloud data.

[0079] Geometric features include the position, normal, and curvature of the points.

[0080] In this embodiment, geometric features are extracted from the highway asset point cloud data, such as the position, normal, and curvature of the points. These features can be automatically extracted using point cloud processing algorithms such as PointNet, PointNet++, etc.

[0081] Step S5, obtain the geographic information system (GIS) data of the to-be-identified highway, and fuse the individualized asset models of each asset of the to-be-identified highway and the GIS data of the to-be-identified highway to construct a real three-dimensional scene.

[0082] The present application extracts the highway asset image recognition data and the highway asset point cloud data, respectively, and obtains the individualized asset models of each asset on the to-be-identified highway based on the fusion strategy of the attention mechanism. The fusion strategy of the attention mechanism continuously optimizes the attention weight and the fusion method during the training process to improve the accuracy and robustness of the fusion result. A one-map management system is realized, which can query, locate, count, and analyze all individual asset components, ensuring effective storage of full-factor information in highway operation. The highway assets are directly identified automatically and accurately based on images and laser point clouds, and there is no need to obtain highway asset data from each operation unit.

[0083] In the description of the present application, it needs to be explained that the terms "upper", "lower" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise expressly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0084] It also needs to be explained that in the description of the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or equipment. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0085] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not a limitation on the embodiments of the present application. For those skilled in the art, on the basis of the above description, other different forms of changes or variations can also be made, and it is impossible to enumerate all the embodiments here. Any obvious changes or variations derived from the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A one-map road asset detection method based on image and laser point cloud, characterized in that: The method comprises: acquiring image data and three-dimensional point cloud data of a to-be-identified highway and both sides of the to-be-identified highway; inputting the image data into a trained full convolutional network model to obtain to-be-identified highway asset image recognition data; classifying the point cloud data according to elevation and transverse position information in the three-dimensional point cloud data and based on an alphashapes algorithm, and screening out highway asset point cloud data; performing fusion processing on the highway asset image recognition data and the highway asset point cloud data based on an attention mechanism-based fusion strategy to obtain individualized asset models of assets on the to-be-identified highway; acquiring GIS data of the to-be-identified highway, and fusing the individualized asset models of the assets of the to-be-identified highway and the GIS data of the to-be-identified highway to construct a real three-dimensional scene; the attention mechanism-based fusion strategy comprises: extracting features of the highway asset image recognition data to obtain asset image features; extracting features of the highway asset point cloud data to obtain asset point cloud features; calculating attention weights between the asset image features and the asset point cloud features according to similarities and correlations of the asset image features and the asset point cloud features; performing weighted fusion on the asset image features and the asset point cloud features according to the attention weights; the asset image features comprise shapes, textures and colors of highway asset images; the asset point cloud features comprise geometric features of highway asset point cloud data; the geometric features comprise positions, normals and curvatures of points.

2. The one-map highway asset detection method based on images and laser point clouds according to claim 1, wherein: the full convolutional network model is a full convolutional network model with 8 times up-sampling capability.

3. The one-map highway asset detection method based on images and laser point clouds according to claim 1, wherein: the training method of the full convolutional network model comprises: acquiring an image set containing highway asset and non-highway asset labels; preprocessing data in the image set; using a cross-entropy loss function as a loss function used when training the full convolutional network model; using an SGD algorithm or an Adam algorithm as an algorithm for optimizing parameters of the full convolutional network model; 4. The method of claim 3, wherein: dividing the preprocessed image set into a training set and a test set, inputting the training set into the full convolutional network model for training, and obtaining the trained full convolutional network model. the preprocessing of data in the image set comprises: uniformly adjusting each image in the image set to an input size required for training the full convolutional network model; performing random cropping, random flipping and random color jittering processing on the images in the image set; normalizing pixel values in each image in the image set to a preset range.

5. The one-map highway asset detection method based on images and laser point clouds according to claim 3, wherein: In the inputting the training set into the full convolutional network model for training, weights of an encoder of the full convolutional network model are initialized by a pre-trained classification network, the weights of the encoder are fixed, and only a decoder of the full convolutional network model is trained until the decoder part is trained.

6. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1-5 when executing the program.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-5. The program is executed by the processor to implement the method of any one of claims 1-5.

Images