One-map road asset detection method based on image and laser point cloud
Through the combination of image and laser point cloud data, the automatic identification and management of highway assets is realized, the problems of low detection efficiency and difficulty in data sharing are solved, and the accuracy and operational efficiency of asset detection are improved.
Patent Information
- Application Number
- CN202410986586.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Highway asset detection is inefficient and prone to errors. Due to the data barrier between operating units, it is difficult to share asset information. It relies on manual point verification and manual filling, and data accuracy depends on the staff's subjective awareness and experience.
Using image and laser point cloud detection methods, by acquiring image data and three-dimensional point cloud data, using a full convolutional network model for asset recognition, and combining the fusion strategy of alphashapes algorithm and attention mechanism, a single-unit asset model is constructed. Implement automated asset detection and management.
It improves the accuracy and robustness of asset testing, realizes automated asset identification and management, ensures effective storage and sharing of all factor information in highway operations, and reduces dependence on manual testing.
Smart Images

Figure CN118967959A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a one-picture highway asset detection method based on images and laser point clouds. Background Art
[0002] Highway asset detection is inefficient and prone to errors. In addition, due to the large number of operating units of various highway resources, there are "data barriers" between the business systems of various operating units, making it impossible to share asset information. This makes it difficult to obtain highway asset data. Traditional asset management mainly uses manual inspection, manual reporting and regular inspections. The accuracy of the data is heavily dependent on the subjective awareness and verification experience of the staff. Summary of the invention
[0003] In view of this, the present invention aims to propose a one-picture highway asset detection method based on images and laser point clouds to solve the problems in the prior art of difficulty in acquiring asset management data and low efficiency of manual detection.
[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:
[0005] A first aspect of the present invention provides a method for detecting highway assets in one picture based on images and laser point clouds, the method comprising:
[0006] Acquire image data and three-dimensional point cloud data of the road to be identified and both sides of the road to be identified;
[0007] Inputting the image data into the trained full convolutional network model to obtain image recognition data of the highway asset to be identified;
[0008] According to the elevation and lateral position information in the 3D point cloud data and based on the alphashapes algorithm, the point cloud data is classified to filter out the highway asset point cloud data;
[0009] The fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain the individual asset model of each asset on the highway to be identified;
[0010] The GIS data of the road to be identified is obtained, and the individual asset models of each asset of the road to be identified are merged with the GIS data of the road to be identified to construct a real three-dimensional scene.
[0011] Furthermore, the fully convolutional network model is a fully convolutional network model with 8 times upsampling capability.
[0012] Furthermore, the training method of the fully convolutional network model includes:
[0013] Obtaining a set of images containing annotations of road assets and non-road assets;
[0014] Preprocess the image data;
[0015] Use the cross entropy loss function as the loss function used when training the fully convolutional network model;
[0016] Use SGD algorithm or Adam algorithm as the algorithm to optimize the parameters of the full convolutional network model;
[0017] The preprocessed image set is divided into a training set and a test set, and the training set is input into the fully convolutional network model for training to obtain a trained fully convolutional network model.
[0018] Furthermore, the preprocessing of the image set data includes:
[0019] Uniformly resize each image in the image set to the input size required for training the full convolutional network model;
[0020] Perform random cropping, random flipping, and random color jittering on the images in the image set;
[0021] Normalizes the pixel values in each image in the image set to a preset range.
[0022] Furthermore, when the training set is input into the fully convolutional network model for training, the weights of the encoder of the fully convolutional network model are initialized by the pre-trained classification network, the weights of the encoder are fixed, and only the decoder of the fully convolutional network model is trained; until the decoder part is trained.
[0023] Furthermore, the fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain a single asset model of each asset on the highway to be identified, including
[0024] Extracting features of highway asset image recognition data to obtain asset image features;
[0025] Extracting features of highway asset point cloud data to obtain asset point cloud features;
[0026] Calculate the attention weight between the asset image features and the asset point cloud features according to the similarity and correlation between the asset image features and the asset point cloud features;
[0027] Perform weighted fusion of asset image features and asset point cloud features according to attention weights;
[0028] Furthermore, the asset image features include shape, texture and color of the highway asset image.
[0029] Further, the asset point cloud features include geometric features of highway asset point cloud data;
[0030] The geometric features include the position, normal and curvature of the point.
[0031] A second aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.
[0032] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0033] Compared with the prior art, the present invention has the following advantages:
[0034] In the present invention, the present invention extracts the image recognition data of the highway asset to be identified and the point cloud data of the highway asset respectively, and obtains the individual asset model of each asset on the highway to be identified based on the fusion strategy of the attention mechanism; the fusion strategy of the attention mechanism continuously optimizes the attention weight and fusion method during the training process to improve the accuracy and robustness of the fusion result; a picture management system is realized, which can query, locate, count, analyze, etc. all the individual asset components to ensure the effective storage of all elements of information in highway operation. Automatically and accurately identify highway assets directly based on images and laser point clouds, and there is no need to obtain highway asset data from various operating units. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0036] Figure 1 A flowchart of a one-picture highway asset detection method based on images and laser point clouds of the present invention;
[0037] Figure 2 Flow chart of the training method of the fully convolutional network model of the present invention;
[0038] Figure 3 A flow chart of preprocessing image collection data according to the present invention;
[0039] Figure 4 A flowchart of obtaining a monomeric asset model of each asset on a road to be identified based on the attention mechanism of the present invention; DETAILED DESCRIPTION
[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0041] In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper", "lower", "inner", "back", etc. are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.
[0042] In addition, in the description of the present invention, unless otherwise clearly defined, the terms "installed", "connected", "connected", and "connector" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in combination with specific circumstances.
[0043] The following will refer to the attached Figures 1 to 4 The present invention is described in detail with reference to embodiments.
[0044] In general, this embodiment provides a method for detecting highway assets in one picture based on images and laser point clouds, and the method includes:
[0045] Step S1, obtaining image data and three-dimensional point cloud data of the road to be identified and both sides of the road to be identified;
[0046] Step S2: input the image data into the trained full convolutional network model to obtain image recognition data of the highway asset to be identified;
[0047] In this embodiment, the fully convolutional network model processes the input image data and outputs a matrix of the same size as the input image. The number of channels in the matrix is 2, corresponding to the predicted probabilities or scores of the highway asset category and the non-highway asset category, respectively. Each value in the matrix is compared with a preset threshold to determine the category of each pixel. Usually, the category with a higher probability (such as the category exceeding the threshold) is regarded as the final category of the pixel, thereby obtaining the highway asset image recognition data;
[0048] More specifically, the fully convolutional network model is a fully convolutional network model (Fully Convolutional Networks-8s, FCN-8s) with 8x upsampling capability.
[0049] More specifically, the training method of the fully convolutional network model includes:
[0050] Step S21, obtaining an image set containing highway assets and non-highway assets annotations;
[0051] Annotation means that each image should have a corresponding pixel-level label to indicate whether each pixel belongs to a road asset.
[0052] Step S22, preprocessing the image data;
[0053] The preprocessing of image data includes:
[0054] Step S221, uniformly adjusting each image in the image set to the input size required for full convolutional network model training;
[0055] Step S222, randomly cropping, randomly flipping, and randomly color dithering the images in the image set; randomly cropping, randomly flipping, and randomly color dithering the images can increase the diversity of the image set to improve the generalization ability of the full convolutional network model.
[0056] Step S223: normalize the pixel values in each image in the image set to a preset range, such as [0, 1] or [-1, 1], to facilitate model training.
[0057] In this embodiment, in the FCN-8s (Fully Convolutional Networks for Semantic Segmentation) model, score_fr, upscore2, fuse_pool4, upscore_pool4, fuse_pool3, score_pool4, score_pool4c, score_pool3, score_pool3c, upscore8, score: these layers can be mainly divided into several categories: feature extraction layer (such as pool3, pool4, etc.), upsampling layer (such as upscore2, upscore_pool4, etc.), fusion layer (such as fuse_pool3, fuse_pool4) and the final prediction layer (score).
[0058] For a binary classification task, you first need to modify the number of output channels of the score layer from the original number of categories (e.g. 20) to 2, which correspond to the predicted scores of road assets and non-road assets, respectively. To ensure that the number of output channels of all layers directly connected to the score layer (e.g. upscore8, fuse_pool3, etc.) is also adjusted accordingly to match the input requirements of the score layer. Therefore, before upsampling or fusing, additional convolutional layers need to be added to reduce or adjust the number of channels.
[0059] The feature extraction layer (such as pool3, pool4) is mainly responsible for extracting useful features from the input image; the number of output channels of the feature extraction layer generally does not need to be changed because they are used to extract more abstract feature representations rather than directly for classification.
[0060] Step S23, using the cross entropy loss function as the loss function used in training the full convolutional network model;
[0061] The loss function used by the FCN-8s model during training is usually the cross-entropy loss function, which measures the difference between the model's predictions and the true labels. In order to better optimize model performance, a weighted cross-entropy loss function is sometimes used to assign different weights to different types of pixels (such as foreground and background) to solve the problem of class imbalance.
[0062] Step S24, using a Stochastic Gradient Descent (SGD) algorithm or an Adaptive Moment Estimation (Adam) algorithm as an algorithm for optimizing the parameters of the full convolutional network model;
[0063] During the training process, optimizers suitable for deep learning tasks are usually selected, such as SGD, Adam, etc. These optimizers can update the network weights according to the gradient information to minimize the loss function.
[0064] Step S25: divide the preprocessed image set into a training set and a test set, and input the training set into the fully convolutional network model for training to obtain a trained fully convolutional network model.
[0065] When the training set is input into the full convolutional network model for training, the weight of the encoder of the full convolutional network model is initialized by the pre-trained classification network, the weight of the encoder is fixed, and only the decoder of the full convolutional network model is trained until the decoder part is trained; after the decoder part is trained, the full convolutional network model is adjusted. The FCN-8s model in the present invention adopts a staged training strategy, which can accelerate the training process and help the model converge to a better solution.
[0066] Step S3, classifying the point cloud data according to the elevation and lateral position information in the three-dimensional point cloud data and based on the alphashapes algorithm, and filtering out the highway asset point cloud data;
[0067] Step S4: The highway asset image recognition data and the highway asset point cloud data are fused and processed based on the fusion strategy of the attention mechanism; and a monomerized asset model of each asset on the highway to be identified is obtained;
[0068] After obtaining the individual asset models of each asset on the road to be identified, the individual asset models of each asset on the road are attribute-encoded and identity-encoded to uniquely represent the constructed identity, which can be used as a primary key for query and identification later.
[0069] The fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain the individual asset models of each asset on the highway to be identified, including:
[0070] Step S41, extracting features of highway asset image recognition data to obtain asset image features;
[0071] Step S42, extracting features of the highway asset point cloud data to obtain asset point cloud features;
[0072] Step S43: Calculate the attention weight between the asset image features and the asset point cloud features according to the similarity and correlation between the asset image features and the asset point cloud features;
[0073] Step S44: weighted fusion of asset image features and asset point cloud features according to attention weights;
[0074] In this implementation, the method for calculating the attention weight between asset image features and asset point cloud features based on the similarity and correlation between the asset image features and the asset point cloud features includes: scoring according to the similarity and correlation between the asset image features and the asset point cloud features; obtaining the similarity and correlation scores between the asset image features and the asset point cloud features through methods such as vector dot product, cosine similarity, neural network, etc.; and normalizing these scores to obtain the final attention weight.
[0075] In this embodiment, weighted fusion can be achieved by simple weighted summation, feature concatenation or more complex fusion methods. In the fusion process, a multi-layer fusion strategy can be considered, that is, multiple fusions are performed at different levels of the network to make full use of feature information at different levels. The attention weights and fusion methods are continuously optimized through the training process to improve the accuracy and robustness of the fusion results. An appropriate loss function can be used to evaluate the quality of the fusion results, and the parameters of the network can be updated through the back propagation algorithm.
[0076] Asset image features include shape, texture, and color of highway asset images.
[0077] In this embodiment, key features are extracted from the highway asset image recognition data, including shape, texture, and color. These features can be automatically extracted using a pre-trained convolutional neural network (CNN) or other deep learning models.
[0078] The asset point cloud features include geometric features of highway asset point cloud data;
[0079] Geometric features include the position, normal, and curvature of a point.
[0080] In this embodiment, geometric features are extracted from the highway asset point cloud data, such as point positions, normals, curvatures, etc. Point cloud processing algorithms (such as PointNet, PointNet++, etc.) can be used to automatically extract these features.
[0081] Step S5: Obtain Geographic Information System (GIS) data of the road to be identified, and fuse the individual asset models of each asset of the road to be identified with the GIS data of the road to be identified to construct a real three-dimensional scene.
[0082] The present invention extracts the image recognition data of the highway assets to be identified and the point cloud data of the highway assets respectively, and obtains the individual asset model of each asset on the highway to be identified based on the fusion strategy of the attention mechanism; the fusion strategy of the attention mechanism continuously optimizes the attention weight and fusion method during the training process to improve the accuracy and robustness of the fusion result; a one-picture management system is implemented, which can query, locate, count, analyze, etc. all the individual asset components, ensuring the effective storage of all-factor information in highway operation. Highway assets are directly identified based on images and laser point clouds automatically and accurately, without the need to obtain highway asset data from various operating units.
[0083] In the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.
[0084] It should also be noted that, in the description of the present invention, relational terms such as first and second, etc., are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0085] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the embodiments here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.
Claims
1. A highway asset detection method based on images and laser point clouds, characterized by: The method comprises: Acquire image data and three-dimensional point cloud data of the road to be identified and both sides of the road to be identified; Inputting the image data into the trained full convolutional network model to obtain image recognition data of the highway asset to be identified; According to the elevation and lateral position information in the 3D point cloud data and based on the alphashapes algorithm, the point cloud data is classified to filter out the highway asset point cloud data; The fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain the individual asset model of each asset on the highway to be identified; The GIS data of the road to be identified is obtained, and the individual asset models of each asset of the road to be identified are merged with the GIS data of the road to be identified to construct a real three-dimensional scene.
2. The one-picture highway asset detection method based on image and laser point cloud according to claim 1, characterized in that: The fully convolutional network model is a fully convolutional network model with 8 times upsampling capability.
3. The one-picture highway asset detection method based on image and laser point cloud according to claim 1, characterized in that: The training method of the fully convolutional network model includes: Obtaining a set of images containing annotations of road assets and non-road assets; Preprocess the image data; Use the cross entropy loss function as the loss function used when training the fully convolutional network model; Use SGD algorithm or Adam algorithm as the algorithm to optimize the parameters of the full convolutional network model; The preprocessed image set is divided into a training set and a test set, and the training set is input into the fully convolutional network model for training to obtain a trained fully convolutional network model.
4. The one-picture highway asset detection method based on image and laser point cloud according to claim 3 is characterized by: The preprocessing of the image set data includes: Uniformly resize each image in the image set to the input size required for training the full convolutional network model; Perform random cropping, random flipping, and random color jittering on the images in the image set; Normalizes the pixel values in each image in the image set to a preset range.
5. The one-picture highway asset detection method based on image and laser point cloud according to claim 3 is characterized by: When the training set is input into the full convolutional network model for training, the weights of the encoder of the full convolutional network model are initialized by the pre-trained classification network, the weights of the encoder are fixed, and only the decoder of the full convolutional network model is trained until the decoder part is trained.
6. The one-picture highway asset detection method based on image and laser point cloud according to claim 1, characterized in that: The fusion strategy based on the attention mechanism fuses the highway asset image recognition data and the highway asset point cloud data to obtain the individual asset models of each asset on the highway to be identified, including Extracting features of highway asset image recognition data to obtain asset image features; Extracting features of highway asset point cloud data to obtain asset point cloud features; Calculate the attention weight between the asset image features and the asset point cloud features according to the similarity and correlation between the asset image features and the asset point cloud features; The asset image features and asset point cloud features are weightedly fused according to the attention weights.
7. The one-picture highway asset detection method based on image and laser point cloud according to claim 6 is characterized by: The asset image features include the shape, texture and color of the highway asset image.
8. The one-picture highway asset detection method based on image and laser point cloud according to claim 1 is characterized by: The asset point cloud features include geometric features of highway asset point cloud data; The geometric features include the position, normal and curvature of the point.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Geographic element identification method fusing remote sensing image and point cloud data
CN116258970A
Road asset information identification method and system
CN116758259A
Traffic sign checking method and system based on multi-source fusion data
CN117746395A
Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics
US20150269438A1
Method and system for asset inspection using unmanned aerial vehicles
US20220083774A1
Cited By
Road asset automatic identification and modeling system based on laser point cloud
CN119538384A
Road asset checking method and system based on three-dimensional Gaussian splashing
CN119940745A
A method and system for road asset inventory based on three-dimensional Gaussian splashing
CN119940745B
