Railway point cloud-oriented active learning weak supervision semantic segmentation method and system

Abstract

The invention provides a railway point cloud-oriented active learning weak supervision semantic segmentation method and system, and the method comprises the steps: obtaining a training data set of a semantic segmentation network model, and initializing the training data set into marked pool data and unmarked pool data; constructing a semantic segmentation network model; performing iterative training on the semantic segmentation network model by using the marked pool data based on an active learning strategy, updating the marked pool data by using the unmarked pool data based on a cyclic output difference and an information entropy theory in the training process, and outputting the trained semantic segmentation network model; and inputting a to-be-processed railway scene point cloud to the trained semantic segmentation network model, and outputting a railway point cloud semantic segmentation result.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of railway scene point cloud data processing, and particularly relates to an active learning weakly supervised semantic segmentation method and system for railway point clouds. BACKGROUND

[0002] A laser radar system carried on a flight platform (such as an airplane or a drone) can quickly obtain large-range ground point cloud data. By performing semantic segmentation on the airborne point cloud, the category attribute of each point can be determined, which can provide important support for downstream applications (such as target ground object extraction, change detection, planning and design, etc.). With the continuous development of deep learning technology, point cloud processing methods based on deep learning have gradually replaced traditional heuristic methods and become the mainstream of current research. Fully supervised deep learning methods usually require a fully labeled data set. A large number of studies have shown that, under sufficient training data, deep learning-based methods can achieve satisfactory semantic segmentation results. However, constructing a high-quality labeled data set requires a large amount of manpower and time cost, and in practical applications, the data set often has the problem of sample imbalance. Therefore, weakly supervised learning methods have gradually become a research hotspot, which reduces the dependence on a large amount of labeled data, reduces the training cost of deep learning, and improves the feasibility and efficiency in practical applications.

[0003] Existing weakly supervised learning methods face various challenges in semantic segmentation of railway scenes. The SQN method proposes a method of generating point-level labels by randomly labeling sparse points, but on large-scale data sets, labeling even 0.1% of the points will bring a high burden, and random labeling exacerbates the class imbalance problem, affecting the model's learning of key facilities. The MPRM method proposes a sub-cloud level weakly supervised labeling method, which can generate sub-cloud data from the scene, but generates a large number of content-repeated sub-clouds, resulting in redundant labeling and difficulty in providing high-quality training data. The OCOC method Based on active learning to reduce the amount of labeled data, however, it fails to effectively solve the sample balance problem, and the labeling method considering only the class center cannot provide representative samples of the class boundary, and cannot fully cope with the complex distribution of facilities and the interweaving relationship between classes in the railway scene. SUMMARY

[0004] In order to overcome the problems of time-consuming and high cost of labeling, limited performance of weak supervision method for railway point cloud semantic segmentation task in the prior art, the present application provides a railway point cloud-oriented active learning weak supervision semantic segmentation method and system. The geometric features are introduced while extracting the point cloud features of the railway scene to meet the particularity of the railway scene. The loss function increases the prototype loss to improve the learning ability of the network model for repeated facilities in the scene and improve the performance of the semantic segmentation model. In the weak supervision environment, the semantic segmentation model is iteratively trained based on the active learning strategy, and the most representative unlabeled sub-cloud samples are dynamically selected based on the cycle output difference and information entropy theory for labeling and training. Only a small amount of points in the massive railway point cloud needs to be labeled, and the semantic segmentation of the whole scene can be realized, reducing the dependence of the deep learning semantic segmentation model on large-scale artificial labeled data, reducing the labeling cost while gradually improving the semantic segmentation performance of the model for the railway point cloud.

[0005] According to an aspect of the present application, a railway point cloud-oriented active learning weak supervision semantic segmentation method is provided, comprising:

[0006] Obtain the training data set of the semantic segmentation network model, and initialize it as labeled pool data and unlabeled pool data;

[0007] Construct a semantic segmentation network model, wherein the main network of the model adopts a KPConv network, and a geometric feature extraction module and a feature fusion module are introduced based on the original encoder framework of the main network. The original encoder extracts the convolutional features of the input point cloud, the geometric feature extraction module extracts the geometric features of the points in the input point cloud, and the feature fusion module obtains the fusion features based on the input point cloud, its convolutional features and the geometric features of the points. The fusion features are input into the original decoder structure, and the point-level semantic prediction results are output;

[0008] Based on the active learning strategy, the labeled pool data is used to iteratively train the semantic segmentation network model. During the training process, the unlabeled pool data is used to update the labeled pool data based on the cycle output difference and the information entropy theory, and the trained semantic segmentation network model is output;

[0009] The railway scene point cloud to be processed is input into the trained semantic segmentation network model, and the railway point cloud semantic segmentation result is output.

[0010] As a further embodiment, the training data set is composed of point clouds in the railway scene, and the initialization process is:

[0011] For each railway scene, a plurality of seed points are randomly selected and all points in a circular region with a preset radius around the seed points are collected to form a sub-cloud, for each sub-cloud, a corresponding point is selected as a semantic internal point for each prediction category and a label is added, the label pool data is initialized using the labels of the sub-cloud and the semantic internal point, and all remaining data in the training dataset is used as unlabeled pool data.

[0012] As a further implementation, the geometric feature extraction module is configured to extract geometric features of the points, and the extraction method comprises:

[0013] For each point in the input point cloud, based on a plurality of points in its local neighborhood, a covariance matrix is constructed based on the three-dimensional coordinates of the point cloud, the eigenvalues of the covariance matrix are extracted, and a geometric distribution index is calculated according to the eigenvalues.

[0014] As a further implementation, the extraction method further comprises: calculating the elevation range and the elevation variance as additional attributes according to the absolute elevation value of each point in the neighbor points, and integrating the geometric distribution index and the additional attributes as the geometric features of the extracted points.

[0015] As a further implementation, the feature fusion module processes the input geometric features to the same dimension as the convolution features extracted by the original encoder of the input main network, and then splices the two, and finally processes through a multi-layer perceptron and splices the original input to obtain the fusion features.

[0016] As a further implementation, the updating process of the labeled pool data comprises:

[0017] After each round of iterative training is completed, the unlabeled pool data is input into the model at this time, the cycle output difference value of each point in the unlabeled pool data is calculated based on the prediction value output by the model, and the high-loss region point cloud is extracted according to the calculation result;

[0018] For each high-loss region point cloud, high-uncertain samples are screened based on the maximum entropy value principle and labeled, and semantic internal points of each prediction category are selected and labeled;

[0019] The high-loss region point cloud, the high-uncertain sample labels and the semantic internal point labels are extracted to the labeled pool data, and the updating of the labeled pool data is completed.

[0020] As a further implementation, the loss function of the semantic segmentation network model comprises a point-level cross-entropy loss, a scene-level loss and a class prototype loss.

[0021] According to another aspect of the present specification, an active learning weakly supervised semantic segmentation system for railway point clouds is provided, comprising:

[0022] The training data collection module is configured to collect a training data set of the semantic segmentation network model, and initialize the training data set as labeled pool data and unlabeled pool data.

[0023] The semantic segmentation network model construction module is configured to construct a semantic segmentation network model, wherein the model main network adopts a KPConv network, a geometric feature extraction module and a feature fusion module are introduced based on an original encoder framework of the main network, the original encoder extracts convolution features of input point clouds, the geometric feature extraction module extracts geometric features of points in the input point clouds, the feature fusion module obtains fusion features based on the input point clouds, convolution features of the input point clouds and the geometric features of the points, and the fusion features are input into an original decoder structure to output point-level semantic prediction results.

[0024] The active learning training module is configured to iteratively train the semantic segmentation network model using the labeled pool data based on an active learning strategy, update the labeled pool data using the unlabeled pool data based on a cycle output difference and information entropy theory during the training process, and output the trained semantic segmentation network model.

[0025] The railway point cloud semantic segmentation module is configured to input a railway scene point cloud to be processed into the trained semantic segmentation network model, and output a railway point cloud semantic segmentation result.

[0026] According to another aspect of the present specification, an electronic device is provided, including a memory and a processor, the memory stores program instructions executed by the processor, and the processor invokes the program instructions to execute a railway point cloud-oriented active learning weakly supervised semantic segmentation method.

[0027] According to another aspect of the present specification, a non-transitory computer readable storage medium is provided, which stores computer instructions, and the computer instructions cause the computer to execute a railway point cloud-oriented active learning weakly supervised semantic segmentation method.

[0028] Compared with the prior art, the present application has the following beneficial effects: the present application introduces geometric features while extracting point cloud features of a railway scene to meet the particularity of the railway scene, adds a prototype loss to a loss function to improve the learning ability of the network model for repeated facilities in the scene and improve the performance of the semantic segmentation model; in a weakly supervised environment, the semantic segmentation model is iteratively trained based on an active learning strategy, and based on a cycle output difference and information entropy theory, the most representative unlabeled sub-cloud samples are dynamically selected for labeling and training, only a small number of points in a large amount of railway point clouds need to be labeled, and the semantic segmentation of the whole scene can be realized, the dependence of a deep learning semantic segmentation model on large-scale artificial labeled data is reduced, the labeling cost is reduced while gradually improving the semantic segmentation performance of the model for railway point clouds, the present application can be applied to various types of railway scenes, and has no special requirements for point cloud density and class balance, and is universal. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the field, other drawings can be obtained based on these drawings without any creative effort.

[0030] Figure 1 A process schematic diagram of a railway point cloud-oriented active learning weakly supervised semantic segmentation method provided by the embodiments of the present application is shown in the figure.

[0031] Figure 2 A railway scene sub-cloud initialization schematic diagram provided by the embodiments of the present application is shown in the figure.

[0032] Figure 3 A schematic diagram of fusing geometric features inside a network model encoder provided by the embodiments of the present application is shown in the figure.

[0033] Figure 4 A process schematic diagram of selecting a sub-cloud according to a cycle output difference provided by the embodiments of the present application is shown in the figure.

[0034] Figure 5 A schematic diagram of an active selection process of sample points inside a sub-cloud provided by the embodiments of the present application is shown in the figure.

[0035] Figure 6 An iterative training schematic diagram based on active learning provided by the embodiments of the present application is shown in the figure.

[0036] Figure 7 An entropy value point effect diagram taking a rail as an example provided by the embodiments of the present application is shown in the figure.

[0037] Figure 8 A structure schematic diagram of a railway point cloud-oriented active learning weakly supervised semantic segmentation system provided by the embodiments of the present application is shown in the figure.

[0038] Figure 9 A structure schematic diagram of an electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0039] It should be noted that:

[0040] The terms "comprising" and "having" and any variations thereof in the specification and claims and the above drawings are intended to cover not exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units, not necessarily limited to which steps or units are clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0041] The block diagrams shown in the drawings are merely functional entities, and do not necessarily correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to actual conditions.

[0042] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other at will to form new technical solutions, and such combination is not restricted by the order of steps and / or structure composition mode, but must be based on the implementation by those of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be implemented, it should be considered that such combination of technical solutions does not exist, nor is it within the scope of protection required by the present application.

[0043] The innovation of the present application is that the semantic segmentation network is trained in a progressive manner based on an active learning method, and in each iteration period, a representative high sample is selected for labeling and updating the training data in the label pool based on a joint strategy of high loss area and high uncertainty sampling. In the design of network structure, the significant geometric structure of railway scene and the characteristics of repeated occurrence of railway facilities are considered. The present application can significantly reduce the number of labeled points required for railway point cloud semantic segmentation, while improving the performance of semantic segmentation.

[0044] As shown in Figure 1 A railway point cloud-oriented active learning weakly supervised semantic segmentation method includes:

[0045] Step 1: Obtain the training data set of the semantic segmentation network model, and initialize it as the labeled pool data and unlabeled pool data.

[0046] Step two, a semantic segmentation network model is constructed, the main network of the model adopts a KPConv network, a geometric feature extraction module and a feature fusion module are introduced based on the original encoder framework of the main network, the original encoder extracts the convolutional features of the input point cloud, the geometric feature extraction module extracts the geometric features of the points in the input point cloud, the feature fusion module obtains the fusion features based on the input point cloud, the convolutional features of the input point cloud and the geometric features of the points, and inputs the fusion features into the original decoder structure to output the point-level semantic prediction results;

[0047] Step three, based on the active learning strategy, the labeled pool data is used to iteratively train the semantic segmentation network model, and during the training process, the unlabeled pool data is used to update the labeled pool data based on the cycle output difference and the information entropy theory, and the trained semantic segmentation network model is output.

[0048] Step four, input the railway scene point cloud to be processed into the trained semantic segmentation network model, and output the railway point cloud semantic segmentation result.

[0049] Further, the training data set in step one is composed of point clouds in the railway scene, and the initialization process is as follows:

[0050] For each railway scene, a plurality of seed points are randomly selected, and all points in the circular region with a predetermined radius around the seed points are collected to form a plurality of sub-clouds, the semantic internal points in the sub-clouds are selected and labeled, the labeled pool data is initialized by using the labels of all sub-clouds and semantic internal points, and the remaining data in the training data set is used as the unlabeled pool data.

[0051] Specifically, the initialization process of the training data set in step one is as follows:

[0052] Traverse the railway scene, randomly select a plurality of seed points in each railway scene, for each seed point, collect all points in the circular region with a predetermined radius in the XOY plane to form a sub-cloud, for each point sub-cloud, select one point corresponding to each predicted category as a semantic internal point according to the predicted category and add a label, initialize the labeled pool data by using the labels of the sub-clouds and the semantic internal points, and initialize the unlabeled pool data by using the remaining data in the training data set.

[0053] Further, in step two, the semantic segmentation network model main network adopts KPConv network, and a geometric feature extraction module and a feature fusion module are introduced on the basis of the original encoder framework to enhance the expression ability of the geometric information of the point cloud data. The network input is a point cloud, and the encoder of the KPConv network and the geometric feature extraction module perform multi-layer down-sampling on the input data and extract convolutional features and point geometric features, respectively. The feature fusion module fuses the extracted convolutional features and point geometric features with the input point cloud, and takes the fused features as the output of the encoder architecture. The decoder receives the fused features, recovers the point number through layer-by-layer up-sampling, learns the semantic features of each point, and finally outputs the point-level semantic prediction result.

[0054] Further, in step two, the geometric feature extraction module is used to extract the geometric features of the points, and the extraction method is as follows:

[0055] For each point in the input point cloud, based on a plurality of points in the local neighborhood of the point, a covariance matrix is constructed based on the three-dimensional coordinates of the point cloud, the eigenvalues of the covariance matrix are extracted, and a geometric distribution index is calculated according to the eigenvalues.

[0056] Further, the extraction method in step two further includes: calculating the elevation range and the elevation variance of each point in the neighbor points as additional attributes according to the absolute elevation value, and integrating the geometric distribution index and the additional attributes as the geometric features of the extracted points.

[0057] As a preferred embodiment, the geometric feature extraction module in step two is designed for typical geometric structure features in the railway scene, such as the linear structure of the catenary, the vertical structure of the support column, and the smooth structure of the track.

[0058] As a preferred embodiment, the geometric distribution index in step two can include a plurality of different indexes, such as neighborhood linearity, planarity, sphericity, total variance, anisotropy, and curvature variation.

[0059] Further, in step two, the feature fusion module upgrades the input geometric features to the same dimension as the convolutional features, then splices them, and then processes them through a plurality of multilayer perceptrons and splices them with the original input to obtain the fused features.

[0060] Specifically, the formula of the fused features is as follows:

[0061]

[0062] In the formula, represents the fused features; represents the convolutional features output by the encoder of the KPConv network; represents the geometric features; represents the features of the original input; denotes a geometry feature projected to the same channel dimension, denotes a feature concatenation operation, and MLP denotes a multi-layer perception module.

[0063] Further, the updating process of the labeled pool data in step three is as follows:

[0064] After each round of iterative training, the unlabeled pool data is input into the model at this time, the cycle output difference value of each point in the unlabeled pool data is calculated based on the predicted value output by the model, and the high-loss area point cloud is extracted according to the calculation result;

[0065] For each high-loss area point cloud, high-uncertain samples are screened and labeled based on the maximum entropy value principle, and semantic internal points of each predicted category are selected and labeled;

[0066] The high-loss area point cloud, the high-uncertain sample label and the semantic internal point label are extracted to the labeled pool data, and the updating of the labeled pool data is completed.

[0067] Specifically, the step of extracting the high-loss area point cloud is as follows: after each round of iterative training, the unlabeled pool data is input into the model at this time, the predicted value output by the model is recorded, the cycle output difference value of each point in the unlabeled pool data is calculated based on the predicted value output by the model, and a preset number of points with the maximum cycle output difference value are selected as new seed points. For each new seed point, all points in a circular area with a preset radius in the XOY plane are collected to form a new high-loss area point cloud;

[0068] The step of adding labels is as follows: for each high-loss area point cloud, the prediction probability of each point distributed in each predicted category is obtained to calculate the entropy value, and the entropy value maximum point in each predicted category is selected as a high-uncertain sample and labeled based on the maximum entropy value principle, and the semantic internal points of each predicted category are selected and labeled;

[0069] Further, the loss function of the semantic segmentation network model includes a point-level cross-entropy loss, a scene-level loss and a class prototype loss, and the network parameters are optimized through back propagation.

[0070] In order to reduce the dependence of railway point cloud semantic segmentation on labels and achieve higher performance of point cloud segmentation results, the present application realizes railway point cloud weakly supervised semantic segmentation based on an active learning method. In addition, the present application also provides an example of railway point cloud weakly supervised semantic segmentation based on an active learning method, mainly including four parts of labeled pool initialization and updating, model training, sample selection and iterative optimization.

[0071] Step 1, labeled pool initialization and updating.

[0072] ​The training data set is composed of point clouds in a railway scene, which is divided into labeled pool data and unlabeled pool data in the training process, wherein the labeled pool data is used for training the semantic segmentation network model, and the unlabeled pool is used for updating the labeled pool. In the initialization stage, part of the points are randomly selected as seed points, and the points around the seed points are sampled as sub-clouds. A human annotator labels a representative sample point as a semantic internal point for each distinguishable class in the sub-cloud. In the updating stage, according to the program prompt, the sub-cloud data to be added is labeled.

[0073] Specifically, taking the ExpressRail vehicle-mounted point cloud data set as an example, the data set is about 400 meters long per scene. In the initialization stage, 30 seed points are randomly selected for each scene, and points within a 5-meter radius centered on the seed points are sampled as a sub-cloud in the XOY plane, as shown in Figure 2 For each sub-cloud, a human annotator arbitrarily labels a point for each predicted class. After completing the labeling, the sub-cloud and the limited point label are put into the labeled pool, ready for the first model training.

[0074] Step 2, model training.

[0075] The semantic segmentation network model is trained or fine-tuned using the data in the labeled pool to improve the accuracy of semantic segmentation. The main network of the semantic segmentation network model uses the KPConv network, and a geometric feature extraction module and a feature fusion module are introduced based on the original encoder framework. In view of the particularity of the railway scene, the geometric features are fused in the encoder of the semantic segmentation network structure. In the loss function design, the prototype loss is increased to improve the learning ability of the network model for repeated facilities in the scene.

[0076] Covariance features based on point cloud three-dimensional coordinates are introduced in the encoder stage. Specifically, for each point, a covariance matrix is constructed based on a number of points in its local neighborhood, and the eigenvalues are extracted and the indicators describing the geometric distribution are calculated, including: neighborhood linearity , flatness , sphericity , total variance , anisotropy and curvature change .

[0077] Specifically, the calculation process of the indicators describing the geometric distribution is as follows:

[0078] Let denote the three eigenvalues of the covariance matrix constructed by the neighborhood point coordinates, and the calculation formulas of the indicators describing the geometric distribution are as follows: neighborhood linearity : ; flatness : ; sphericity : ; total variance : ; anisotropy : ; curvature variation : .

[0079] In addition to the above features, two additional attributes, elevation range and elevation variance , are calculated based on the absolute elevation values of each point in the local neighborhood, where the formula for calculating the elevation range is:

[0080]

[0081] where and represent the maximum and minimum absolute elevation values of the points in the local neighborhood, respectively.

[0082] Final geometric features are defined as an 8-dimensional feature vector, denoted as:

[0083] .

[0084] In the network feature fusion stage, see Figure 3 , first calculate the convolutional features of the points based on the KPConv convolution kernel , then project the geometric features to the same dimension as by a linear projection module, concatenate the two, pass them through a multi-layer perceptron, and concatenate them with the original input features , as the output feature of the encoder, which is expressed as follows:

[0085]

[0086] where represents projecting the geometric features to the same channel dimension as , represents the feature concatenation operation, and MLP represents the multi-layer perceptron module.

[0087] Step 3, sample selection.

[0088] After completing each iteration cycle of active learning, select sub-cloud data from the high-loss area. According to the prediction probability of each point in the sub-cloud in each category, select points with high uncertainty based on information entropy theory, and hand them over to human annotators for labeling.

[0089] Specifically, the following steps are included:

[0090] (1) Selecting high-loss areas

[0091] After completing the network training of each cycle of active learning, based on the cyclic output discrepancy (COD), a preset number of points with the maximum cyclic output discrepancy value are selected as new seed points, and for each new seed point, all points in the circular area with a preset radius in the XOY plane form a new high-loss area point cloud; the COD value of a sample point p can be expressed as:

[0092]

[0093] In the formula, represents the prediction value of the dth iteration training, represents the prediction value of the (d-1)th iteration training,

[0094] Therefore, the COD value is expressed as the sum of the squares of the differences between the prediction outputs of adjacent two iteration trainings, b points with the maximum COD are selected as seed points, and points within a radius of 5 meters are collected as new sub-cloud data, see Figure 4 .

[0095] (2) Selecting high-uncertainty samples

[0096] According to the prediction probability of each point in the sub-cloud distributed in different categories, the entropy value of the point is calculated to represent the uncertainty of the point. Following the maximum entropy value principle, a point with the maximum entropy value in each prediction category is selected as a high-uncertainty sample and is labeled by an annotator. The calculation formula of the entropy value is as follows:

[0097]

[0098] In the formula, represents the probability of the point being predicted as category , and C represents the total number of categories.

[0099] (3) Manual labeling

[0100] For the newly selected high-loss area point cloud, an entropy value maximum point (high-uncertainty point) and a semantic internal point are labeled by a manual annotator. The active selection process of the points to be labeled in the sub-cloud is shown in Figure 5 .

[0101] (4) Updating the labeled pool

[0102] The newly selected high-loss area sub-cloud is extracted from the unlabeled pool without replacement, and the label obtained by manual labeling is added to the labeled pool to update the training data.

[0103] Step 4, iterative optimization.

[0104] After updating the labeled pool data, the next iteration training cycle begins. The sample selection and model update steps are repeated until the preset accuracy or number of iterations is met, achieving a gradual improvement in model performance.

[0105] In this implementation example, a total of 5 iterations of training were set. After each iteration cycle, predictions were made on the unlabeled data in a loop, and seed points were selected based on the differences in the loop predictions. Then, points within a 5-meter radius of the seed points were sampled as high-loss region sub-clouds. Based on the maximum entropy principle, high-uncertainty samples were actively selected and handed over to human annotators for labeling. The newly added point clouds and labels were added to the label pool data, and the label pool data was updated. The semantic segmentation model was then trained again using the label pool data. The above steps were repeated until the set number of iterations was met, resulting in the final network model used for railway point cloud semantic segmentation. The iterative learning process is described in [link to documentation]. Figure 6 .

[0106] Specifically, in designing the loss function, in addition to using point-level cross-entropy loss and scene-level loss, a class prototype loss is added to improve the network model's ability to learn repetitive facilities in the scene. Class Prototype In the feature space, it is represented as the class centroid, which is continuously learned and updated using a moving average during iteration:

[0107]

[0108] In the formula, Point The predicted probability distribution, where h is a linear mapping with vector normalization. To prevent gradient backpropagation, this is used to freeze gradient updates for the class prototype. It is a point The tag, This represents the number of points with label 'c' in the batch of data. This is the momentum coefficient. The value is usually set between 0.7 and 0.99 to appropriately incorporate the feature distribution of new samples while maintaining the stability of historical information.

[0109] Prototype loss represents prediction bias by measuring the cosine similarity between predicted and prototype values. For each entropy point and manual point in each batch, the prototype loss... Represented as:

[0110]

[0111] In the formula, denotes the cosine similarity between the model’s predicted output and the corresponding class prototype, and τ is a hyperparameter, denotes the cosine similarity between the model’s predicted output and the corresponding class prototype, and τ is a hyperparameter, aims to learn more discriminative embedding features by enhancing intra-class compactness and inter-class separation.

[0112] Step 5, Experimental results and analysis.

[0113] The experimental results of different methods are shown in Table 1, including three fully supervised semantic segmentation methods: PointNet++, KPConv, and RandLA-Net, and three weakly supervised methods: SQN, PSD, and OCOC. All methods are evaluated on the same dataset, and the predicted classes include: Rail (rail), Support (support arm), Pillar (contact net support column), Overhead (contact net), Fence (fence), Bed (track bed), Veget (vegetation), Ground (ground), and Others (others). A consistent input format, batch division, and data augmentation strategy is used. In terms of model configuration, the default training parameters and hierarchical structure of the KPConv network are used. The proposed weakly supervised semantic segmentation method is implemented in the PyTorch framework.

[0114] The final results show that the proposed method achieves the best performance in both mIoU (Mean Intersection over Union) and OA (Overall Accuracy) with only about 0.06‰ of point labels.

[0115] Table 1 Experimental results of different methods on the ExpressRail dataset

[0116]

[0117] Further, an ablation experiment on entropy points was conducted on the current embodiment. Table 2 shows the semantic segmentation results with and without entropy points. It can be seen that the addition of entropy points improves the overall segmentation accuracy, especially for major railway facilities such as rail (Rail), support arm (Support), and support column (Pillar). For fence (Fence) and vegetation (Vegetation) categories, the segmentation accuracy decreases slightly, mainly because the uncertainty sampling method based on entropy tends to focus on the boundary region, and these two types of targets have more blurred and overlapping boundaries in the point cloud, which can easily cause recognition confusion. However, considering the lower importance of these two categories in the railway scene, this result is acceptable in practical applications.

[0118] Table Prediction results without entropy maximum points

[0119]

[0120] To further intuitively show the advantages of the method, the qualitative segmentation results are given for the steel rail as an example, see Figure 7 It can be seen that the introduction of the entropy value point significantly improves the segmentation effect of the steel rail boundary and effectively reduces the misclassification phenomenon in the boundary area.

[0121] The implementation basis of each embodiment of the present application is realized by the programmed processing of a system with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of the above-mentioned embodiments, the embodiments of the present application provide an active learning weakly supervised semantic segmentation system for railway point clouds, which is used to execute one of the active learning weakly supervised semantic segmentation methods for railway point clouds in the above-mentioned method embodiments.

[0122] Referring to Figure 8 , the system comprises:

[0123] A training data acquisition module is configured to acquire a training data set of a semantic segmentation network model, and initialize the training data set as labeled pool data and unlabeled pool data. A semantic segmentation network model construction module is configured to construct a semantic segmentation network model, wherein the model main network adopts a KPConv network, and a geometric feature extraction module and a feature fusion module are introduced on the basis of the original encoder framework of the main network. The original encoder extracts convolutional features of input point clouds, the geometric feature extraction module extracts geometric features of points in the input point clouds, and the feature fusion module obtains fusion features based on the input point clouds, the convolutional features of the input point clouds and the geometric features of the points. The fusion features are input into the original decoder structure, and point-level semantic prediction results are output. An active learning training module is configured to use the labeled pool data to iteratively train the semantic segmentation network model based on an active learning strategy. In the training process, the unlabeled pool data is used to update the labeled pool data based on the cycle output difference and the information entropy theory, and a trained semantic segmentation network model is output. A railway point cloud semantic segmentation module is configured to input a railway scene point cloud to be processed into the trained semantic segmentation network model, and output a railway point cloud semantic segmentation result.

[0124] It should be noted that the system embodiments provided by the present application are used to implement the methods in the above method embodiments, and are also used to implement the methods in other method embodiments provided by the present application, the difference is only that the corresponding function modules are set, and the principle is basically the same as that of the above system embodiments provided by the present application, as long as the person skilled in the art improves the system in the above system embodiments on the basis of the above system embodiments, refers to the specific technical solutions in other method embodiments, obtains the corresponding technical means by combining technical features, and the technical solutions composed of these technical means, on the premise of ensuring the practicability of the technical solutions, the corresponding system class embodiments are obtained by improving the system in the above system embodiments, and are used to implement the methods in other method class embodiments.

[0125] The method of the embodiment of the present application is realized by relying on an electronic device, and therefore it is necessary to introduce the related electronic device. For this purpose, the embodiment of the present application provides an electronic device, as shown in the figure, which comprises at least one processor, a communications interface, at least one memory and a communications bus, wherein the at least one processor, the communications interface and the at least one memory complete the communication among each other through the communications bus. The at least one processor calls the logic instructions in the at least one memory to execute all or part of the steps of the method provided by the above various method embodiments. Figure 9

[0126] In addition, the logic instructions in the at least one memory described above are realized in the form of a software function unit and sold or used as an independent product when the logic instructions in the at least one memory described above are realized in the form of a software function unit and sold or used as an independent product, stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the part of the technical solutions in the form of a software product is embodied, and the computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (a personal computer, a server or a network device) execute all or part of the steps of the method described in the various method embodiments of the present application. The storage medium described above includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk various storage program code medium.

[0127] ​The embodiments of systems described above are only illustrative, where the components that are described as separate means or not, are or also not physically separated, the components that are shown as units are or also not physical units, located in one place, or also distributed to multiple network units. According to the actual selection of part or all of the modules, the purpose of the embodiment scheme is realized. Those skilled in the art can understand and implement without creative labor.

[0128] Those skilled in the art will appreciate that embodiments of the application can be a method, a system, or a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0129] The application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions that are executed by the processor of the computer or other programmable data processing apparatus generate means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The flow or multiple flows and / or blocks Figure 1 The means for performing the functions specified in the flow or multiple flows and / or blocks.

[0130] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The flow or multiple flows and / or blocks Figure 1 The means for performing the functions specified in the flow or multiple flows and / or blocks.

[0131] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The flow or multiple flows and / or blocks Figure 1 The means for performing the functions specified in the flow or multiple flows and / or blocks.

[0132] To sum up, the method firstly selects a sub-cloud sample from the original point cloud data set randomly, which is used to construct an initial label set and train an initial model. In the subsequent iterative training process, a method combining active learning strategy, cycle output difference and information entropy theory is adopted to construct a prediction loss and uncertainty joint evaluation index, to actively select representative samples from unmarked data for manual annotation, and to update the label pool. Through continuous iterative training, the performance of the model in the semantic segmentation task is gradually improved.

[0133] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A method for active learning weakly supervised semantic segmentation for railway point clouds, characterized in that, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

2. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 1, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

3. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 1, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

4. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 3, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

5. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 1, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

6. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 1, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

7. The railway point cloud oriented active learning weakly supervised semantic segmentation method of claim 1, wherein, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device.

8. An active learning weakly supervised semantic segmentation system for railway point clouds, characterized in that, The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene point cloud semantic segmentation device. The application relates to a railway scene point cloud semantic segmentation method based on active learning and a railway scene The semantic segmentation network model construction module is configured to construct a semantic segmentation network model, wherein the model main network adopts a KPConv network, a geometric feature extraction module and a feature fusion module are introduced on the basis of an original encoder framework of the main network, the original encoder extracts convolution features of input point clouds, the geometric feature extraction module extracts geometric features of points in the input point clouds, the feature fusion module obtains fused features based on the input point clouds, convolution features of the input point clouds and the geometric features of the points, the fused features are input into an original decoder structure, and a point-level semantic prediction result is output; The active learning training module is configured to iteratively train the semantic segmentation network model based on a labeling pool data using an active learning strategy, update the labeling pool data based on a loop output difference and information entropy theory using unlabeled pool data during the training process, and output the trained semantic segmentation network model. The railway point cloud semantic segmentation module is configured to input a railway scene point cloud to be processed into the trained semantic segmentation network model, and output a railway point cloud semantic segmentation result.

9. An electronic device, comprising: The railway point cloud active learning weakly supervised semantic segmentation method comprises a memory and a processor, the memory stores program instructions executed by the processor, and the processor invokes the program instructions to execute the railway point cloud active learning weakly supervised semantic segmentation method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the railway point cloud active learning weakly supervised semantic segmentation method according to any one of claims 1 to 7.

Images