An end-to-end scatterer profile modeling method based on online hard example mining

Abstract

The application provides an end-to-end scatterer profile modeling method based on online difficult example mining, comprising: obtaining point cloud group data to be modeled and a pre-trained profile modeling network; wherein the profile modeling network comprises a backbone network, a point cloud group detection network, a vertex detection network and an edge detection network, and the point cloud group detection network is a detection network integrated with an online difficult example mining module; inputting the point cloud group data to be modeled into the backbone network to obtain point cloud feature data; inputting the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices; performing vertex detection and edge detection on the point cloud group containing the vertices based on the vertex detection network and the edge detection network to generate a wireframe model; and generating a watertight plane based on the wireframe model to obtain a scatterer profile model. Not only the recognition accuracy of positive samples is improved, but also the robustness of negative samples is enhanced, and the performance problem caused by the imbalance between positive and negative samples in traditional point cloud group detection is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of contour modeling, and in particular to an end-to-end scatterer contour modeling method based on online hard example mining. BACKGROUND

[0002] Three-dimensional reconstruction refers to the process of reconstructing three-dimensional information of an object from single-view images, multi-view images, or radar point clouds, etc. It is an important research direction in computer vision and graphics, and plays an important role in many fields such as city planning, emergency rescue, channel modeling, virtual reality, and medical imaging.

[0003] In recent years, with the development of laser scanning technology, it has become more convenient to obtain LiDAR data, making Light Detection And Ranging (LiDAR) the main means of obtaining large-scale three-dimensional information of urban scenes. The point cloud obtained by LiDAR is usually dense and relatively regular, so it is very suitable for building reconstruction. Traditional contour modeling methods rely on geometric primitive extraction, which is difficult to handle data missing, and steps such as normal estimation and plane extraction involved may cause cumulative errors. Some methods obtain implicit representation through neural networks, and reconstruct a small triangular mesh model of the object. Since a large amount of vertex and face information is stored, the reconstruction result obtained by this method often occupies a large storage space. In contrast, wireframes focus on edge structures and are very suitable for describing buildings with a small number of obvious crease edges and smoothness. Some other methods attempt to recover the wireframe model of the object from the point cloud, but it is very time-consuming to detect points one by one in large-scale point cloud data. PC2WF first proposed to quickly screen point cloud groups containing vertices through point cloud group detection, which can be regarded as positive samples, while the remaining point cloud groups not containing key vertices are regarded as negative samples. PC2WF adopts a balanced positive and negative sample strategy when training the model. However, in actual application scenarios, the number of negative samples not containing key vertices is usually much larger than that of positive samples containing vertices. This imbalance in sample distribution leads to insufficient generalization ability of the trained network to negative samples in actual application. SUMMARY

[0004] The present application provides an end-to-end scatterer contour modeling method based on online hard example mining, which solves the problem of imbalance of positive and negative samples in existing three-dimensional reconstruction methods, improves the recognition accuracy of positive samples, i.e. roof point clouds, and enhances the robustness to negative samples such as background point clouds and noise. The technical solution proposed by the present application is as follows:

[0005] In the first aspect, the present application provides an end-to-end scatterer contour modeling method based on online hard example mining, comprising:

[0006] obtain point cloud group data to be modeled and a pre-trained contour modeling network; wherein the contour modeling network comprises a backbone network, a point cloud group detection network, a vertex detection network and an edge detection network, and the point cloud group detection network is a detection network integrated with an online difficult example mining module;

[0007] input the point cloud group data to be modeled into the backbone network to obtain point cloud feature data;

[0008] input the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices;

[0009] perform vertex detection and edge detection on the point cloud group containing vertices based on the vertex detection network and the edge detection network to generate a wireframe model;

[0010] generate a watertight plane based on the wireframe model to obtain a scatterer contour model.

[0011] Optionally, the contour modeling network is trained in the following manner:

[0012] input a point cloud group sample into a pre-established contour modeling network to calculate a first loss, a second loss and a third loss respectively; wherein the first loss is the loss of the point cloud group detection network, the second loss is the loss of the vertex detection network, and the third loss is the loss of the edge detection network;

[0013] calculate an overall loss according to the first loss, the second loss and the third loss;

[0014] update network parameters of the point cloud group detection network, the vertex detection network and the edge detection network according to the overall loss to obtain the pre-trained contour modeling network.

[0015] Optionally, the point cloud group detection network comprises a read-only network and a read-write network;

[0016] the first loss is determined in the following manner:

[0017] input a point cloud group sample into the read-only network to obtain a first prediction result; wherein the first prediction result is the probability that the point cloud group sample is a positive sample;

[0018] calculate a sample loss based on the true label of the point cloud group sample and the first prediction result;

[0019] select a plurality of difficult example point cloud group samples from all samples according to the sample loss of all samples and a pre-set difficult example ratio;

[0020] input the difficult example point cloud group sample into the read-write network to obtain a second prediction result; wherein the second prediction result is the probability that the difficult example point cloud group sample is a positive sample.

[0021] The first loss is calculated according to the true labels of all point cloud group samples and the second prediction result.

[0022] Optionally, the second loss is determined in the following manner:

[0023] The vertex samples are input into the vertex detection network to obtain vertex detection results, wherein the vertex detection results include vertex prediction positions and vertex prediction quantities.

[0024] The true vertex positions of the vertex samples are obtained, and the second loss is calculated according to the vertex prediction positions of all vertex samples, the true vertex positions and the vertex prediction quantities.

[0025] Optionally, the third loss is determined in the following manner:

[0026] Each edge sample is input into an edge detection network to obtain a corresponding edge detection result, wherein the edge detection result is a probability that the edge sample is a positive sample.

[0027] The third loss is calculated according to the true labels of all edge samples and the edge detection results.

[0028] Optionally, the method further comprises:

[0029] The number of correctly predicted vertices, the number of incorrectly predicted vertices and the number of vertices incorrectly predicted as counterexamples are obtained, wherein, for a predicted vertex, if the distance between the vertex and the vertex in the reference true value closest to the vertex is less than a predefined threshold, the vertex is regarded as a correctly predicted vertex.

[0030] The vertex detection accuracy is calculated according to the number of correctly predicted vertices and the number of incorrectly predicted vertices.

[0031] The vertex detection recall rate is calculated according to the number of correctly predicted vertices and the number of vertices incorrectly predicted as counterexamples.

[0032] The vertex detection effect of the vertex detection network is evaluated using the vertex detection accuracy and the vertex detection recall rate.

[0033] In a second aspect, the present application further provides an end-to-end scatterer profile modeling device based on online difficult example mining, comprising the following modules:

[0034] A data acquisition module is configured to acquire point cloud group data to be modeled and a pre-trained profile modeling network; wherein the profile modeling network comprises a backbone network, a point cloud group detection network, a vertex detection network and an edge detection network, and the point cloud group detection network is a detection network integrated with an online difficult example mining module.

[0035] a feature extraction module, configured to input the point cloud group data to be modeled into the backbone network to obtain point cloud feature data;

[0036] a first detection module, configured to input the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices;

[0037] a second detection module, configured to perform vertex detection and edge detection on the point cloud group containing vertices based on the vertex detection network and the edge detection network to generate a wireframe model;

[0038] a contour modeling module, configured to generate a watertight plane based on the wireframe model to obtain a scatterer contour model.

[0039] In a third aspect, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the online-difficult-example-mining-based end-to-end scatterer contour modeling method according to the first aspect when executing the computer program.

[0040] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program implements the online-difficult-example-mining-based end-to-end scatterer contour modeling method according to the first aspect when executed by a processor.

[0041] In a fifth aspect, the present application also provides a computer program product, which comprises a computer program, and the computer program implements the online-difficult-example-mining-based end-to-end scatterer contour modeling method according to the first aspect when executed by a processor.

[0042] Based on the above technical solutions, the present application has the following beneficial effects compared with the prior art:

[0043] The application provides an end-to-end scatterer profile modeling method based on online hard example mining, which processes input point cloud data by using a point cloud group detection network integrated with an online hard example mining module. The online hard example mining module can dynamically identify and focus on those difficult-to-identify samples, i.e. hard examples, which are roof point clouds with complex shapes, occlusions or low contrast. Guided by the online hard example mining module, the point cloud group detection network can more effectively learn the key features of the building profile. This process not only improves the recognition accuracy of positive samples (i.e. roof point clouds), but also enhances its robustness to negative samples (such as background point clouds, noise, etc.), effectively solving the performance problems caused by the imbalance between positive and negative samples in traditional point cloud group detection. The trained vertex detection and edge prediction networks are used to further process the point cloud group containing key vertices. The two networks work together to accurately identify the vertex position and edge information of the roof, thereby generating a lightweight roof wireframe model. This step makes full use of the key features extracted by the point cloud group detection network, further simplifying the representation of the roof profile. After obtaining the roof wireframe model, a watertight plane is generated based on the wireframe model. This plane not only covers all the vertices, edges and line segments in the wireframe model, but also ensures the continuity and imperforation of the plane, finally obtaining an accurate and complete scatterer profile model.

[0044] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the description, claims and drawings.

[0045] In order to make the above-mentioned objects, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0047] Figure 1 is a flowchart of the end-to-end scatterer profile modeling method based on online hard example mining provided by the present application.

[0048] Figure 2 is the overall network architecture diagram of the profile modeling network provided by the present application.

[0049] Figure 3is an example diagram of a SimulateRoof_Wireframe dataset provided by the present application.

[0050] Figure 4 is a structural schematic diagram of a point cloud group detection network provided by the present application.

[0051] Figure 5 is a schematic diagram of an ablation experiment visualization result provided by the present application.

[0052] Figure 6 is a schematic diagram of a qualitative evaluation result on a SimulateRoof_Wireframe dataset provided by the present application.

[0053] Figure 7 is a structural schematic diagram of an end-to-end scatterer profile modeling device based on online difficult example mining provided by the present application.

[0054] Figure 8 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] In the research of recovering three-dimensional building models from airborne LiDAR data, there are mainly two methods: model-driven method and data-driven method. The model-driven method reconstructs the model by matching the geometric shape of the roof, which can ensure the topological structure of the reconstructed model to be correct. Specifically, it relies on a pre-defined model library consisting of basic building shapes, such as flat roof, triangular roof, asymmetric double slope roof, double slope roof, pyramid roof and complex roof, etc. Reconstruction is performed by selecting the model that best matches the point cloud data from the library. However, if some roof shapes are missing from the model library, or the roof structure is too complex, this method may encounter difficulties. In addition, due to the limited number of shapes in the model library, this limits the production accuracy and may not be able to accurately model complex roof structures.

[0057] Data-driven methods usually require two steps to be performed: geometry primitive extraction and building structure inference. This approach starts with extracting basic geometric shapes, such as planes, cylinders, cones, spheres, or tori, and then analyzes the topological relationships of these primitives in two or three dimensions. In the urban scene, the plane is a major geometric structure, and therefore, various plane extraction algorithms for point clouds have been developed, such as the RANdom SAmple Consensus (RANSAC) algorithm and the region growing algorithm. Nan and Wonka proposed a data-driven method named Polyfit in 2017, which combines the RANSAC algorithm and contextual knowledge for reconstructing lightweight polygonal surfaces from point clouds. However, this reconstruction process relies on the extraction of geometric primitives, and the extraction of some geometric primitives relies on the normal information of the point cloud, which is sometimes difficult to obtain and the normal estimation process can introduce errors. Obviously, these traditional methods are not end-to-end and cannot avoid the cumulative errors generated at different stages, resulting in the final 3D roof model may not be optimal.

[0058] With the rapid development of deep learning in the field of point cloud processing, several three-dimensional reconstruction methods based on deep learning have emerged. These methods can construct triangular meshes from point clouds by learning the implicit representation of surfaces, better recovering irregularly shaped surfaces. However, the models recovered by these methods are often large, and for buildings with regular shapes and large point cloud sizes, lightweight 3D wireframe models are more suitable. Lightweight 3D wireframe models provide a comprehensive representation of the shape and structure information of 3D objects, and the model is more concise, facilitating subsequent processing and analysis. Some methods attempt to reconstruct wireframe models from point clouds, such as Point2Roof, which judges vertices point by point from all input point clouds, and PC2WF innovatively introduces point cloud group detection to quickly screen point cloud group data containing vertices, but the imbalance of positive and negative samples limits the performance of point cloud group detection.

[0059] In order to solve the problems of data set missing for wireframe reconstruction of buildings and the limitation of point cloud group detection performance by the imbalance of positive and negative samples in existing three-dimensional reconstruction methods, the present application constructs a large-scale synthetic data set for training and testing of building wireframe reconstruction. At the same time, the present application proposes an innovative end-to-end building contour modeling method. In the training process, all losses can be backpropagated, effectively avoiding the generation of cumulative errors. In addition, by introducing a point cloud group detection network, dense point cloud data can be quickly processed, and combined with an online hard example mining module to allow the network to fully learn hard example point cloud group samples, improving detection efficiency.

[0060] The following will be described in conjunction with Figures 1-6 The end-to-end scatterer contour modeling method based on online hard example mining of the present application is described.

[0061] Referring to Figure 1 As shown in the method comprises the following:

[0062] Step S101, obtaining the point cloud group data to be modeled and the pre-trained contour modeling network.

[0063] The above-mentioned point cloud group data to be modeled is usually obtained by laser radar (LiDAR), unmanned aerial vehicle photography or other 3D scanning equipment, and contains three-dimensional point cloud information of the building roof.

[0064] The above-mentioned pre-trained contour modeling network is a deep learning model, referring to Figure 2 As shown, it includes a backbone network (FCFG), a point cloud group detection network, a vertex detection network and an edge detection network. These networks have been trained through a large amount of data and have the ability to extract the roof contour from the point cloud. The point cloud group detection network integrates an online hard example mining (OHEM) module to improve the detection ability of complex roof contours. The online hard example mining module can dynamically select samples that are difficult to identify for additional training during the training process, thereby improving the detection accuracy. The introduction of the online hard example mining module enables the model to conduct targeted training on difficult-to-identify samples, thereby improving the overall detection performance.

[0065] The core of the contour modeling network is the online hard example mining-point cloud detection (OHEM-PCD) module, i.e. the above-mentioned point cloud group detection network, which is responsible for detecting point cloud groups containing key vertices. The point cloud group detection network performs classification tasks through a series of efficient convolution and normalization operations, which are designed to identify point cloud groups containing vertices. In order to further improve the detection accuracy, the point cloud group detection network will filter out samples with larger loss after each detection, and perform additional forward and backward propagation on these samples. This strategy enables the network to focus more on difficult-to-identify point cloud groups, thereby improving the overall detection performance.

[0066] After preliminary screening by the point cloud group detection network, only the point cloud groups identified as containing vertices will be passed to the subsequent processing modules. These modules include the vertex detection network and the edge detection network, which are designed as lightweight convolutional networks specifically for restoring the wireframe model of the roof from the screened point cloud groups.

[0067] In terms of vertex detection, the input of the vertex detection network is the point cloud group feature and the point cloud coordinates, and the output is the vertex coordinates. The vertex detection network is responsible for accurately locating the vertex of the roof, providing a basis for subsequent edge detection and model reconstruction. The edge detection network is used to determine the edges of the roof, with the input being edge features and the output being the probability that the edge is a valid edge. These edges define the outline of the roof. Through the cooperative work of the two networks, a three-dimensional wireframe model of the roof can be reconstructed. Finally, in order to generate a complete three-dimensional model of the roof, the invention also includes a step of converting the wireframe model into a watertight plane. This step ensures the continuity and integrity of the roof model, making the final roof model suitable for various subsequent applications such as urban planning, building analysis and virtual reality, etc.

[0068] Step S102, input the point cloud group data to be modeled into the backbone network to obtain point cloud feature data.

[0069] The backbone network is responsible for extracting rich feature information from the input point cloud data. These feature information is crucial for subsequent point cloud group detection, vertex detection and edge detection. The backbone network converts the original point cloud data into high-dimensional feature vectors through a series of convolution, pooling and other operations. The extracted feature information can more accurately reflect the spatial structure and texture information of the point cloud data, providing strong support for subsequent detection tasks.

[0070] In the present invention, the Fully Convolutional Geometric Features (FCGF) network is used as the backbone network, which is responsible for extracting rich feature information from the point cloud. The FCGF network processes the point cloud data and outputs the feature representation of each point in the point cloud. These feature representations are compact and capture a wide range of spatial context information, so they can be used for subsequent point cloud group detection, classification and other tasks. For each point in the input point cloud data, the FCGF network outputs a 32-dimensional feature vector. These feature vectors contain rich information such as the position, shape and texture of the point in space.

[0071] Step S103, input the point cloud feature data into the point cloud group detection network to obtain the point cloud group containing the vertex.

[0072] The point cloud group detection network uses the feature information extracted from the backbone network to group the point cloud data and identify the point cloud group containing the vertex. Through a series of efficient convolution and normalization operations, as well as the assistance of the OHEM module, this network can accurately identify the point cloud group containing the vertex. This reduces the complexity of subsequent processing and improves the efficiency of the overall algorithm.

[0073] In step S104, vertex detection and edge detection are performed on the point cloud set containing vertices based on the vertex detection network and the edge detection network, and a wireframe model is generated.

[0074] Taking roof contour modeling as an example, the vertex detection network is responsible for identifying specific roof vertices from the point cloud set containing roof vertices. These vertices represent key positions of the roof, such as ridges, eaves, etc. The edge detection network is responsible for detecting the edges of the roof contour, i.e., the line segments connecting the roof vertices. Through the cooperative work of the two networks, a preliminary roof wireframe model can be generated.

[0075] In step S105, a watertight plane is generated based on the wireframe model, and a scatterer contour model is obtained.

[0076] After obtaining the wireframe model, a series of geometric processing operations (such as interpolation, smoothing, etc.) are performed to convert the wireframe model into a watertight plane. The watertight plane is a continuous, non-porous plane that can more accurately reflect the true shape of the roof. Finally, the watertight plane is output as the roof contour model, and the generated roof contour model is more accurate, continuous, and smooth.

[0077] The end-to-end scatterer contour modeling method based on online hard example mining provided by the present application uses a point cloud set detection network integrated with an online hard example mining module to process input point cloud data. The online hard example mining module can dynamically identify and focus on samples that are difficult to classify, i.e., hard examples. In building contour modeling, these hard examples correspond to roof point clouds with complex shapes, occlusions, or low contrast. They are often the key to the problem of imbalance between positive and negative samples. Through the guidance of the online hard example mining module, the point cloud set detection network can more effectively learn the key features of the building contour, even if these features are unevenly distributed in positive and negative samples. This process not only improves the recognition accuracy of the model for positive samples (i.e., roof point clouds), but also enhances its robustness to negative samples (such as background point clouds, noise, etc.). Next, the trained vertex detection and edge prediction networks are used to further process the point cloud set containing key vertices. Taking roof contour modeling as an example, the two networks work together to accurately identify the vertex positions and edge information of the roof, thereby generating a lightweight roof wireframe model. This step fully utilizes the key features extracted by the point cloud set detection network, further simplifying the representation of the roof contour. After obtaining the roof wireframe model, a watertight plane is further generated based on the wireframe model through geometric processing and optimization algorithms. This plane not only covers all vertices, edges, and line segments in the wireframe model, but also ensures the continuity and non-porousness of the plane. Through this step, an accurate and complete roof contour model is finally obtained.

[0078] Throughout the whole process, the application of the online difficult example mining module enables the model to focus more on those difficult example point cloud group samples that are crucial for performance improvement. At the same time, through the cooperative work of the vertex detection and edge prediction network, as well as the subsequent geometric processing and optimization steps, the accuracy and robustness of the roof contour model are further improved, solving the problem of the limitation of the imbalance between positive and negative samples on the detection performance of the point cloud group.

[0079] The present application is tested on the synthetic dataset simulateRoof_Wireframe to verify its performance and accuracy. Wichmann et al. created a publicly available 3D building reconstruction dataset called RoofN3D in 2018 and 2019, which covers three basic roof types: pyramid-shaped, hip-shaped, and cable-shaped roofs. Subsequently, Li et al. expanded a larger-scale roof dataset SimulateRoof based on RoofN3D in 2022, increasing the number of roof types to 16. However, the SimulateRoof dataset only provides sparse point clouds and mesh models, failing to fully meet the needs of roof wireframe reconstruction.

[0080] To solve this problem and better restore the wireframe model of the building, the present application further develops a new synthetic dataset SimulateRoof_Wireframe based on SimulateRoof. The dataset contains 17,600 roof models, divided into training and test sets in a ratio of 10:1 to ensure the adequacy of model training and the accuracy of testing. Each roof model contains point clouds, wireframe models, and mesh models, providing comprehensive reference data for roof wireframe reconstruction. Through testing on the SimulateRoof_Wireframe dataset, the present application demonstrates its excellent performance and high accuracy in the task of roof wireframe reconstruction. Referring to Figure 3 The SimulateRoof_Wireframe dataset is shown in the figure.

[0081] In the wireframe reconstruction process of three-dimensional building models, the first step is to identify the position of the vertex from the vast amount of point cloud data. However, the point cloud data of buildings is often large in scale and mainly composed of regular planar structures. Compared with the point cloud data of indoor objects, directly detecting vertices will greatly increase the computational burden, resulting in low efficiency. To solve this problem, the present application refers to the PatchNet network proposed by PC2WF to pre-filter out a large number of irrelevant or noisy point clouds, thereby reducing the computational pressure of subsequent processing.

[0082] For building point clouds, most point cloud groups do not contain vertices. Training a point cloud group detection network with balanced positive and negative samples can result in insufficient generalization of the model to point cloud groups that do not contain vertices in actual applications. In addition, since the training data may also contain a large number of simple samples, i.e., samples that the model can easily classify correctly, the model spends most of the time processing these simple samples during training, and cannot fully learn the characteristics of difficult example point cloud group samples, thereby affecting the generalization ability of the model. To this end, the point cloud group detection network is designed to avoid the model from being overly biased towards the majority class or simple samples.

[0083] The online difficult example mining module is responsible for detecting point cloud groups containing key vertices. The point cloud group detection network performs classification tasks through a series of efficient convolution and normalization operations, which are designed to identify point cloud groups containing vertices. To further improve detection accuracy, the point cloud group detection network filters out samples with large losses after each detection and performs additional forward and backward propagation on these samples. This strategy enables the network to focus more on difficult-to-identify point cloud groups, thereby improving overall detection performance.

[0084] In an optional embodiment, Figure 4 The details of the point cloud group detection network are shown. The point cloud group detection network described above includes a read-only network and a read-write network. During training, the loss of the point cloud group detection network is determined in the following way:

[0085] S201, input the point cloud group sample into the read-only network to obtain a first prediction result;

[0086] First, the pre-prepared point cloud group sample is input into the read-only network. The main function of the read-only network is to make a preliminary prediction to determine the probability that the point cloud group sample is a positive sample (i.e., a point cloud containing key roof information). The read-only network outputs the first prediction result, i.e., the probability value of each point cloud group sample being a positive sample.

[0087] Specifically, the read-only network described above includes Convolutional Layer 1 (Conv1), Convolutional Layer 2 (Conv2), Max Pooling Layer 2D (MaxPool2D), and Convolutional Layer 3 (Conv3).

[0088] The read-only network processing procedure is as follows: the point cloud group sample is sent into the Conv1 layer of the read-only network as input data. The Conv1 layer performs convolution operation on the input point cloud data to extract a preliminary feature map. After the convolution operation, the feature map is normalized by the BatchNorm layer to speed up the training process and improve the stability of the model. Then, the ReLU activation function is applied to the feature map to increase the nonlinearity of the network. The Conv2 layer receives the feature map output by the Conv1 layer and performs further convolution operation to extract higher-level features. Similarly, the BatchNorm and ReLU processing are applied to the output of the Conv2 layer. The MaxPool2D layer performs maximum pooling operation on the output of the Conv2 layer to reduce the size of the feature map while retaining the most important feature information. The pooling operation helps to reduce the amount of calculation and overfitting. The Conv3 layer receives the output of the MaxPool2D layer and continues to extract features. The output of the Conv3 layer is taken as the final output of the read-only network, which is used to output the probability of the point cloud group sample being a positive sample, i.e., the first prediction result.

[0089] In S202, a sample loss is calculated based on the true label of the point cloud group sample and the first prediction result.

[0090] The true label of the point cloud group sample (i.e., the sample category manually labeled) and the first prediction result output by the read-only network are used to calculate the sample loss of each sample to measure the difference between the prediction result and the true label. The sample loss is calculated by the present application using the following formula:

[0091]

[0092] wherein, is the loss of the i-th sample, is the true label of the i-th point cloud group sample, the positive sample takes the value of 1, and the negative sample takes the value of 0, is the probability of the OHEM-PCD module predicting the i-th point cloud group as a positive sample.

[0093] In S203, a plurality of hard example point cloud group samples are selected from all samples according to the sample loss of all samples and a pre-set hard example ratio.

[0094] ​​​The OHEM module is configured to dynamically select difficult instance point cloud group samples according to loss values for training, so as to improve the performance of the model. The read-only network provides preliminary classification prediction and loss calculation, and the read-write network is configured to perform targeted training on the difficult instance point cloud group samples. The two networks together constitute the application of the OHEM module in the point cloud detection task. The OHEM module is configured to select difficult instance point cloud group samples from all samples according to sample losses of the samples and a preset difficult instance ratio. The difficult instance ratio is a hyperparameter configured to control the number and difficulty of the difficult instance point cloud group samples. Samples with higher sample losses are regarded as difficult instance point cloud group samples.

[0095] Let be a set of all samples, be the i-th sample, be the total number of samples, be a difficult instance ratio, then the samples with the highest loss values are selected, and a difficult instance point cloud group sample set is obtained.

[0096]

[0097] S204, input the difficult instance point cloud group samples into the read-write network to obtain a second prediction result.

[0098] The selected difficult instance point cloud group samples are input into the read-write network. Unlike the read-only network, the read-write network is trainable and can update weights according to input data. The read-write network outputs a second prediction result, i.e., the probability that the difficult instance point cloud group sample is a positive sample.

[0099] Specifically, the read-write network also includes a convolutional layer 1 (Convolutional Layer 1, Conv1), a convolutional layer 2 (Convolutional Layer 2, Conv2), a max pooling layer 2D (Max Pooling Layer 2D, MaxPool2D), and a convolutional layer 3 (Convolutional Layer 3, Conv3).

[0100] The read-write network processing process is as follows: the input of the read-write network is the difficult instance point cloud group sample selected by the OHEM module. The Conv1, Conv2, and Conv3 layers of the read-write network are similar to the corresponding layers in the read-only network, and perform convolution operations, BatchNorm, and ReLU processing. These layers continue to extract and convert features. The MaxPool2D layer is configured to further reduce the size of the feature map. The read-write network is configured to output a layer after the Conv3 layer, and the output layer can be a softmax layer configured to output the probability that the difficult instance point cloud group sample is a positive sample, i.e., the second prediction result described above.​​

[0101] S205: Calculate a first loss based on the true labels of all point cloud group samples and the second prediction results.

[0102] The first loss is calculated using the true labels of all point cloud group samples and the second prediction results output by the read-write network. This loss value will be used for subsequent network training and optimization.

[0103] Point cloud group detection is a binary classification task. The present invention uses binary cross entropy to calculate the first loss , calculate the loss for each point cloud group sample and average the losses of all samples to get the loss of the entire point cloud group detection task. The loss function is as follows:

[0104]

[0105] in, is the number of point cloud groups, For the The true label of the point cloud group samples, the positive sample value is 1, the negative sample value is 0, Predict the first The probability that a point cloud group is a positive sample.

[0106] This loss function is not only used to evaluate model performance, but also to screen difficult point cloud group samples for further training in the read-write network. During training, the weights of the read-write network are updated through the backpropagation algorithm to minimize the loss function and improve task performance.

[0107] By introducing an online hard example mining module, the present invention enables the point cloud group detection network to focus more on samples that are difficult to classify, thereby improving the detection accuracy of complex roof profiles. Through in-depth learning and weight updates of hard example point cloud group samples, the read-write network can learn more feature information about complex roof profiles, thereby enhancing the model's generalization ability. The combined use of a read-only network and a read-write network enables the point cloud group detection network to improve training efficiency while maintaining accuracy. The read-only network is used to quickly screen hard example point cloud group samples, while the read-write network focuses on in-depth learning of these hard examples. This method is particularly suitable for scenarios with an imbalance between positive and negative samples. By dynamically screening hard example point cloud group samples and performing targeted training, the point cloud group detection network can better adapt to this imbalance, thereby improving overall detection performance. In summary, the point cloud group detection network, by introducing a read-only network and a read-write network, combined with an online hard example mining module, achieves efficient and accurate detection of complex roof profiles. This method not only improves detection accuracy and generalization ability, but also optimizes training efficiency, making it particularly suitable for scenarios with an imbalance between positive and negative samples.

[0108] In an optional embodiment, vertex detection is a regression task. The goal of the vertex detection task is to accurately predict the vertex positions of geometric shapes in the point cloud, which is crucial for applications such as shape reconstruction and 3D printing. The present invention calculates the loss for each predicted vertex position and averages the losses of all predicted vertices to obtain the loss of the entire vertex detection task. The loss of the vertex detection network is referred to as the second loss. , calculated using mean square error (MSE). The second loss Determined by:

[0109] S301: Input a vertex sample into the vertex detection network to obtain a vertex detection result, wherein the vertex detection result includes a predicted vertex position and a predicted vertex number.

[0110] S302: Acquire the actual vertex position of the vertex sample, and calculate the second loss based on the predicted vertex positions of all vertex samples, the actual vertex positions, and the predicted number of vertices:

[0111]

[0112] Among them, P is the number of vertex predictions, Predict the position of the vertex, is the actual vertex position.

[0113] In an optional embodiment, the goal of the edge detection task is to identify edge points in the point cloud, which is very important for subsequent tasks such as geometric shape analysis and reconstruction. Edge detection is also a binary classification task loss function, also calculated using binary cross entropy. It calculates the loss for each edge sample and averages the losses of all edge samples to obtain the loss of the entire edge detection task, hereinafter referred to as the third loss. The specific calculation process is as follows:

[0114] S401 , input each edge sample into an edge detection network to obtain corresponding edge detection results; wherein the edge detection result is the probability that the edge sample is a positive sample.

[0115] S402: Calculate the third loss based on the true labels of all edge samples and the edge detection results. :

[0116]

[0117] in, is the number of edge samples, For the The true label of the edge samples, the positive sample value is 1, the negative sample value is 0, Predict the first the probability that an edge is a positive sample.

[0118] The structures of the above-mentioned vertex detection network and edge detection network can refer to the description in the prior art, and will not be described here again.

[0119] In an optional embodiment, when training the complete contour modeling network, a global loss function is used to minimize the loss of all tasks. The global loss is composed of the point cloud group detection loss (i.e., the first loss described above), the vertex detection loss (i.e., the second loss described above), and the edge detection loss (i.e., the third loss described above), and the relative importance between them is adjusted by balancing the weights and . The contour modeling network training process is as follows:

[0120] S501, input the point cloud group sample into the pre-established contour modeling network, and calculate the first loss, the second loss and the third loss respectively.

[0121] The point cloud group sample is used as input data, which contains the three-dimensional shape and contour information of the object. The point cloud group detection loss (first loss), vertex detection loss (second loss) and edge detection loss (third loss) are calculated respectively.

[0122] The first loss is used to evaluate the recognition ability of the point cloud group detection network for the input point cloud group sample, to ensure that the network can accurately detect and extract the key features in the point cloud. The vertex detection network is responsible for identifying the vertex information in the point cloud, and the second loss is used to measure the performance of the network in the vertex detection task, to ensure the accuracy of the vertex position. The edge detection network is used to detect the edge information in the point cloud, and the third loss is used to evaluate the effect of the network in the edge detection task, to ensure the clarity and continuity of the edge.

[0123] S502, calculate the global loss according to the first loss, the second loss and the third loss. This global loss value comprehensively reflects the overall performance of the three network modules in the current training state.

[0124] When training the complete network, the global loss is minimized by balancing the weights , and the global loss function is:

[0125]

[0126] S503, update the network parameters of the point cloud group detection network, the vertex detection network and the edge detection network according to the global loss, to obtain the pre-trained contour modeling network.

[0127] ​According to the overall loss value, the network parameters of the point cloud group detection network, the vertex detection network, and the edge detection network are updated using optimization methods such as backpropagation algorithm and gradient descent method. This step aims to reduce the overall loss value by adjusting the network parameters, thereby improving the performance of the model. The above steps S501-S503 are repeated for multiple iterations until the preset training rounds are reached or the model performance no longer improves significantly.

[0128] Specifically, during the training process, the point cloud group samples are first predicted and loss calculated by the read-only network. Difficult example point cloud group samples are selected according to the sample loss, and these samples are sent to the read-write network for further training. Meanwhile, the edge detection task and the vertex detection task are also trained under the guidance of their respective loss functions. The network weights are updated through the backpropagation algorithm to minimize the overall loss. The training process usually includes multiple iteration cycles, and the training strategy is adjusted according to the current network weights and loss value in each cycle.

[0129] The present application comprehensively considers the loss of point cloud group detection, vertex detection, and edge detection, and the training process can more comprehensively evaluate and improve the performance of the model. This helps the model accurately identify the contour and shape features of objects in complex scenes. Since multiple loss functions and different network modules are used in the training process, the model can better adapt to different types of point cloud data and contour features. This enhances the generalization ability of the model, enabling it to perform well in unknown or new scenarios. The training process gradually reduces the loss value of the model by iteratively training and optimizing the network parameters, thereby improving the performance of the model. This helps optimize the use of computing resources, reducing unnecessary computational overhead and time cost.

[0130] In an optional embodiment, in the vertex detection task, in order to comprehensively evaluate the performance of the vertex detection network, the present application introduces two key indicators, vertex detection precision (VP) and vertex detection recall rate (VR), to evaluate the effect of vertex detection. The specific definitions of VP and VR are as follows:

[0131]

[0132]

[0133] where TP is the number of correctly predicted vertices, i.e., the vertices found in both the benchmark ground truth and the prediction results. FP is the number of incorrectly predicted vertices. FN is the number of vertices that are incorrectly predicted as counterexamples. The higher the VP and VR values, the better the vertex prediction results. For each predicted vertex, if the distance between the vertex and its nearest vertex in the benchmark ground truth is less than a predefined threshold , the vertex is considered to be a correctly predicted vertex. ​

[0134] The application can also use patch precision (PP) and patch recall (PR) to evaluate the patch detection effect, and use edge precision (EP) and edge recall (ER) to evaluate the edge detection effect. The calculation of PP, PR, EP and ER is similar to VP and VR.

[0135] The application takes vertex detection precision (VP) and vertex detection recall as an example to illustrate the calculation process:

[0136] S601, obtain the number of correctly predicted vertices, the number of incorrectly predicted vertices, and the number of vertices incorrectly predicted as counterexamples; wherein, for a predicted vertex, if the distance between the vertex and the vertex closest to it in the benchmark true value is less than a predefined threshold, the vertex is considered to be a correctly predicted vertex.

[0137] First, obtain all the vertex positions predicted by the vertex detection network, as well as the corresponding benchmark true vertex positions, which are obtained through manual labeling or high-precision sensors. For each predicted vertex, find the vertex closest to it in the benchmark true value and calculate the distance between them. According to the predefined threshold (which is determined according to the task requirements and the accuracy of the point cloud data), if the distance between a predicted vertex and its corresponding benchmark true vertex is less than the threshold, the vertex is considered to be correctly predicted. Count the number of all correctly predicted vertices, denoted as TP (True Positives).

[0138] For those predicted vertices whose distance exceeds the threshold, they are considered to be incorrectly predicted vertices. Count the number of all incorrectly predicted vertices, denoted as FP (False Positives) and FN (False Negatives). Among them, FP represents the number of negative samples incorrectly predicted as positive examples, and FN represents the number of positive samples incorrectly predicted as counterexamples, i.e. those vertices that actually exist but are not detected.

[0139] S602, calculate the vertex detection precision according to the number of correctly predicted vertices and the number of incorrectly predicted vertices, which reflects the accuracy of the vertex detection network in predicting vertices.

[0140] Vertex detection precision is the ratio of the number of correctly predicted vertices to the number of all predicted vertices (including correctly and incorrectly predicted vertices), i.e.

[0141]

[0142] S603、According to the number of correctly predicted vertices and the number of vertices incorrectly predicted as counterexamples, calculate the vertex detection recall rate, which reflects the ability of the vertex detection network in detecting real vertices.

[0143] Vertex detection recall rate is the ratio of the number of correctly predicted vertices to the number of all real vertices (including correctly predicted and incorrectly predicted as counterexamples), that is:

[0144]

[0145] S604, use vertex detection accuracy and vertex detection recall rate to evaluate the vertex detection effect of the vertex detection network.

[0146] Using vertex detection accuracy and vertex detection recall rate, the performance of the vertex detection network can be evaluated comprehensively. High precision indicates that the network has high accuracy in predicting vertices, while high recall rate indicates that the network can detect more real vertices. By adjusting the network structure, optimizing the algorithm or increasing the training data, etc., the values of these two indicators can be further improved, thereby improving the performance of the vertex detection network.

[0147] By introducing vertex detection accuracy and recall rate, the performance of the vertex detection network can be evaluated more comprehensively, not just relying on a single accuracy or recall rate indicator. These two indicators can provide strong guidance for network optimization. For example, when the accuracy is low, it may be necessary to increase the regularization term or adjust the network structure to reduce overfitting; while when the recall rate is low, it may be necessary to increase the training data or improve the data enhancement strategy to improve the network's ability to detect real vertices. By continuously optimizing the vertex detection network, its effectiveness in practical applications can be improved. For example, in the 3D reconstruction task, more accurate vertex detection can generate more detailed 3D models.

[0148] All the codes of the present application are implemented based on the PyTorch framework, and experiments are conducted on a server configured with a GeForce RTX 4090, with a device memory of 24GB, an operating system version of Ubuntu22.04, a cuda version of 11.1, and a pytorch version of 1.9.0.

[0149] The Adam optimizer is adopted in the training process, with the learning rate set to 0.001, and a weight decay of 1e-4 is applied to prevent overfitting. To further refine the learning rate adjustment strategy, a learning rate scheduler is set so that the learning rate will automatically halve every 10 epochs to gradually refine the weight adjustment of the model. The entire training process is set to 20 epochs, and the batch size is set to 512. In each epoch of training, the model will traverse the entire training data set, perform forward propagation on each small batch of samples, calculate the loss function, and then perform back propagation and weight optimization update. At the end of each epoch, in addition to automatically updating the learning rate, the current model state is also saved for subsequent training or model evaluation.

[0150] To comprehensively evaluate the actual effect of online hard example mining (OHEM) in point cloud group detection network, the present application carries out ablation experiments on SimulateRoof_Wireframe dataset. These experiments are specially designed to compare the performance of point cloud group detection network in accuracy and recall rate when online hard example mining function is turned on and off.

[0151] Table 1 is the ablation experiment results, and the ablation experiment results on SimulateRoof_Wireframe dataset show that when OHEM is not used, the average precision (PP) of point cloud group detection is 0.22, and the average recall rate (PR) is 0.95. When the OHEM-PCD module integrated with the OHEM mechanism is applied, the average precision is improved to 0.92, and the average recall rate is improved to 0.96. This result shows that the OHEM mechanism significantly improves the model's ability to identify difficult samples, especially when dealing with unbalanced data sets, OHEM can effectively improve the model performance.

[0152]

[0153] Table 1

[0154] In order to more intuitively see the effect of the OHEM module in point cloud group detection, Figure 5The visualization results of PatchNet and OHEM-PCD in point cloud group detection are shown. Specifically, PatchNet not only identifies positive samples containing vertices, but also incorrectly labels a large number of negative samples as positive samples, which indicates that it has certain limitations in distinguishing positive and negative samples. In contrast, the OHEM-PCD method exhibits superior performance, as it not only successfully identifies positive samples containing vertices, but also effectively filters out negative samples that do not contain vertices, thereby obtaining more accurate detection results. This phenomenon proves the effectiveness of the OHEM mechanism in improving the model's ability to identify negative samples.

[0155] Table 2 shows the quantitative evaluation results of different models on the SimulateRoof_Wireframe dataset. It can be seen that different models have different performance in the four key indicators of VP, VR, EP and ER. .

[0156] In all four indicators, the method of the present application shows significant performance advantages, with vertex detection accuracy and vertex detection recall reaching 0.97 and 0.98, and edge detection accuracy and edge detection recall reaching 0.98 and 0.92, respectively. These results show that the method of the present application can effectively detect vertex positions from point cloud data and accurately construct complete wireframe structures. This performance improvement is due to the introduction of the online hard example mining (OHEM) mechanism, which enables the model to focus more on difficult-to-identify samples, thereby improving the model's generalization ability and overall performance.

[0157] In order to more intuitively show the effect of the method of the present application, Figure 6 The visualization results of the proposed method are shown.

[0158]

[0159] Table 2

[0160] The present application constructs a large-scale synthetic dataset SimulateRoof_Wireframe for building wireframe reconstruction training and testing. At the same time, the present application proposes a new end-to-end building contour modeling method, which effectively improves the point cloud group detection accuracy by adding an online hard example mining-based point cloud group detection module, thereby improving the subsequent vertex detection and edge detection accuracy, and obtaining a clean building wireframe model, providing a new solution for efficient processing of building point clouds.

[0161] The end-to-end scatterer profile modeling device based on online difficult example mining provided by the present invention is described below. The end-to-end scatterer profile modeling device based on online difficult example mining described below and the end-to-end scatterer profile modeling method based on online difficult example mining described above can be referenced to each other.

[0162] The end-to-end scatterer profile modeling device based on online hard case mining provided by the present invention refers to Figure 7 Shown, including:

[0163] A data acquisition module 701 is used to acquire point cloud group data to be modeled and a pre-trained contour modeling network; wherein the contour modeling network includes a backbone network, a point cloud group detection network, a vertex detection network, and an edge detection network; the point cloud group detection network is a detection network integrated with an online hard example mining module;

[0164] A feature extraction module 702 is used to input the point cloud group data to be modeled into the backbone network to obtain point cloud feature data;

[0165] A first detection module 703 is configured to input the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices;

[0166] A second detection module 704 is configured to perform vertex detection and edge detection on the point cloud group containing vertices based on the vertex detection network and the edge detection network to generate a wireframe model;

[0167] The contour modeling module 705 is configured to generate a watertight plane based on the wireframe model to obtain a scatterer contour model.

[0168] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to execute an end-to-end scatterer profile modeling method based on online hard example mining.

[0169] Moreover, the logic instructions in the memory 830 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0170] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the online difficult example mining based end-to-end scatterer profile modeling method provided by the above-mentioned methods.

[0171] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the online difficult example mining based end-to-end scatterer profile modeling method provided by the above-mentioned methods.

[0172] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0173] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the technical solutions described above essentially or the parts that make contributions to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0174] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An end-to-end scatterer profile modeling method based on online hard example mining, characterized in that, The method comprises the following steps: obtaining point cloud group data to be modeled and a pre-trained contour modeling network; wherein the contour modeling network comprises a backbone network, a point cloud group detection network, a vertex detection network and an edge detection network, and the point cloud group detection network is a detection network integrated with an online difficult case mining module; inputting the point cloud group data to be modeled into the backbone network to obtain point cloud feature data; inputting the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices; performing vertex detection and edge detection on the point cloud group containing vertices based on the vertex detection network and the edge detection network to generate a wireframe model; generating a watertight plane based on the wireframe model to obtain a scatterer contour model; the contour modeling network is trained in the following manner: inputting a point cloud group sample into a pre-established contour modeling network to calculate a first loss, a second loss and a third loss respectively; wherein the first loss is the loss of the point cloud group detection network, the second loss is the loss of the vertex detection network, and the third loss is the loss of the edge detection network; calculating an overall loss according to the first loss, the second loss and the third loss; updating the network parameters of the point cloud group detection network, the vertex detection network and the edge detection network according to the overall loss to obtain the pre-trained contour modeling network; the point cloud group detection network comprises a read-only network and a read-write network; the first loss is determined in the following manner: inputting a point cloud group sample into the read-only network to obtain a first prediction result; wherein the first prediction result is the probability that the point cloud group sample is a positive sample; calculating a sample loss based on the true label of the point cloud group sample and the first prediction result; the sample loss is calculated by the following formula: wherein, is the loss for the i-th sample, is the true label for the i-th point cloud group sample, is the probability that the point cloud group detection network predicts the i-th point cloud group as a positive sample; selecting a plurality of difficult point cloud group samples from all samples according to the sample loss of all samples and a pre-set difficult case ratio; inputting the difficult point cloud group sample into the read-write network to obtain a second prediction result; wherein the second prediction result is the probability that the difficult point cloud group sample is a positive sample; calculating the first loss according to the true label of all point cloud group samples and the second prediction result; wherein, is a first loss, N pat is the number of point cloud groups, is a true label of the i-th point cloud group sample, is a probability that the i-th point cloud group is a positive sample predicted by the point cloud group detection network.

2. The online hard example mining based end-to-end scatterer profile modeling method according to claim 1, characterized in that, the second loss is determined in the following manner: inputting a vertex sample into the vertex detection network to obtain a vertex detection result, wherein the vertex detection result comprises a vertex predicted position and a vertex predicted number; obtaining the true vertex position of the vertex sample, and calculating the second loss according to the vertex predicted position of all vertex samples, the true vertex position and the vertex predicted number; wherein, is a second loss, P is a number of vertex predictions, v i is a vertex prediction position, is a real vertex position.

3. The online hard example mining based end-to-end scatterer profile modeling method according to claim 1, wherein, the third loss is determined in the following manner: inputting each edge sample into the edge detection network to obtain a corresponding edge detection result; wherein the edge detection result is the probability that the edge sample is a positive sample; calculating the third loss according to the true label of all edge samples and the edge detection result; wherein, is a third loss, N edge is the number of edge samples, is the true label of the i-th edge sample, is the probability that the edge detection network predicts the i-th edge as a positive sample.

4. The online hard example mining based end-to-end scatterer profile modeling method according to claim 1, characterized in that, the method further comprises: obtaining a number of correctly predicted vertices, a number of wrongly predicted vertices, and a number of vertices wrongly predicted as counterexamples; wherein, for a predicted vertex, if a distance between the vertex and a vertex in the ground truth closest to the vertex is less than a predefined threshold, the vertex is considered as a correctly predicted vertex; calculating vertex detection precision according to the number of correctly predicted vertices and the number of wrongly predicted vertices; calculating vertex detection recall according to the number of correctly predicted vertices and the number of vertices wrongly predicted as counterexamples; evaluating vertex detection effect of the vertex detection network using the vertex detection precision and the vertex detection recall.

5. An online hard example mining based end-to-end scatterer profile modeling apparatus, characterized in that, comprise: a data acquisition module configured to acquire point cloud group data to be modeled and a pre-trained contour modeling network; wherein the contour modeling network comprises a backbone network, a point cloud group detection network, a vertex detection network, and an edge detection network, and the point cloud group detection network is a detection network integrated with an online difficult example mining module; a feature extraction module configured to input the point cloud group data to be modeled into the backbone network to obtain point cloud feature data; a first detection module configured to input the point cloud feature data into the point cloud group detection network to obtain a point cloud group containing vertices; a second detection module configured to perform vertex detection and edge detection on the point cloud group containing vertices based on the vertex detection network and the edge detection network to generate a wireframe model; a contour modeling module configured to generate a water-tight plane based on the wireframe model to obtain a scatterer contour model; the contour modeling network is trained in the following manner: inputting a point cloud group sample into a pre-established contour modeling network to calculate a first loss, a second loss, and a third loss respectively; wherein the first loss is a loss of the point cloud group detection network, the second loss is a loss of the vertex detection network, and the third loss is a loss of the edge detection network; calculating an overall loss according to the first loss, the second loss, and the third loss; updating network parameters of the point cloud group detection network, the vertex detection network, and the edge detection network according to the overall loss to obtain the pre-trained contour modeling network; the point cloud group detection network comprises a read-only network and a read-write network; the first loss is determined in the following manner: inputting a point cloud group sample into the read-only network to obtain a first prediction result; wherein the first prediction result is a probability that the point cloud group sample is a positive sample; calculating a sample loss based on a true label of the point cloud group sample and the first prediction result; the sample loss is calculated using the following formula: wherein, is the loss for the i-th sample, is the true label for the i-th point cloud group sample, is the probability predicted by the point cloud group detection network that the i-th point cloud group is a positive sample; selecting a plurality of difficult example point cloud group samples from all samples according to the sample loss of all samples and a pre-set difficult example ratio; inputting the difficult example point cloud group samples into the read-write network to obtain a second prediction result; wherein the second prediction result is a probability that the difficult example point cloud group sample is a positive sample; calculating the first loss according to true labels of all point cloud group samples and the second prediction result; wherein, is a first loss, N pat is the number of point cloud groups, is a true label of the i-th point cloud group sample, is a probability that the i-th point cloud group is a positive sample predicted by the point cloud group detection network.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the processor implements the computer program to realize the end-to-end scatterer contour modeling method based on online difficult example mining according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the end-to-end scatterer profile modeling method based on online hard example mining according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the end-to-end scatterer profile modeling method based on online hard example mining according to any one of claims 1 to 4.

Images