Sample Data Preprocessing Method, Apparatus and Computer Readable Storage Medium

Abstract

Embodiments of the present disclosure disclose a method, an apparatus, and a computer-readable storage medium for preprocessing sample data. The method includes: extracting a surface model of a scene in space from three-dimensional point cloud data of the space; wherein the surface model is represented by N triangular meshes; extracting M feature planes from the N triangular meshes; determining class labels of each of the M feature planes according to the three-dimensional point cloud data; determining a training point set according to the M feature planes; determining class labels of each point in the training point set according to the class labels of each of the M feature planes; wherein the training point set and the class labels of each point in the training point set are used as samples for training a neural network. Embodiments of the present disclosure can avoid the adverse effects caused by inaccurate and incomplete manually labeled information on the training of the neural network, thereby facilitating improving the accuracy and reliability of the trained neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and particularly to a method and apparatus for preprocessing sample data and a computer-readable storage medium. Background Art

[0002] With the improvement of hardware level and data acquisition technology, the threshold for obtaining large-scale indoor scene point cloud data (such as three-dimensional point cloud data of a house) is getting lower and lower.

[0003] In some cases, after obtaining the three-dimensional point cloud data of a house, it is necessary to use the three-dimensional point cloud data in which each point is manually labeled with a class label to train the required neural network (such as a neural network for feature extraction). It should be noted that the manually labeled information in the three-dimensional point cloud data may not be accurate and complete, which will have an adverse impact on the training of the neural network. Summary of the Invention

[0004] To solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method and apparatus for preprocessing sample data and a computer-readable storage medium.

[0005] According to one aspect of the embodiments of the present disclosure, a method for preprocessing sample data is provided, including:

[0006] Extracting a surface model of a scene in space from three-dimensional point cloud data of the space; wherein, the surface model is represented by N triangular meshes;

[0007] Extracting M feature planes from the N triangular meshes;

[0008] Determining class labels of the M feature planes respectively according to the three-dimensional point cloud data;

[0009] Determining a training point set according to the M feature planes;

[0010] Determining class labels of each point in the training point set respectively according to the class labels of the M feature planes; wherein, the training point set and the class labels of each point in the training point set are used as samples for training a neural network.

[0011] In an optional example, the determining class labels of the M feature planes respectively according to the three-dimensional point cloud data includes:

[0012] Determining class labels of the N triangular meshes respectively according to the three-dimensional point cloud data;

[0013] Determining class labels of the M feature planes respectively according to the class labels of the N triangular meshes.

[0014] In an alternative example, determining the class label of each of the N triangular meshes according to the three-dimensional point cloud data includes:

[0015] Selecting, from each point in the three-dimensional point cloud data, the point closest to a target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes;

[0016] Obtaining the class label of the selected point;

[0017] Determining the class label of the target triangular mesh as the class label of the selected point.

[0018] In an alternative example, determining the class label of each of the M feature planes according to the class labels of the N triangular meshes includes:

[0019] Determining a class label set; wherein, the class label set includes the class labels of each of the triangular meshes among the N triangular meshes that are distributed on a target feature plane, and the target feature plane is any one of the M feature planes;

[0020] Selecting non-repeated class labels in the class label set;

[0021] Counting the number of occurrences of each of the selected class labels in the class label set;

[0022] Determining the class label with the most occurrences among the selected class labels as the class label of the target feature plane.

[0023] In an alternative example, determining a training point set according to the M feature planes includes:

[0024] For each of the M feature planes, uniformly sampling on each of the triangular meshes distributed on it among the N triangular meshes to obtain corresponding sampled points;

[0025] Determining a training point set composed of all the obtained sampled points.

[0026] In an alternative example, determining the class label of each point in the training point set according to the class labels of the M feature planes includes:

[0027] Determining the class label of the target feature plane where the target sampled point is located as the class label of the target sampled point; wherein, the target sampled point is any one of the sampled points in the training point set.

[0028] In an optional example, before extracting the surface model of the scene in the space from the three-dimensional point cloud data of the space, the method further includes:

[0029] Invoking a depth camera and / or a lidar to collect three-dimensional point cloud data of the space.

[0030] According to another aspect of the embodiments of the present disclosure, there is provided a sample data preprocessing device, including:

[0031] A first extraction module, configured to extract the surface model of the scene in the space from the three-dimensional point cloud data of the space; wherein, the surface model is represented by N triangular meshes;

[0032] A second extraction module, configured to extract M feature planes from the N triangular meshes;

[0033] A first determination module, configured to determine the class label of each of the M feature planes according to the three-dimensional point cloud data;

[0034] A second determination module, configured to determine a training point set according to the M feature planes;

[0035] A third determination module, configured to determine the class label of each point in the training point set according to the class label of each of the M feature planes; wherein, the training point set and the class label of each point in the training point set are used as samples for training a neural network.

[0036] In an optional example, the first determination module includes:

[0037] A first determination sub-module, configured to determine the class label of each of the N triangular meshes according to the three-dimensional point cloud data;

[0038] A second determination sub-module, configured to determine the class label of each of the M feature planes according to the class label of each of the N triangular meshes.

[0039] In an optional example, the first determination sub-module includes:

[0040] A first screening unit, configured to screen, from each point in the three-dimensional point cloud data, the point closest to a target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes;

[0041] An acquisition unit, configured to acquire the class label of the screened point;

[0042] A first determination unit, configured to determine the class label of the screened point as the class label of the target triangular mesh.

[0043] In an alternative example, the second determination sub-module includes:

[0044] A second determination unit, configured to determine a set of class labels; wherein, the set of class labels includes the class labels of each of the N triangular meshes distributed on a target feature plane among the N triangular meshes, and the target feature plane is any one of the M feature planes;

[0045] A second screening unit, configured to screen out the non-repeating class labels in the set of class labels;

[0046] A statistics unit, configured to count the number of occurrences of each of the screened class labels in the set of class labels;

[0047] A third determination unit, configured to determine the class label with the most occurrences among the screened class labels as the class label of the target feature plane.

[0048] In an alternative example, the second determination module includes:

[0049] A sampling sub-module, configured to perform uniform sampling on each of the M feature planes, respectively, on each of the triangular meshes distributed thereon among the N triangular meshes, to obtain corresponding sampling points;

[0050] A third determination sub-module, configured to determine a training point set composed of all the obtained sampling points.

[0051] In an alternative example, the third determination module is specifically configured to:

[0052] Determine the class label of the feature plane where the target sampling point is located as the class label of the target sampling point; wherein, the target sampling point is any one of the sampling points in the training point set.

[0053] In an alternative example, the apparatus further includes:

[0054] An acquisition module, configured to call a depth camera and / or a lidar to acquire three-dimensional point cloud data of a space before extracting a surface model of a scene in the space from the three-dimensional point cloud data of the space.

[0055] According to another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the above sample data preprocessing method.

[0056] According to yet another aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0057] A processor;

[0058] A memory for storing the processor-executable instructions;

[0059] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above sample data preprocessing method.

[0060] In an embodiment of the present disclosure, a surface model representing a scene in space using N triangular meshes may be first extracted from the three-dimensional point cloud data of the space; next, M feature planes may be extracted from the N triangular meshes, and according to the three-dimensional point cloud data, the class labels of each of the M feature planes may be determined, and a training point set may be determined according to the M feature planes; thereafter, according to the class labels of each of the M feature planes, the class labels of each point in the training point set may be determined, and the training point set and the class labels of each point in the training point set may be used as samples for training a neural network, so as to train the required neural network. It can be seen that in the embodiments of the present disclosure, the neural network is not directly trained using the three-dimensional point cloud data after artificial annotation, but first, based on the three-dimensional point cloud data, M feature planes and the class labels of each of the M feature planes are obtained, and then, based on the M feature planes and the class labels of each of the M feature planes, a suitable training point set is obtained, and accurate class labels are obtained for each point in the training point set, and then the obtained training point set and class labels are used for training the neural network. Therefore, the embodiments of the present disclosure can avoid the adverse effects caused by inaccurate and incomplete artificial annotation information on the training of the neural network when directly using the three-dimensional point cloud data after artificial annotation for training the neural network, thereby being beneficial to improving the accuracy and reliability of the trained neural network.

[0061] The technical solution of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0063] Figure 1 It is a flowchart of a sample data preprocessing method provided by an exemplary embodiment of the present disclosure.

[0064] Figure 2 It is a flowchart of a sample data preprocessing method provided by another exemplary embodiment of the present disclosure.

[0065] Figure 3It is a schematic diagram of the original three-dimensional point cloud data in the embodiments of the present disclosure.

[0066] Figure 4 It is a schematic diagram of the surface model of the in-scene in the three-dimensional point cloud data extracted in the embodiments of the present disclosure.

[0067] Figure 5 It is a schematic diagram of the determination result after determining the class label for each feature plane by adopting a majority voting mechanism in the embodiments of the present disclosure.

[0068] Figure 6 It is a schematic diagram of the processing result after performing uniform sampling processing and determining the class label for each sampling point in the embodiments of the present disclosure.

[0069] Figure 7 It is a schematic diagram of the structure of a sample data preprocessing device provided by an exemplary embodiment of the present disclosure.

[0070] Figure 8 It is a schematic diagram of the structure of a sample data preprocessing device provided by another exemplary embodiment of the present disclosure.

[0071] Figure 9 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. Detailed implementation manners

[0072] Next, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0073] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0074] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.

[0075] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0076] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, without clear definition or contrary indication in the context, it can generally be understood as one or more.

[0077] In addition, the term "and / or" in this disclosure is merely a description of the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0078] It should also be understood that the descriptions of the various embodiments in this disclosure emphasize the differences between the various embodiments, and their similarities can be referred to each other. For the sake of brevity, they will not be elaborated one by one.

[0079] At the same time, it should be understood that, for the convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationship.

[0080] The following description of at least one exemplary embodiment is actually merely illustrative and in no way restrictive of this disclosure or its application or use.

[0081] Well-known technologies, methods, and devices for those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said technologies, methods, and devices should be regarded as part of the specification.

[0082] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0083] The embodiments of this disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0084] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules may include routines, programs, target programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0085] Exemplary method

[0086] Figure 1 It is a schematic flowchart of a sample data preprocessing method provided by an exemplary embodiment of the present disclosure. Figure 1 The method shown includes steps 101, 102, 103, 104, and 105, which will be described separately below.

[0087] Step 101, extract the surface model of the scene in the space from the three-dimensional point cloud data of the space; wherein, the surface model is represented by N triangular meshes.

[0088] It should be noted that before step 101, the three-dimensional point cloud data of the space can be obtained. In a specific implementation, before step 101, the method further includes:

[0089] Call a depth camera and / or a lidar to collect the three-dimensional point cloud data of the space.

[0090] Here, the depth camera can be an RGBD camera, where RGBD = RGB + Depth Map, RGB represents the colors of the three channels of red, green, and blue, and Depth Map represents the depth map; the lidar can also be called a laser lidar.

[0091] In this implementation, for a space (such as the indoor space of a house), the three-dimensional point cloud data of the space can be generated by the depth camera through the call of the depth camera, and / or, the three-dimensional point cloud data of the space can be obtained by scanning with the lidar through the call of the lidar.

[0092] Next, the Poisson reconstruction algorithm can be used to extract the surface model of the scene in the space from the three-dimensional point cloud data of the space. The surface model can specifically be a continuous triangular mesh model. At this time, the surface model can be represented by a large number of triangular meshes, and the number of triangular meshes can be represented as N.

[0093] Step 102, extract M feature planes from the N triangular meshes.

[0094] Here, the Random Sample Consensus (RANSAC) algorithm or region growing algorithm can be used to extract feature planes for N triangular meshes, so as to extract M feature planes therefrom; among them, several triangular meshes of the N triangular meshes can be distributed on each of the M feature planes.

[0095] Step 103: Determine the class labels of each of the N triangular meshes according to the three-dimensional point cloud data.

[0096] Here, the class label of each point in the three-dimensional point cloud data can be obtained first. For example, the class label of each point in the manually annotated three-dimensional point cloud data can be obtained. The class label of any point in the three-dimensional point cloud data is used to represent the specific class of this point. For example, it is used to represent that the specific class of this point is furniture, wall, ceiling, door, etc.

[0097] Next, based on the class labels of each point in the obtained three-dimensional point cloud data, the class labels of each of the N triangular meshes can be determined. It should be noted that there are various ways to determine the class labels of each of the N triangular meshes here. For the sake of clear layout, examples will be introduced later.

[0098] Step 104: Determine the training point set according to the M feature planes.

[0099] Here, according to the M feature planes, a large number of points can be determined, and further, the training point set composed of these points can be determined. The training point set can be a point set suitable for subsequent neural network training.

[0100] In a specific embodiment, step 104 includes:

[0101] For each of the M feature planes, perform uniform sampling on each of the triangular meshes distributed thereon among the N triangular meshes to obtain corresponding sampling points;

[0102] Determine the training point set composed of all the obtained sampling points.

[0103] Here, for any one of the M feature planes, the triangular meshes distributed on this feature plane among the N triangular meshes can be determined first. Suppose K triangular meshes are determined here, then uniform sampling processing can be performed on these K triangular meshes. In this way, several sampling points can be obtained from each of the K triangular meshes, and these sampling points can form the sampling point set corresponding to this feature plane.

[0104] In the manner described above, M sets of sampling points corresponding one-to-one to the M feature planes can be obtained. After that, a training point set can be formed by all the sampling points included in the M sets of sampling points.

[0105] In this implementation manner, by performing uniform sampling processing on the corresponding triangular meshes for each feature plane, a training point set can be obtained conveniently and reliably.

[0106] Of course, the specific implementation manner of step 104 is not limited to this. For example, after determining K triangular meshes for any one of the M feature planes, the sampling processing performed on these K triangular meshes can be non-uniform sampling processing. After obtaining the M sets of sampling points, points that do not meet the requirements can be filtered out from all the sampling points included in these M sets of sampling points, and a training point set can be formed by the remaining points, which is also feasible.

[0107] Step 105: Determine the class labels of each point in the training point set according to the class labels of the M feature planes respectively; wherein, the training point set and the class labels of each point in the training point set are used as samples for the training of the neural network.

[0108] Here, for any one point in the training point set, the class label of the feature plane where this point is located can be obtained first, and then according to the class label of the feature plane where this point is located, an accurate class label can be determined for this point. Optionally, when the training point set is obtained by performing uniform sampling processing on the corresponding triangular meshes for each feature plane, step 105 includes:

[0109] Determine the class label of the target sampling point as the class label of the target sampling point; wherein, the target sampling point is any sampling point in the training point set.

[0110] In this implementation manner, based on the class labels of the M feature planes respectively, and combined with which feature plane each sampling point in the training point set is distributed in, the class labels of each point in the training point set can be determined conveniently and reliably.

[0111] After determining the class labels of each point in the training point set, the class labels of each point in the training point set can be used to label each point in the training point set respectively, so that each point in the training point set is labeled with the corresponding class label. After that, the labeled training point set can be used as training data (which is equivalent to a sample) for the training of the neural network to obtain the required neural network. The obtained neural network can be used to extract global features, local features, etc. from the input 3D point cloud data.

[0112] It should be noted that the execution subject of the operation of training a neural network based on samples and the execution subject of the sample data preprocessing method provided by the embodiments of the present disclosure can be either the same execution subject (for example, the same electronic device) or different execution subjects.

[0113] In the embodiments of the present disclosure, first, a surface model representing a scene in space and using N triangular meshes can be extracted from the three-dimensional point cloud data in space; next, M feature planes can be extracted from the N triangular meshes, and according to the three-dimensional point cloud data, the class labels of each of the M feature planes can be determined, and according to the M feature planes, a training point set can be determined; then, according to the class labels of each of the M feature planes, the class labels of each point in the training point set can be determined. The training point set and the class labels of each point in the training point set can be used as samples for training a neural network, so as to train the required neural network. It can be seen that in the embodiments of the present disclosure, the neural network is not directly trained using the three-dimensional point cloud data after manual annotation, but first, based on the three-dimensional point cloud data, M feature planes and the class labels of each of the M feature planes are obtained, and then based on the M feature planes and the class labels of each of the M feature planes, a suitable training point set is obtained, and accurate class labels are obtained for each point in the training point set, and then the obtained training point set and class labels are used for training the neural network. Therefore, the embodiments of the present disclosure can avoid the adverse effects on the training of the neural network caused by inaccurate and incomplete manual annotation information when directly using the three-dimensional point cloud data after manual annotation for training the neural network, thereby being beneficial to improving the accuracy and reliability of the trained neural network.

[0114] In Figure 1 On the basis of the shown embodiment, as Figure 2 shown, step 103 includes:

[0115] Step 1031, according to the three-dimensional point cloud data, determine the class labels of each of the N triangular meshes;

[0116] Step 1032, according to the class labels of each of the N triangular meshes, determine the class labels of each of the M feature planes.

[0117] Here, first, the class labels of each of the N triangular meshes can be determined according to the three-dimensional point cloud data. In a specific implementation manner, determining the class labels of each of the N triangular meshes according to the three-dimensional point cloud data includes:

[0118] From each point in the three-dimensional point cloud data, screen out the point closest to the target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes;

[0119] Obtain the class label of the screened point;

[0120] Determine the class label of the selected points as the class label of the target triangular mesh.

[0121] In this implementation, the distances between each point in the 3D point cloud data and the target triangular network can be calculated separately first. Then, all the calculated distances are compared pairwise. According to the obtained comparison results, the points closest to the target triangular mesh can be screened out from each point in the 3D point cloud data. Next, the class label of the selected points can be obtained. For example, the class label of the manually labeled selected points can be obtained, and then the obtained class label of the points is used as the class label of the target triangular mesh. In a similar way, the class label of each triangular mesh in the N triangular meshes can be determined.

[0122] It can be seen that in this implementation, the points closest to each triangular mesh in the 3D point cloud data can be found, and the class label of this point can be assigned to the corresponding triangular mesh, so that the class label of each triangular mesh can be determined very conveniently and reliably.

[0123] Of course, the specific implementation of determining the respective class labels of the N triangular meshes based on the 3D point cloud data is not limited to this. For example, other factors besides the distance factor can be referred to, and the corresponding points for the target triangular mesh can be screened out from each point in the 3D point cloud data, and the class label of the selected points is assigned to the target triangular mesh.

[0124] After determining the respective class labels of the N triangular meshes, the respective class labels of the M feature planes can be determined according to the respective class labels of the N triangular meshes. Here, for any one of the M feature planes, the triangular meshes distributed on this feature plane among the N triangular meshes can be determined first, and then the class label of this feature plane can be determined according to the respective class labels of the triangular meshes distributed on this feature plane.

[0125] In a specific implementation, determining the respective class labels of the M feature planes according to the respective class labels of the N triangular meshes includes:

[0126] Determine a set of class labels; wherein, the set of class labels includes the respective class labels of the triangular meshes distributed on the target feature plane among the N triangular meshes, and the target feature plane is any one of the M feature planes;

[0127] Screen out the non-repeating class labels in the set of class labels;

[0128] Count the number of occurrences of the screened class labels in the set of class labels;

[0129] Among the selected category labels of each type, the category label with the highest corresponding occurrence frequency is determined as the category label of the target feature plane.

[0130] In this implementation manner, it is possible to first determine each triangular mesh distributed on the target feature plane among the N triangular meshes, and add the category labels of the determined triangular meshes to the category label set one by one. In this way, the category label set can include the category labels of each triangular mesh distributed on the target feature plane.

[0131] Next, it is possible to traverse all the category labels in the category label set to filter out non-repeating category labels of each type. The number of filtered category labels may be 1, 2, 3, 4, or more than 4, which will not be listed one by one here.

[0132] After that, it is possible to count the occurrence frequencies of the selected category labels in the category label set. Suppose the category label set specifically includes N1 category labels of type 1, N2 category labels of type 2, N3 category labels of type 3, and N4 category labels of type 4. Then, by traversing all the category labels in the category label set, it is possible to filter out the 4 category labels of type 1, type 2, type 3, and type 4. And by counting the occurrence frequencies of the selected category labels in the category label set, it is possible to determine that the occurrence frequency corresponding to the category label of type 1 is N1, the occurrence frequency corresponding to the category label of type 2 is N2, the occurrence frequency corresponding to the category label of type 3 is N3, and the occurrence frequency corresponding to the category label of type 4 is N4.

[0133] After counting the occurrence frequencies (i.e., N1 to N4) corresponding to the category labels from type 1 to type 4 respectively, it is possible to compare these counted occurrence frequencies pairwise to determine the maximum occurrence frequency value. The category label corresponding to the maximum occurrence frequency value can be used as the category label of the target feature plane. Here, it is equivalent to adopting a majority voting mechanism to determine the category label of the target feature plane based on the category labels of each triangular mesh distributed on the target feature plane.

[0134] It can be seen that in this implementation manner, by adopting the majority voting mechanism, it is possible to very conveniently and reliably determine the category label of the target feature plane.

[0135] In the embodiments of the present disclosure, according to the three-dimensional point cloud data, it is possible to conveniently and reliably determine the category labels of each of the N triangular meshes. After that, according to the category labels of each of the N triangular meshes, it is possible to conveniently and reliably determine the category labels of each of the M feature planes, so as to perform subsequent processing based on the category labels of each of the M feature planes.

[0136] In an optional example, the three-dimensional point cloud data of a space (such as the indoor space of a house) can be obtained first. In the three-dimensional point cloud data, class labels of each point can be manually marked. Among them, different class labels can be represented by different colors. At this time, the three-dimensional point cloud data can be as Figure 3 shown. It should be noted that Figure 3 there are a few cases of incorrect point markings in the three-dimensional point cloud data in

[0137] The embodiments of the present disclosure can sequentially perform the following operations:

[0138] (1) First, use the Poisson reconstruction algorithm to extract the surface model of the indoor scene from the discrete three-dimensional point cloud data. For example, Figure 4 the surface model shown. The surface model can be represented by N triangular meshes; find the point in the three-dimensional point cloud data that is closest to each triangular mesh, and assign the class label of this point to the corresponding triangular mesh. This is beneficial to eliminating the influence of local noise, holes, uneven distribution, etc. existing in the original three-dimensional point cloud data on the subsequent neural network training.

[0139] (2) Use the RANSAC algorithm or the region growing algorithm to extract M feature planes from the N triangular meshes. Since most indoor scenes are composed of planes, it can be considered that the triangular meshes distributed on any feature plane have the same class label. Then, a majority voting mechanism can be adopted to count the number of label classes of all the triangular meshes distributed on each feature plane, and select the label with the largest number of labels as the label of this feature plane (which is equivalent to determining the class label of the target feature plane as the class label that appears the most times in the above text). For details, please refer to Figure 5 .

[0140] (3) After determining the class labels of each feature plane, uniform sampling processing can be performed on the triangular meshes corresponding to each feature plane, and the label class of the feature plane is assigned to the sampling points distributed thereon. For details, please refer to Figure 6 . In this way, it is possible to eliminate the mislabeling or missing labeling of the class labels of some points generated during the manual marking of the original three-dimensional point cloud data. By using the resampled points for the training of the neural network, the accuracy and generalization of the trained neural network can be greatly increased.

[0141] In summary, in the embodiments of the present disclosure, in view of the defects existing in the original three-dimensional point cloud data, by generating a continuous triangular mesh model based on the discrete three-dimensional point cloud data and uniformly sampling on the triangular mesh, it is possible to make up for the noise, holes, uneven distribution, etc. in the original three-dimensional point cloud data. In addition, by extracting feature planes on the triangular mesh model and using a majority voting method for the points in each feature plane to determine the class label of each point, it is possible to greatly reduce the mislabeling or missing labeling of the class labels of some points generated during the manual annotation process.

[0142] Any of the sample data preprocessing methods provided by the embodiments of the present disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Alternatively, any of the sample data preprocessing methods provided by the embodiments of the present disclosure can be executed by a processor. For example, the processor executes any of the sample data preprocessing methods mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be elaborated below.

[0143] Exemplary device

[0144] Figure 7 is a schematic structural diagram of a sample data preprocessing device provided by an exemplary embodiment of the present disclosure. Figure 7 The illustrated device includes a first extraction module 701, a second extraction module 702, a first determination module 703, a second determination module 704, and a third determination module 705.

[0145] The first extraction module 701 is configured to extract a surface model of the scene in the space from the three-dimensional point cloud data in the space; wherein, the surface model is represented by N triangular meshes.

[0146] The second extraction module 702 is configured to extract M feature planes from the N triangular meshes.

[0147] The first determination module 703 is configured to determine the class label of each of the M feature planes according to the three-dimensional point cloud data.

[0148] The second determination module 704 is configured to determine a training point set according to the M feature planes.

[0149] The third determination module 705 is configured to determine the class label of each point in the training point set according to the class label of each of the M feature planes; wherein, the training point set and the class label of each point in the training point set are used as samples for the training of the neural network.

[0150] In an alternative example, as Figure 8 shown, the first determination module 703 includes:

[0151] The first determination sub-module 7031 is configured to determine the class label of each of the N triangular meshes according to the three-dimensional point cloud data;

[0152] The second determination sub-module 7032 is configured to determine the class label of each of the M feature planes according to the class labels of the N triangular meshes.

[0153] In an optional example, the first determination sub-module 7031 includes:

[0154] The first screening unit is configured to screen, from each point in the three-dimensional point cloud data, the point closest to the target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes;

[0155] The acquisition unit is configured to acquire the class label of the screened point;

[0156] The first determination unit is configured to determine the class label of the screened point as the class label of the target triangular mesh.

[0157] In an optional example, the second determination sub-module 7032 includes:

[0158] The second determination unit is configured to determine a set of class labels; wherein, the set of class labels includes the class labels of each of the triangular meshes distributed on the target feature plane among the N triangular meshes, and the target feature plane is any one of the M feature planes;

[0159] The second screening unit is configured to screen the non-repeating class labels in the set of class labels;

[0160] The statistics unit is configured to count the number of occurrences of each of the screened class labels in the set of class labels;

[0161] The third determination unit is configured to determine the class label with the most occurrences among the screened class labels as the class label of the target feature plane.

[0162] In an optional example, as Figure 8 shown, the second determination module 704 includes:

[0163] The sampling sub-module 7041 is configured to perform uniform sampling on each of the triangular meshes distributed on each of the M feature planes among the N triangular meshes, respectively, to obtain corresponding sampling points;

[0164] The third determination sub-module 7042 is configured to determine a training point set composed of all the obtained sampling points.

[0165] In an optional example, the third determination module 705 is specifically configured to:

[0166] Determine the class label of the feature plane where the target sampling point is located as the class label of the target sampling point; wherein, the target sampling point is any sampling point in the training point set.

[0167] In an optional example, the device further includes:

[0168] An acquisition module, configured to call a depth camera and / or a lidar to acquire three-dimensional point cloud data of the space before extracting the surface model of the scene in the space from the three-dimensional point cloud data of the space.

[0169] Exemplary electronic device

[0170] Next, refer to Figure 9 to describe the electronic device according to an embodiment of the present disclosure. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the acquired input signals from them.

[0171] Figure 9 The block diagram of the electronic device 90 according to an embodiment of the present disclosure is illustrated.

[0172] As Figure 9 shown, the electronic device 90 includes one or more processors 901 and a memory 902.

[0173] The processor 901 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0174] The memory 902 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may run the program instructions to implement the sample data preprocessing method of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0175] In an example, the electronic device 90 may further include: an input device 903 and an output device 904, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0176] For example, when the electronic device 90 is the first device or the second device, the input device 903 can be a microphone or a microphone array. When the electronic device 90 is a stand-alone device, the input device 903 can be a communication network connector for receiving the collected input signals from the first device and the second device.

[0177] In addition, the input device 903 may further include, for example, a keyboard, a mouse, and the like.

[0178] The output device 904 can output various information to the outside. The output device 904 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto, and the like.

[0179] Of course, for simplicity, Figure 9 only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, and the like are omitted. In addition, according to specific application scenarios, the electronic device 90 may further include any other appropriate components.

[0180] Exemplary computer program product and computer-readable storage medium

[0181] In addition to the above methods and devices, embodiments of the present disclosure may also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps in the sample data preprocessing method according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0182] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] In addition, embodiments of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when run by a processor, cause the processor to execute the steps in the sample data preprocessing method according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0184] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0185] The basic principles of the present disclosure have been described in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. Additionally, the specific details disclosed above are only for illustrative and facilitating understanding purposes and are not limitations. The above details do not limit the present disclosure to necessarily implement using the above specific details.

[0186] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and reference can be made to the relevant parts of the method embodiments for the related content.

[0187] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or" and can be used interchangeably with each other, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to" and can be used interchangeably with each other.

[0188] The methods and apparatuses of the present disclosure may be implemented in many ways. For example, the methods and apparatuses of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0189] It should also be noted that in the apparatuses, devices, and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0190] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0191] The above description has been presented for purposes of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. A method for preprocessing sample data, characterized in that, Including: Extracting a surface model of a scene in space from three-dimensional point cloud data of the space; wherein, the surface model is represented by N triangular meshes; Extracting M feature planes from the N triangular meshes; Determining a class label for each of the M feature planes according to the three-dimensional point cloud data; Determining a training point set according to the M feature planes; Determining a class label for each point in the training point set according to the class label of each of the M feature planes; wherein, the training point set and the class label of each point in the training point set are used as samples for training a neural network; The determining a class label for each of the M feature planes according to the three-dimensional point cloud data includes: Determining a class label for each of the N triangular meshes according to the three-dimensional point cloud data; Determining a class label for each of the M feature planes according to the class label of each of the N triangular meshes; The determining a class label for each of the N triangular meshes according to the three-dimensional point cloud data includes: Selecting, from each point in the three-dimensional point cloud data, the point closest to a target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes; Obtaining the class label of the selected point; Determining the class label of the selected point as the class label of the target triangular mesh.

2. The method according to claim 1, wherein The determining a class label for each of the M feature planes according to the class label of each of the N triangular meshes includes: Determining a class label set; wherein, the class label set includes the class labels of each of the triangular meshes distributed on a target feature plane among the N triangular meshes, and the target feature plane is any one of the M feature planes; Selecting the non-repeated class labels in the class label set; Counting the number of occurrences of each of the selected class labels in the class label set; Determining the class label with the most occurrences among the selected class labels as the class label of the target feature plane.

3. The method according to claim 1, wherein The determining a training point set according to the M feature planes includes: For each of the M feature planes, performing uniform sampling on each of the triangular meshes distributed on it among the N triangular meshes to obtain corresponding sampling points; Determining a training point set composed of all the obtained sampling points.

4. The method according to claim 3, characterized in that, The determining a class label for each point in the training point set according to the class label of each of the M feature planes includes: Determining the class label of the feature plane where a target sampling point is located as the class label of the target sampling point; wherein, the target sampling point is any sampling point in the training point set.

5. The method according to any one of claims 1 to 4, characterized in that, Before the extracting a surface model of a scene in space from three-dimensional point cloud data of the space, the method further includes: Invoking a depth camera and / or a lidar to collect three-dimensional point cloud data of the space.

6. A sample data preprocessing device, characterized in that, Including: A first extraction module, configured to extract a surface model of a scene in space from three-dimensional point cloud data of the space; wherein, the surface model is represented by N triangular meshes; A second extraction module, configured to extract M feature planes from the N triangular meshes; A first determination module, configured to determine the class label of each of the M feature planes according to the three-dimensional point cloud data; A second determination module, configured to determine a training point set according to the M feature planes; A third determination module, configured to determine the class label of each point in the training point set according to the class label of each of the M feature planes; wherein, the training point set and the class label of each point in the training point set are used as samples for the training of a neural network; The first determination module includes: A first determination sub-module, configured to determine the class label of each of the N triangular meshes according to the three-dimensional point cloud data; A second determination sub-module, configured to determine the class label of each of the M feature planes according to the class label of each of the N triangular meshes; The first determination sub-module includes: A first screening unit, configured to screen, from each point in the three-dimensional point cloud data, the point closest to a target triangular mesh; wherein, the target triangular mesh is any one of the N triangular meshes; An acquisition unit, configured to acquire the class label of the screened point; A first determination unit, configured to determine the class label of the screened point as the class label of the target triangular mesh.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to execute the sample data preprocessing method according to any one of claims 1 to 5 above.

8. An electronic device, characterized in that, It includes: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the sample data preprocessing method according to any one of claims 1 to 5 above.

Images