Point cloud interactive semantic annotation method and device based on implicit neural field network

Abstract

The present invention provides a method and device for interactive semantic annotation of point clouds based on an implicit neural field network, wherein the method comprises: in response to obtaining a target point cloud to be annotated, determining multiple supervoxels of the target point cloud, as well as the spatial coordinates, appearance features, and geometric features of each point; based on a preset position encoding rule, determining the position encoding corresponding to the spatial coordinates of each point; in response to receiving manual semantic annotation information about a selected point in the target point cloud, based on the position encoding, appearance features, geometric features, and multiple supervoxels corresponding to the spatial coordinates of each point, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic annotation prediction results of each point matched by the output of the optimized implicit neural field network. The method of the present invention can effectively improve the efficiency of semantic annotation of point clouds.

Description

Technical Field

[0001] The present invention relates to the field of point cloud data processing technology, and in particular to a point cloud interactive semantic annotation method, device, electronic device, non-transient computer-readable storage medium, and computer program product based on an implicit neural field network. Background Art

[0002] Point cloud semantic segmentation is a key step in realizing digital twin transportation systems and is widely used in intelligent transportation systems such as street mapping, autonomous driving, and road scene reconstruction. Point cloud data, acquired through devices such as lidar and depth cameras, is high-dimensional, sparse, and disordered, making its processing and analysis challenging.

[0003] In recent years, deep learning methods have made significant progress in image processing and computer vision. Convolutional neural networks (CNNs), in particular, have demonstrated outstanding performance in tasks such as image classification, object detection, and semantic segmentation. However, existing deep learning-based point cloud semantic segmentation methods often rely on large amounts of manually annotated data, which is time-consuming and costly to obtain. In practical applications, ensuring the accuracy and consistency of data annotation is difficult, especially when processing large-scale point cloud data, which can be a daunting task.

[0004] Therefore, how to efficiently perform semantic annotation of point cloud data and reduce the workload of manual annotation has become a problem that needs to be solved. Summary of the Invention

[0005] The present invention provides a point cloud interactive semantic annotation method, device, electronic device, non-transitory computer-readable storage medium, and computer program product based on an implicit neural field network to address the deficiencies in the prior art.

[0006] The present invention provides a point cloud interactive semantic annotation method based on an implicit neural field network, comprising:

[0007] In response to acquiring a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels; wherein each supervoxel includes a plurality of points;

[0008] Based on the preset position coding rules, determine the position coding corresponding to the spatial coordinates of each point;

[0009] In response to receiving manual semantic annotation information about selected points in the target point cloud, performing online iterative optimization on the implicit neural field network established offline based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels using a preset iterative optimization rule;

[0010] In response to the completion of the online iterative optimization of the implicit neural field network, the appearance feature prediction result, geometric feature prediction result and semantic annotation prediction result of each matching point output by the optimized implicit neural field network are obtained.

[0011] The present invention also provides a point cloud interactive semantic annotation device based on an implicit neural field network, comprising:

[0012] a data preprocessing module configured to: in response to acquiring a target point cloud to be annotated, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into a plurality of supervoxels; wherein each supervoxel includes a plurality of points;

[0013] The position coding module is configured to: determine the position coding corresponding to the spatial coordinates of each point based on a preset position coding rule;

[0014] An online interactive optimization module is configured to: in response to receiving manual semantic annotation information about selected points in the target point cloud, perform online iterative optimization on the implicit neural field network established offline based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels using a preset iterative optimization rule;

[0015] The result acquisition module is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, obtain the appearance feature prediction result, geometric feature prediction result and semantic annotation prediction result of each matching point output by the optimized implicit neural field network

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, it implements any of the above-described point cloud interactive semantic annotation methods based on implicit neural field networks.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described point cloud interactive semantic annotation methods based on implicit neural field networks.

[0018] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described point cloud interactive semantic annotation methods based on implicit neural field networks.

[0019] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss (described in the following embodiments) and a supervoxel-based local consistency loss (described in the following embodiments), so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is applicable to point clouds with different sensor types, different acquisition scenarios, and different point numbers and area ranges, effectively improving the efficiency of point cloud semantic annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is one of the flow charts of the method for interactive semantic annotation of point clouds based on implicit neural field networks provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of the implicit neural field network structure provided by an embodiment of the present invention;

[0023] Figure 3 This is the second flow chart of the method for interactive semantic annotation of point clouds based on implicit neural field networks provided by an embodiment of the present invention;

[0024] Figure 4 This is the third flow chart of the method for interactive semantic annotation of point clouds based on implicit neural field networks provided by an embodiment of the present invention;

[0025] Figure 5 Schematic diagram of an interactive interface for adding semantic annotation information to a selected point in the target point cloud provided by an embodiment of the present invention;

[0026] Figure 6 Schematic diagram of the structure of the point cloud interactive semantic annotation device based on the implicit neural field network provided by an embodiment of the present invention;

[0027] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] Summary of the invention concept:

[0030] Implicit neural fields (abbreviated as fields) are a technology that uses neural networks to represent continuous attributes in space. Significant progress has been made in 3D reconstruction and representation learning in recent years. Specifically, implicit neural fields can represent continuous attributes in space through neural networks, effectively handling the high-dimensional, sparse, and disordered nature of point cloud data. In view of this, the present invention considers taking the spatial coordinates of points in the point cloud as the input of the implicit neural field, which can output corresponding attribute values ​​such as color, geometric features, etc.; thereby processing point clouds with complex geometric shapes, and this processing method has advantages in memory efficiency and computational efficiency.

[0031] Example:

[0032] The following describes the implementation details of the point cloud interactive semantic annotation solution based on implicit neural field network of the present invention in conjunction with the accompanying drawings.

[0033] Figure 1 This is a flow chart of a method for interactive semantic annotation of point clouds based on an implicit neural network, provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices (e.g., servers or cloud computing platforms), such as Figure 1 As shown in FIG, the point cloud interactive semantic annotation method based on implicit neural field network includes the following steps:

[0034] S110 : In response to acquiring a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels.

[0035] Each of the supervoxels includes multiple points.

[0036] S120 . Determine the position code corresponding to the spatial coordinates of each point based on a preset position coding rule.

[0037] S130. In response to receiving manual semantic annotation information about the selected points in the target point cloud, based on the position encoding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, the implicit neural field network established offline is optimized online using preset iterative optimization rules.

[0038] S140. In response to the completion of the online iterative optimization of the implicit neural field network, the appearance feature prediction result, the geometric feature prediction result, and the semantic annotation prediction result of each matching point output by the optimized implicit neural field network are obtained.

[0039] Among them, assuming that a target point cloud to be labeled is given , for the i-th point in the point cloud, its spatial coordinates can be expressed as , the appearance can be expressed as (such as color or intensity), geometric features can be represented as (e.g. normal or size features), the semantic label can be expressed as (Initially empty, meaning it is waiting to be marked).

[0040] Optionally, a voxel-based point cloud segmentation algorithm (Voxel Cloud Connectivity Segmentation, VCCS) is used to divide the target point cloud into A subset of perceptually similar points, i.e., generating This processing can eliminate errors inside objects and enhance the network's segmentation ability at object boundaries.

[0041] Among them, the "online iterative optimization of the implicit neural field network established by the line" is the reconstruction of the implicit neural field. Specifically, the implicit neural field is a function whose input is a 3D space coordinate , the output is the attribute of the corresponding point. Field reconstruction refers to optimizing the loss function by minimizing the distance between the output and the point attribute. The spatial distribution of the appearance, geometry and semantics of the point cloud can be represented by three implicit neural fields respectively: appearance field , geometric fields and semantic field The appearance and geometric features of point clouds are dense, while the semantic labels are sparse because only a few points are annotated through human-computer interaction. Based on the assumption that points with similar appearance and geometric features are more likely to belong to the same semantic class, this paper uses an implicit neural field with a shared backbone To jointly reconstruct the exterior scene , geometric fields and semantic field Implicit Neural Field The input is the encoded spatial coordinates, and the output is the appearance feature prediction value , geometric feature prediction value and semantic label prediction values ​​(i.e., semantic probability prediction values) Implicit neural field The reconstruction is based on the appearance feature prediction value , geometric feature prediction value and semantic label prediction values ​​(i.e., semantic probability prediction values) Compared with the ground truth (i.e., the appearance features collected from the target point cloud and geometric features and semantic tags given by users ), use the relevant loss function to calculate the function value of the loss function; then based on the function value of the loss function, it is passed back to the implicit neural field Perform network parameter tuning; the details will be described below.

[0042] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss (described in the following embodiments) and a supervoxel-based local consistency loss (described in the following embodiments), so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is applicable to point clouds with different sensor types, different acquisition scenarios, and different point numbers and area ranges, effectively improving the efficiency of point cloud semantic annotation.

[0043] exist Figure 1 Based on the embodiment, as an optional implementation method, refer to Figure 2 , the implicit neural field network includes a cascaded backbone network ( Figure 2 The light grey rectangular box shown in the figure), the appearance feature network branch (i.e. Figure 2 Appearance shown), geometric feature network branch (i.e. Figure 2 Geometry) and the semantic probability network branch (i.e. Figure 2 The backbone network is cascaded with the appearance feature network branch, the geometric feature network branch, and the semantic probability network branch respectively; and the appearance feature network branch, the geometric feature network branch, and the semantic probability network branch are connected in parallel. Figure 2 The PE shown represents the position code.

[0044] As an optional example, the backbone network and the appearance feature network branch constitute an appearance field network, the backbone network and the geometric feature network branch constitute a geometric field network, and the backbone network and the semantic probability network branch constitute a semantic field network.

[0045] As an optional example, the input end of the backbone network serves as the input end of the implicit neural field network, the output end of the appearance feature network branch is used to output the appearance feature prediction value (for example, appearance vector, such as color, intensity) of each point in the target point cloud, the output end of the geometric feature network branch is used to output the geometric feature prediction value (for example, geometric feature vector, such as normal, size) of each point in the target point cloud, and the output end of the semantic probability network branch is used to output the semantic probability prediction value (for example, semantic probability vector) of each point in the target point cloud.

[0046] As an optional example, the backbone network includes four cascaded first units, each of which includes an activation function ReLU and a 128-channel fully connected layer;

[0047] As an optional example, the appearance feature network branch, the geometric feature network branch, and the semantic probability network branch each include an activation function ReLU and a fully connected layer.

[0048] As an optional example, the position encoding PE input to the backbone network is also synchronously input to the activation function ReLU of the third of the four first units.

[0049] exist Figure 1 Based on the embodiment, as an optional implementation method, refer to Figure 3 and Figure 4 In response to receiving manual semantic annotation information about selected points in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels, using a preset iterative optimization rule, an offline established implicit neural field network is online iteratively optimized, including:

[0050] S410, based on the position code, appearance features and geometric features corresponding to the spatial coordinates of each point, using the first stage iterative optimization rule, perform the first stage iterative optimization on the appearance field network and the geometric field network to obtain the first stage appearance field network (i.e., Figure 3 "Appearance Field Reconstruction") and the first-stage geometry field network (i.e., Figure 3 "Geometric Field Reconstruction" shown).

[0051] Reference Figure 3 The first stage is the corresponding stage shown in Figure (a). Since the appearance feature and geometric feature data are relatively dense, it is preferably necessary to perform the first stage iterative optimization of the appearance field network and the geometric field network in advance, so as to improve the efficiency of the overall joint optimization in the next stage.

[0052] S420. In response to receiving manual semantic annotation information about selected points (sparse) in the target point cloud, based on the position encoding, appearance features, geometric features and the multiple supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using preset iterative optimization rules to obtain the optimized final appearance field network, final geometric field network and final semantic field network.

[0053] Reference Figure 5 , you can use the interactive interface to add semantic annotation information to the selected point in the target point cloud. The highlighted white dot in the figure is the selected point.

[0054] Reference Figure 3 The joint optimization stage is the corresponding stage shown in Figure (b). In this stage, the uncertain semantic entropy loss function and the supervoxel-based local consistency loss function are introduced, which can enable the optimized implicit neural field network to produce deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries.

[0055] As an alternative example, refer to Figure 3 , step S410 may include:

[0056] 1) The spatial coordinates of each point Corresponding position code The appearance field network and the geometry field network are input respectively, and the appearance field network outputs the corresponding first appearance feature prediction value (for example ), the geometric field network outputs the corresponding first geometric feature prediction value (for example ).

[0057] 2) Using the preset comprehensive loss function in the first stage, based on the appearance features corresponding to each point (For example, Figure 3 Color, reflection intensity) and the corresponding first appearance feature prediction value, the geometric features corresponding to each point (For example, Figure 3 The normal and dimensional features shown) and the corresponding first geometric feature prediction value are used to obtain the function value of the preset comprehensive loss function of the first stage.

[0058] Optionally, step 2) can be implemented as follows:

[0059] 21) Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value.

[0060] The following formulas (1) to (2) can be used to calculate the function value of the appearance feature loss function (including RGB color feature loss , reflection intensity loss ), that is, Figure 3 The “appearance field reconstruction loss” shown;

[0061] (1)

[0062] (2)

[0063] in, Indicates the number of points contained in the target point cloud; Represents the predicted RGB color of the i-th point in the target point cloud; Represents the predicted reflection intensity of the i-th point in the target point cloud; Represents the original RGB color of the i-th point in the target point cloud; Here, the original RGB color and original reflection intensity are the RGB color and reflection intensity inherent in the original data collected from the RGB-D camera or lidar.

[0064] 22) Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding first geometric feature prediction value.

[0065] The following formulas (3) to (4) can be used to calculate the function value of the geometric feature loss function (including the normal vector loss , dimensional feature loss ), that is, Figure 3 The “geometric field reconstruction loss” shown;

[0066] (3)

[0067] (4)

[0068] in, Indicates the number of points contained in the target point cloud; Represents the predicted value of the normal direction of the i-th point in the target point cloud; Represents the predicted value of the i-th point dimension feature in the target point cloud; represents the normal direction of the i-th point in the target point cloud determined by step 110; represents the dimensional feature of the i-th point in the target point cloud determined by step 110. That is, and Constitutes the first geometric feature prediction value of the current stage.

[0069] 23) Using the first preset weight, the function value of the appearance feature loss function and the function value of the geometric feature loss function are weightedly fused to obtain the function value of the preset comprehensive loss function of the first stage.

[0070] The preset comprehensive loss function of the first stage can be obtained by the following calculation formula (5): L1 The function value of ;

[0071] (5)

[0072] in, express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of .

[0073] 3) Based on the function value of the preset comprehensive loss function in the first stage, the appearance field network and the network parameters of the appearance field network are optimized in the first stage until the preset first stage iterative optimization completion conditions are met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network.

[0074] Optionally, step 3) can be implemented by the following steps:

[0075] 31) Feeding back the function value of the first-stage preset comprehensive loss function to the appearance field network and the appearance field network; 32) The appearance field network and the appearance field network adjust the first-stage network parameters of each network layer according to the function value of the first-stage preset comprehensive loss function; 33) Iterating the steps of obtaining the function value of the first-stage preset comprehensive loss function to adjusting the first-stage network parameters of each network layer until the preset first-stage iterative optimization completion conditions are met.

[0076] The preset completion condition of the first stage iterative optimization can be that the number of iterations meets the preset number threshold, or the first stage preset comprehensive loss function L1 convergence.

[0077] As an alternative example, refer to Figure 3 , step S420 may include:

[0078] I) The spatial coordinates of each point Corresponding position code , respectively input the first stage appearance field network, the first stage geometric field network and the semantic field network, and output the corresponding second appearance feature prediction value through the first stage appearance field network, the corresponding second geometric feature prediction value through the first stage geometric field network, and the corresponding semantic probability prediction value through the semantic field network (that is, Figure 3 shown ).

[0079] II) Using the preset comprehensive loss function in the second stage, based on the appearance features corresponding to each point (For example, Figure 3 Color, reflection intensity) and the corresponding second appearance feature prediction value, the geometric features corresponding to each point (For example, Figure 3 The normal, dimensional features shown) and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point and the manual semantic annotation information of the selected point (ie, Figure 3 The “sparse annotations given by users” or semantic labels shown ), obtain the function value of the preset comprehensive loss function of the second stage.

[0080] Optionally, step II) can be achieved by:

[0081] II1) Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value.

[0082] Optionally, the specific implementation method may refer to the above step 21).

[0083] II2) Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding second geometric feature prediction value.

[0084] Optionally, the specific implementation method may refer to the above step 22).

[0085] II3) Based on the semantic probability prediction value corresponding to each point and the manual semantic annotation information of the selected point, the semantic reconstruction loss function (ie, Figure 3 The function value of the "semantic field reconstruction loss" shown in Figure 2 is used.

[0086] Optionally, the function value of the semantic reconstruction loss function is calculated by the following formula (6);

[0087] (6)

[0088] in, represents the number of the selected points; Indicates the number of semantic categories involved in the manual semantic annotation information of the selected point; when the i-th selected point belongs to the j-th semantic category The value is 1; when the i-th selected point does not belong to the j-th semantic category The value is 0; Represents the semantic probability prediction value corresponding to the i-th selected point.

[0089] II4) Based on the semantic probability prediction value corresponding to each point, calculate the uncertainty semantic entropy loss function (ie, Figure 3 The function value of "Uncertainty-based Semantic Entropy Loss" is shown in Figure 2.

[0090] Optionally, refer to Figure 3 , calculate the function value of the uncertainty semantic entropy loss function through the following calculation formula (7);

[0091] (7)

[0092] in, Indicates the number of points contained in the target point cloud; Represents the semantic probability prediction value corresponding to the i-th selected point; Indicates the number of semantic categories involved in the manual semantic annotation information of the selected point.

[0093] II5) Based on the semantic probability prediction value corresponding to each point belonging to each supervoxel, a local consistency loss function is calculated for each supervoxel (i.e., Figure 3 The function value of “Supervoxel-based Local Consistency Loss” is shown in Figure 2.

[0094] Optionally, refer to Figure 3 , calculate the function value of the local consistency loss function of each supervoxel by the following calculation formulas (8)~(9);

[0095] (8)

[0096] (9)

[0097] in, Represents supervoxel the number of midpoints; Represents supervoxel The semantic probability prediction value of the t-th point belonging to the j-th semantic category; Indicates the number of supervoxels of the target point cloud division; Indicates the number of semantic categories involved in the manual semantic annotation information of the selected point.

[0098] II6) Using the second preset weight, the function value of the appearance feature loss function, the function value of the geometric feature loss function, the function value of the semantic reconstruction loss function, the function value of the semantic entropy loss function and the function value of the local consistency loss function are weightedly fused to obtain the function value of the second-stage preset comprehensive loss function.

[0099] Optionally, the function value of the preset comprehensive loss function in the second stage is obtained by the following calculation formula (10);

[0100] (10)

[0101] in, express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of .

[0102] III) Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, and the final appearance field network is obtained from the first stage appearance field network, the final geometric field network is obtained from the first stage geometric field network, and the final semantic field network is obtained from the semantic field network (that is, Figure 3 “Semantic Field Reconstruction” shown).

[0103] Optionally, step III) can be achieved by the following steps:

[0104] III-1) The function value of the second-stage preset comprehensive loss function is passed back to the first-stage appearance field network, the first-stage geometric field network and the semantic field network; III-2) The first-stage appearance field network, the first-stage geometric field network and the semantic field network adjust the second-stage network parameters of each network layer according to the function value of the second-stage preset comprehensive loss function; III-3) Obtain the stage semantic probability prediction value output by the semantic field network after adjusting the second-stage network parameters; III-4) In response to receiving the user's updated manual semantic annotation information in the target point cloud based on the stage semantic probability prediction value; III-5) Iterate the step of obtaining the function value of the second-stage preset comprehensive loss function to the step of updating the manual semantic annotation information in the target point cloud until the preset second-stage iterative optimization completion condition is met.

[0105] The preset second-stage iterative optimization completion condition may be an instruction input by the user to end the iterative optimization.

[0106] Based on the above embodiment, as an optional implementation method, in response to obtaining the target point cloud to be labeled, the geometric features of each point in the target point cloud are determined, including: first, determining the neighboring points of each point in the target point cloud; second, based on the neighboring points, using preset feature calculation rules, determining the normal vector and dimensional features of each point.

[0107] Specifically, for each point, find the neighboring points within the radius r and use these neighboring points to calculate the covariance matrix; calculate the eigenvalues ​​by eigenvalue decomposition of the covariance matrix and corresponds to The eigenvector of ; As point-by-point normal direction ; Point-by-point dimension features are calculated using the following formula:

[0108]

[0109] in, Indicates linearity; Indicates surface quality; Indicates physicality.

[0110] Based on the above embodiment, as an optional implementation, determining the position code corresponding to the spatial coordinates of each point based on a preset position coding rule includes:

[0111] Use the following formula to calculate the position code corresponding to the spatial coordinates of each point;

[0112]

[0113] in, represents the position encoding function; represents the three-dimensional coordinates of the i-th point in the target point cloud; D represents a constant.

[0114] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss and a supervoxel-based local consistency loss, so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is applicable to point clouds with different sensor types, different acquisition scenarios, and different number of points and area ranges, effectively improving the efficiency of point cloud semantic annotation.

[0115] The point cloud interactive semantic annotation device based on the implicit neural field network provided by the present invention is described below. The point cloud interactive semantic annotation device based on the implicit neural field network described below and the point cloud interactive semantic annotation method based on the implicit neural field network described above can be referenced to each other.

[0116] Figure 6 FIG is a schematic diagram of the structure of a point cloud interactive semantic annotation device based on an implicit neural field network provided by an exemplary embodiment of the present invention. Figure 6 As shown, the point cloud interactive semantic annotation device based on implicit neural field network includes:

[0117] The data preprocessing module 610 is configured to: in response to obtaining the target point cloud to be labeled, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into multiple supervoxels; wherein each supervoxel includes multiple points; the position encoding module 620 is configured to: based on a preset position encoding rule, determine the position encoding corresponding to the spatial coordinates of each point; the online interactive optimization module 630 is configured to: in response to receiving manual semantic annotation information about the selected point in the target point cloud, based on the position encoding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, use the preset iterative optimization rules to perform online iterative optimization on the implicit neural field network established offline; the result acquisition module 640 is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, obtain the appearance feature prediction results, geometric feature prediction results and semantic annotation prediction results of each point matched by the output of the optimized implicit neural field network.

[0118] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7As shown, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute an interactive semantic labeling method for point clouds based on an implicit neural field network, the method including: in response to obtaining a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each supervoxel includes multiple points; based on a preset position coding rule, determining the position coding corresponding to the spatial coordinates of each point; in response to receiving manual semantic labeling information about a selected point in the target point cloud, based on the position coding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic labeling prediction results of each point matched by the output of the optimized implicit neural field network.

[0119] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the point cloud interactive semantic annotation method based on the implicit neural field network provided by the above methods, the method including: in response to obtaining a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each supervoxel includes multiple points; based on a preset position encoding rule, determining the position code corresponding to the spatial coordinates of each point; in response to receiving manual semantic annotation information about a selected point in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction result, geometric feature prediction result and semantic annotation prediction result of each point matched by the output of the optimized implicit neural field network.

[0121] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the point cloud interactive semantic annotation method based on the implicit neural field network provided by the above-mentioned methods, the method comprising: in response to obtaining a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each of the supervoxels includes multiple points; based on a preset position encoding rule, determining the position encoding corresponding to the spatial coordinates of each point; in response to receiving manual semantic annotation information about a selected point in the target point cloud, based on the position encoding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic annotation prediction results of each point matched by the output of the optimized implicit neural field network.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0123] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A point cloud interactive semantic annotation method based on implicit neural field network, characterized in that: include: In response to acquiring a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; Based on the preset position coding rule, determine the position coding corresponding to the spatial coordinates of each point; In response to receiving manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels, the implicit neural field network established offline is optimized online using a preset iterative optimization rule; In response to the completion of the online iterative optimization of the implicit neural field network, the appearance feature prediction result, the geometric feature prediction result and the semantic annotation prediction result of each matching point output by the optimized implicit neural field network are obtained.

2. The point cloud interactive semantic annotation method according to claim 1, characterized in that: The implicit neural field network includes a cascaded backbone network, an appearance feature network branch, a geometric feature network branch, and a semantic probability network branch; The backbone network is cascaded with the appearance feature network branch, the geometric feature network branch and the semantic probability network branch respectively; The appearance feature network branch, the geometric feature network branch and the semantic probability network branch are connected in parallel; The backbone network and the appearance feature network branch form an appearance field network, the backbone network and the geometric feature network branch form a geometric field network, and the backbone network and the semantic probability network branch form a semantic field network. The input end of the backbone network serves as the input end of the implicit neural field network, the output end of the appearance feature network branch is used to output the appearance feature prediction value of each point in the target point cloud, the output end of the geometric feature network branch is used to output the geometric feature prediction value of each point in the target point cloud, and the output end of the semantic probability network branch is used to output the semantic probability prediction value of each point in the target point cloud; The backbone network includes four cascaded first units, each of which includes an activation function ReLU and a 128-channel fully connected layer; Among them, the appearance feature network branch, the geometric feature network branch and the semantic probability network branch respectively include an activation function ReLU and a fully connected layer.

3. The point cloud interactive semantic annotation method according to claim 2, characterized in that: In response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance feature, geometric feature and the plurality of supervoxels corresponding to the spatial coordinates of each point, the implicit neural field network established offline is optimized online using a preset iterative optimization rule, including: Based on the position coding, appearance features and geometric features corresponding to the spatial coordinates of each point, the first-stage iterative optimization rule is used to perform the first-stage iterative optimization on the appearance field network and the geometric field network to obtain the first-stage appearance field network and the first-stage geometric field network; In response to receiving manual semantic annotation information about selected points in the target point cloud, based on the position encoding, appearance features, geometric features and the multiple supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using preset iterative optimization rules to obtain the optimized final appearance field network, final geometric field network and final semantic field network.

4. The point cloud interactive semantic annotation method according to claim 3, characterized in that: Based on the position coding, appearance features and geometric features corresponding to the spatial coordinates of each point, the appearance field network and the geometric field network are iteratively optimized in the first stage using the first stage iterative optimization rule to obtain the first stage appearance field network and the first stage geometric field network, including: Inputting the position code corresponding to the spatial coordinates of each point into the appearance field network and the geometric field network respectively, outputting the corresponding first appearance feature prediction value through the appearance field network, and outputting the corresponding first geometric feature prediction value through the geometric field network; Using the first-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value, the geometric feature corresponding to each point and the corresponding first geometric feature prediction value, obtaining the function value of the first-stage preset comprehensive loss function; Based on the function value of the preset comprehensive loss function in the first stage, the appearance field network and the network parameters of the appearance field network are subjected to the first stage iterative optimization until the preset first stage iterative optimization completion conditions are met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network.

5. The point cloud interactive semantic annotation method according to claim 4, characterized in that: Using the first-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value, the geometric feature corresponding to each point and the corresponding first geometric feature prediction value, obtaining the function value of the first-stage preset comprehensive loss function, including: Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value; Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding first geometric feature prediction value; The function value of the appearance feature loss function and the function value of the geometric feature loss function are weightedly fused using the first preset weight to obtain the function value of the preset comprehensive loss function of the first stage.

6. The point cloud interactive semantic annotation method according to claim 4, characterized in that: Based on the function value of the preset comprehensive loss function in the first stage, the appearance field network and the network parameters of the appearance field network are subjected to the first stage iterative optimization until the preset first stage iterative optimization completion condition is met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network, including: Passing back the function value of the preset comprehensive loss function of the first stage to the appearance field network and the appearance field network; The appearance field network and the appearance field network adjust the first-stage network parameters of each network layer according to the function value of the preset comprehensive loss function of the first stage; Iteratively execute the step of obtaining the function value of the preset comprehensive loss function of the first stage to the step of adjusting the first stage network parameters of each network layer until the preset first stage iterative optimization completion condition is met.

7. The point cloud interactive semantic annotation method according to claim 3, characterized in that: In response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features and the plurality of supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using a preset iterative optimization rule to obtain the optimized final appearance field network, the final geometric field network and the final semantic field network, including: The position code corresponding to the spatial coordinates of each point is input into the first-stage appearance field network, the first-stage geometric field network and the semantic field network respectively, and the first-stage appearance field network outputs the corresponding second appearance feature prediction value, the first-stage geometric field network outputs the corresponding second geometric feature prediction value, and the semantic field network outputs the corresponding semantic probability prediction value; Utilizing the second-stage preset comprehensive loss function, based on the appearance features corresponding to each point and the corresponding second appearance feature prediction value, the geometric features corresponding to each point and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point, and the manual semantic annotation information of the selected point, obtaining the function value of the second-stage preset comprehensive loss function; Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, and the final appearance field network is obtained from the first stage appearance field network, the final geometric field network is obtained from the first stage geometric field network, and the final semantic field network is obtained from the semantic field network.

8. The point cloud interactive semantic annotation method according to claim 7, characterized in that: Using the second-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value, the geometric feature corresponding to each point and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point, and the manual semantic annotation information of the selected point, obtaining the function value of the second-stage preset comprehensive loss function, including: Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value; Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding second geometric feature prediction value; Calculating the function value of the semantic reconstruction loss function based on the semantic probability prediction value corresponding to each point and the manual semantic annotation information of the selected point; Based on the semantic probability prediction value corresponding to each point, calculating the function value of the uncertain semantic entropy loss function; Calculate the function value of the local consistency loss function of each supervoxel based on the semantic probability prediction value corresponding to each point belonging to each supervoxel; Using the second preset weight, the function value of the appearance feature loss function, the function value of the geometric feature loss function, the function value of the semantic reconstruction loss function, the function value of the semantic entropy loss function and the function value of the local consistency loss function are weightedly fused to obtain the function value of the second stage preset comprehensive loss function.

9. The point cloud interactive semantic annotation method according to claim 7, characterized in that: Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, including: Passing back the function value of the preset comprehensive loss function of the second stage to the appearance field network of the first stage, the geometric field network of the first stage, and the semantic field network; The first-stage appearance field network, the first-stage geometric field network, and the semantic field network adjust the second-stage network parameters of each network layer according to the function value of the second-stage preset comprehensive loss function; Obtaining a stage semantic probability prediction value output by the semantic field network after adjusting the second stage network parameters; In response to receiving the user's prediction value of the semantic probability of the stage, updating the manual semantic annotation information in the target point cloud; Iteratively execute the step of obtaining the function value of the preset comprehensive loss function of the second stage to the step of updating the artificial semantic annotation information in the target point cloud until the preset second stage iterative optimization completion condition is met.

10. A point cloud interactive semantic annotation device based on implicit neural field network, characterized in that: include: The data preprocessing module is configured to: in response to acquiring a target point cloud to be annotated, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; The position coding module is configured to: determine the position coding corresponding to the spatial coordinates of each point based on a preset position coding rule; The online interactive optimization module is configured to: in response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance feature, geometric feature and the plurality of supervoxels corresponding to the spatial coordinates of each point, perform online iterative optimization on the implicit neural field network established offline by using a preset iterative optimization rule; The result acquisition module is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, obtain the appearance feature prediction results, geometric feature prediction results and semantic annotation prediction results of each matching point output by the optimized implicit neural field network.

Images