Point cloud interactive semantic annotation method and device based on implicit neural field network

Abstract

The invention provides a point cloud interactive semantic annotation method and device based on an implicit neural field network, and the method comprises the steps: determining a plurality of super voxels of a target point cloud, and the space coordinates, appearance features and geometric features of each point in response to the obtained to-be-annotated target point cloud; determining a position code corresponding to the space coordinate of each point based on a preset position coding rule; in response to the received artificial semantic annotation information about the selected point in the target point cloud, performing online iterative optimization on the implicit neural field network established offline by using a preset iterative optimization rule based on the position code, the appearance feature, the geometric feature and the plurality of super voxels corresponding to the space coordinate of each point; and in response to the ending of the online iterative optimization of the implicit neural field network, obtaining an appearance feature prediction result, a geometric feature prediction result and a semantic annotation prediction result matched with each point output by the optimized implicit neural field network. According to the method, the semantic annotation efficiency of the point cloud can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of point cloud data processing technology, and in particular to a point cloud interactive semantic annotation method, device, electronic device, non-transitory computer-readable storage medium, and computer program product based on an implicit neural field network. Background Art

[0002] Point cloud semantic segmentation is a key step in realizing digital twin transportation systems and is widely used in intelligent transportation systems such as street map drawing, autonomous driving, and road scene reconstruction. Point cloud data is obtained through devices such as lidar or depth cameras, and is high-dimensional, sparse, and disordered, so processing and analyzing this data is challenging.

[0003] In recent years, deep learning methods have made significant progress in the fields of image processing and computer vision. In particular, convolutional neural networks (CNNs) have performed well in tasks such as image classification, object detection, and semantic segmentation. However, existing deep learning-based point cloud semantic segmentation methods usually rely on a large amount of manually annotated data, which is time-consuming and costly to obtain. In practical applications, the accuracy and consistency of data annotation are difficult to guarantee, especially when processing large-scale point cloud data, where the annotation workload is huge.

[0004] Therefore, how to efficiently perform semantic annotation of point cloud data and reduce the workload of manual annotation has become a problem that needs to be solved. Summary of the invention

[0005] The present invention provides a point cloud interactive semantic annotation method, device, electronic device, non-transitory computer-readable storage medium, and computer program product based on an implicit neural field network to solve the defects in the prior art.

[0006] The present invention provides a point cloud interactive semantic annotation method based on an implicit neural field network, comprising: In response to acquiring a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; Based on the preset position coding rules, determine the position coding corresponding to the spatial coordinates of each point; In response to receiving manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels, the implicit neural field network established offline is optimized online using a preset iterative optimization rule; In response to the completion of the online iterative optimization of the implicit neural field network, the appearance feature prediction result, the geometric feature prediction result and the semantic annotation prediction result of each matching point output by the optimized implicit neural field network are obtained.

[0007] The present invention also provides a point cloud interactive semantic annotation device based on an implicit neural field network, comprising: The data preprocessing module is configured to: in response to acquiring a target point cloud to be annotated, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; The position coding module is configured to: determine the position coding corresponding to the spatial coordinates of each point based on a preset position coding rule; The online interactive optimization module is configured to: in response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance feature, geometric feature and the plurality of supervoxels corresponding to the spatial coordinates of each point, perform online iterative optimization on the implicit neural field network established offline by using a preset iterative optimization rule; The result acquisition module is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, acquire the appearance feature prediction result, geometric feature prediction result and semantic annotation prediction result of each matching point output by the optimized implicit neural field network; The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-described point cloud interactive semantic annotation methods based on implicit neural field networks.

[0008] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described point cloud interactive semantic annotation methods based on an implicit neural field network.

[0009] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned point cloud interactive semantic annotation methods based on implicit neural field networks.

[0010] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss (described in the following embodiments) and a supervoxel-based local consistency loss (described in the following embodiments), so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is applicable to point clouds with different sensor types, different acquisition scenarios, and different number of points and area ranges, effectively improving the efficiency of point cloud semantic annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is one of the flow diagrams of the method for interactive semantic annotation of point cloud based on implicit neural field network provided by an embodiment of the present invention; Figure 2 is a schematic diagram of the implicit neural field network structure provided by an embodiment of the present invention; Figure 3 This is the second flow chart of the method for interactive semantic annotation of point clouds based on implicit neural field networks provided by an embodiment of the present invention; Figure 4 This is the third flow chart of the method for interactive semantic annotation of point cloud based on implicit neural field network provided by an embodiment of the present invention; Figure 5 is a schematic diagram of an interactive interface for adding semantic annotation information to a selected point in the target point cloud provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of a point cloud interactive semantic annotation device based on an implicit neural field network provided by an embodiment of the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] Summary of the invention concept: Implicit neural field (field for short) is a technology that uses neural networks to represent continuous attributes in space, and has made significant progress in 3D reconstruction and representation learning in recent years. Specifically, implicit neural fields can represent continuous attributes in space through neural networks, effectively dealing with the high-dimensional, sparse and disordered problems of point cloud data. In view of this, the scheme of the present invention considers taking the spatial coordinates of points in the point cloud as the input of the implicit neural field, and the implicit neural field can output corresponding attribute values, such as color, geometric features, etc.; thereby processing point clouds with complex geometric shapes, and this processing method has advantages in memory efficiency and computational efficiency.

[0015] Example: The following describes the implementation details of the point cloud interactive semantic annotation solution based on implicit neural field network of the present invention in conjunction with the accompanying drawings.

[0016] Figure 1 1 is a flow chart of a method for interactive semantic annotation of point clouds based on an implicit neural field network provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices (e.g., servers or cloud computing platforms), such as Figure 1 As shown in FIG, the point cloud interactive semantic annotation method based on implicit neural field network includes the following steps: S110 , in response to acquiring a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels.

[0017] Each of the supervoxels includes a plurality of points.

[0018] S120. Determine the position code corresponding to the spatial coordinates of each point based on a preset position coding rule.

[0019] S130. In response to receiving manual semantic annotation information about selected points in the target point cloud, based on the position encoding, appearance features, geometric features and the multiple supervoxels corresponding to the spatial coordinates of each point, the implicit neural field network established offline is optimized online using preset iterative optimization rules.

[0020] S140. In response to the completion of the online iterative optimization of the implicit neural field network, an appearance feature prediction result, a geometric feature prediction result, and a semantic annotation prediction result matching each point output by the optimized implicit neural field network are obtained.

[0021] Among them, assuming that a target point cloud to be labeled is given , for the i-th point in the point cloud, its spatial coordinates can be expressed as , the appearance can be expressed as (such as color or intensity), geometric features can be represented as (e.g. normal or size features), the semantic label can be expressed as (Initially empty, meaning it is waiting to be marked).

[0022] Optionally, a supervoxel-based point cloud segmentation algorithm (Voxel Cloud Connectivity Segmentation, VCCS) is used to divide the target point cloud into A subset of perceptually similar points, i.e., generating This processing can eliminate errors inside objects and enhance the network's segmentation ability at object boundaries.

[0023] Among them, the "online iterative optimization of the implicit neural field network established by the line" is the reconstruction of the implicit neural field. Specifically, the implicit neural field is a function whose input is a 3D space coordinate , the output is the attribute of the corresponding point. The reconstruction of the field is to optimize the loss function by minimizing the distance between the output of the loss function and the point attribute. The spatial distribution of the appearance, geometry and semantics of the point cloud can be represented by three implicit neural fields respectively: appearance field , Geometric Field and semantic field The appearance and geometric features of point clouds are dense, while the semantic labels are sparse because only a few points are annotated through human-computer interaction. Based on the assumption that points with similar appearance and geometric features are more likely to belong to the same semantic class, the present invention uses an implicit neural field with a shared backbone To jointly reconstruct the exterior scene , Geometric Field and semantic field Implicit Neural Field The input is the encoded spatial coordinates, and the output is the appearance feature prediction value , Geometric feature prediction value and semantic label predictions (i.e., semantic probability predictions) Implicit Neural Field The reconstruction is based on the appearance feature prediction value , Geometric feature prediction value and semantic label predictions (i.e., semantic probability predictions) Compared with the ground truth (i.e., the appearance features collected from the target point cloud and geometric features And the semantic tags given by the user ), use the relevant loss function to calculate the function value of the loss function; then based on the function value of the loss function, it is passed back to the implicit neural field Perform network parameter tuning; the specific details will be described below.

[0024] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss (described in the following embodiments) and a supervoxel-based local consistency loss (described in the following embodiments), so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is applicable to point clouds with different sensor types, different acquisition scenarios, and different number of points and area ranges, effectively improving the efficiency of point cloud semantic annotation.

[0025] exist Figure 1 Based on the embodiment, as an optional implementation method, refer to Figure 2 , the implicit neural field network includes a cascaded backbone network ( Figure 2 The light grey rectangular box shown in the figure), the appearance feature network branch (i.e. Figure 2 Appearance), geometric feature network branch (i.e. Figure 2 Geometry) and the semantic probability network branch (i.e. Figure 2 The backbone network is cascaded with the appearance feature network branch, the geometric feature network branch and the semantic probability network branch respectively; the appearance feature network branch, the geometric feature network branch and the semantic probability network branch are connected in parallel. Figure 2 The PE shown represents the position encoding.

[0026] As an optional example, the backbone network and the appearance feature network branch constitute an appearance field network, the backbone network and the geometric feature network branch constitute a geometric field network, and the backbone network and the semantic probability network branch constitute a semantic field network.

[0027] As an optional example, the input end of the backbone network serves as the input end of the implicit neural field network, the output end of the appearance feature network branch is used to output the appearance feature prediction value (for example, appearance vector, such as color, intensity) for each point in the target point cloud, the output end of the geometric feature network branch is used to output the geometric feature prediction value (for example, geometric feature vector, such as normal, size) for each point in the target point cloud, and the output end of the semantic probability network branch is used to output the semantic probability prediction value (for example, semantic probability vector) for each point in the target point cloud.

[0028] As an optional example, the backbone network includes four cascaded first units, each of which includes an activation function ReLU and a 128-channel fully connected layer; As an optional example, the appearance feature network branch, the geometric feature network branch, and the semantic probability network branch respectively include an activation function ReLU and a fully connected layer.

[0029] As an optional example, the position encoding PE input to the backbone network is also synchronously input to the activation function ReLU of the third of the four first units.

[0030] exist Figure 1 Based on the embodiment, as an optional implementation method, refer to Figure 3 and Figure 4 In response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance feature, geometric feature and the plurality of supervoxels corresponding to the spatial coordinates of each point, using a preset iterative optimization rule, an implicit neural field network established offline is online iteratively optimized, including: S410, based on the position code, appearance features and geometric features corresponding to the spatial coordinates of each point, using the first stage iterative optimization rule, perform the first stage iterative optimization on the appearance field network and the geometric field network to obtain the first stage appearance field network (that is, Figure 3 The "appearance field reconstruction" shown) and the first-stage geometry field network (i.e., Figure 3 "Geometric Field Reconstruction" shown).

[0031] Reference Figure 3 The first stage is the corresponding stage as shown in Figure (a). Since the appearance feature and geometric feature data are relatively dense, it is preferably necessary to perform the first stage of iterative optimization on the appearance field network and the geometric field network in advance, so as to improve the efficiency of the overall joint optimization in the next stage.

[0032] S420. In response to receiving manual semantic annotation information about selected points (sparse) in the target point cloud, based on the position coding, appearance features, geometric features and the multiple supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using preset iterative optimization rules to obtain the optimized final appearance field network, final geometric field network and final semantic field network.

[0033] Reference Figure 5 , you can use the graphical interactive interface to add semantic annotation information to the selected point in the target point cloud. The highlighted white dot in the figure is the selected point.

[0034] Reference Figure 3 The joint optimization stage is the corresponding stage as shown in the figure (b). In this stage, the uncertain semantic entropy loss function and the supervoxel-based local consistency loss function are introduced, which can make the optimized implicit neural field network produce deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries.

[0035] As an alternative example, refer to Figure 3 , step S410 may include: 1) The spatial coordinates of each point The corresponding position code The appearance field network and the geometry field network are respectively input, and the appearance field network outputs the corresponding first appearance feature prediction value (for example ), and the corresponding first geometric feature prediction value (e.g. ).

[0036] 2) Using the preset comprehensive loss function in the first stage, based on the appearance features corresponding to each point (For example, Figure 3 The color, reflection intensity shown) and the corresponding first appearance feature prediction value, the geometric feature corresponding to each point (For example, Figure 3 The normal and dimensional features shown in the figure) and the corresponding first geometric feature prediction value are used to obtain the function value of the preset comprehensive loss function of the first stage.

[0037] Optionally, step 2) can be implemented as follows: 21) Based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value, calculate the function value of the appearance feature loss function.

[0038] The following formulas (1) to (2) can be used to calculate the function value of the appearance feature loss function (including RGB color feature loss , reflection intensity loss ), that is, Figure 3 The "appearance field reconstruction loss" shown; (1) (2) in, Indicates the number of points contained in the target point cloud; Represents the predicted RGB color of the i-th point in the target point cloud; Represents the predicted reflection intensity of the i-th point in the target point cloud; Represents the original RGB color of the i-th point in the target point cloud; Represents the original reflection intensity of the i-th point in the target point cloud. Here, the original RGB color and original reflection intensity are the RGB color and reflection intensity inherent in the original data collected from the RGB-D camera or lidar.

[0039] 22) Based on the geometric features corresponding to each point and the corresponding first geometric feature prediction value, calculate the function value of the geometric feature loss function.

[0040] The following calculation formulas (3) to (4) can be used to calculate the function value of the geometric feature loss function (including the normal vector loss , dimensional feature loss ), that is, Figure 3 The "geometry field reconstruction loss" shown; (3) (4) in, Indicates the number of points contained in the target point cloud; Represents the predicted value of the normal direction of the i-th point in the target point cloud; Represents the predicted value of the feature dimension of the i-th point in the target point cloud; represents the normal direction of the i-th point in the target point cloud determined by step 110; represents the dimensional feature of the i-th point in the target point cloud determined by step 110. That is, and Constitutes the first geometric feature prediction value of the current stage.

[0041] 23) Using the first preset weight, weighted fusion is performed on the function value of the appearance feature loss function and the function value of the geometric feature loss function to obtain the function value of the preset comprehensive loss function of the first stage.

[0042] The preset comprehensive loss function of the first stage can be obtained by the following calculation formula (5): L1 The function value of ; (5) in, express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of .

[0043] 3) Based on the function value of the preset comprehensive loss function of the first stage, the appearance field network and the network parameters of the appearance field network are optimized in the first stage until the preset first stage iterative optimization completion conditions are met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network.

[0044] Optionally, step 3) can be implemented by the following steps: 31) Feeding back the function value of the first-stage preset comprehensive loss function to the appearance field network and the appearance field network; 32) The appearance field network and the appearance field network adjust the first-stage network parameters of each network layer according to the function value of the first-stage preset comprehensive loss function; 33) Iterating the step of obtaining the function value of the first-stage preset comprehensive loss function to the step of adjusting the first-stage network parameters of each network layer until the preset first-stage iterative optimization completion conditions are met.

[0045] The preset first stage iterative optimization completion condition may be that the number of iterations meets the preset number threshold, or the first stage preset comprehensive loss function L1 convergence.

[0046] As an alternative example, refer to Figure 3 , step S420 may include: I) The spatial coordinates of each point The corresponding position code , respectively input the first stage appearance field network, the first stage geometric field network and the semantic field network, and output the corresponding second appearance feature prediction value through the first stage appearance field network, the corresponding second geometric feature prediction value through the first stage geometric field network, and the corresponding semantic probability prediction value through the semantic field network (that is, Figure 3 Shown ).

[0047] II) Using the preset comprehensive loss function in the second stage, based on the appearance features corresponding to each point (For example, Figure 3The color, reflection intensity shown) and the corresponding second appearance feature prediction value, the geometric feature corresponding to each point (For example, Figure 3 The normal, dimensional features shown) and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point and the artificial semantic annotation information of the selected point (ie, Figure 3 The “sparse annotations given by users” or semantic labels shown ), obtain the function value of the preset comprehensive loss function of the second stage.

[0048] Optionally, step II) can be implemented as follows: II1) Based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value, calculate the function value of the appearance feature loss function.

[0049] Optionally, the specific implementation method may refer to the above step 21).

[0050] II2) Based on the geometric features corresponding to each point and the corresponding second geometric feature prediction value, calculate the function value of the geometric feature loss function.

[0051] Optionally, the specific implementation method may refer to the above step 22).

[0052] II3) Based on the semantic probability prediction value corresponding to each point and the manual semantic annotation information of the selected point, the semantic reconstruction loss function is calculated (ie, Figure 3 The function value of the "semantic field reconstruction loss" shown in Figure 2.

[0053] Optionally, the function value of the semantic reconstruction loss function is calculated by the following calculation formula (6); (6) in, represents the number of the selected points; Indicates the number of semantic categories involved in the artificial semantic annotation information of the selected point; when the i-th selected point belongs to the j-th semantic category The value is 1; when the i-th selected point does not belong to the j-th semantic category The value is 0; Represents the semantic probability prediction value corresponding to the i-th selected point.

[0054] II4) Based on the semantic probability prediction value corresponding to each point, the uncertainty semantic entropy loss function is calculated (ie, Figure 3 The function value of "Uncertainty-based Semantic Entropy Loss") is shown in Figure 2.

[0055] Optionally, refer to Figure 3, calculate the function value of the uncertain semantic entropy loss function through the following calculation formula (7); (7) in, Indicates the number of points contained in the target point cloud; Represents the semantic probability prediction value corresponding to the i-th selected point; Indicates the number of semantic categories involved in the manual semantic annotation information of the selected point.

[0056] II5) Based on the semantic probability prediction value corresponding to each point belonging to each supervoxel, the local consistency loss function of each supervoxel is calculated (i.e., Figure 3 The function value of “Supervoxel-based local consistency loss”) is shown.

[0057] Optionally, refer to Figure 3 , calculate the function value of the local consistency loss function of each supervoxel by the following calculation formulas (8)~(9); (8) (9) in, Supervoxel the number of midpoints; Supervoxel The semantic probability prediction value that the t-th point belongs to the j-th semantic category; Indicates the number of supervoxels into which the target point cloud is divided; Indicates the number of semantic categories involved in the manual semantic annotation information of the selected point.

[0058] II6) Using the second preset weight, the function value of the appearance feature loss function, the function value of the geometric feature loss function, the function value of the semantic reconstruction loss function, the function value of the semantic entropy loss function and the function value of the local consistency loss function are weightedly fused to obtain the function value of the second stage preset comprehensive loss function.

[0059] Optionally, the function value of the preset comprehensive loss function in the second stage is obtained by the following calculation formula (10); (10) in, express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of express The weight coefficient of .

[0060] III) Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, and the final appearance field network is obtained from the first stage appearance field network, the final geometric field network is obtained from the first stage geometric field network, and the final semantic field network is obtained from the semantic field network (that is, Figure 3 “Semantic Field Reconstruction” shown).

[0061] Optionally, step III) can be achieved by the following steps: III-1) Feeding back the function value of the preset comprehensive loss function of the second stage to the appearance field network of the first stage, the geometric field network of the first stage and the semantic field network; III-2) The appearance field network of the first stage, the geometric field network of the first stage and the semantic field network adjust the second stage network parameters of each network layer according to the function value of the preset comprehensive loss function of the second stage; III-3) Obtaining the stage semantic probability prediction value output by the semantic field network after adjusting the second stage network parameters; III-4) In response to receiving the user's updated artificial semantic annotation information in the target point cloud based on the stage semantic probability prediction value; III-5) Iterating the step of obtaining the function value of the preset comprehensive loss function of the second stage to the step of updating the artificial semantic annotation information in the target point cloud until the preset second stage iterative optimization completion condition is met.

[0062] The preset second-stage iterative optimization completion condition may be an instruction input by the user to end the iterative optimization.

[0063] Based on the above embodiment, as an optional implementation, in response to obtaining a target point cloud to be labeled, determining the geometric features of each point in the target point cloud, including: first, determining the neighboring points of each point in the target point cloud; second, based on the neighboring points, using preset feature calculation rules, determining the normal vector and dimensional features of each point.

[0064] Specifically, for each point, find the neighboring points within radius r and use these neighboring points to calculate the covariance matrix; calculate the eigenvalues ​​by eigenvalue decomposition of the covariance matrix and corresponds to The eigenvector of ; As point-by-point normal direction ; The point-by-point dimensional features are calculated by the following formula:

[0065] in, Indicates linearity; Indicates surface quality; Indicates body.

[0066] Based on the above embodiment, as an optional implementation manner, determining the position code corresponding to the spatial coordinates of each point based on a preset position coding rule includes: Use the following calculation formula to calculate the position code corresponding to the spatial coordinates of each point;

[0067] in, represents the position encoding function; represents the three-dimensional coordinates of the i-th point in the target point cloud; D represents a constant.

[0068] As described above, the present invention jointly characterizes the appearance, geometric distribution, and semantics of the point cloud through an implicit neural field with a shared backbone, thereby achieving accurate propagation of the user's sparse manual annotations throughout the point cloud. In addition, the present invention designs an uncertainty-based semantic entropy loss and a supervoxel-based local consistency loss, so that the implicit neural field network produces deterministic predictions with local consistency, thereby obtaining smoother and more accurate segmentation boundaries. Compared with the existing state-of-the-art methods, the present invention has a faster response speed and higher segmentation accuracy, and is suitable for point clouds with different sensor types, different acquisition scenarios, and different number of points and area ranges, effectively improving the efficiency of point cloud semantic annotation.

[0069] The point cloud interactive semantic labeling device based on implicit neural field network provided by the present invention is described below. The point cloud interactive semantic labeling device based on implicit neural field network described below and the point cloud interactive semantic labeling method based on implicit neural field network described above can be referenced to each other.

[0070] Figure 6 Schematic diagram of the structure of a point cloud interactive semantic annotation device based on an implicit neural field network provided by an exemplary embodiment of the present invention. Figure 6 As shown, the point cloud interactive semantic annotation device based on implicit neural field network includes: The data preprocessing module 610 is configured to: in response to obtaining the target point cloud to be labeled, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into multiple supervoxels; wherein each of the supervoxels includes multiple points; the position encoding module 620 is configured to: based on a preset position encoding rule, determine the position encoding corresponding to the spatial coordinates of each point; the online interactive optimization module 630 is configured to: in response to receiving manual semantic annotation information about the selected point in the target point cloud, based on the position encoding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, use preset iterative optimization rules to perform online iterative optimization on the implicit neural field network established offline; the result acquisition module 640 is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, obtain the appearance feature prediction results, geometric feature prediction results, and semantic annotation prediction results of each point matched by the output of the optimized implicit neural field network.

[0071] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute an interactive semantic labeling method for point clouds based on an implicit neural field network, the method comprising: in response to obtaining a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each of the supervoxels includes multiple points; based on a preset position coding rule, determining the position coding corresponding to the spatial coordinates of each point; in response to receiving manual semantic labeling information about a selected point in the target point cloud, based on the position coding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic labeling prediction results matching each point output by the optimized implicit neural field network.

[0072] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the point cloud interactive semantic labeling method based on the implicit neural field network provided by the above methods, the method including: in response to obtaining a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each of the supervoxels includes multiple points; based on a preset position coding rule, determining the position coding corresponding to the spatial coordinates of each point; in response to receiving manual semantic labeling information about a selected point in the target point cloud, based on the position coding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using preset iterative optimization rules, the implicit neural field network established offline is online iteratively optimized; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic labeling prediction results matching each point output by the optimized implicit neural field network.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the point cloud interactive semantic labeling method based on an implicit neural field network provided by the above-mentioned methods, the method comprising: in response to obtaining a target point cloud to be labeled, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into multiple supervoxels; wherein each of the supervoxels includes multiple points; based on a preset position coding rule, determining the position coding corresponding to the spatial coordinates of each point; in response to receiving manual semantic labeling information about a selected point in the target point cloud, based on the position coding, appearance features, geometric features corresponding to the spatial coordinates of each point and the multiple supervoxels, using a preset iterative optimization rule, performing online iterative optimization on the implicit neural field network established offline; in response to the completion of the online iterative optimization of the implicit neural field network, obtaining the appearance feature prediction results, geometric feature prediction results, and semantic labeling prediction results matching each point output by the optimized implicit neural field network.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud interactive semantic annotation method based on implicit neural field network, characterized in that: include: In response to acquiring a target point cloud to be annotated, determining the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and dividing the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; Based on the preset position coding rule, determine the position coding corresponding to the spatial coordinates of each point; In response to receiving manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels, the implicit neural field network established offline is optimized online using a preset iterative optimization rule; In response to the completion of the online iterative optimization of the implicit neural field network, the appearance feature prediction result, the geometric feature prediction result and the semantic annotation prediction result of each matching point output by the optimized implicit neural field network are obtained.

2. The point cloud interactive semantic annotation method according to claim 1, characterized in that: The implicit neural field network includes a cascaded backbone network, an appearance feature network branch, a geometric feature network branch, and a semantic probability network branch; The backbone network is cascaded with the appearance feature network branch, the geometric feature network branch and the semantic probability network branch respectively; The appearance feature network branch, the geometric feature network branch and the semantic probability network branch are connected in parallel; The backbone network and the appearance feature network branch form an appearance field network, the backbone network and the geometric feature network branch form a geometric field network, and the backbone network and the semantic probability network branch form a semantic field network. The input end of the backbone network serves as the input end of the implicit neural field network, the output end of the appearance feature network branch is used to output the appearance feature prediction value of each point in the target point cloud, the output end of the geometric feature network branch is used to output the geometric feature prediction value of each point in the target point cloud, and the output end of the semantic probability network branch is used to output the semantic probability prediction value of each point in the target point cloud; The backbone network includes four cascaded first units, each of which includes an activation function ReLU and a 128-channel fully connected layer; Among them, the appearance feature network branch, the geometric feature network branch and the semantic probability network branch respectively include an activation function ReLU and a fully connected layer.

3. The point cloud interactive semantic annotation method according to claim 2, characterized in that: In response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features corresponding to the spatial coordinates of each point and the plurality of supervoxels, the implicit neural field network established offline is optimized online using a preset iterative optimization rule, including: Based on the position coding, appearance features and geometric features corresponding to the spatial coordinates of each point, the first-stage iterative optimization rule is used to perform the first-stage iterative optimization on the appearance field network and the geometric field network to obtain the first-stage appearance field network and the first-stage geometric field network; In response to receiving manual semantic annotation information about selected points in the target point cloud, based on the position encoding, appearance features, geometric features and the multiple supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using preset iterative optimization rules to obtain the optimized final appearance field network, final geometric field network and final semantic field network.

4. The point cloud interactive semantic annotation method according to claim 3, characterized in that: Based on the position coding, appearance features and geometric features corresponding to the spatial coordinates of each point, the appearance field network and the geometric field network are iteratively optimized in the first stage using the first stage iterative optimization rule to obtain the first stage appearance field network and the first stage geometric field network, including: Inputting the position code corresponding to the spatial coordinates of each point into the appearance field network and the geometric field network respectively, outputting the corresponding first appearance feature prediction value through the appearance field network, and outputting the corresponding first geometric feature prediction value through the geometric field network; Using the first-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value, the geometric feature corresponding to each point and the corresponding first geometric feature prediction value, obtaining the function value of the first-stage preset comprehensive loss function; Based on the function value of the preset comprehensive loss function in the first stage, the appearance field network and the network parameters of the appearance field network are subjected to the first stage iterative optimization until the preset first stage iterative optimization completion conditions are met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network.

5. The point cloud interactive semantic annotation method according to claim 4, characterized in that: Using the first-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value, the geometric feature corresponding to each point and the corresponding first geometric feature prediction value, obtaining the function value of the first-stage preset comprehensive loss function, including: Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding first appearance feature prediction value; Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding first geometric feature prediction value; The function value of the appearance feature loss function and the function value of the geometric feature loss function are weightedly fused using the first preset weight to obtain the function value of the preset comprehensive loss function of the first stage.

6. The point cloud interactive semantic annotation method according to claim 4, characterized in that: Based on the function value of the preset comprehensive loss function in the first stage, the appearance field network and the network parameters of the appearance field network are subjected to the first stage iterative optimization until the preset first stage iterative optimization completion condition is met, and the first stage appearance field network is obtained from the appearance field network, and the first stage geometric field network is obtained from the geometric field network, including: Passing back the function value of the preset comprehensive loss function of the first stage to the appearance field network and the appearance field network; The appearance field network and the appearance field network adjust the first-stage network parameters of each network layer according to the function value of the preset comprehensive loss function of the first stage; Iteratively execute the step of obtaining the function value of the preset comprehensive loss function of the first stage to the step of adjusting the first stage network parameters of each network layer until the preset first stage iterative optimization completion condition is met.

7. The point cloud interactive semantic annotation method according to claim 3, characterized in that: In response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance features, geometric features and the plurality of supervoxels corresponding to the spatial coordinates of each point, the first-stage appearance field network, the first-stage geometric field network and the semantic field network are jointly optimized using a preset iterative optimization rule to obtain the optimized final appearance field network, the final geometric field network and the final semantic field network, including: The position code corresponding to the spatial coordinates of each point is input into the first-stage appearance field network, the first-stage geometric field network and the semantic field network respectively, and the first-stage appearance field network outputs the corresponding second appearance feature prediction value, the first-stage geometric field network outputs the corresponding second geometric feature prediction value, and the semantic field network outputs the corresponding semantic probability prediction value; Utilizing the second-stage preset comprehensive loss function, based on the appearance features corresponding to each point and the corresponding second appearance feature prediction value, the geometric features corresponding to each point and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point, and the manual semantic annotation information of the selected point, obtaining the function value of the second-stage preset comprehensive loss function; Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, and the final appearance field network is obtained from the first stage appearance field network, the final geometric field network is obtained from the first stage geometric field network, and the final semantic field network is obtained from the semantic field network.

8. The point cloud interactive semantic annotation method according to claim 7, characterized in that: Using the second-stage preset comprehensive loss function, based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value, the geometric feature corresponding to each point and the corresponding second geometric feature prediction value, the semantic probability prediction value corresponding to each point, and the manual semantic annotation information of the selected point, obtaining the function value of the second-stage preset comprehensive loss function, including: Calculating a function value of an appearance feature loss function based on the appearance feature corresponding to each point and the corresponding second appearance feature prediction value; Calculating a function value of a geometric feature loss function based on the geometric feature corresponding to each point and the corresponding second geometric feature prediction value; Calculating the function value of the semantic reconstruction loss function based on the semantic probability prediction value corresponding to each point and the manual semantic annotation information of the selected point; Based on the semantic probability prediction value corresponding to each point, calculating the function value of the uncertain semantic entropy loss function; Calculate the function value of the local consistency loss function of each supervoxel based on the semantic probability prediction value corresponding to each point belonging to each supervoxel; Using the second preset weight, the function value of the appearance feature loss function, the function value of the geometric feature loss function, the function value of the semantic reconstruction loss function, the function value of the semantic entropy loss function and the function value of the local consistency loss function are weightedly fused to obtain the function value of the second stage preset comprehensive loss function.

9. The point cloud interactive semantic annotation method according to claim 7, characterized in that: Based on the function value of the preset comprehensive loss function of the second stage, the first stage appearance field network, the first stage geometric field network and the semantic field network are jointly optimized until the preset second stage iterative optimization completion conditions are met, including: Passing back the function value of the preset comprehensive loss function of the second stage to the appearance field network of the first stage, the geometric field network of the first stage, and the semantic field network; The first-stage appearance field network, the first-stage geometric field network, and the semantic field network adjust the second-stage network parameters of each network layer according to the function value of the second-stage preset comprehensive loss function; Obtaining a stage semantic probability prediction value output by the semantic field network after adjusting the second stage network parameters; In response to receiving the user's prediction value of the semantic probability of the stage, updating the manual semantic annotation information in the target point cloud; Iteratively execute the step of obtaining the function value of the preset comprehensive loss function of the second stage to the step of updating the artificial semantic annotation information in the target point cloud until the preset second stage iterative optimization completion condition is met.

10. A point cloud interactive semantic annotation device based on implicit neural field network, characterized in that: include: The data preprocessing module is configured to: in response to acquiring a target point cloud to be annotated, determine the spatial coordinates, appearance features, and geometric features of each point in the target point cloud, and divide the target point cloud into a plurality of supervoxels; wherein each of the supervoxels includes a plurality of points; The position coding module is configured to: determine the position coding corresponding to the spatial coordinates of each point based on a preset position coding rule; The online interactive optimization module is configured to: in response to receiving the manual semantic annotation information about the selected point in the target point cloud, based on the position code, appearance feature, geometric feature and the plurality of supervoxels corresponding to the spatial coordinates of each point, perform online iterative optimization on the implicit neural field network established offline by using a preset iterative optimization rule; The result acquisition module is configured to: in response to the completion of the online iterative optimization of the implicit neural field network, obtain the appearance feature prediction results, geometric feature prediction results and semantic annotation prediction results of each matching point output by the optimized implicit neural field network.

Images