Point cloud instance segmentation method and related system and storage medium

Abstract

The embodiment of the present application provides a point cloud instance segmentation method and related systems and storage media. It relates to the field of artificial intelligence, and specifically to the field of computer vision. The method includes: processing the point cloud in the target scene to obtain M superpoints and the feature vector of each superpoint in the M superpoints, where M is an integer not less than 2; obtaining a semantic superpoint tree and the feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints; dividing the semantic superpoint tree into K subtrees according to the feature vector of each node, where K is a positive integer; obtaining the semantic categories of the K instances in the point cloud according to the K subtrees, and the K subtrees correspond one-to-one to the K instances. Compared with the prior art, the entire process of this solution does not require manual setting of parameters, reduces the human factors that affect the segmentation accuracy, and effectively improves the segmentation accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a point cloud instance segmentation method and related systems and storage media. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a science about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Computer vision can be applied to 3D instance segmentation to distinguish scenes. Among them, 3D instance segmentation can distinguish various individuals in each category in the scene. At present, 3D instance segmentation has attracted more and more attention, and corresponding methods have been proposed one after another. The ideas of many methods are mainly divided into two categories: instance segmentation based on candidate regions (proposal-based) and instance segmentation without candidate regions (proposal-free). Among them, proposal-based first obtains the candidate region of interest in the scene, and further predicts the 3D data in the candidate region to obtain the instance label. Considering that proposal-based instance segmentation usually requires two processes (first obtain the candidate region, then instance segmentation), the segmentation process is cumbersome, and proposal-free abandons the candidate region-based method and directly obtains the instance segmentation result through data features or combined with semantic segmentation results.

[0004] Currently, proposal-free clustering uses the adjacent coordinate features of points belonging to the same instance to obtain the semantic category of the point cloud of interest and determine the instance to which each point belongs. However, since the region growing algorithm used in clustering requires setting the radius parameter, this parameter will have a greater impact on the clustering result and significantly reduce the segmentation accuracy. Summary of the invention

[0005] The present application discloses a point cloud instance segmentation method and related system and storage medium, which can effectively improve the segmentation accuracy without manually setting parameters.

[0006] In a first aspect, an embodiment of the present application provides a point cloud instance segmentation method, comprising: processing a point cloud in a target scene to obtain M super points and a feature vector of each of the M super points, where M is an integer not less than 2; obtaining a semantic super point tree and a feature vector of each node (not) in the semantic super point tree based on the feature vectors of the M super points; dividing the semantic super point tree into K subtrees based on the feature vector of each node, where K is a positive integer; obtaining semantic categories of K instances in the point cloud based on the K subtrees, where the K subtrees correspond one-to-one to the K instances.

[0007] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0008] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0009] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0010] As an optional implementation, the method also includes: obtaining the number of points corresponding to each of the at least two subnodes; and obtaining the feature vector of each non-leaf node based on the number of points corresponding to each subnode and the feature vector of each subnode.

[0011] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, and the non-leaf nodes include a root node, and the semantic superpoint tree is divided into K subtrees according to the feature vector of each node, including: obtaining the score of each non-leaf node according to the feature vector of each child node in the at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continuing to traverse other nodes until all leaf nodes have subordinate subtrees and stopping the traversal to obtain K subtrees.

[0012] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0013] As an optional implementation manner, obtaining the score of each non-leaf node according to the feature vector of each child node in at least two child nodes corresponding to each non-leaf node includes:

[0014] The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0015] As an optional implementation, the scoring model is learned based on the feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample, and the score of each node sample is obtained based on the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0016] As an optional implementation, the method also includes: processing each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; deleting the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0017] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0018] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtaining the score of each subtree based on the feature vector of the point in the leaf node of each subtree; deleting the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

[0019] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0020] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the K subtrees; obtaining the score of each subtree according to the feature vector of the point in the leaf node of each subtree; deleting the subtrees in the K subtrees whose scores are less than a fourth preset score to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0021] In a second aspect, an embodiment of the present application provides a method for determining an object based on a three-dimensional scene, comprising: receiving a request sent by a user, the request carrying a first target, and the request being used to indicate acquisition of the first target; acquiring a point cloud in a target scene, wherein the target scene includes the first target; performing instance segmentation processing on the point cloud in the target scene to obtain the first target, wherein the instance segmentation processing is performed based on a tree structure; and displaying the first target to the user.

[0022] By adopting the method for determining objects based on three-dimensional scenes provided by this solution, complete object segmentation and good segmentation effect can be achieved.

[0023] Among them, the above-mentioned instance segmentation processing includes: processing the point cloud in the target scene to obtain M super points (super point) and the feature vector of each super point in the M super points, M is an integer not less than 2; obtaining a semantic super point tree and the feature vector of each node (not) in the semantic super point tree according to the feature vectors of the M super points; dividing the semantic super point tree into K subtrees according to the feature vector of each node, K is a positive integer; obtaining the semantic categories of K instances in the point cloud according to the K subtrees, and the K subtrees correspond one-to-one to the K instances.

[0024] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0025] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0026] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0027] As an optional implementation, the method also includes: obtaining the number of points corresponding to each of the at least two subnodes; and obtaining the feature vector of each non-leaf node based on the number of points corresponding to each subnode and the feature vector of each subnode.

[0028] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, and the non-leaf nodes include a root node, and the semantic superpoint tree is divided into K subtrees according to the feature vector of each node, including: obtaining the score of each non-leaf node according to the feature vector of each child node in the at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continuing to traverse other nodes until all leaf nodes have subordinate subtrees and stopping the traversal to obtain K subtrees.

[0029] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0030] As an optional implementation manner, obtaining the score of each non-leaf node according to the feature vector of each child node in at least two child nodes corresponding to each non-leaf node includes:

[0031] The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0032] As an optional implementation, the scoring model is learned based on the feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample, and the score of each node sample is obtained based on the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0033] As an optional implementation, the method also includes: processing each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; deleting the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0034] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0035] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtaining the score of each subtree based on the feature vector of the point in the leaf node of each subtree; deleting the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

[0036] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0037] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the K subtrees; obtaining the score of each subtree according to the feature vector of the point in the leaf node of each subtree; deleting the subtrees in the K subtrees whose scores are less than a fourth preset score to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0038] In a third aspect, an embodiment of the present application provides a point cloud instance segmentation device, comprising: a processing module, used to process the point cloud in the target scene to obtain M superpoints and a feature vector of each of the M superpoints, where M is an integer not less than 2; a tree building module, used to obtain a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree based on the feature vectors of the M superpoints; a segmentation module, used to segment the semantic superpoint tree into K subtrees based on the feature vector of each node, where K is a positive integer; and a determination module, used to obtain semantic categories of K instances in the point cloud based on the K subtrees, and the K subtrees correspond one-to-one to the K instances.

[0039] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0040] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0041] As an optional implementation method, the tree building module is also used to: obtain the number of points corresponding to each of the at least two child nodes; and obtain the feature vector of each non-leaf node based on the number of points corresponding to each child node and the feature vector of each child node.

[0042] As an optional implementation method, the segmentation module is used to: obtain the score of each non-leaf node based on the feature vector of each child node in at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traverse the semantic superpoint tree from the root node to the leaf node, and when the score of the node exceeds a second preset score, treat the node and the branch corresponding to the node as a subtree, and stop traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continue traversing other nodes until all leaf nodes have subordinate subtrees, and obtain K subtrees.

[0043] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0044] As an optional implementation method, the segmentation module is also used to: input the feature vectors of at least two child nodes corresponding to each non-leaf node into the scoring model for processing to obtain the score of each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0045] As an optional implementation method, the device also includes a first update module, which is used to: process each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; delete the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0046] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0047] As another optional implementation method, the device also includes a second updating module, which is used to: obtain the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0048] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0049] As another optional implementation method, the device also includes a third update module, which is used to: obtain the feature vector of each point in the leaf node of each subtree in the K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees in the K subtrees whose scores are less than a fourth preset score, and obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0050] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0051] In a fourth aspect, the present application provides a device for determining an object based on a three-dimensional scene, comprising: a receiving module, used to receive a request sent by a user, the request carrying a first target, and the request being used to indicate the acquisition of the first target; an acquisition module, used to acquire a point cloud in a target scene, wherein the target scene includes the first target; a processing module, used to perform instance segmentation processing on the point cloud in the target scene to obtain the first target, wherein the instance segmentation processing is performed based on a tree structure; and a display module, used to display the first target to the user.

[0052] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0053] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0054] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0055] As an optional implementation method, the acquisition module is also used to: obtain the number of points corresponding to each sub-node of the at least two sub-nodes; and obtain the feature vector of each non-leaf node based on the number of points corresponding to each sub-node and the feature vector of each sub-node.

[0056] As an optional implementation method, the processing module is used to: obtain the score of each non-leaf node based on the feature vector of each child node in at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traverse the semantic superpoint tree from the root node to the leaf node, and when the score of the node exceeds a second preset score, treat the node and the branch corresponding to the node as a subtree, and stop traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continue traversing other nodes until all leaf nodes have subordinate subtrees, and obtain K subtrees.

[0057] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0058] As an optional implementation, the processing module is further used to:

[0059] The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0060] As an optional implementation, the scoring model is learned based on the feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample, and the score of each node sample is obtained based on the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0061] As an optional implementation method, the processing module is also used to: process each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; delete the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0062] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0063] As another optional implementation method, the processing module is also used to: obtain the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0064] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0065] As another optional implementation method, the processing module is also used to: obtain the feature vector of each point in the leaf node of each subtree in the K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees in the K subtrees whose scores are less than a fourth preset score, and obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0066] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0067] Among them, the above instance segmentation processing can refer to any one of the implementation methods provided in the first aspect above.

[0068] By adopting the device for determining objects based on three-dimensional scenes provided by the present solution, complete object segmentation and good segmentation effect can be achieved.

[0069] In a fifth aspect, the present application provides a point cloud instance segmentation device, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the described method.

[0070] In a sixth aspect, the present application provides a device for determining objects in a three-dimensional scene, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method for determining objects in a three-dimensional scene.

[0071] In a seventh aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a method provided in any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0072] In an eighth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute a method provided in any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0073] In a ninth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute a method provided by any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0074] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method provided by any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0075] In a tenth aspect, an electronic device is provided, comprising the point cloud instance segmentation device of any one of the third to fifth aspects.

[0076] The electronic device may also include the device for determining objects in a three-dimensional scene according to any one of the sixth aspects mentioned above.

[0077] It can be understood that the device described in the third aspect, the device described in the fourth aspect, the device described in the fifth aspect, the device described in the sixth aspect, the computer storage medium described in the seventh aspect, the computer program product described in the eighth aspect, or the chip provided in the ninth aspect are all used to execute any method provided in the first aspect and any method provided in the second aspect.

[0078] Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The following is an introduction to the drawings used in the embodiments of the present application.

[0080] Figure 1 It is a structural diagram of the system architecture provided by the embodiment of the present application;

[0081] Figure 2 It is a schematic diagram of performing point cloud instance segmentation processing according to a CNN model provided in an embodiment of the present application;

[0082] Figure 3 It is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0083] Figure 4 It is a flow chart of a point cloud instance segmentation method provided in an embodiment of the present application;

[0084] Figure 5 is a schematic diagram of point cloud processing provided by an embodiment of the present application;

[0085] Figure 6 is a schematic diagram of a semantic superpoint tree provided in an embodiment of the present application;

[0086] Figure 7 is a schematic diagram of determining a node score provided in an embodiment of the present application;

[0087] Figure 8 It is a schematic diagram of a subtree processing provided by an embodiment of the present application;

[0088] Fig. 9 It is a structural schematic diagram of a point cloud instance segmentation device provided in an embodiment of the present application;

[0089] Fig.10 It is a structural schematic diagram of another point cloud instance segmentation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0091] The point cloud instance segmentation method provided in the embodiment of the present application can be applied to the scene of augmented reality (AR) map, and can also be applied to robots, point cloud editing scenes, etc. Among them, this solution can also be applied to vehicle control scenes such as autonomous driving.

[0092] The following is an introduction to the scenarios of augmented reality AR maps:

[0093] For example, a user puts on AR glasses and hopes that the virtual dog will go near a toy duck based on the scene he sees. The electronic device receives a request input by the user, and the request carries a first target, such as a toy duck, and the request is used to instruct to find the toy duck.

[0094] At present, the existing technology cannot achieve accurate point cloud segmentation, so the target it obtains may be incomplete, such as missing a part, or having an extra part, or even not being found or finding the wrong one. However, the present solution can achieve good segmentation effect and high segmentation accuracy.

[0095] The following is an introduction to the robot scene:

[0096] For example, the user sends a command to the robot to bring an apple over. The robot needs to find the apple first. Therefore, the point cloud of the target scene needs to be processed.

[0097] At present, the existing technology cannot achieve accurate point cloud segmentation, so the apple obtained may be incomplete, such as missing a part, or having an extra part, or even not being found or being found incorrectly. However, the present solution can achieve good segmentation effect and high segmentation accuracy.

[0098] The following is an introduction to the point cloud editing scene:

[0099] For example, if the user wants to replace the sofa in the living room with a new one, he wants to see how the new sofa will look first. Therefore, he needs to find the original sofa in the living room, remove it from the point cloud, and put the new sofa in.

[0100] At present, the existing technology cannot achieve accurate point cloud segmentation, so the sofa obtained may be incomplete, such as missing a part, or having an extra part, or even not found or found wrong. However, the present solution can achieve good segmentation effect and high segmentation accuracy.

[0101] The solution provided in this application can also be applied to vehicle control scenarios such as autonomous driving.

[0102] For example, if you need to reach the position corresponding to an object in autonomous driving, you need to perform instance segmentation on the point cloud in the current scene to determine the position corresponding to the object.

[0103] To facilitate understanding, relevant terms and other related concepts involved in the embodiments of the present application are first introduced below.

[0104] (1) Point Cloud refers to a collection of points obtained after obtaining the spatial coordinates of each sampling point on the surface of an object. The point cloud in this solution can be a point cloud obtained based on the principle of laser measurement, or a point cloud obtained based on the principle of photogrammetry, or any other point cloud, which is not specifically limited in this solution.

[0105] (2) Super point refers to a set of three-dimensional points with similar geometric features. A super point contains multiple points. The similarity of geometric features can be color, shape, texture, etc.

[0106] (3) A tree structure is a nested structure. The outer and inner layers of a tree structure have similar structures, so this structure can often be represented recursively. Various tree diagrams in classical data structures are typical tree structures, such as binary trees and ternary trees.

[0107] (4) Node: refers to a data element in a tree. Nodes include leaf nodes and non-leaf nodes. Leaf nodes are also called terminal nodes, nodes without subtrees, etc. Except for leaf nodes, all other nodes can be called non-leaf nodes.

[0108] Among them, non-leaf nodes include the root node.

[0109] It should be noted that a non-leaf node in this solution corresponds to at least two child nodes.

[0110] The child node refers to the node to which it is subordinate. For example, two nodes a and b are subordinate to node 1, that is, node 1 corresponds to two nodes a and b, and these two nodes are the child nodes of node 1.

[0111] (5) Instance refers to the specificity of an abstract concept class to a physical object of that class. In other words, an instance is a precise distinction between different individuals in the same class, and can refer to any physical object.

[0112] (6) Semantics: people, animals, tables, etc. belong to different semantics.

[0113] (7) Neural Network

[0114] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0115]

[0116] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0117] (8) Deep Neural Networks

[0118] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0119] (9) Convolutional Neural Networks

[0120] Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0121] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0122] (10) Recurrent Neural Networks (RNN) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this common neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network of the current step, but also on the state of the network of the previous steps. This learning algorithm is called Back Propagation Through Time (BPTT).

[0123] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0124] (11) Loss function

[0125] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0126] (12) Back propagation algorithm

[0127] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0128] (13) Pixel value

[0129] The pixel value of an image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For a grayscale image, the pixel value can be a grayscale value.

[0130] The following introduces the system architecture provided by the embodiments of the present application.

[0131] See attached Figure 1, an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, a point cloud instance segmentation execution device 110 is used to perform instance segmentation on a point cloud. The architecture includes an execution device 110, a client device 140, and a data acquisition device 160. Among them, the data acquisition device 160 is used to collect data, such as a point cloud in a target scene. The execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, etc., and can also be a server or a cloud. In the attached Figure 1 In the embodiment, the execution device 110 is configured with an I / O interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data in the embodiment of the present application may include: reaching a specific instance in the target scene, etc. The input data may be input by the user or in other forms, which is not specifically limited in the present solution.

[0132] The preprocessing module 113 is used to perform preprocessing according to the input data (such as point cloud, etc.) received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 can be used to process the point cloud to obtain super points.

[0133] The preprocessing module 114 is used to preprocess the input data received by the I / O interface 112, such as the point cloud. In the embodiment of the present application, the preprocessing module 114 can be used to process the point cloud to obtain the feature vector of each point, etc.

[0134] The execution device 110 pre-processes the input data, or the instance segmentation module 111 of the execution device 110 performs instance segmentation and other related processing. Specifically, the instance segmentation module 111 processes the point cloud in the target scene to obtain M superpoints and a feature vector of each superpoint in the M superpoints, where M is an integer not less than 2; obtains a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints; divides the semantic superpoint tree into K subtrees according to the feature vector of each node, where K is a positive integer; obtains semantic categories of K instances in the point cloud according to the K subtrees, where the K subtrees correspond one-to-one to the K instances.

[0135] Furthermore, the result obtained by the instance segmentation module 111 can also be output to the preprocessing module 113 for processing (not shown in the figure), so as to obtain the final processing result that meets the user's request.

[0136] Finally, the I / O interface 112 returns the processing result, such as the navigation route to a specific instance that the user wants to obtain, to the client device 140, so as to provide it to the user.

[0137] In the attached Figure 1 In the case shown in , the user can manually give input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data, it is necessary to obtain the user's authorization, and the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be display, sound, action and other specific methods.

[0138] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 1 In the embodiment, the execution device 110 is an external device relative to the client device 140. In other cases, the execution device 110 may also be placed in the client device 140. Further, the data collection device 160 may also be placed in the client device 140, or in the execution device 110, etc.

[0139] As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the point cloud input into it.

[0140] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230 .

[0141] Convolutional layer / pooling layer 220:

[0142] Convolutional Layer:

[0143] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0144] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0145] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0146] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0147] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0148] Pooling layer:

[0149] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 2 Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0150] Neural Network Layer 230:

[0151] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 2 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0152] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0153] It should be noted that if Figure 2 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0154] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0155] Figure 3 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 50. The chip can be arranged as follows: Figure 1 In the execution device 110 shown in FIG. 1 , the processing work of the instance segmentation module 111 is completed. The chip can also be set in Figure 1 The preprocessing modules 113 and 114 shown in FIG. 1 are used to complete related preprocessing tasks. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 is implemented in the chip shown.

[0156] The neural network processor NPU 50NPU is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0157] In some implementations, the operation circuit 503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0158] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 508.

[0159] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0160] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified memory 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0161] The unified memory 506 is used to store input data and output data.

[0162] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the storage unit access controller 505 (Direct Memory Access Controller, DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0163] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0164] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0165] The controller 504 is used to call the instructions cached in the instruction fetch memory 509 to control the working process of the computing accelerator.

[0166] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated as DDR SDRAM), a high bandwidth memory (High Bandwidth Memory, HBM) or other readable and writable memory.

[0167] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0168] Reference Figure 4 As shown, it is a flow chart of a point cloud instance segmentation method provided by an embodiment of the present application. Figure 4 As shown, the method includes steps 401-404, which are specifically as follows:

[0169] 401. Process the point cloud in the target scene to obtain M super points and a feature vector of each of the M super points, where M is an integer not less than 2;

[0170] The above target scene can be an AR map scene, a robot application scene, or a point cloud editing scene, etc.

[0171] As an optional implementation, the above processing of the point cloud in the target scene to obtain M super points and a feature vector of each super point in the M super points may include the following steps:

[0172] The point cloud in the target scene is subjected to super-segmentation processing to obtain M super-points. Optionally, the super-points may be generated by using a division method based on a normal similarity measure of each point. The M super-points do not overlap with each other.

[0173] On the other hand, the method further includes voxelizing the point cloud in the target scene to obtain a processed point cloud. The voxelization process refers to dividing the space where the model is located into grids. Voxelization can simplify the model and obtain a uniform grid.

[0174] Then, the processed point cloud is input into the semantic segmentation network U-Net for processing to obtain the feature vector f of each point in the point cloud. Further, the feature vector f of each point is input into the multi-layer perceptron (MLP) neural network for processing to obtain the semantic score a and predicted offset b of each point.

[0175] Mean pooling is performed on each of the M super points according to the feature vector f, semantic score a and predicted offset b of each point to obtain the first feature vector f', semantic score a' and offset of each super point.

[0176] The position coordinates of the predicted center of each super point are obtained according to the offset of each super point and the pooled value of the origin coordinates.

[0177] The instance attribute a" of each superpoint can be obtained by combining the semantic score a' of each superpoint and the position coordinates of the predicted center of each superpoint.

[0178] Specifically, Figure 5 As shown, the point set P of N×3 point cloud coordinates and N×3 RGB color information is voxelized according to the preset voxel size d=2cm, wherein N×3 means that each point in N points has 3 values, such as the 3 values ​​are x, y, and z, representing the three-dimensional space coordinate value of the point; then a U-Net with submanifold sparse convolution and sparse convolution is constructed, and the feature vector of each point is obtained by using U-Net.

[0179] For the semantic branch, MLP is used to generate semantic scores a for K categories of N points. For the offset branch, the features of each point are encoded to obtain N offsets of N points.

[0180] After obtaining the above information of each point, the feature vector, semantic score and offset of each super point can be obtained by performing mean pooling on each super point.

[0181] Among them, the feature vector of the super point in the embodiment of the present application can be the above-mentioned first feature vector, or it can be the above-mentioned instance attribute a", and it can also be the feature vector obtained by combining the first feature vector of the super point with the instance attribute a" of the super point. This solution does not make specific limitations on this.

[0182] 402. Obtain a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints;

[0183] The above semantic superpoint tree is constructed by gradually merging nodes whose similarity exceeds a preset threshold according to the feature vectors of the M superpoints and repeating the process until a root node is obtained by merging.

[0184] The above semantic super-point tree may be a binary tree or a ternary tree, etc., and this solution does not make any specific limitation on this.

[0185] Among them, merging two nodes whose similarity exceeds the preset threshold can obtain a binary tree, such as Figure 6 , which is a schematic diagram of a semantic superpoint tree provided by an embodiment of the present application. Accordingly, by merging three nodes whose similarity exceeds a preset threshold, a ternary tree or the like can be obtained.

[0186] The nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes.

[0187] Specifically, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes;

[0188] For any non-leaf node A1, any of the at least two child nodes corresponding to the non-leaf node A1 is a super node;

[0189] Alternatively, any of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node;

[0190] Alternatively, at least one of the at least two child nodes corresponding to the non-leaf node A1 is a super node, and at least one of the child nodes is a non-leaf node;

[0191] The leaf nodes in the semantic superpoint tree correspond to the M superpoints one-to-one, and the feature vectors of the leaf nodes correspond to the feature vectors of the M superpoints one-to-one.

[0192] That is, a leaf node corresponds to the above-mentioned super node. A non-leaf node may be obtained by merging at least two super nodes, or at least two non-leaf nodes, or at least one super node and at least one non-leaf node.

[0193] like Figure 6 In the schematic diagram of the semantic superpoint tree shown, the leaf nodes in the semantic superpoint tree are p0, p1, p2, p3, p4, p5, p6, p7, p8 and p9, and the non-leaf nodes are t0, t1, t2, t3, t4, t5, t6, t7, t8.

[0194] Among them, node t0 is the root node, node t1 and node t2 are two child nodes of node t0. Correspondingly, node t3 and node t4 are two child nodes of node t1. Node P0 and node p1 are two child nodes of node t3.

[0195] As an optional implementation method, the feature vector of each non-leaf node mentioned above can be obtained by obtaining the number of points corresponding to each sub-node of the at least two sub-nodes; and then obtaining the feature vector of each non-leaf node based on the number of points corresponding to each sub-node and the feature vector of each sub-node.

[0196] For example, for node A1, it corresponds to two child nodes, namely S1 and S2, then the feature vector F of node A1 can be expressed as: Among them, N1 is the number of points corresponding to the child node S1, N2 is the number of points corresponding to the child node S2, and F1 and F2 are the feature vectors of the child node S1 and the child node S2 respectively.

[0197] The number of points corresponding to the above subnodes can be understood as the total number of points in the leaf nodes contained in the branch corresponding to the subnode. A leaf node is a superpoint, where each superpoint corresponds to multiple points. The number of points corresponding to the above subnodes can be obtained by counting the number of points in the leaf nodes.

[0198] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0199] 403. Divide the semantic superpoint tree into K subtrees according to the feature vector of each node, where K is a positive integer;

[0200] As an optional implementation, the score of each non-leaf node is obtained according to the feature vector of each child node in at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score;

[0201] The nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, and the non-leaf nodes include the root node.

[0202] Traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the subnodes of the node, wherein the first preset score is greater than the second preset score;

[0203] Continue traversing other nodes until all leaf nodes have subordinate subtrees, and then stop traversing to obtain K subtrees.

[0204] That is, the score of each non-leaf node is obtained based on the feature vector of its corresponding child node. Figure 6 In the example, the score of node t3 is based on p0 and p1, the score of t1 is based on t3 and t4, and so on.

[0205] The score is used to indicate whether the two child nodes belong to the same instance or subtree. If they belong to the same subtree, the score is high, otherwise the score is low.

[0206] Then, starting from the root node of the semantic superpoint tree, it is determined whether the score of each node is greater than a second preset score. If it is less, the determination of other nodes continues.

[0207] like Figure 6 The semantic superpoint tree shown starts from t0 and determines whether the score of t0 is greater than the second preset score. If it is less than the second preset score, t1 and t2 are determined in sequence. If the score of t1 is greater than the second preset score, t1-t3-t4-p0-p1-p2-p3 is used as a subtree and the traversal of each node of the branch below t1 is stopped. If the score of t2 is less than the second preset score, the scores of t5 and t6 are continued to be determined.

[0208] And so on, until all superpoints have subordinate subtrees.

[0209] Among them, by traversing from top to bottom, when the node score is small, it is judged that the two subtrees corresponding to the node do not belong to the same instance. On the contrary, when the node score is high, it is judged that the two subtrees corresponding to the node are likely to belong to the same instance. Therefore, the tree where the node is located can be taken out as a subtree or instance.

[0210] By traversing from the root node to the leaf node, when a node score is higher than the preset score, the node and its branches are taken as a subtree, and the traversal of the nodes under the branches of the node is stopped; other nodes are continued to be traversed until the leaf node is reached, and K subtrees can be obtained.

[0211] The K subtrees correspond to the K instances one by one, that is, each subtree corresponds to one instance.

[0212] For example, subtree 1 corresponds to the first instance, subtree 2 corresponds to the second instance, subtree 3 corresponds to the third instance, etc. The three instances are different instances.

[0213] As an optional implementation manner, the feature vectors of at least two child nodes of each node are input into a scoring model for processing, thereby obtaining a score for each node.

[0214] Optionally, the scoring model may be obtained based on supervised learning. Specifically, the scoring model is obtained by learning according to feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample.

[0215] As an optional implementation manner, the score of each node sample is obtained according to the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0216] For example, Figure 7As shown, assuming that there are three different instances, the instance proportion vector of superpoint P0 is (1, 0, 0), and the instance proportion vector of superpoint P1 is (0.8, 0.2, 0), then the score of node t3 is S = P0*P1, which is 0.8.

[0217] The instance proportion vector of the superpoint P0 above indicates that all points in P0 belong to the first instance; the instance proportion vector of the superpoint P1 indicates that 80% of the points in P1 belong to the first instance and 20% belong to the second instance.

[0218] The above-mentioned method of calculating the score is only an example, and other methods are also possible, and this solution does not make any specific limitation on this.

[0219] The above-mentioned scoring model can also be obtained based on unsupervised learning, etc., and this solution does not make any specific limitations on this.

[0220] The first preset score may be, for example, 1, and the second preset score may be, for example, 0.5, etc. This solution does not make any specific limitation on this.

[0221] 404. Obtain semantic categories of K instances in the point cloud according to the K subtrees, wherein the K subtrees correspond to the K instances one-to-one.

[0222] As an optional implementation method, by inputting the above K subtrees into a preset neural network, the semantic category corresponding to each subtree can be obtained.

[0223] As another optional implementation, the semantic category of each subtree is obtained based on the semantic score of each point. For example, by counting the semantic scores of all points in each subtree, the semantics with a higher score is obtained, which is the semantics corresponding to the subtree.

[0224] The acquisition of the semantic score of each point mentioned above can refer to the specific description in the aforementioned step 401, which will not be repeated here.

[0225] Based on the K subtrees obtained in step 403, it can be known that the K subtrees correspond to the K instances one by one. That is, each subtree corresponds to an instance. Through step 404, the specific semantic category of the instance is further known.

[0226] For example, based on the above step 403, it can be known that the target scene includes a first instance, a second instance, and a third instance. Through step 404, it can be known that the first instance is chair 1, the second instance is chair 2, the third instance is table 1, and so on.

[0227] Based on the semantic categories corresponding to each subtree obtained above, the robot's navigation and grasping, the interaction between the virtual image and the actual scene in AR or Virtual Reality (VR), the deletion of specific instances in the point cloud editing tool and other editing operations can be realized.

[0228] By using the point cloud instance segmentation method provided in this embodiment, the semantic category of each instance can be obtained based on the point cloud in the target scene. This solution uses a semantic superpoint tree to obtain a subtree, that is, to obtain different instances. This method is more intuitive, and compared with the prior art, this solution does not require manual setting of parameters, thereby improving segmentation accuracy.

[0229] As an optional implementation, after step 404, the method further includes:

[0230] Processing each of the K subtrees separately to obtain a score of a leaf node of each subtree, wherein the leaf node of each subtree is a node corresponding to a superpoint in the subtree;

[0231] The leaf nodes whose scores are lower than the third preset score are deleted to obtain updated K subtrees.

[0232] The above-mentioned processing of each subtree in the K subtrees separately may be inputting each subtree into a preset neural network for processing, thereby obtaining the score of the leaf node of each subtree.

[0233] Specifically, for a subtree consisting of nodes t1, t3, t4, p0, p1, p2, and p3, the subtree is converted into Figure 8 The connection structure shown.

[0234] By flattening the tree structure, deleting the connection relationship shown by the dotted line, and retaining only t1, p0, p1, p2, and p3, the leaf nodes and the root node of the subtree are obtained, and the corresponding adjacency matrix can be obtained. The adjacency matrix is ​​input into the convolutional neural network to obtain the score of each leaf node, and the leaf nodes with low scores are deleted from the subtree to achieve the refinement of the subtree.

[0235] Furthermore, the semantic categories of the K instances in the point cloud are obtained according to the updated K subtrees, so as to optimize the point cloud instance segmentation of this scheme.

[0236] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0237] Based on the above embodiment, as an optional implementation, the method further includes:

[0238] Obtain a feature vector of each point in a leaf node of each subtree in the updated K subtrees;

[0239] Obtaining a score for each subtree according to a feature vector of a point in a leaf node of each subtree;

[0240] The subtrees with scores less than a fourth preset score in the updated K subtrees are deleted to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

[0241] Furthermore, semantic categories of K' instances in the point cloud are obtained according to the processed K' subtrees, so as to optimize the point cloud instance segmentation of this solution.

[0242] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0243] Specifically, for each updated subtree, the feature vector of the midpoint of each subtree is obtained, each subtree is voxelized to obtain the voxel of each subtree, and the voxel is used as the input of the score evaluation network to obtain the score of the midpoint of each subtree.

[0244] The above is obtained by processing the updated K subtrees, and it can also be processed by processing the K subtrees obtained in step 403, for example:

[0245] Obtain a feature vector of each point in a leaf node of each subtree in the K subtrees;

[0246] Obtaining a score for each subtree according to a feature vector of a point in a leaf node of each subtree;

[0247] The subtrees with scores less than a fourth preset score in the K subtrees are deleted to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

[0248] That is to say, the update may be performed on the originally obtained K subtrees, or may be performed again on the basis of the above updated subtrees, and this solution does not make any specific limitation on this.

[0249] The feature vector of each point in the leaf node of each subtree can be obtained based on the description in step 401, which will not be repeated here.

[0250] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0251] The following introduces the application of this solution to AR map scenarios.

[0252] For example, a user puts on AR glasses and sends a request to an electronic device based on the scene he sees, hoping that the virtual dog will go near a toy duck. The electronic device receives the request input by the user, and the request carries a first target, such as a toy duck, and the request is used to instruct to find the toy duck.

[0253] The electronic device obtains the point cloud in the target scene, and then performs the above-mentioned point cloud instance segmentation processing on the point cloud. Specifically, by processing the point cloud in the target scene, M superpoints and a feature vector of each of the M superpoints are obtained, and a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree are obtained according to the feature vectors of the M superpoints. According to the feature vector of each node, the semantic superpoint tree is segmented into K subtrees, and then the semantic categories of the K instances in the point cloud are obtained according to the K subtrees, and the K subtrees correspond one-to-one to the K instances.

[0254] That is to say, the semantics of each instance in the target scene is obtained through the above processing. Then, the location of the toy duck in the scene can be determined. Based on this, the navigation route from the virtual puppy to the toy duck can be obtained, and then the virtual puppy can be brought near the toy duck.

[0255] The following introduces the application of this solution in robot scenarios.

[0256] For example, the user sends a request to the robot, asking the robot to bring the apples from the living room.

[0257] Among them, after the robot receives the request input by the user, it obtains the point cloud in the target scene, and then performs the above-mentioned point cloud instance segmentation processing on the point cloud. Specifically, by processing the point cloud in the target scene, M super points and the feature vector of each super point in the M super points are obtained, and a semantic super point tree and a feature vector of each node in the semantic super point tree are obtained according to the feature vectors of the M super points. According to the feature vector of each node, the semantic super point tree is segmented into K subtrees, and then the semantic categories of the K instances in the point cloud are obtained according to the K subtrees, and the K subtrees correspond to the K instances one by one.

[0258] That is to say, the semantics of each instance in the target scene is obtained through the above processing. Then, the location of the apple in the scene can be determined. Based on this, the robot can get the apple.

[0259] The following introduces the application of this solution in point cloud editing scenarios.

[0260] For example, the user wants to replace the sofa in the living room and wants to know the effect of putting the new sofa in place. The electronic device receives a request input by the user, and the request carries a first target, which may be a sofa.

[0261] The electronic device obtains the point cloud in the target scene, and then performs the above-mentioned point cloud instance segmentation processing on the point cloud. Specifically, by processing the point cloud in the target scene, M superpoints and a feature vector of each of the M superpoints are obtained, and a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree are obtained according to the feature vectors of the M superpoints. According to the feature vector of each node, the semantic superpoint tree is segmented into K subtrees, and then the semantic categories of the K instances in the point cloud are obtained according to the K subtrees, and the K subtrees correspond one-to-one to the K instances.

[0262] That is to say, the semantics of each instance in the target scene is obtained through the above processing. Then, the location of the sofa in the scene can be determined. Based on this, the sofa can be displayed to the user so that the user can select it and perform operations such as editing, such as deleting, or moving.

[0263] Furthermore, the user can also move and place the new sofa in various positions to facilitate viewing of the placement effect.

[0264] By adopting the technical means of this solution, it is possible to realize the interaction between the virtual image and the actual scene in AR / VR, perform robot navigation and grasping, or perform editing such as deletion of specific instances in the point cloud editing tool.

[0265] Reference Fig. 9As shown, an embodiment of the present application provides a point cloud instance segmentation device. The device includes a processing module 901, a tree building module 902, a segmentation module 903 and a determination module 904, which are specifically as follows:

[0266] A processing module 901 is used to process the point cloud in the target scene to obtain M super points and a feature vector of each of the M super points, where M is an integer not less than 2;

[0267] A tree building module 902, configured to obtain a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints;

[0268] A segmentation module 903 is used to segment the semantic superpoint tree into K subtrees according to the feature vector of each node, where K is a positive integer;

[0269] The determination module 904 is used to obtain semantic categories of K instances in the point cloud according to the K subtrees, and the K subtrees correspond to the K instances one by one.

[0270] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0271] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0272] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0273] As an optional implementation method, the tree building module 902 is also used to: obtain the number of points corresponding to each of the at least two child nodes; and obtain the feature vector of each non-leaf node based on the number of points corresponding to each child node and the feature vector of each child node.

[0274] As an optional implementation method, the segmentation module 903 is used to: obtain the score of each non-leaf node based on the feature vector of each child node in at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traverse the semantic superpoint tree from the root node to the leaf node, and when the score of the node exceeds the second preset score, treat the node and the branch corresponding to the node as a subtree, and stop traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continue traversing other nodes until all leaf nodes have subordinate subtrees, and obtain K subtrees.

[0275] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0276] As an optional implementation method, the segmentation module 903 is also used to: input the feature vectors of at least two child nodes corresponding to each non-leaf node into the scoring model for processing to obtain the score of each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0277] As an optional implementation method, the device also includes a first update module, which is used to: process each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; delete the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0278] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0279] As another optional implementation method, the device also includes a second updating module, which is used to: obtain the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0280] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0281] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the K subtrees; obtaining the score of each subtree according to the feature vector of the point in the leaf node of each subtree; deleting the subtrees in the K subtrees whose scores are less than a fourth preset score to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0282] The present application provides a method for determining an object based on a three-dimensional scene, including:

[0283] receiving a request sent by a user, wherein the request carries a first target, and the request is used to instruct to obtain the first target;

[0284] Acquire a point cloud in a target scene, wherein the target scene includes the first target;

[0285] Performing instance segmentation processing on the point cloud in the target scene to obtain the first target, wherein the instance segmentation processing is performed based on a tree structure;

[0286] The first goal is presented to the user.

[0287] By adopting the method for determining objects based on three-dimensional scenes provided by this solution, complete object segmentation and good segmentation effect can be achieved.

[0288] Among them, the above-mentioned instance segmentation processing includes: processing the point cloud in the target scene to obtain M super points (super point) and the feature vector of each super point in the M super points, M is an integer not less than 2; obtaining a semantic super point tree and the feature vector of each node (not) in the semantic super point tree according to the feature vectors of the M super points; dividing the semantic super point tree into K subtrees according to the feature vector of each node, K is a positive integer; obtaining the semantic categories of K instances in the point cloud according to the K subtrees, and the K subtrees correspond one-to-one to the K instances.

[0289] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0290] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0291] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0292] As an optional implementation, the method also includes: obtaining the number of points corresponding to each of the at least two subnodes; and obtaining the feature vector of each non-leaf node based on the number of points corresponding to each subnode and the feature vector of each subnode.

[0293] As an optional implementation method, the semantic superpoint tree is divided into K subtrees according to the feature vector of each node, including: obtaining the score of each non-leaf node according to the feature vector of each of the at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continuing to traverse other nodes until all leaf nodes have subordinate subtrees, and obtaining K subtrees.

[0294] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0295] As an optional implementation manner, obtaining the score of each non-leaf node according to the feature vector of each child node in at least two child nodes corresponding to each non-leaf node includes:

[0296] The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0297] As an optional implementation, the scoring model is learned based on the feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample, and the score of each node sample is obtained based on the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0298] As an optional implementation, the method also includes: processing each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; deleting the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0299] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0300] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtaining the score of each subtree based on the feature vector of the point in the leaf node of each subtree; deleting the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

[0301] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0302] As another optional implementation, the method also includes: obtaining the feature vector of each point in the leaf node of each subtree in the K subtrees; obtaining the score of each subtree according to the feature vector of the point in the leaf node of each subtree; deleting the subtrees in the K subtrees whose scores are less than a fourth preset score to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0303] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0304] The present application also provides a device for determining an object based on a three-dimensional scene, comprising:

[0305] A receiving module, used for receiving a request sent by a user, wherein the request carries a first target, and the request is used for instructing to obtain the first target;

[0306] An acquisition module, configured to acquire a point cloud in a target scene, wherein the target scene includes the first target;

[0307] A processing module, configured to perform instance segmentation processing on the point cloud in the target scene to obtain the first target, wherein the instance segmentation processing is performed based on a tree structure;

[0308] A display module is used to display the first target to the user.

[0309] Through the embodiments of the present application, the point cloud is processed to obtain superpoints, and then a semantic superpoint tree is obtained based on the superpoints, and then the semantic superpoint tree is divided into different subtrees, and the semantic categories of each instance in the point cloud are obtained based on the different subtrees. This scheme reduces the complexity of point cloud instance segmentation and avoids fragmented segmentation by aggregating points with geometric similarity into superpoints. At the same time, this scheme uses the semantic superpoint tree to obtain subtrees, that is, to obtain different instances. Using this method to perform instance segmentation is more intuitive. Moreover, compared with the prior art, the entire process of this scheme does not require manual setting of parameters, which reduces the human factors that affect the segmentation accuracy and effectively improves the segmentation accuracy.

[0310] As an optional implementation, the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained based on the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints.

[0311] This solution merges nodes with large similarity to obtain a semantic superpoint tree. By adopting this method, different nodes can be classified, and then similar nodes can be merged, which can reflect the relationship between nodes. This method is more intuitive and three-dimensional.

[0312] As an optional implementation method, the acquisition module is also used to: obtain the number of points corresponding to each sub-node of the at least two sub-nodes; and obtain the feature vector of each non-leaf node based on the number of points corresponding to each sub-node and the feature vector of each sub-node.

[0313] As an optional implementation method, the processing module is used to: obtain the score of each non-leaf node based on the feature vector of each child node in at least two child nodes corresponding to each non-leaf node, wherein the score of each leaf node is a first preset score; traverse the semantic superpoint tree from the root node to the leaf node, and when the score of the node exceeds a second preset score, treat the node and the branch corresponding to the node as a subtree, and stop traversing the child nodes of the node, wherein the first preset score is greater than the second preset score; continue traversing other nodes until all leaf nodes have subordinate subtrees, and obtain K subtrees.

[0314] This solution uses a semantic superpoint tree to obtain subtrees, that is, to obtain different instances. This method is more intuitive for instance segmentation.

[0315] As an optional implementation, the processing module is further used to:

[0316] The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

[0317] As an optional implementation, the scoring model is learned based on the feature vectors of at least two sub-node samples corresponding to each node sample and the score of each node sample, and the score of each node sample is obtained based on the instance proportion vector of at least two sub-node samples corresponding to each node sample.

[0318] As an optional implementation method, the processing module is also used to: process each of the K subtrees separately to obtain the score of the leaf node of each subtree, wherein the leaf node of each subtree is the node corresponding to the super point in the subtree; delete the leaf nodes whose scores are lower than the third preset score to obtain the updated K subtrees.

[0319] Since there may be misclassified super points in the subtree, this method is used to update the subtree by calculating the score of each leaf node and then deleting the leaf nodes that are not the subtree, thereby further improving the accuracy of point cloud instance segmentation.

[0320] As another optional implementation method, the processing module is also used to: obtain the feature vector of each point in the leaf node of each subtree in the updated K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees with scores less than a fourth preset score in the updated K subtrees to obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0321] By adopting this method, the subtrees are updated by calculating the scores of each subtree and then deleting the subtrees with lower scores, which further improves the accuracy of point cloud instance segmentation.

[0322] As another optional implementation method, the processing module is also used to: obtain the feature vector of each point in the leaf node of each subtree in the K subtrees; obtain the score of each subtree according to the feature vector of the point in the leaf node of each subtree; delete the subtrees in the K subtrees whose scores are less than a fourth preset score, and obtain processed K' subtrees, K' is not less than 1, and K' is not greater than the K.

[0323] Among them, the above instance segmentation processing can refer to any one of the implementation methods provided in the first aspect above.

[0324] By adopting the device for determining objects based on three-dimensional scenes provided by the present solution, complete object segmentation and good segmentation effect can be achieved.

[0325] It should be noted that the above Fig. 9 The processing module 901 , the tree building module 902 , the segmentation module 903 and the determination module 904 shown are used to execute the relevant steps of the above-mentioned point cloud instance segmentation method.

[0326] For example, the processing module 901 is used to execute the relevant content of step 401 , the tree building module 902 is used to execute the relevant content of step 402 , the segmentation module 903 is used to execute the relevant content of step 403 , and the determination module 904 is used to execute the relevant content of step 404 .

[0327] In this embodiment, the point cloud instance segmentation device is presented in the form of a module. The "module" here may refer to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0328] In addition, the above processing module 901, tree building module 902, segmentation module 903 and determination module 904 can be Fig.10 The point cloud instance segmentation device shown is implemented by the processor 1002.

[0329] Fig.10 It is a schematic diagram of the hardware structure of the point cloud instance segmentation device provided in an embodiment of the present application. Fig.10The point cloud instance segmentation device 1000 shown (the device 1000 may be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002 and the communication interface 1003 are connected to each other through the bus 1004.

[0330] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0331] The memory 1001 can store programs. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the point cloud instance segmentation method of the embodiment of the present application.

[0332] Processor 1002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the point cloud instance segmentation device of the embodiment of the present application, or to execute the point cloud instance segmentation method of the method embodiment of the present application.

[0333] The processor 1002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the point cloud instance segmentation method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1002. The above-mentioned processor 1002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and combines its hardware to complete the functions required to be performed by the units included in the point cloud instance segmentation device of the embodiment of the present application, or executes the point cloud instance segmentation method of the method embodiment of the present application.

[0334] The communication interface 1003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1000 and other devices or a communication network. For example, data can be acquired through the communication interface 1003 .

[0335] The bus 1004 may include a path for transmitting information between various components of the device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0336] It should be noted that although Fig.10 The device 1000 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 1000 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1000 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.10 All devices shown in .

[0337] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0338] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0339] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the specific descriptions of the corresponding steps in the aforementioned method embodiments, and will not be repeated here.

[0340] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0341] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0342] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0343] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0344] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A point cloud instance segmentation method, characterized in that: include: Processing the point cloud in the target scene to obtain M super points and a feature vector of each of the M super points, where M is an integer not less than 2; A semantic superpoint tree and a feature vector of each node in the semantic superpoint tree are obtained according to the feature vectors of the M superpoints; the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained according to the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints; According to the feature vector of each node, the semantic superpoint tree is divided into K subtrees, where K is a positive integer; Semantic categories of K instances in the point cloud are obtained according to the K subtrees, and the K subtrees correspond to the K instances one by one.

2. The method according to claim 1, characterized in that The method further comprises: Obtain the number of points corresponding to each of the at least two child nodes; The feature vector of each non-leaf node is obtained according to the number of points corresponding to each child node and the feature vector of each child node.

3. The method according to claim 1 or 2, characterized in that: The nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, and the non-leaf nodes include a root node, and the semantic superpoint tree is divided into K subtrees according to the feature vector of each node, including: Obtaining a score for each non-leaf node according to a feature vector of each of the at least two child nodes corresponding to each non-leaf node, wherein the score for each leaf node is a first preset score; Traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the subnodes of the node, wherein the first preset score is greater than the second preset score; Continue traversing other nodes until all leaf nodes have subordinate subtrees, and then stop traversing to obtain K subtrees.

4. The method according to claim 3, characterized in that The step of obtaining the score of each non-leaf node according to the feature vector of each child node in at least two child nodes corresponding to each non-leaf node includes: The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

5. The method according to claim 1, characterized in that The method further comprises: Processing each of the K subtrees separately to obtain a score of a leaf node of each subtree, wherein the leaf node of each subtree is a node corresponding to a superpoint in the subtree; The leaf nodes whose scores are lower than the third preset score are deleted to obtain updated K subtrees.

6. The method according to claim 5, characterized in that The method further comprises: Obtain a feature vector of each point in a leaf node of each subtree in the updated K subtrees; Obtaining a score for each subtree according to a feature vector of a point in a leaf node of each subtree; The subtrees with scores less than a fourth preset score in the updated K subtrees are deleted to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

7. A point cloud instance segmentation device, characterized in that: include: A processing module, used for processing the point cloud in the target scene to obtain M super points and a feature vector of each of the M super points, where M is an integer not less than 2; A tree building module, used for obtaining a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints; the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained according to the feature vectors of the at least two child nodes; for any non-leaf node A1, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of the at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of the at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein the leaf nodes in the semantic superpoint tree correspond one-to-one to the M superpoints, and the feature vectors of the leaf nodes correspond one-to-one to the feature vectors of the M superpoints; A segmentation module, used for segmenting the semantic superpoint tree into K subtrees according to the feature vector of each node, where K is a positive integer; A determination module is used to obtain semantic categories of K instances in the point cloud according to the K subtrees, and the K subtrees correspond to the K instances one by one.

8. The device according to claim 7, characterized in that The tree building module is also used for: Obtain the number of points corresponding to each of the at least two child nodes; The feature vector of each non-leaf node is obtained according to the number of points corresponding to each child node and the feature vector of each child node.

9. The device according to claim 7 or 8, characterized in that The nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, and the non-leaf nodes include a root node, and the segmentation module is used to: Obtaining a score for each non-leaf node according to a feature vector of each of the at least two child nodes corresponding to each non-leaf node, wherein the score for each leaf node is a first preset score; Traversing the semantic superpoint tree from the root node to the leaf node, when the score of the node exceeds a second preset score, treating the node and the branch corresponding to the node as a subtree, and stopping traversing the subnodes of the node, wherein the first preset score is greater than the second preset score; Continue traversing other nodes until all leaf nodes have subordinate subtrees, and then stop traversing to obtain K subtrees.

10. The device according to claim 9, characterized in that The segmentation module is further used for: The feature vectors of at least two child nodes corresponding to each non-leaf node are input into a scoring model for processing to obtain a score for each non-leaf node, wherein the scoring model is obtained based on machine learning.

11. The device according to claim 7, characterized in that The device also includes a first updating module, configured to: Processing each of the K subtrees separately to obtain a score of a leaf node of each subtree, wherein the leaf node of each subtree is a node corresponding to a superpoint in the subtree; The leaf nodes whose scores are lower than the third preset score are deleted to obtain updated K subtrees.

12. The device according to claim 11, characterized in that The device also includes a second updating module, configured to: Obtain a feature vector of each point in a leaf node of each subtree in the updated K subtrees; Obtaining a score for each subtree according to a feature vector of a point in a leaf node of each subtree; The subtrees with scores less than a fourth preset score in the updated K subtrees are deleted to obtain processed K' subtrees, where K' is not less than 1 and K' is not greater than the K.

13. A method for determining an object based on a three-dimensional scene, characterized in that: include: receiving a request sent by a user, wherein the request carries a first target, and the request is used to instruct to obtain the first target; Acquire a point cloud in a target scene, wherein the target scene includes the first target; Perform instance segmentation processing on the point cloud in the target scene to obtain the first target, wherein the instance segmentation processing is performed based on a tree structure; the instance segmentation processing includes: processing the point cloud in the target scene to obtain M superpoints and a feature vector of each superpoint in the M superpoints, where M is an integer not less than 2; obtaining a semantic superpoint tree and a feature vector of each node in the semantic superpoint tree according to the feature vectors of the M superpoints; the nodes of the semantic superpoint tree include leaf nodes and non-leaf nodes, wherein the non-leaf nodes correspond to at least two child nodes, and each non-leaf node in the semantic superpoint tree is obtained by merging at least two child nodes, and the similarity between the at least two child nodes exceeds a preset threshold, wherein the feature vector of each non-leaf node is obtained according to the feature vectors of the at least two child nodes. The semantic superpoint tree is obtained by obtaining the feature vectors of at least two child nodes corresponding to the non-leaf node A1; for any non-leaf node A1, any child node of at least two child nodes corresponding to the non-leaf node A1 is a superpoint; or, any child node of at least two child nodes corresponding to the non-leaf node A1 is a non-leaf node; or, at least one child node of at least two child nodes corresponding to the non-leaf node A1 is a superpoint, and at least one child node is a non-leaf node; wherein, the leaf nodes in the semantic superpoint tree correspond to the M superpoints one-to-one, and the feature vectors of the leaf nodes correspond to the feature vectors of the M superpoints one-to-one; according to the feature vector of each node, the semantic superpoint tree is divided into K subtrees, K is a positive integer; according to the K subtrees, the semantic categories of the K instances in the point cloud are obtained, and the K subtrees correspond to the K instances one-to-one; The first goal is presented to the user.

14. A point cloud instance segmentation device, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 6 and claim 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1 to 6 and claim 13.

16. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 6 and claim 13.

Images